Multi-sensor intelligent cooperative control method, device, equipment and medium

Abstract

The application discloses a multi-sensor intelligent collaborative control method and device, equipment and a medium, and the method comprises the steps of establishing a reinforcement learning agent model, wherein the state space of the reinforcement learning agent model comprises a global comprehensive situation expression and a single sensor state embedding representation, and the action space of the reinforcement learning agent model comprises action output values abstracted from different tasks performed by the multi-sensor; the reinforcement learning agent model is trained through sampling, and the reinforcement learning agent model is guided to learn through a reward shaping project, wherein the training comprises centralized training and decentralized execution; and the multi-sensor is collaboratively controlled by loading the trained reinforcement learning agent model. The application realizes cross-means and cross-region collaborative control of the multi-sensor by using the reinforcement learning technology.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and in particular relates to multi-sensor intelligent collaborative control methods, devices, equipment and media. Background Technology

[0002] Currently, the electromagnetic spectrum space exhibits strong adversarial characteristics. When facing a large number of complex radiation sources with different characteristics conducting detection activities on key signals of targets in the electromagnetic spectrum space, it is necessary to coordinate and dispatch multiple sensors of the same or different types (such as ultra-shortwave, microwave, electronic and portable radar, etc.) across regions to give full play to their respective advantages, achieve the need for real-time detection and location of multiple targets, and achieve long-term continuous tracking.

[0003] Traditional manual decision-making control methods suffer from high latency, low fault tolerance, inability to comprehensively consider high-dimensional complex signal situations and optimal coordinated scheduling of multi-sensor resources, and the short bursts characteristic of most key target signals, making it difficult to achieve effective and continuous localization and tracking of radiation sources.

[0004] Deep reinforcement learning faces challenges such as sparse reward signals and low sampling rates, leading to low sample utilization, slow learning speed, and even difficulty in training convergence. These drawbacks are particularly pronounced in complex electromagnetic spectrum environments with multiple radiation sources and strong adversarial characteristics. Furthermore, because some key signals possess characteristics such as concealment, suddenness, and phased nature, and different signals have different characteristics that cannot be detected by a single means (i.e., a single type of sensor), radiation source localization and tracking tasks become obscure and difficult to break down. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art by providing a multi-sensor intelligent collaborative control method, device, equipment and medium, which can use artificial intelligence technology to collaboratively control multiple sensors in complex multi-radiation source environments, thereby achieving real-time detection and positioning of multiple targets and meeting the need for long-term continuous tracking of targets.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A multi-sensor intelligent collaborative control method, the method comprising:

[0008] Establish a reinforcement learning agent model for each sensor. The state space of the reinforcement learning agent model includes a global integrated situational representation and a single sensor state embedding representation. The action space of the reinforcement learning agent model includes action output values ​​abstracted from different tasks performed by multiple sensors.

[0009] The reinforcement learning agent model is trained by sampling and guided to learn by reward shaping engineering. The training includes centralized training and decentralized execution.

[0010] The reinforcement learning agent model obtained through training coordinates the control of multiple sensors.

[0011] Furthermore, the step of training the reinforcement learning agent model through sampling specifically includes:

[0012] Multiple sampling threads are started to perform independent sampling in multiple parallel simulation environments or parallel real environments with different configuration scenarios;

[0013] The sampled data is uniformly placed into the sampling experience cache pool;

[0014] When the training conditions are met, the sampled data is retrieved from the sampled experience cache pool and the agent model is updated by centralized training according to the corresponding reinforcement learning algorithm. Then, the updated model parameters are put into the model parameter cache pool.

[0015] Furthermore, the process of guiding the reinforcement learning agent model to learn through reward shaping engineering specifically includes:

[0016] Set up rewards based on user design, endgame mode rewards, and curiosity rewards;

[0017] The artificially designed rewards include corresponding reward values ​​given when a preset task is completed or when a task is failed.

[0018] The final mode reward includes a reward based on the overall signal detection performance;

[0019] The curiosity reward includes a reward given when exploring unknown spaces.

[0020] Furthermore, the confidence region policy optimization method used when establishing the reinforcement learning agent model specifically includes:

[0021] The confidence region policy optimization algorithm is simplified using a first-order approximation, and the simplified algorithm is as follows:

[0022]

[0023] The corresponding constraints are:

[0024]

[0025] Where π represents the new strategy; π old The old strategy; s represents the state; a represents the action. The advantage function of the old strategy;

[0026] State action value function:

[0027] State value function:

[0028] Advantage function:

[0029] Where γ is the attenuation factor, This represents the average KL divergence between the old and new strategies;

[0030] The simplified confidence region policy optimization algorithm is approximated using the Monte Carlo method, resulting in...

[0031]

[0032] make The ratio of the old and new strategies is used to obtain...

[0033]

[0034] The constraint condition of the confidence region policy optimization algorithm is approximated as r t (θ)∈[1-∈,1+∈], where ∈ is the clip coefficient, then the objective function of the constrained confidence region policy optimization algorithm can be expressed as the unconstrained objective function:

[0035]

[0036] Adding the objective function of the state-value function and the entropy of the policy model to the unconstrained objective function yields the complete objective function:

[0037]

[0038] in, C1 and C2 are the pre-set coefficients for the corresponding items.

[0039] Furthermore, the centralized training and decentralized execution specifically include:

[0040] A central controller collects the global state of all agents and makes unified decisions.

[0041] Each sensor asynchronously executes its own task based on its current state.

[0042] Furthermore, the method also includes guiding the learning of the reinforcement learning agent model with an expert policy based on generative adversarial imitation learning.

[0043] Furthermore, the expert policy based on generative adversarial imitation learning guides the learning of the reinforcement learning agent model, specifically including:

[0044] Repeat the following steps until the optimal strategy is obtained:

[0045] The trajectory of the agent is generated by interacting with the environment using the reinforcement learning agent model corresponding to the current sensor.

[0046] The agent generates both the trajectory and the demonstration trajectory, which are then input into the discriminator, and the discriminator parameters are updated using supervised learning.

[0047] The updated discriminator outputs a new discriminant reward function;

[0048] The updated reward function is used to provide reward signals to further update the agent's policy.

[0049] On the other hand, the present invention also provides a multi-sensor intelligent collaborative control device, the device comprising:

[0050] The agent model building module builds a reinforcement learning agent model for each sensor. The state space of the reinforcement learning agent model includes a global comprehensive situational expression and a single sensor state embedding representation. The action space of the reinforcement learning agent model includes action output values ​​abstracted from different tasks performed by multiple sensors.

[0051] The agent model training module trains the reinforcement learning agent model through sampling and guides the learning of the reinforcement learning agent model through reward shaping engineering. The training includes centralized training and decentralized execution.

[0052] A sensor control module that collaboratively controls multiple sensors through a trained reinforcement learning agent model.

[0053] On the other hand, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, which is loaded and executed by the processor to implement any of the above-described multi-sensor intelligent collaborative control methods.

[0054] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement any of the above-described multi-sensor intelligent collaborative control methods.

[0055] The beneficial effects of this invention are as follows:

[0056] (1) This invention achieves the coordinated control of multiple sensors by establishing a reinforcement learning agent model, which enables multiple sensors to have the corresponding processing capabilities for complex signal situations in complex working environments.

[0057] (2) This invention utilizes reinforcement learning technology and generative adversarial learning technology based on expert knowledge to achieve continuous localization and tracking of key signals in complex electromagnetic spectrum environments.

[0058] (3) The present invention can control multiple sensors across regions to asynchronously execute their respective tasks, has the ability to capture and locate short burst signals, and has a certain continuous positioning and tracking capability. Attached Figure Description

[0059] Figure 1 This is the multi-sensor intelligent collaborative control method provided in the embodiments of the present invention;

[0060] Figure 2 This is a schematic diagram of the state space of an embodiment of the present invention;

[0061] Figure 3 This is a logic diagram of the intelligent agent training module in an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of the intelligent agent training system architecture according to an embodiment of the present invention;

[0063] Figure 5 This is a distributed architecture framework diagram of an embodiment of the present invention;

[0064] Figure 6 This is a schematic diagram of a multi-agent learning system according to an embodiment of the present invention;

[0065] Figure 7 This is a schematic diagram of an expert strategy-guided learning method for generating adversarial imitation learning according to an embodiment of the present invention;

[0066] Figure 8 This is a structural block diagram of the multi-sensor intelligent collaborative control device provided in an embodiment of the present invention. Detailed Implementation

[0067] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0068] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Traditional manual decision-making and control methods suffer from high latency, low fault tolerance, inability to comprehensively consider high-dimensional complex signal situations and optimal coordinated scheduling of multi-sensor resources, and the short bursts characteristic of most key target signals, making it difficult to achieve effective and continuous localization and tracking of radiation sources.

[0070] Deep reinforcement learning faces challenges such as sparse reward signals and low sampling rates, leading to low sample utilization, slow learning speed, and even difficulty in training convergence. These drawbacks are particularly pronounced in complex electromagnetic spectrum environments with multiple radiation sources and strong adversarial characteristics. Furthermore, because some key signals possess characteristics such as concealment, suddenness, and phased nature, and different signals have different characteristics that cannot be detected by a single means (i.e., a single type of sensor), radiation source localization and tracking tasks become obscure and difficult to break down.

[0071] To address the aforementioned technical problems, the following embodiments of the multi-sensor intelligent collaborative control method, apparatus, device, and medium of the present invention are proposed.

[0072] Example 1

[0073] This embodiment addresses the localization and tracking of key radiation source signals in complex electromagnetic spectrum environments. It utilizes reinforcement learning technology to achieve cross-means and cross-regional collaborative control of multiple sensors, optimizes resource scheduling and decision-making speed, improves positioning accuracy and precision, and further enhances the continuous positioning capability for some short burst signal radiation sources.

[0074] Reference Figure 1 ,like Figure 1 The diagram shown is a flowchart of the multi-sensor intelligent collaborative control method provided in this embodiment. The method specifically includes the following steps:

[0075] Step 1: Establish a reinforcement learning agent model for each sensor.

[0076] Specifically, the state space of a reinforcement learning agent model includes a global integrated situational representation and a single sensor state embedding representation, while the action space of a reinforcement learning agent model includes action output values ​​abstracted from different tasks performed by multiple sensors.

[0077] In this embodiment, the state space of the reinforcement learning agent model consists of state vectors of sensors, targets, and signals. In this scheme, the state space is an N×D matrix. N represents the number of elements in the current environment, including sensors, targets, and signals. When the number of environmental elements is less than N, the remaining positions are padded with zeros. D represents the length of the vectors; vectors of different elements are padded with zeros to reach the same length.

[0078] The following sections of this invention will describe specific embodiments using the coordinated control of three detection device sensors located in three different regions. It should be understood that the number of sensors and regions is not intended to limit this invention.

[0079] Reference Figure 2 ,like Figure 2 The diagram shows the state space of this embodiment. The state space consists of two parts: a global comprehensive situational awareness representation and a single sensor state embedding representation. The global situational awareness representation includes targets, signals, and global operational status, respectively. In the target comprehensive situational awareness embedding representation, 0 represents that the current agent has not successfully located the signal, therefore there is no latitude and longitude; 1 represents that the target has been located. Taking three detection sensors in three different regions as an example, since positioning requires at least two sensors in different regions, the sensor IDs supporting positioning are as follows: 0 represents invalid data; 1 represents sensors between regions (1,2) performing positioning; 2 represents sensors between regions (1,3) performing positioning; 3 represents sensors between regions (2,3) performing positioning; and 4 represents sensors between regions (1,2,3) performing positioning. The signal comprehensive situational awareness embedding representation consists of the situational awareness of all currently detected signals, specifically including frequency, bandwidth, amplitude, azimuth, and key signal identifiers. The global operational status discrete embedding representation consists of the current working status of all sensors, specifically including task type, task status, current wide-scan frequency band (fill in 0 if none), and current monitored frequency point (fill in 0 if none). The embedded representation of a single sensor status includes the current sensor signal status data (frequency, bandwidth, amplitude, azimuth, key signal identifier), historical fixed frequency data (frequency, bandwidth, identifier), the current working status of the sensor (task type, task status, current wide-scan frequency band, current control frequency point), and the set of wide-scan frequency points of similar sensors (consisting of a binary array, the first dimension being frequency, and the second dimension being whether it is a signal identifier scanned by the sensor itself).

[0080] This embodiment uses an ultra-shortwave detection sensor as an example. The action space of the reinforcement learning agent is shown in Table 1. For continuous actions, the corresponding specific action output value is obtained by sampling from the corresponding Gaussian distribution.

[0081]

[0082]

[0083] Table 1 Action Space of Reinforcement Learning Agent

[0084] Step 2: Train the reinforcement learning agent model through sampling, and guide the learning of the reinforcement learning agent model through reward shaping engineering. The training includes centralized training and decentralized execution.

[0085] The multi-sensor intelligent cooperative control method for radiation source localization and tracking consists of a task planning module and an agent training module. The task planning module provides different sensor cooperative schemes by loading different agent models, such as resource priority, localization priority, and tracking priority.

[0086] Agent training is the core module of this embodiment, which mainly trains the corresponding agent by interacting with simulation software or the real environment.

[0087] Reference Figure 3 and 4 ,like Figure 3 The diagram shown is the operational logic diagram of the intelligent agent training module in this embodiment. Figure 4 The diagram shown is a schematic of the intelligent agent training system architecture in this embodiment.

[0088] In this embodiment, the agent training is mainly divided into a sampling part and a training part. The blue side is the radiation source side, and its corresponding blue side trajectory generation submodule (how the radiation source moves along its trajectory) and blue side decision submodule (when the signal carried by the radiation source switches on and off) are both generated by rules. The red side is the sensor agent side. The red side model is the agent model, and the blue side model is the rule model.

[0089] This embodiment addresses the low sampling rate problem faced by deploying PPO reinforcement learning agents in complex electromagnetic spectrum environments. Based on the gRPC distributed architecture, it adopts a distributed sampling centralized training scheme for the agent training module. (Refer to...) Figure 5 ,like Figure 5 The diagram shows the distributed architecture framework of this embodiment. Multiple sampling threads are started to independently sample in multiple parallel simulation environments with different configurations. The sampled data is then uniformly placed into a sampling experience cache pool. When the training conditions are met, the sampled data is retrieved from the sampling experience cache pool and used to centrally train the agent model according to the corresponding reinforcement learning algorithm to update it. The updated model parameters are then placed into the model parameter cache pool. The sampling module periodically retrieves the latest model parameters from the model parameter cache pool to update the agent model, thereby updating the agent policy, and using the updated policy for sampling.

[0090] Thus, the reinforcement learning training technique based on distributed sampling and centralized training has completed the cycle of sampling, training, and model iteration. By maximizing the use of physical hardware resources to improve the sampling rate, it greatly enhances the learning efficiency of the agent.

[0091] To address the challenges of cross-regional multi-sensor collaborative control, and considering the asynchronous execution of tasks by multiple sensors across regions, a semi-multi-agent PPO reinforcement learning algorithm with centralized training and decentralized execution (CTDE) is proposed based on the concept of multi-agent reinforcement learning.

[0092] Reinforcement learning algorithms generally exhibit some volatility, which significantly impacts the training process and final performance. To ensure monotonically improving policy models during optimization, the Trust Region Policy Optimization (TRPO) algorithm uses KL divergence to measure the difference between new and old policies, constructing an objective formula similar to the natural gradient method. This objective is used to continuously optimize the policy, effectively preventing large fluctuations caused by noise in the policy gradient. Furthermore, the conjugate gradient method is used to reduce the computational cost of the Fisher information matrix. However, as a second-order method, TRPO still requires substantial computational resources. The PPO algorithm further simplifies the objective function of TRPO, using a first-order approximation, thus accelerating training while maintaining accuracy.

[0093] First, the objective function of the TRPO algorithm can be simplified as follows:

[0094]

[0095] The corresponding constraints are:

[0096]

[0097] Where π represents the new strategy; π old The old strategy; s represents the state; a represents the action. Let γ be the advantage function of the old strategy, defined as follows (γ is the decay factor):

[0098] State action value function:

[0099] State value function:

[0100] Advantage function:

[0101] This represents the average KL divergence between the old and new strategies.

[0102] Although TRPO uses the conjugate gradient method to minimize the computational cost of solving this complex and constrained objective function, it still requires significant computational resources, resulting in low algorithm efficiency. Therefore, the PPO algorithm further optimizes the objective function.

[0103] In practical applications, the calculation of the expectation is often approximated using the Monte Carlo method; therefore, the objective function of the TRPO algorithm becomes:

[0104]

[0105] make The objective function of the TRPO algorithm, used to represent the ratio of the old and new strategies, further becomes:

[0106]

[0107] The constraint condition of TRPO is approximated as r t (θ)∈[1-∈,1+∈], where ∈ are clip coefficients, then the constrained TRPO objective function can be expressed as an unconstrained objective function:

[0108]

[0109] Therefore, the common gradient descent method is used for solving. While retaining the advantage of the TRPO algorithm in that it can stably improve the policy, the use of a first-order method greatly reduces the computational cost and improves the algorithm's efficiency. Furthermore, the objective function of the state-value function (VF) and the entropy (S) of the policy model are added to the final objective function, so the complete objective function of PPO becomes:

[0110]

[0111] in, These are the coefficients of the corresponding terms. In this embodiment, C1 = 0.001 and C2 = 0 by default.

[0112] Target signal localization and tracking in complex electromagnetic spectrum environments requires collaboration among multiple sensors across different regions. Common multi-agent reinforcement learning frameworks include three types: Centralized Training and Centralized Execution (CTCE), Centralized Training and Decentralized Execution (DTCE), and Decentralized Training and Decentralized Execution (DTDE). Centralized training requires a central controller to collect the global states of all agents and make unified decisions, thus ensuring good collaborative performance. However, sensors located in different regions have varying observation ranges, and the number and type of signals they observe differ. Therefore, each sensor asynchronously executes its task based on its current state. (See reference...) Figure 6 ,like Figure 6 The diagram shown is a schematic of the multi-agent learning system in this embodiment. This embodiment uses a centralized training and decentralized execution (DTCE) framework to deploy the multi-agent learning system.

[0113] To further leverage expert prior knowledge (i.e., expert policies) to accelerate agent learning and address the low sample utilization problem of reinforcement learning methods, this invention proposes an expert policy-guided learning method based on Generative Adversarial Imitation Learning (GAIL). Inspired by the successful application of nonlinear loss functions in generative adversarial networks and inverse reinforcement learning, GAIL can directly learn policies from expert trajectories. Specific implementation methods are as follows... Figure 7 As shown.

[0114] The goal of inverse reinforcement learning is to learn from the latent family of functions C:R S×A We fit a loss function c to the interval {c: S×A→R}, such that the expected cumulative loss is minimized on the expert-demonstrated trajectory, but maximized for trajectories generated by any other strategy. Considering the complex multi-source electromagnetic spectrum environment, due to the high environmental dimensionality, there exists a huge set of loss functions. With a limited sample dataset, inverse reinforcement learning is prone to overfitting. Therefore, a loss function regularizer ψ(c) is used to avoid overfitting. ψ(c) applies a slight penalty to the loss function if the action state assigned to the expert sample has a small loss value, and a larger penalty otherwise. Thus, in this invention, the objective function of inverse reinforcement learning can be expressed as:

[0115]

[0116] Further, taking the form of the expectation of expert data, ψ(c) can be expressed as:

[0117]

[0118] in

[0119]

[0120] And define the occupancy of the strategy as:

[0121]

[0122] Where γ is the decay factor in the Markov decision process. Occupancy can be interpreted as the distribution of state-action pairs encountered by the agent when interacting with the environment under policy π. Therefore, the expected decay of the reward for the trajectory generated by interacting with the environment under different policies can be expressed as:

[0123]

[0124] Therefore, it can be proven that first recovering a loss function through inverse reinforcement learning, and then learning a policy using reinforcement learning, can be expressed as:

[0125]

[0126] Where λ is the weight of the policy entropy H(π); It is the convex conjugate form of ψ(c); ρ π and It is strategy π and expert strategy π E Occupancy rate.

[0127] In GAIL, the loss function is set as follows:

[0128] c(s,a)=log(D(s,a))

[0129] Where D:S×A→(0,1) is a discriminator. c(s,a) provides the reward signal for updating the agent's policy. It can be further proven that:

[0130]

[0131]

[0132] Ultimately, the expert policy-guided learning method based on Generative Adversarial Imitation Learning (GAIL) can be summarized as solving the saddle point in the above equation.

[0133] Wherein, the discriminator D(s,a) is minimized The form is used to train the expert policy to sample trajectories τ E The state-action pair (s,a)~τ E And the agent generates the trajectory τ agent The state-action pair (s,a)~τ agent Distinguish; the generator (i.e., the agent's policy π) maximizes E π [log(D(s,a))] causes the discriminator to "misjudge" the agent's state-action pair as an expert-sampled trajectory state-action pair; Let γ be the decaying causal entropy of the strategy.

[0134] Reference Figure 7 ,like Figure 7The diagram illustrates the expert policy-guided learning method for adversarial imitation learning in this embodiment. In each new iteration: (1) the generator interacts with the environment using the current sensor agent policy to obtain the agent's generated trajectory; (2) the agent's generated trajectory and the demonstration trajectory are input into the discriminator and the discriminator parameters are updated in a supervised learning manner; (3) the updated discriminator outputs a new discrimination reward function; (4) the updated reward function is used to provide a reward signal to further update the agent policy (i.e., the generator). The above steps are repeated continuously, and the generator and discriminator continuously optimize their respective performance through adversarial training until they learn the ideal policy.

[0135] This embodiment addresses the difficulty of breaking down the localization and tracking task. Based on the concept of hierarchical reinforcement learning, it guides the agent's learning through a reward-based shaping process. Specific reward methods are shown in Table 2.

[0136]

[0137] Table 2 Summary of Rewards

[0138] Rewards 1-7 are artificially designed, set around the balance between signal discovery, target location, and control / detection. The specific reward values ​​and calculation processes will be continuously adjusted during actual testing. Reward 8 is the final mode reward, calculated based on the overall signal detection performance evaluation. Reward 9 is special, a curiosity reward. From the perspective of the agent's exploration of the state space in reinforcement learning, the following additional reward and penalty content is provided to enable the agent to better search the state space. Traditional reinforcement learning algorithms have extremely low sample utilization in environments with sparse feedback, resulting in slow learning speeds and difficulty in convergence. In this invention, sensors in different regions are in a signal search state in most states; only when a target signal is scanned can a corresponding positive reward be obtained. Furthermore, in some scenarios, there exists a set of states with a high transition probability, but a low transition rate between these states and other states. Therefore, a curiosity reward is designed to provide additional rewards for states with low exploration rates, incentivizing exploration of unknown spaces or preventing some spaces from remaining unexplored.

[0139] Specifically, design rewards and punishments r(S) known ,S novel As a curiosity reward function, this function rewards the agent for exploring unknown states based on the currently explored and unexplored state spaces. Firstly, during agent training, a dedicated memory storage module is designed to record each state s during the training process. t Probability of occurrence: φ(s) t;m), where m represents the parameters of the memory storage module, and the parameters of the memory storage network are updated after training. When curiosity reward is enabled, a corresponding reward or penalty is given each time the agent makes an action and transitions to a new state: Asign(k-φ(s) t ,m -l )), where A is the reward weight; k is the reward threshold. When the corresponding state transition probability is greater than k, a penalty is imposed, otherwise a reward is imposed.

[0140] Step 3: The trained reinforcement learning agent model collaboratively controls multiple sensors.

[0141] By loading different intelligent agent models through the task planning module, different sensor collaboration schemes are provided to achieve collaborative control of multiple sensors. In complex multi-radiation source environments, by executing different working modes (broadband scanning, fixed frequency control and guarding, direction finding and positioning), the real-time detection and continuous localization and tracking of radiation source targets can be achieved.

[0142] The multi-sensor intelligent collaborative control method provided in this embodiment has the capability to process complex signal situations. Addressing the need for locating and tracking key radiation source signals in complex electromagnetic spectrum environments, it combines deep learning and reinforcement learning to achieve comprehensive processing of the overall global situation (target situation, signal situation, and global operational status) and the status of individual sensors. By fine-grained settings of the action spaces of different sensors and the internal representations of individual actions, and by comprehensively utilizing reinforcement learning techniques and generative adversarial learning techniques based on expert knowledge, it achieves continuous location and tracking of key signals in complex electromagnetic spectrum environments. It possesses the capability to capture and locate short burst signals and has a certain degree of continuous location and tracking capability.

[0143] Example 2

[0144] Reference Figure 8 ,like Figure 8 The diagram shown is a structural block diagram of the multi-sensor intelligent collaborative control device provided in this embodiment. The device specifically includes:

[0145] The agent model building module builds a reinforcement learning agent model for each sensor. The state space of the reinforcement learning agent model includes a global comprehensive situational expression and a single sensor state embedding representation. The action space of the reinforcement learning agent model includes action output values ​​abstracted from different tasks performed by multiple sensors.

[0146] The agent model training module trains the reinforcement learning agent model through sampling and guides the learning of the reinforcement learning agent model through reward shaping engineering. The training includes centralized training and decentralized execution.

[0147] The sensor control module uses a trained reinforcement learning agent model to collaboratively control multiple sensors.

[0148] The multi-sensor intelligent collaborative control device provided in this embodiment achieves collaborative control of multiple sensors by establishing a reinforcement learning agent model. This enables multiple sensors to process complex signal situations in complex working environments. Utilizing reinforcement learning techniques and generative adversarial learning techniques based on expert knowledge, it achieves continuous localization and tracking of key signals in complex electromagnetic spectrum environments. It can control multiple sensors across regions to asynchronously execute their respective tasks, possessing the ability to capture and locate short burst signals, and exhibiting a certain degree of continuous localization and tracking capability.

[0149] Example 3

[0150] This preferred embodiment provides a computer device that can implement the steps of any embodiment of the multi-sensor intelligent cooperative control method provided in this application. Therefore, it can achieve the beneficial effects of the multi-sensor intelligent cooperative control method provided in this application. For details, please refer to the previous embodiments, which will not be repeated here.

[0151] Example 4

[0152] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the multi-sensor intelligent cooperative control method provided by the present invention.

[0153] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0154] Since the instructions stored in the storage medium can execute the steps in any of the multi-sensor intelligent cooperative control method embodiments provided by the present invention, the beneficial effects that any of the multi-sensor intelligent cooperative control methods provided by the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0155] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-sensor intelligent collaborative control method, characterized in that, The method includes: Establish a reinforcement learning agent model for each sensor. The state space of the reinforcement learning agent model includes a global integrated situational representation and a single sensor state embedding representation. The action space of the reinforcement learning agent model includes action output values ​​abstracted from different tasks performed by multiple sensors. The reinforcement learning agent model is trained by sampling and guided to learn through a reward shaping process. The training includes centralized training and decentralized execution. The reward shaping process specifically guides the learning of the reinforcement learning agent model by setting artificially designed rewards, endgame rewards, and curiosity rewards. The artificially designed rewards include corresponding reward values ​​given when a preset task is completed or when a task is failed. The endgame rewards include rewards given based on the overall signal detection performance. The curiosity rewards include rewards given when an unknown space is explored. The reinforcement learning agent model obtained through training coordinates the control of multiple sensors.

2. The multi-sensor intelligent collaborative control method as described in claim 1, characterized in that, The specific steps of training the reinforcement learning agent model through sampling include: Multiple sampling threads are started to perform independent sampling in multiple parallel simulation environments or parallel real environments with different configuration scenarios; The sampled data is uniformly placed into the sampling experience cache pool; When the training conditions are met, the sampled data is retrieved from the sampled experience cache pool and the agent model is updated by centralized training according to the corresponding reinforcement learning algorithm. Then, the updated model parameters are put into the model parameter cache pool.

3. The multi-sensor intelligent collaborative control method as described in claim 1, characterized in that, The confidence region policy optimization method used when establishing the reinforcement learning agent model specifically includes: The confidence region policy optimization algorithm is simplified using a first-order approximation, and the simplified algorithm is as follows: ； The corresponding constraints are: ； Among them For the new strategy; This is the old strategy; State; For action, The advantage function of the old strategy; State action value function: State value function: Advantage function: in, As the attenuation factor, This represents the average KL divergence between the old and new strategies; The simplified confidence region policy optimization algorithm is approximated using the Monte Carlo method, resulting in... ； make The ratio of the old and new strategies is used to obtain... ； The constraints of the confidence region policy optimization algorithm are approximated as follows: ,in for The coefficients then allow the constrained confidence region policy optimization algorithm's objective function to be expressed as an unconstrained objective function: ； Adding the objective function of the state-value function and the entropy of the policy model to the unconstrained objective function yields the complete objective function: ； in, , , These are the coefficients for the corresponding pre-set items.

4. The multi-sensor intelligent collaborative control method as described in claim 1, characterized in that, The centralized training and decentralized execution specifically include: A central controller collects the global state of all agents and makes unified decisions. Each sensor asynchronously executes its own task based on its current state.

5. The multi-sensor intelligent collaborative control method as described in claim 1, characterized in that, The method also includes guiding the learning of the reinforcement learning agent model with an expert policy based on generative adversarial imitation learning.

6. The multi-sensor intelligent collaborative control method as described in claim 5, characterized in that, The expert strategy based on generative adversarial imitation learning guides the learning of the reinforcement learning agent model, specifically including: Repeat the following steps until the optimal strategy is obtained: The trajectory of the agent is generated by interacting with the environment using the reinforcement learning agent model corresponding to the current sensor. The agent generates both the trajectory and the demonstration trajectory, which are then input into the discriminator, and the discriminator parameters are updated using supervised learning. The updated discriminator outputs a new discriminant reward function; The updated reward function is used to provide reward signals to further update the agent's policy.

7. A multi-sensor intelligent collaborative control device, characterized in that, The device includes: The agent model building module builds a reinforcement learning agent model for each sensor. The state space of the reinforcement learning agent model includes a global comprehensive situational expression and a single sensor state embedding representation. The action space of the reinforcement learning agent model includes action output values ​​abstracted from different tasks performed by multiple sensors. The intelligent agent model training module trains the reinforcement learning intelligent agent model through sampling and guides its learning through a reward shaping process. The training includes centralized training and decentralized execution. The reward shaping process specifically guides the learning of the reinforcement learning intelligent agent model by setting artificially designed rewards, end-game rewards, and curiosity rewards. The artificially designed rewards include corresponding reward values ​​given when a preset task is completed or when a task is failed. The end-game rewards include rewards based on overall signal detection performance. The curiosity rewards include rewards given when exploring unknown spaces. A sensor control module that collaboratively controls multiple sensors through a trained reinforcement learning agent model.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, which is loaded and executed by the processor to implement the multi-sensor intelligent collaborative control method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by a processor to implement the multi-sensor intelligent collaborative control method as described in any one of claims 1 to 6.

Images