Sea-air multi-target multi-sensor scheduling method based on deep reinforcement learning

Through the multi-sensor scheduling method of deep reinforcement learning, the problem that traditional technology in sea-air multi-target collaborative detection is difficult to detect electromagnetic silent targets, and efficient target detection and task completion in dynamic environments is achieved, ensuring platform security and resources Utilization efficiency.

CN118940629APending Publication Date: 2024-11-12SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411007173.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the coordinated detection of sea and air multi-targets, traditional single-platform sensors are difficult to detect large water surface targets in electromagnetic silence state, and the unknown dynamic system model leads to uncertainty in sensor scheduling parameters, which cannot effectively solve the multi-target under unknown system model. Sensor scheduling issues.

Method used

Adopt the multi-target multi-sensor scheduling method based on deep reinforcement learning, and generates a scheduling strategy through deep Q network (DQN) training, combining the value of target detection and platform security optimization indicators, designing state space and reward functions to achieve model-free sensor scheduling , ensure platform security and task completion rate.

Benefits of technology

It significantly improves the value of target detection and task completion rate, reduces the amount of computing, realizes effective sensor scheduling under unknown conditions of the system model, and ensures platform security and resource utilization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118940629A_ABST
    Figure CN118940629A_ABST
Patent Text Reader

Abstract

A sea-air multi-target multi-sensor scheduling method based on deep reinforcement learning comprises the following steps: in an offline training stage, constructing an environment feature state sk through feature extraction, training according to a deep Q network (DQN), obtaining all Q values about the state, and selecting a distribution action ak by using an epsilon-greedy strategy; the environment gives an effectiveness evaluation reward rk and a next observed value according to the distribution action ak, namely a state sk + 1; in the test stage, the trained deep Q network is adopted to carry out the same environment feature state construction process, and finally the optimal sensor-target scheduling action is output, so that the Q-factor is maximized. According to the method, the scheduling strategy is generated through experience-driven autonomous learning framework and environment interactive training, and the detection efficiency of the sensor can be improved on the premise of ensuring the platform safety.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of multi-sensor scheduling, specifically a sea and air multi-target multi-sensor scheduling method based on deep reinforcement learning. Background Art

[0002] The existing collaborative detection mode for multiple targets in the sea and air mainly studies the single same-platform collaboration and mostly studies the information processing methods. It is mainly based on the synthetic aperture radar (SAR) images carried by air-based and space-based platforms. However, due to the ocean environment full of waves, sea fog and other complex factors, traditional single-platform sensors have difficulty in discovering and detecting large surface targets in an electromagnetically silent state. Summary of the invention

[0003] In view of the fact that the existing sensor scheduling method based on dynamic system models will bring parameter uncertainty in the dynamic environment of collaborative detection of large surface targets, resulting in the defect of unknown system model, and the multi-sensor scheduling problem in the existing collaborative detection tasks when the system model is unknown, the present invention proposes a sea and air multi-target multi-sensor scheduling method based on deep reinforcement learning. The scheduling strategy is generated through an experience-driven autonomous learning framework and environmental interactive training, which can improve the sensor detection efficiency while ensuring the safety of the platform.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a method for scheduling multi-targets and multi-sensors in the sea and air based on deep reinforcement learning. In the offline training stage, feature extraction is used to construct the environment feature state s k , after training the deep Q network (DQN) and obtaining all the Q values ​​about the state, the ε-greedy strategy is used to select the assigned action a k ; The environment is assigned action a k Give the effectiveness evaluation reward r k and the next observation, i.e., state s k+1 ; In the testing phase, the trained deep Q network is used to perform the same environmental feature state construction process, and the final output is the optimal sensor-target scheduling action to maximize the Q-factor.

[0006] The deep Q network includes: an estimation network and a target network, wherein: the estimation network fits an estimated value function through a neural network according to the target state and current action information to obtain a current state value estimation result; the target network fits a target value function through a neural network according to the target state information to obtain a current optimal action state value target value result.

[0007] The estimation network includes: a signal measurement unit and a state value estimation unit, wherein: the signal measurement unit performs discretization processing according to the target motion state and the sensor scheduling action information to obtain the target state and scheduling action measurement results; the state value estimation unit performs neural network estimation value function fitting processing according to the target state and scheduling action measurement information to obtain the current state value estimation result.

[0008] The target network includes: a signal measurement unit and an action state value estimation unit, wherein: the signal measurement unit performs discretization processing according to the target motion state information to obtain a target state measurement result, and the action state value estimation unit performs a neural network to perform target value function fitting processing according to the target motion state information to obtain the current optimal action state value target value result.

[0009] The estimation network calculates the gradient of the loss function through the optimizer to achieve parameter update.

[0010] The target network achieves parameter update by periodically copying the parameters of the estimation network.

[0011] In the offline training phase, preferably, small samples are randomly extracted from the buffer for training, thereby improving learning efficiency. Technical Effects

[0012] The present invention significantly reduces the amount of calculation through the discrete state vector composed of target detection value ranking index and cumulative radiation statistics, models sensor scheduling as a Markov decision process, designs state space and reward function in combination with target detection value and platform safety optimization index, and uses a model-free deep Q-network algorithm to train the sensor scheduling strategy, thereby achieving sensor scheduling under the condition of unknown system model, and improving target detection value and task completion rate while ensuring platform safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the present invention;

[0014] Figure 2 This is a diagram of the DQN training process;

[0015] Figure 3 This is a diagram of the DQN-based sensor scheduling training results;

[0016] Figure 4 Training scene graphs for multiple targets and platforms;

[0017] Figure 5 It is the relative position trajectory diagram of multiple targets and platform 1;

[0018] Figure 6 It is the relative position trajectory diagram of multiple targets and platform 2;

[0019] Figure 7 It is the relative position trajectory diagram of multiple targets and platform 3;

[0020] Figure 8 Sensor scheduling diagram for three platforms based on learning;

[0021] Fig. 9 This is a kernel density diagram of target detection value comparing the present invention with other strategies. DETAILED DESCRIPTION

[0022] like Figure 1 As shown, this embodiment relates to a sea and air multi-target multi-sensor scheduling method based on deep reinforcement learning, including:

[0023] Step 1: Construct a sensor scheduling action model: N platforms (each equipped with an active sensor) perform detection in the mission area, and M targets are avoiding platform detection in their own movement forms, then the sensor scheduling action Where: i = 1, 2, ..., N, j = 1, 2, ..., M, or 0 indicates whether the sensor of platform i is assigned to target j at time k.

[0024] The constraints of the sensor scheduling action model include: 1) Each sensor can be assigned to the same target at most once at time k, that is, 2) Each sensor can be assigned at most L different targets at the same time at time k, that is, 3) When target j is within the detection range of platform i at time k, that is, Then the sensor will d (i,j) detects the target. Only when platform i detects target j at time k can the allocation action be performed, where: is the distance between target j and sensor i at time k, r i and p d (i, j) is the detection radius and detection efficiency of the sensor.

[0025] Step 2: Design optimization indicators for heterogeneous targets: Divide the target detection value into tactical value and strategic value, that is, the four target detection value influencing factors, namely, the tactical value is calculated based on the target's relative platform distance, speed and heading, and the strategic value is calculated based on the target type. The analytic hierarchy process is used to determine the weights of tactical value and strategic value, so as to scientifically estimate the target detection value, including:

[0026] 2.1) Calculate the target tactical value index: Calculate the distance value function, speed value function and heading value function of target j relative to platform i respectively, and use four labels: absolutely high, high, low or no value at all to distinguish the value level, including:

[0027] a) When the target is within the detection range of the platform, for example, the target is very close or very far from the platform, the distance value level label is absolutely high or completely absent, and the value is assigned to 1 or 0. When the distance is between the maximum and minimum values ​​of the value level threshold, the value level label is high or low, and a function is used to map it to the distance value label, and a quadratic function is used to construct the distance value function in: is the distance between target j and platform i, d m (m=1,2,3) is the distance threshold between the target and the platform, corresponding to the threshold of the value level, and ρ1 and ρ2 are the value weights.

[0028] b) When the target speed is extremely high, its speed value level label is absolutely high, and the value is close to 1. When the target speed is between the maximum and minimum values ​​of the value level threshold, the value increases with the speed. A function is used to map it to the speed value label, and an exponential function is used to construct the speed value function. in: is the velocity of target j, V max is the upper limit threshold of the target speed value label, and ρ3 is the curvature adjustment factor of the speed value function.

[0029] c) When the platform is not within the target heading range, its heading value tag is completely absent, and the heading value is assigned 0. When the platform is within the target heading range, the more the target heading is biased toward the platform, the higher the heading value is. A function is used to map the heading value tag, and a quadratic function is used to construct the heading value function. in: is the heading angle between target j and platform i, α max is the upper threshold of the target heading value label.

[0030] 2.2) Calculate the target strategic value: Among them: ω1, ω2, ω3 and ω4 are the weights of the value influencing factors, and the fuzzy factors are used to construct the target type value function I is the target type.

[0031] Step 3: Construct a Markov decision process quadruple (S, A, P, r) for the sensor scheduling problem, where S is the state space, A is the action space that can be taken, and P is the transition probability, that is, P(s k ,s k+1 ; a k ) is in state s k Take action a k When the state is transferred to s k+1 The probability of k is a single-step reward function, which includes:

[0032] 3.1) Construct state space: According to the actual needs of collaborative detection of multiple targets in the sea and air, construct the target detection value optimization index and platform survival optimization index, specifically: state vector Among them: Target detection value ranking index It is obtained by the following method: Calculate the value of all targets as V k The index number of each element in is recorded as The resulting index number permutation set is the first part of the state space, where: Indicates that at each moment k, the corresponding labels are sorted in descending order of target detection value, and the target detection value sorting index The dimension is The second part of the state space cumulative radiation statistics It is obtained by calculating the cumulative risk value of each sensor i to each target j, which is in: is the cumulative value of the electromagnetic radiation of target j to platform i at time k, and is quantified as a random variable in a finite discrete state Cumulative radiation statistics The dimension is (E max ) NM .

[0033] 3.2) Constructing the action space: multi-sensor scheduling action Its dimension is {0,1} NM ,in: or 0 indicates whether the sensor of platform i is assigned to target j at time k.

[0034] 3.3) Constructing a single-step reward function: The feedback for evaluating the execution effect of the multi-sensor scheduling action is a single-step reward. At time k, the evaluation of the multi-sensor scheduling action by the detected target detection value and the sensor cumulative radiation is: Among them: a1, b1, c>0, a2, b2<0 are constants. Here each reward is a real number, positive numbers indicate encouragement, and negative numbers indicate discouragement. At time k, the reward function is constructed as Where: V ,ω E is the weight of each part of the index and satisfies ω T +ω E =1.

[0035] Step 4: DQN-based scheduling strategy selection, including:

[0036] 4.1) Evaluate the quality of a state s or a state-action pair (s, a) through the value function, and use the optimal action value Q * (s k ,a k ) means in state s k Take action k , the maximum expected return is: Where: π is the strategy for mapping sequences to actions, and β is the discount coefficient. When β is close to 1, the decision becomes long-sighted; conversely, when β is close to 0, the decision becomes short-sighted.

[0037] 4.2) Compare and determine the optimal strategy, specifically: When at the next moment, the sequence s k+1 The optimal value of Q * (s k+1 ,a k+1 ) For all possible actions a k+1 are all known, then the optimal strategy is to choose an action a k+1 , so that the expected value r k+1 +βQ * (s k+1 ,a k+1 ) is the largest, where: Bellman equation ε is the system dynamics, and the optimal strategy π is obtained by iteratively updating the action-value function of the Bellman equation * .

[0038] The action-value function is solved by function approximation through a deep neural network. Specifically, the neural network transforms the state s k , and for each possible action a k , output Q(s k ,a k ,θ t ) to achieve Q(s k ,a k ,θ t )≈Q * (s k ,a k ).

[0039] 4.3) Train DQN by minimizing a sequence of loss functions that change over time, specifically: Where: θ l is the parameter of the Q network at iteration ι, θ′ l It is the network parameter used to calculate the target during iteration. Only Q network parameter θ is updated every C steps. l , remains unchanged between updates, where the weights θ lUse samples for updating, that is, at each time period, use the ε-greedy method to select actions.

[0040] Preferably, experience replay is used to perform single-step small sample gradient descent, specifically: in the data set D k ={e1,...,e k}For each time step k, the previous K experiences e k =(s k ,a k ,r k ,s k+1 ), which is a small sample randomly drawn from the experience replay D.

[0041] After simulation experiments, the processor in the simulation experiment is 12th Gen Intel(R)Core(TM)i7-12700F, and the operating system is win11 system. The development tool used is PyCharm 2022.2.3. And the mainstream environmental simulation method is adopted to simulate the collaborative detection of multi-target multi-sensor scheduling problems in the sea and air.

[0042] The simulation experiment scenario is constructed as a 500km×500km water surface. The Red side performs a coordinated detection mission, the Blue side's two bombers move back and forth, and a ship sails in the southwest, southeast, and southwest directions in turn. The Red side's two reconnaissance aircraft (each equipped with a radar with a detection range of 300km) make regional circular motions, and a ship sails in a straight line (equipped with a radar with a detection range of 400km) for detection. The sensor resources are limited and can be allocated to a maximum of 2 targets in each detection cycle.

[0043] By randomly initializing the platform and target positions, a variety of scenarios are generated for 5,000 offline trainings, each of which contains 100 time steps. During the training process, each training is to execute a complete sensor scheduling process, including 100 sensor scheduling actions. Based on the constructed MDP reward function, the total reward changes of 5,000 offline trainings are plotted, as shown in the figure below: Figure 3 As shown in the figure. The horizontal axis represents the number of training times, and the vertical axis represents the total benefit of a single complete sensor scheduling. As can be seen from the figure, with the increase in the number of training times, the total benefit of sensor scheduling shows an upward trend. At 2100 training times, the training curve converges to about 165, and the offline training is successfully completed.

[0044] In the above experiment, a multi-sensor scheduling strategy based on deep reinforcement learning is used to select sensor scheduling actions. A complete test platform and target test trajectory are shown in Figure 2. Figure 4 As shown in the figure, the relative position trajectory of multiple targets and the platform is as follows Figure 5 , Figure 6 and Figure 7The sensor scheduling results are shown in Figure 8 As shown in the figure, the scheduling results of the three platforms (Agents) are shown. The horizontal axis represents the simulation time step, and the vertical axis represents the target number (No. 1 and No. 2 are bomber targets, and No. 3 is a large surface target). The black circle indicates that the platform detects the target at that moment. Combining the relative position diagram and the scheduling result diagram, it can be found that the three platforms dynamically schedule the targets in the area in real time during the collaborative detection mission. Under the condition of limited detection resources, they complete the continuous detection of all targets in the area, and at the same time ensure that the cumulative radiation value of any single target is within a given range, that is, meet the platform survival index requirements, reduce the platform's mission risk, and improve the platform's mission safety. In addition, target No. 3 is scheduled to have the most detection times, which meets the requirement of giving priority to scheduling resources for targets with high detection value, so as to fully obtain target information.

[0045] The traditional multi-sensor scheduling method mentioned in the problem modeling only considers the maximization of information advantage, which is not suitable for the multi-sensor scheduling goal of the present invention and loses the meaning of comparison. Therefore, the random strategy and polling scheduling strategy are selected to compare with the scheduling strategy based on deep reinforcement learning proposed in the present invention.

[0046] Random strategy: The sensors are randomly scheduled to detect the target each time. Under this strategy, multiple sensors can be assigned at each moment. In principle, relatively more measurements can be obtained, but it will cause the sensor to switch targets continuously, the cumulative number of continuous detections is small, the platform survival index is improved, and the target detection value cannot be guaranteed.

[0047] Polling scheduling strategy: The sensors are scheduled in turns to detect the target using a fixed division of labor strategy, that is, the sensors are periodically scheduled to the target. The sensor accumulates a large number of continuous detections, and the platform survival index cannot be guaranteed.

[0048] The target detection value index and task completion rate are selected as the measurement criteria, namely Where: N Com is the number of times the multi-sensor scheduling task is completed, N MC is the number of Monte Carlo trials.

[0049] 1) Target detection value comparison

[0050] Conduct 100 Monte Carlo simulation experiments, and count the target detection values ​​obtained from the scheduling results of each time step in each test to obtain a probability density diagram as shown in the figure below: Fig. 9As shown. The horizontal axis represents the target detection value of a single time step, and the vertical axis represents the kernel density of the scheduling action that generates the target detection value. The red, green, and blue curves are the target detection value kernel density curves of the RL scheduling strategy, the random scheduling strategy, and the polling scheduling strategy, respectively. Because it is a kernel density curve, the absolute value of the vertical axis can only represent the probability of the target detection value under a certain strategy, and cannot represent the probability of the target detection value of the two strategies. For example, when the horizontal axis is 0.4, the kernel density of the random scheduling strategy and the polling scheduling strategy is 3.4 and 1.6. Due to the inconsistency of the horizontal axis range, it cannot be said that the probability of the random scheduling strategy is higher than that of the polling scheduling strategy. When the horizontal axis is 0.4 and 0.6, the kernel density of the random strategy is 3.4 and 1.4, indicating that when using the random strategy, the probability of the target detection value of 0.4 obtained by the scheduling action taken is higher than the probability of 0.6.

[0051] Depend on Figure 8 It can be seen that, using the RL strategy of the present invention, the target detection value obtained by executing the scheduling action at all time steps is greater than 0.4, and is mainly concentrated around 0.7. Using the random scheduling strategy, the target detection value obtained is only 0.8 at most, and is mainly concentrated around 0.4, which is significantly lower than the RL strategy; while using the polling scheduling strategy, the target detection value obtained is more evenly distributed between 0.1 and 0.65, which is lower than the RL strategy as a whole. Therefore, the target detection value brought by the multi-sensor scheduling using the RL strategy is better than the other two strategies, and has higher stability.

[0052] 2) Comparison of task completion rate

[0053] 100 Monte Carlo simulation experiments were conducted, and the target detection value and task completion rate obtained from all scheduling results in each test were statistically analyzed, and the test comparison results are shown in Table 1. The DRL scheduling strategy can maintain high platform survivability and ensure a high task completion rate while having high target detection value in collaborative detection tasks.

[0054] Table 1. Comparison of test results of various strategies index DRL Scheduling Random Scheduling Polling Scheduling Target detection value 76.51 41.13 38.56 Task completion rate 93% 45% 31%

[0055] Compared with the random scheduling strategy and polling scheduling strategy in the prior art, the present invention adopts a model-free deep Q-network algorithm to effectively solve the problems of unknown dynamic system models and limited sensor resources. It is significantly superior to the existing traditional technologies in terms of target detection value and task completion rate. It realizes a multi-sensor scheduling method that can ensure the security of the platform and make full use of the sensor detection efficiency, and can well solve the multi-sensor scheduling problem in dynamic unknown scenarios.

[0056] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multi-target and multi-sensor scheduling method for sea and air based on deep reinforcement learning, It is characterized in that In the offline training phase, the environment feature state s is constructed by feature extraction. k , after training the deep Q network (DQN) and obtaining all the Q values ​​about the state, the ε-greedy strategy is used to select the assigned action a k ; The environment is assigned action a k Give the effectiveness evaluation reward r k and the next observation, i.e., state s k+1 ; In the testing phase, the trained deep Q network is used to perform the same environmental feature state construction process, and the final output is the optimal sensor-target scheduling action to maximize the Q-factor.

2. According to the deep reinforcement learning-based sea and air multi-target multi-sensor scheduling method of claim 1, Its characteristics are: The deep Q network includes: an estimation network and a target network, wherein: the estimation network performs estimation value function fitting through a neural network according to the target state and current action information to obtain the current state value estimation result; the target network performs target value function fitting through a neural network according to the target state information to obtain the current optimal action state value target value result; The estimation network includes: a signal measurement unit and a state value estimation unit, wherein: the signal measurement unit performs discretization processing according to the target motion state and the sensor scheduling action information to obtain the target state and scheduling action measurement results, and the state value estimation unit performs neural network estimation value function fitting processing according to the target state and scheduling action measurement information to obtain the current state value estimation result; The target network includes: a signal measurement unit and an action state value estimation unit, wherein: the signal measurement unit performs discretization processing according to the target motion state information to obtain a target state measurement result, and the action state value estimation unit performs a neural network to perform target value function fitting processing according to the target motion state information to obtain the current optimal action state value target value result.

3. According to claim 2, the sea and air multi-target multi-sensor scheduling method based on deep reinforcement learning, Its characteristics are: The estimation network calculates the gradient of the loss function through the optimizer to achieve parameter update; The target network achieves parameter update by periodically copying the parameters of the estimation network.

4. The sea and air multi-target multi-sensor scheduling method based on deep reinforcement learning according to claim 1, Its characteristics are: In the offline training phase, small samples are further randomly extracted from the buffer for training.

5. The sea and air multi-target multi-sensor scheduling method based on deep reinforcement learning according to any one of claims 1 to 4, Its characteristics are that include: Step 1: Construct a sensor scheduling action model: N platforms (each equipped with an active sensor) perform detection in the mission area, and M targets are avoiding platform detection in their own movement forms, then the sensor scheduling action Where: i = 1, 2, ..., N, j = 1, 2, ..., M, or 0 indicates whether the sensor of platform i is assigned to target j at time k; The constraints of the sensor scheduling action model include: 1) Each sensor can be assigned to the same target at most once at time k, that is, 2) Each sensor can be assigned at most L different targets at the same time at time k, that is, 3) When target j is within the detection range of platform i at time k, that is, Then the sensor will d (i,j) detects the target. Only when platform i detects target j at time k can the allocation action be performed, where: is the distance between target j and sensor i at time k, r i and p d (i, j) is the detection radius and detection efficiency of the sensor; Step 2: Design optimization indicators for heterogeneous targets: Divide the target detection value into tactical value and strategic value, that is, calculate the tactical value based on the target's relative platform distance, speed and heading, and calculate the strategic value based on the target type. Use the hierarchical analysis method to determine the weights of tactical value and strategic value, so as to scientifically estimate the target detection value. Step 3: Construct a Markov decision process quadruple (S, A, P, r) for the sensor scheduling problem, where S is the state space, A is the action space that can be taken, and P is the transition probability, that is, P(s k ,s k+1 ; a k ) is in state s k Take action a k When the state is transferred to s k+1 The probability of k is a single-step reward function, which includes: Step 4: Scheduling strategy selection based on DQN.

6. The method for scheduling multi-targets and multi-sensors at sea and in the air based on deep reinforcement learning according to claim 5, Its characteristics are: The step 2 specifically includes: 2.1) Calculate the target tactical value index: Calculate the distance value function, speed value function and heading value function of target j relative to platform i respectively, and use four labels: absolutely high, high, low or no value at all to distinguish the value level; 2.2) Calculate the target strategic value: Where: 1 ,ω 2 ,ω 3 and ω 4 is the weight of the value influencing factor, and the fuzzy factor is used to construct the target type value function I is the target type.

7. The method for scheduling multi-targets and multi-sensors at sea and in the air based on deep reinforcement learning according to claim 6, Its characteristics are: The value levels mentioned include: a) When the target is within the detection range of the platform, when the target is very close or very far from the platform, the distance value level label is absolutely high or completely absent, and the value is assigned to 1 or 0. When the distance is between the maximum and minimum values ​​of the value level threshold, the value level label is high or low, and a function is used to map it to the distance value label. The distance value function is constructed using a quadratic function. in: is the distance between target j and platform i, d m (m=1,2,3) is the distance threshold between the target and the platform, corresponding to the threshold of the value level, ρ1 and ρ2 are the value weights; b) When the target speed is extremely high, its speed value level label is absolutely high, and the value is close to 1. When the target speed is between the maximum and minimum values ​​of the value level threshold, the value increases with the speed. A function is used to map it to the speed value label, and an exponential function is used to construct the speed value function. in: is the velocity of target j, V max is the upper threshold of the target speed value label, ρ3 is the curvature adjustment factor of the speed value function; c) When the platform is not within the target heading range, its heading value label is completely absent and the heading value is assigned 0. When the platform is within the target heading range, the more the target heading is biased towards the platform, the higher the heading value is. A function is used to map the heading value label and a quadratic function is used to construct the heading value function. in: is the heading angle between target j and platform i, α max is the upper threshold of the target heading value label.

8. The method for scheduling multi-targets and multi-sensors at sea and in the air based on deep reinforcement learning according to claim 5, Its characteristics are: The step three specifically includes: 3.1) Construct state space: According to the actual needs of collaborative detection of multiple targets in the sea and air, construct the target detection value optimization index and platform survival optimization index, specifically: state vector Among them: Target detection value ranking index It is obtained by the following method: Calculate the value of all targets as V k The index number of each element in is recorded as The resulting index number permutation set is the first part of the state space, where: Indicates that at each moment k, the corresponding labels are sorted in descending order of target detection value, and the target detection value sorting index The dimension is The second part of the state space cumulative radiation statistics It is obtained by calculating the cumulative risk value of each sensor i to each target j, which is in: is the cumulative value of the electromagnetic radiation of target j to platform i at time k, and is quantified as a random variable in a finite discrete state Cumulative radiation statistics The dimension is (E max ) NM ; 3.2) Constructing the action space: multi-sensor scheduling action Its dimension is {0,1} NM ,in: or 0 indicates whether the sensor of platform i is assigned to target j at time k; 3.3) Constructing a single-step reward function: The feedback for evaluating the execution effect of the multi-sensor scheduling action is a single-step reward. At time k, the evaluation of the multi-sensor scheduling action by the detected target detection value and the sensor cumulative radiation is: Among them: a 1 ,b 1 ,c>0,a 2 ,b 2 <0 is a constant. Here, each reward is a real number. A positive number indicates encouragement, while a negative number indicates discouragement. At time k, the reward function is constructed as follows: Where: V ,ω E is the weight of each part of the index and satisfies ω T +ω E =1.

9. The method for scheduling multi-targets and multi-sensors at sea and in the air based on deep reinforcement learning according to claim 5, Its characteristics are: The step 4 specifically includes: 4.1) Evaluate the quality of a state s or a state-action pair (s, a) through the value function, and use the optimal action value Q * (s k ,a k ) means in state s k Take action k , the maximum expected return is: Where: π is the strategy for mapping sequences to actions, β is the discount coefficient, when β is close to 1, the decision becomes long-sighted; conversely, when β is close to 0, the decision becomes short-sighted; 4.2) Compare and determine the optimal strategy, specifically: When at the next moment, the sequence s k+1 The optimal value of Q * (s k+1 ,a k+1 ) For all possible actions a k+1 are all known, then the optimal strategy is to choose an action a k+1 , so that the expected value r k+1 +βQ * (s k+1 ,a k+1 ) is the largest, where: Bellman equation ε is the system dynamics, and the optimal strategy π is obtained by iteratively updating the action-value function of the Bellman equation * ; The action-value function is solved by function approximation through a deep neural network. Specifically, the neural network transforms the state s k , and for each possible action a k , output Q(s k ,a k ,θ t ) to achieve Q(s k ,a k ,θ t )≈Q * (s k ,a k ); 4.3) Train DQN by minimizing a sequence of loss functions that change over time, specifically: Where: θ l is the parameter of the Q network at iteration ι, θ′ l It is the network parameter used to calculate the target during iteration. Only Q network parameter θ is updated every C steps. l , remains unchanged between updates, where the weights θ l Use samples for updating, that is, at each time period, use the ε-greedy method to select actions.

10. The method for scheduling multi-targets and multi-sensors at sea and in the air based on deep reinforcement learning according to claim 9, Its characteristics are: Use experience replay to perform single-step small sample gradient descent, specifically: in the dataset D k ={e 1 ,…,a k }For each time step k, the previous K experiences e k =(s k ,a k ,r k ,s k+1 ), which is a small sample randomly drawn from the experience replay D.

Citation Information

Cited By

  • Multi-temperature-salinity-depth sensor cooperative detection task planning method

    CN119720815A

  • Mission planning method for collaborative exploration of multi-temperature, salt and depth sensors

    CN119720815B