Unmanned aerial vehicle cluster flight path deception method based on deep reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle cluster flight path deception method based on deep reinforcement learning. The method comprises the following steps: establishing a single unmanned aerial vehicle flight path deception model; establishing an unmanned aerial vehicle intelligent body; designing a deep reinforcement learning reward function; according to the method, a centralized training and distributed execution architecture is adopted, and an unmanned aerial vehicle cluster agent is trained by using a multi-agent dual-delay depth deterministic policy gradient algorithm (Multi-agent Twin Delayed Description Policy Gradient, MATD3). According to the invention, the unmanned aerial vehicle cluster is controlled by using multiple agents, and intelligent flight path deception of the unmanned aerial vehicle cluster is realized.

Description

Technical Field

[0001] The present invention relates to a drone cluster control technology, and in particular to a drone cluster track deception method based on deep reinforcement learning. Background Art

[0002] The emergence and application of networked radars have greatly improved the anti-interference characteristics of radar systems. Simple distance deception, angle deception, and speed deception are difficult to effectively interfere with enemy networked radars. As a comprehensive deception technology that combines the above deception methods, drone cluster track deception technology has received widespread attention. UAVs modulate and delay the intercepted enemy radar signals to make the enemy radar receive one or more false target signals that are farther than the actual distance of the target. Multiple continuous false targets with dynamic characteristics can form a false track to complete the track deception of the enemy radar. However, networked radars use the rank K criterion for information fusion. The false target needs to be detected by multiple radars at the same time to be identified as a real target. Therefore, when using the main lobe track deception, the drone needs to be in the line of sight (LOS) of each radar at all times to forward the deception signal. Therefore, it is necessary to reasonably design the drone track and deception signal so that the drone cluster can implement collaborative track deception.

[0003] At present, many scholars at home and abroad have conducted in-depth research on drone swarm trajectory deception technology. However, traditional algorithms mostly use optimization methods to plan drone trajectories. The intelligence level of drone swarms is low, making it difficult to cope with emergencies. In addition, the iterative optimization algorithm is time-consuming and it is difficult to achieve real-time optimal trajectory planning. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method for UAV cluster track deception based on deep reinforcement learning, so as to realize the intelligent and autonomous control of UAV cluster to implement track deception on networked radar.

[0005] Technical solution: The method for spoofing the track of a drone cluster based on deep reinforcement learning of the present invention comprises the following steps:

[0006] Establish a track deception model and deception signal model for a single UAV to a single radar;

[0007] Establish the observation space, state space and action space of the UAV agent, the strategy network, value network and corresponding target network of the UAV agent;

[0008] Based on the UAV dynamics performance constraints and track deception mission requirements, a deep reinforcement learning reward function is constructed, including the UAV acceleration reward function, the UAV altitude reward function, the UAV speed reward function, the UAV yaw angle reward function, the UAV pitch angle reward function, the UAV spacing reward function and the UAV agent total reward function;

[0009] A centralized training and distributed execution architecture is adopted, and the multi-agent double-delay deep deterministic policy gradient algorithm MATD3 is used to train the drone swarm agents to achieve autonomous control of drone swarms for track deception.

[0010] Furthermore, the track deception model of a single UAV to a single radar is expressed as:

[0011]

[0012] Among them, u k =[x u,k ,y u,k ,z u,k ] T and p k =[x p,k ,y p,k ,z p,k ] T denote the spatial coordinates of the UAV and the false target at time k respectively; r = [x r ,y r ,z r ] T Indicates the spatial coordinates of the drone deceiving the radar;

[0013] The deception signal model expression is:

[0014]

[0015] Where D(t) represents the deception signal; t represents time; A represents the amplitude of the radar transmission signal; △t represents the DRFM forwarding delay; f0 represents the center frequency of the radar transmission signal; f d represents the Doppler modulation parameter; μ represents the frequency modulation slope; T represents the pulse width.

[0016] Furthermore, the observation space expression of the UAV agent is:

[0017]

[0018] Among them, i is the observation space of the i-th UAV agent, represents the spatial coordinates of the i-th UAV at time k; represents the spatial coordinates of the false target generated by the i-th UAV at time (k+1); Represents the spatial coordinates of the radar deceived by the i-th drone.

[0019] Furthermore, the state space expression of the drone cluster is:

[0020]

[0021] Among them, s is the state space of the drone cluster, u i,k represents the spatial coordinates of the i-th UAV at time k; p l,k+1 represents the spatial coordinates of the lth false target at time k+1; r q Represents the spatial coordinates of the qth radar.

[0022] Furthermore, the action space expression of the UAV agent is:

[0023] a i =[△h i ]

[0024] Among them, a i is the action space of the i-th UAV agent, △h i represents the change in the flight altitude of the i-th UAV; the flight altitude expression of the i-th UAV at the next moment is:

[0025]

[0026] in, is the flight altitude of the i-th UAV at time k+1.

[0027] Furthermore, each drone agent contains a policy network, a target policy network, two value networks and two target value networks; the policy network and the target policy network contain two hidden layers, the activation functions of the input layer and the hidden layer are ReLU functions, and the activation function of the output layer is tanh function; the value network and the target value network also contain two hidden layers, the activation functions of the input layer and the hidden layer are ReLU functions, but no activation function is added to the output layer.

[0028] Furthermore, the drone acceleration reward function expression is:

[0029]

[0030] Among them, a u,min and a u,max Respectively represent the minimum and maximum acceleration of the drone; a u,k represents the acceleration of the UAV at time k;

[0031] The drone altitude reward function is a cosine roll-off function, and its expression is:

[0032]

[0033] Among them, h u,min and h u,max They represent the minimum and maximum flight altitude of the UAV respectively; h th,min =h u,min -2.5 and h th,max =h u,max +2.5 indicates the threshold at which the reward function reaches the minimum or maximum value; z u,k represents the flight altitude of the UAV at time k;

[0034] The drone speed reward function expression is:

[0035]

[0036] Among them, v u,min and v u,max Respectively represent the minimum and maximum speed of the drone; v u,k represents the flight speed of the UAV at time k;

[0037] The reward function of the drone’s yaw angle is:

[0038] J yew =10exp(-△α u,k 2 / 2)

[0039] Among them, △α u,k represents the yaw angle of the UAV at time k;

[0040] The reward function expression of the drone pitch angle is:

[0041] J pitch =10exp(-β u,k 2 / 2)

[0042] Among them, β u,k represents the pitch angle of the UAV at time k;

[0043] The reward function expression of the drone spacing is:

[0044]

[0045] Among them, d safe Indicates the safe distance between drones; represents the distance between UAVs u1 and u2 at time k;

[0046] The reward function expression of deep reinforcement learning is:

[0047] J=J a +Jh +J v +J yew +J pitch +J d

[0048] Among them, J a is the drone acceleration reward function, J h is the drone altitude reward function, J v is the drone speed reward function, J yew is the reward function for the yaw angle of the drone, J pitch is the reward function for the pitch angle of the drone, J d is the reward function for the distance between drones.

[0049] Furthermore, during the training phase, the value network and target value network of each drone can receive the action, status and reward information of all drones for network updates; the policy network and target policy network of each drone only update the network based on their own action, observation and reward information; during the execution phase, the policy network of each drone outputs actions based on its own observation information.

[0050] Furthermore, the multi-agent double-delay deep deterministic policy gradient algorithm MATD3 is used to train each neural network. The specific steps are as follows:

[0051] (1) Initialize the two value networks of drone agent i A strategic network And the corresponding target value network and strategy network Network parameters Initialize the number of training rounds N t , discount rate γ, delayed update round t d , learning rates η1 and η2 of the value network and policy network, soft update rate τ, noise variance σ, number of drones N u and experience replay pool;

[0052] (2) The observation space o of the i-th UAV agent i Get the action space with noise added The noise ε follows a normal distribution with a mean of 0 and a variance of σ;

[0053] (3) Execute the action and transform the current state space s of the drone cluster, the next state space s' of the drone cluster, and the observation space o of the i-th drone agent at the current moment i And the observation space o of the i-th UAV agent at the next moment i ' and reward J i , forming a tuple [o i ,s,a i ,oi ',s',J i ] is stored in the experience replay pool;

[0054] (4) M tuples are sampled from the experience pool to form a batch, and the target policy network calculates the action of each tuple And calculate the time difference target y for each tuple i :

[0055]

[0056] (5) Update two value network parameters:

[0057]

[0058] Among them, θ n,i and They represent the nth value network parameters before and after the update of the i-th UAV agent; η1 represents the value network learning rate; Represents about θ n,i The gradient of y i represents the time difference target of each tuple;

[0059] (6) If t mod t d =0, mod represents the remainder operation; then the deterministic policy gradient is used to update the policy network parameters:

[0060]

[0061] Among them, φ i and They represent the nth policy network parameters before and after the update of the i-th UAV agent; η2 represents the value network learning rate; and Respectively represent φ i and a i The gradient of

[0062] Perform a soft update on the target network:

[0063]

[0064] in, and They represent the nth target value network parameters before and after the update of the i-th UAV agent; and They represent the target strategy network parameters before and after the update of the i-th UAV agent; τ represents the soft update rate;

[0065] (7) Check whether the above operations are performed on each UAV agent: if satisfied, proceed to the next round of training; otherwise, return to step (2) and train the next UAV agent;

[0066] (8) Check whether the number of training rounds N has been reached t : If satisfied, the training ends; otherwise, return to step (2) for the next round of training;

[0067] (9) Output the policy network of all drone agents

[0068] The UAV swarm track deception system based on deep reinforcement learning of the present invention comprises:

[0069] A model building unit, used to build a track deception model and a deception signal model of a single UAV against a single radar;

[0070] The UAV intelligent agent establishment unit is used to establish the observation space, state space and action space of the UAV intelligent agent, the strategy network, value network and corresponding target network of the UAV intelligent agent;

[0071] A deep reinforcement learning reward function construction unit is used to construct a deep reinforcement learning reward function based on the UAV dynamic performance constraints and the track deception task requirements, including a UAV acceleration reward function, a UAV altitude reward function, a UAV speed reward function, a UAV yaw angle reward function, a UAV pitch angle reward function, a UAV spacing reward function, and a UAV agent total reward function;

[0072] The drone swarm agent training unit is used to train drone swarm agents using a centralized training and distributed execution architecture, using the multi-agent double-delay deep deterministic policy gradient algorithm MATD3 algorithm to achieve autonomous control of drone swarms for track deception.

[0073] Beneficial effects: Compared with the prior art, the significant advantages of the present invention are: (1) The present invention constructs a drone cluster intelligent agent and a deep reinforcement learning reward function, adopts a centralized training and distributed execution architecture, and uses the MATD3 algorithm to train the drone cluster intelligent agent, thereby achieving the purpose of the drone cluster's intelligent and autonomous control to implement track deception on the networked radar; (2) Each drone can make autonomous decisions through its own observation and reinforcement learning strategy network, and perform real-time track planning by controlling its own flight altitude to achieve track deception on the networked radar, thereby improving the intelligence level of the drone cluster and enabling the drone cluster to intelligently and autonomously complete the track deception task. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a flow chart of the drone cluster trajectory deception method based on deep reinforcement learning;

[0075] Figure 2 Schematic diagram of the angle between the velocity vector of the UAV and the false target and the radar line of sight;

[0076] Figure 3 This is the structure diagram of the UAV agent strategy network;

[0077] Figure 4 This is the value network structure diagram of the UAV agent;

[0078] Figure 5 Diagram of the distributed execution architecture for centralized training;

[0079] Figure 6 This is the reward function convergence curve of the simulation experiment;

[0080] Figure 7 This is the delay parameter diagram of each UAV in the simulation experiment;

[0081] Figure 8 Doppler parameter diagram of each UAV in the simulation experiment;

[0082] Fig. 9 This is a 2D drone track deception map for simulation experiments;

[0083] Fig.10 This is the speed diagram of each UAV in the simulation experiment;

[0084] Fig.11 This is the pitch angle diagram of each UAV in the simulation experiment;

[0085] Fig.12 This is the heading angle diagram of each UAV in the simulation experiment. DETAILED DESCRIPTION

[0086] The results and working process of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments.

[0087] There is no UAV swarm track deception method based on deep reinforcement learning in the prior art, and from the perspective of improving the intelligence level of UAV swarms, it is of great significance to carry out research on using intelligent agents to control UAVs to implement track deception on networked radars. Starting from the actual combat scenario, the present invention proposes a UAV swarm track deception method based on deep reinforcement learning. First, a track deception model and a deception signal model of a single UAV to a single radar are established. Secondly, the observation space, state space and action space of the UAV intelligent agent, the strategy network, value network and corresponding target network of the UAV intelligent agent are established. Thirdly, a deep reinforcement learning reward function is constructed based on the UAV dynamic performance constraints and the track deception task requirements. Finally, a centralized training and distributed execution architecture is adopted, and the MATD3 algorithm is used to train the UAV swarm to achieve the purpose of autonomous control of the UAV intelligent agent to achieve track deception of the networked radar.

[0088] like Figure 1 As shown, the drone cluster track deception method based on deep reinforcement learning of the present invention includes the following steps:

[0089] S1. Establish a track deception model of a single UAV to a single radar, as shown in formula (1):

[0090]

[0091] Among them, u k =[x u,k ,y u,k ,z u,k ] T and p k =[x p,k ,y p,k ,z p,k ] T denote the spatial coordinates of the UAV and the false target at time k respectively; r = [x r ,y r ,z r ] T Represents the spatial coordinates of the drone deceiving the radar.

[0092] Assuming that the radar transmission signal is a linear frequency modulation (LFM) signal, the spoofing signal forwarded by the digital radio frequency memory (DRFM) carried by the drone in step S1 is shown in formula (2):

[0093]

[0094] Where D(t) represents the deception signal; t represents time; A represents the radar transmission signal amplitude; △t represents the DRFM forwarding delay; f0 represents the radar transmission signal center frequency; f d represents the Doppler modulation parameter; μ represents the frequency modulation slope; T represents the pulse width.

[0095] Where △t is calculated by formula (3):

[0096] △t=2(R p -R u ) / c (3)

[0097] Among them, R p Indicates the distance between the false target and the radar; R u represents the distance between the UAV and the radar; c represents the speed of light.

[0098] Among them, f d Calculated by formula (4):

[0099]

[0100] Where, λ represents the wavelength of the radar transmission signal; v p and v u Represent the speed of false targets and drones respectively; β p and β u They represent the angles between the false target and the UAV velocity vector and the radar line of sight (LOS), respectively. Figure 2 shown.

[0101] S2. Establish UAV agent. Construct the observation space o of the i-th UAV agent i for:

[0102]

[0103] in, represents the spatial coordinates of the i-th UAV at time k; represents the spatial coordinates of the false target generated by the i-th UAV at time (k+1); Represents the spatial coordinates of the i-th drone deceiving the radar.

[0104] The state space s of the drone cluster is constructed as:

[0105]

[0106] Among them, u i,k represents the spatial coordinates of the i-th UAV at time k; p l,k+1 represents the spatial coordinates of the lth false target at time k+1; r qRepresents the spatial coordinates of the drone deceiving the qth radar.

[0107] The action space of the i-th UAV agent is constructed as:

[0108] a i =[△h i ] (7)

[0109] Among them, △h i represents the change in the flight altitude of the ith UAV. The flight altitude of the ith UAV at the next moment is calculated by formula (8):

[0110]

[0111] Each drone agent contains a policy network, a target policy network, two value networks and two target value networks. The policy network and the target policy network contain two hidden layers with 64 nodes. The activation function of the input layer and the hidden layer is the ReLU function, and the activation function of the output layer is the tanh function. The network structure is as follows: Figure 3 The value network and target value network also contain two hidden layers with 64 nodes. The activation functions of the input layer and hidden layer are ReLU functions, but no activation function is added to the output layer. The network structure is as follows Figure 4 shown.

[0112] S3. Construct a deep reinforcement learning reward function. The drone acceleration reward function is expressed as formula (9):

[0113]

[0114] Among them, a u,min and a u,max Respectively represent the minimum and maximum acceleration of the drone; a u,k represents the acceleration of the UAV at time k.

[0115] The drone altitude reward function is a cosine roll-off function, expressed as formula (10):

[0116]

[0117] Among them, h u,min and h u,max They represent the minimum and maximum flight altitude of the UAV respectively; h th,min =h u,min -2.5 and h th,max =h u,max +2.5 indicates the threshold at which the reward function reaches the minimum or maximum value; z u,k represents the flight altitude of the UAV at time k.

[0118] The drone speed reward function is expressed as formula (11):

[0119]

[0120] Among them, v u,min and v u,max Respectively represent the minimum and maximum speed of the drone; v u,k represents the flight speed of the UAV at time k.

[0121] The drone yaw angle reward function is expressed as formula (12):

[0122] J yew =10exp(-△α u,k 2 / 2) (12)

[0123] Among them, △α u,k represents the yaw angle of the UAV at time k.

[0124] The reward function of the drone pitch angle is expressed as formula (13):

[0125] J pitch =10exp(-β u,k 2 / 2) (13)

[0126] Among them, β u,k represents the pitch angle of the UAV at time k.

[0127] The drone spacing reward function is expressed as formula (14):

[0128]

[0129] Among them, d safe Indicates the safe distance between drones; Represents the distance between UAVs u1 and u2 at time k.

[0130] The total reward function of the UAV agent is expressed as formula (15):

[0131] J=J a +J h +J v +J yew +J pitch +J d (15)

[0132] S4, using centralized training and distributed execution architecture, using the Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm to train the drone swarm agents. Figure 5 As shown in the figure, during the training phase, the value network and target value network of each drone can receive the action, state and reward information of all drones for network update. The policy network and target policy network of each drone only update the network based on its own action, observation and reward information. During the execution phase, the policy network of each drone outputs actions based on its own observation information.

[0133] The proposed method uses the MATD3 algorithm to train each neural network. In the first 500 generations, only the drone altitude reward is considered to train the drone cluster. After the drone altitude reward converges, other rewards are added to train the drone cluster agent. The specific steps are as follows:

[0134] (1) Initialize the two value networks of the i-th drone agent A strategic network And the corresponding target value network and strategy network Network parameters Initialize the number of training rounds N t , discount rate γ, delayed update round t d , learning rates η1 and η2 of the value network and policy network, soft update rate τ, noise variance σ, number of drones N u and experience replay pool;

[0135] (2) The observation space o of the i-th UAV agent i Get the action space with noise added The noise ε follows a normal distribution with a mean of 0 and a variance of σ;

[0136] (3) Execute the action and transform the current state space s of the drone cluster, the next state space s' of the drone cluster, and the observation space o of the i-th drone agent at the current moment i And the observation space o of the i-th UAV agent at the next moment i ' and reward J i , forming a tuple [o i ,s,a i ,o i ',s',J i ] is stored in the experience replay pool;

[0137] (4) M tuples are sampled from the experience pool to form a batch, and the target policy network calculates the action of each tuple. And calculate the temporal difference (TD) target y for each tuple i :

[0138]

[0139] (5) Update two value network parameters:

[0140]

[0141] Among them, θ n,i and They represent the nth value network parameters before and after the update of the i-th UAV agent; η1 represents the value network learning rate; Represents about θ n,i The gradient of y i represents the TD target of each tuple;

[0142] (6) If t mod t d =0, mod represents the remainder operation; then the deterministic policy gradient is used to update the policy network parameters:

[0143]

[0144] Among them, φ i and They represent the nth policy network parameters before and after the update of the i-th UAV agent; η2 represents the value network learning rate; and Respectively represent φ i and a i The gradient of

[0145] Perform a soft update on the target network:

[0146]

[0147]

[0148] in, and Respectively represent the nth target value network parameters before and after the update of the i-th UAV agent; φ i * and They represent the target strategy network parameters before and after the update of the i-th UAV agent; τ represents the soft update rate;

[0149] (7) Check whether the above operations are performed on each UAV agent: if satisfied, proceed to the next round of training; otherwise, return to step (2) and train the next UAV agent;

[0150] (8) Check whether the number of training rounds N has been reached t : If satisfied, the training ends; otherwise, return to step (2) for the next round of training;

[0151] (9) Output the policy network of all drone agents

[0152] The system corresponding to the above method includes:

[0153] A model building unit, used to build a track deception model and a deception signal model of a single UAV against a single radar;

[0154] The UAV intelligent agent establishment unit is used to establish the observation space, state space and action space of the UAV intelligent agent, the strategy network, value network and corresponding target network of the UAV intelligent agent;

[0155] A deep reinforcement learning reward function construction unit is used to construct a deep reinforcement learning reward function based on the UAV dynamic performance constraints and the track deception task requirements, including a UAV acceleration reward function, a UAV altitude reward function, a UAV speed reward function, a UAV yaw angle reward function, a UAV pitch angle reward function, a UAV spacing reward function, and a UAV agent total reward function;

[0156] The drone swarm agent training unit is used to train drone swarm agents using a centralized training and distributed execution architecture, using the multi-agent double-delay deep deterministic policy gradient algorithm MATD3 algorithm to achieve autonomous control of drone swarms for track deception.

[0157] Simulation results:

[0158] In order to verify the feasibility and superiority of the method proposed in this invention, the following simulation scenario is designed: Consider a drone cluster consisting of 9 drones to implement track spoofing on a distributed network radar system consisting of 3 pulse radars, where the coordinates of each radar are (0, 0, 0.3) km, (-20, 9, 0.5) km, (10, 20, 0.4) km, the wavelength of the radar transmission signal λ = 0.5 m, and the safety distance d between drones. safe =50m. The UAV dynamic performance constraints are shown in Table 1:

[0159] Table 1 UAV dynamic performance constraint parameters

[0160]

[0161] The reinforcement learning training hyperparameters are shown in Table 2:

[0162] Table 2 Reinforcement learning training hyperparameters

[0163]

[0164] Figure 6 The reward curve for reinforcement learning shows that from the 1st round to the 250th round, the experience replay pool did not collect enough sample tuples, so training was not started and the drone cluster reward was in an oscillating state. In addition, since only relatively simple height rewards were used to train the drones in the first 500 rounds, the drone cluster reward curve converged quickly and stably, and then other rewards were added to train the drone cluster. The drone cluster reward basically converged in about 1000 rounds. Figure 7 and Figure 8 The forwarding delay parameters and Doppler modulation parameters of the UAV forwarding deception signals are shown respectively. The results show that according to the kinematic characteristics of the UAV and the false target, DRFM can forward the deception signals with corresponding parameters. Fig. 9 A two-dimensional drone cluster track deception map is shown. It can be seen that 9 drones generated a total of 4 false tracks. Figures 10 to 12 The flight speed, pitch angle and direction angle diagrams of the 9 drones are shown respectively. It can be seen from the figure that the drones meet the pre-set dynamic performance constraints and the generated track has high feasibility. Since the initial height of the drone is given in advance, the drone speed is relatively high at the initial moment. As the track deception task continues, the drone agent makes the best decision based on the observation of the environment, and continuously reduces the speed to about 70m / s while ensuring that the heading angle and pitch angle changes are as small as possible, and makes decisions in the direction of maximizing the overall reward. Since the false target setting heading is constantly changing, the corresponding drone heading angle continues to fluctuate, but it still meets the yaw angle constraint.

[0165] The working principle and working process of the invention are as follows:

[0166] The present invention considers the track deception of drone swarm based on deep reinforcement learning, and the intelligent agent autonomously controls the drone to complete the track deception. First, a single drone track deception model and a deception signal model are established; secondly, the observation space, state space and action space of the drone intelligent agent, as well as the policy network, value network and corresponding target network of the drone intelligent agent are constructed; thirdly, a deep reinforcement learning reward function is constructed; finally, a centralized training and distributed execution architecture is adopted, and the multi-agent double-delay deep deterministic policy gradient algorithm is used to train the drone swarm intelligent agent. During the execution process, the policy network of each drone intelligent agent is trained through its own observation o i Output action a iControl the UAV, complete track planning, and achieve track deception of the networked radar.

[0167] The invention of the present invention: The present invention considers the drone cluster track deception based on reinforcement learning. To achieve this goal, a drone intelligent agent is constructed, including the drone observation space, state space and action space, as well as the drone policy network, value network and corresponding target network. At the same time, a deep reinforcement learning reward function is designed to train the drone cluster according to the drone dynamics performance constraints and track deception task requirements. During the training process, a centralized training and distributed execution architecture is considered, and a multi-agent double-delay deep deterministic policy gradient algorithm is used to train the drone cluster intelligent agent. By training the policy network and state network of the drone intelligent agent, the policy network is trained according to its own observations. i Output action a that satisfies the UAV's dynamic performance constraints and maximizes the long-term reward i .

Claims

1. A drone cluster track deception method based on deep reinforcement learning, characterized in that: The following steps are involved: Establish a track deception model and deception signal model for a single UAV to a single radar; Establish the observation space, state space and action space of the UAV agent, the strategy network, value network and corresponding target network of the UAV agent; Based on the UAV dynamics performance constraints and track deception mission requirements, a deep reinforcement learning reward function is constructed, including the UAV acceleration reward function, the UAV altitude reward function, the UAV speed reward function, the UAV yaw angle reward function, the UAV pitch angle reward function, the UAV spacing reward function and the UAV agent total reward function; Based on the centralized training and distributed execution architecture, the multi-agent double-delay deep deterministic policy gradient MATD3 algorithm is used to train the drone swarm agents to achieve autonomous control of drone swarms for track deception.

2. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: The track deception model expression of a single UAV to a single radar is: Among them, u k =[x u,k ,y u,k ,z u,k ] T and p k =[x p,k ,y p,k ,z p,k ] T denote the spatial coordinates of the UAV and the false target at time k respectively; r = [x r ,y r ,z r ] T Indicates the spatial coordinates of the drone deceiving the radar; The deception signal model expression is: Where D(t) represents the deception signal; t represents time; A represents the radar transmission signal amplitude; △t represents the DRFM forwarding delay; f0 represents the radar transmission signal center frequency; f d represents the Doppler modulation parameter; μ represents the frequency modulation slope; T represents the pulse width.

3. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: The observation space expression of the UAV agent is: Among them, i is the observation space of the i-th UAV agent, represents the spatial coordinates of the i-th UAV at time k; represents the spatial coordinates of the false target generated by the i-th UAV at time (k+1); Represents the spatial coordinates of the radar deceived by the i-th drone.

4. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: The state space expression of the drone cluster is: Among them, s is the state space of the drone cluster, u i,k represents the spatial coordinates of the i-th UAV at time k; p l,k+1 represents the spatial coordinates of the lth false target at time k+1; r q Represents the spatial coordinates of the qth radar.

5. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: The action space expression of the UAV agent is: a i =[△h i ] Among them, a i is the action space of the i-th UAV agent, △h i represents the change in the flight altitude of the i-th UAV; the flight altitude expression of the i-th UAV at the next moment is: in, is the flight altitude of the i-th UAV at time k+1.

6. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: Each drone agent contains a policy network, a target policy network, two value networks and two target value networks; the policy network and the target policy network contain two hidden layers, the activation function of the input layer and the hidden layer is the ReLU function, and the activation function of the output layer is the tanh function; the value network and the target value network also contain two hidden layers, the activation function of the input layer and the hidden layer is the ReLU function, but no activation function is added to the output layer.

7. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: The expression of the drone acceleration reward function is: Among them, a u,min and a u,max Respectively represent the minimum and maximum acceleration of the drone; a u,k represents the acceleration of the UAV at time k; The drone altitude reward function is a cosine roll-off function, and its expression is: Among them, h u,min and h u,max Respectively represent the minimum and maximum flight altitude of the UAV; h th,min =h u,min -2.5 and h th,max =h u,max +2.5 indicates the threshold at which the reward function reaches the minimum or maximum value; z u,k represents the flight altitude of the UAV at time k; The drone speed reward function expression is: Among them, v u,min and v u,max Respectively represent the minimum and maximum speed of the drone; v u,k represents the flight speed of the UAV at time k; The reward function of the drone’s yaw angle is: I yew =10exp(-△α u,k 2 / 2) Among them, △α u,k represents the yaw angle of the UAV at time k; The reward function expression of the drone pitch angle is: J pitch =10exp(-β u,k 2 / 2) Among them, β u,k represents the pitch angle of the UAV at time k; The reward function expression of the drone spacing is: Among them, d safe Indicates the safe distance between drones; represents the distance between UAVs u1 and u2 at time k; The reward function expression of deep reinforcement learning is: J=J a +J h +J v +J yew +J pitch +J d Among them, J a is the drone acceleration reward function, J h is the drone altitude reward function, J v is the drone speed reward function, J yew is the reward function for the yaw angle of the drone, J pitch is the reward function for the pitch angle of the drone, J d is the reward function for the distance between drones.

8. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: During the training phase, the value network and target value network of each drone can receive the action, state and reward information of all drones for network update; the policy network and target policy network of each drone only update the network based on its own action, observation and reward information; In the execution phase, the policy network of each drone outputs actions based on its own observation information.

9. The method for spoofing the track of drone clusters based on deep reinforcement learning according to claim 1 is characterized in that: The multi-agent dual-delay deep deterministic policy gradient algorithm MATD3 is used to train each neural network. The specific steps are as follows: (1) Initialize the two value networks of drone agent i A strategic network And the corresponding target value network and strategy network Network parameters Initialize the number of training rounds N t , discount rate γ, delayed update round t d , learning rates η1 and η2 of the value network and policy network, soft update rate τ, noise variance σ, number of drones N u and experience replay pool; (2) The observation space o of the i-th UAV agent i Get the action space with noise added The noise ε follows a normal distribution with a mean of 0 and a variance of σ; (3) Execute the action and transform the current state space s of the drone cluster, the next state space s' of the drone cluster, and the observation space o of the i-th drone agent at the current moment i And the observation space o of the i-th UAV agent at the next moment i ' and reward J i , forming a tuple [o i ,s,a i ,o i ',s',J i ] is stored in the experience replay pool; (4) M tuples are sampled from the experience pool to form a batch, and the target policy network calculates the action of each tuple. And calculate the time difference target y for each tuple i : (5) Update two value network parameters: Among them, θ n,i and They represent the nth value network parameters before and after the update of the i-th UAV agent; η1 represents the value network learning rate; Represents about θ n,i The gradient of y i represents the time difference target of each tuple; (6) If t mod t d =0, mod represents the remainder operation; then the deterministic policy gradient is used to update the policy network parameters: Among them, φ i and They represent the nth policy network parameters before and after the update of the i-th UAV agent; η2 represents the value network learning rate; and Respectively represent φ i and a i The gradient of Perform a soft update on the target network: in, and They represent the nth target value network parameters before and after the update of the i-th UAV agent; and They represent the target strategy network parameters before and after the update of the i-th UAV agent; τ represents the soft update rate; (7) Check whether the above operations are performed on each UAV agent: if satisfied, proceed to the next round of training; otherwise, return to step (2) and train the next UAV agent; (8) Check whether the number of training rounds N has been reached t : If satisfied, the training ends; otherwise, return to step (2) for the next round of training; (9) Output the policy network of all drone agents 10. The UAV swarm track deception system based on deep reinforcement learning is characterized by: include: A model building unit, used to build a track deception model and a deception signal model of a single UAV against a single radar; The UAV intelligent agent establishment unit is used to establish the observation space, state space and action space of the UAV intelligent agent, the strategy network, value network and corresponding target network of the UAV intelligent agent; A deep reinforcement learning reward function construction unit is used to construct a deep reinforcement learning reward function based on the UAV dynamic performance constraints and the track deception task requirements, including a UAV acceleration reward function, a UAV altitude reward function, a UAV speed reward function, a UAV yaw angle reward function, a UAV pitch angle reward function, a UAV spacing reward function, and a UAV agent total reward function; The drone swarm agent training unit is used to train drone swarm agents using a centralized training and distributed execution architecture, using the multi-agent double-delay deep deterministic policy gradient algorithm MATD3 algorithm to achieve autonomous control of drone swarms for track deception.

Images