A method for cooperative path planning of unmanned swarm facing sudden threats

Abstract

The application discloses a method for unmanned aerial vehicle swarm cooperative path planning facing sudden threats. The application belongs to the field of unmanned aerial vehicle control and mainly solves the problem of real-time obstacle avoidance and autonomous path planning of unmanned aerial vehicles in complex environments with sudden threats and dense obstacles. The proposed algorithm models the unmanned aerial vehicle path planning problem as a Markov decision process and solves the problem by using deep reinforcement learning. By classifying and normalizing the complex environment threats, the DDQN algorithm with a "target network-evaluation network" architecture is introduced for the unmanned aerial vehicle action state space, and the reward function is reshaped to realize the perception and obstacle avoidance functions of complex threats. At the same time, a dynamic epsilon decay strategy is designed to balance the exploration-exploitation balance. The algorithm ensures that the unmanned aerial vehicle can autonomously complete path planning while avoiding environmental threats. A large number of simulation experiments prove the effectiveness and robustness of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) control, and specifically relates to the design of a real-time UAV trajectory planning algorithm based on deep reinforcement learning. Background Technology

[0002] In recent years, drones have gained widespread attention in the civilian sector due to their low cost, small size, high maneuverability, and strong environmental adaptability. Currently, mainstream applications include agricultural plant protection, power line inspection, logistics and express delivery, emergency rescue, environmental monitoring, and intelligent transportation. In complex application environments, where human intervention is difficult and operating costs are high, drones are more suitable for completing tasks. However, weather changes and terrain obstacles in complex environments are significant factors affecting the normal operation of drones. Threats in the operating environment also impact drone operations, and many threats are dynamic. For safe operation, constraints such as drone flight time, energy carrying capacity, and actual maneuverability must be considered. Considering these multiple threat factors, autonomous navigation capabilities for drones are particularly important. Drone autonomous navigation mainly includes three aspects: positioning and mapping, trajectory planning, and flight control. Among these, trajectory planning, as a crucial research area of ​​drone autonomous navigation, is the technical guarantee for drones to avoid threats and successfully reach their target location.

[0003] In the current research status both domestically and internationally, UAV trajectory planning is mainly divided into non-learning traditional trajectory planning algorithms and artificial intelligence-based learning trajectory planning algorithms. Commonly used algorithms in traditional trajectory planning include A* algorithm, D* algorithm, Artificial Potential Field (APF) algorithm, Probabilistic Path Graph (PRM) algorithm, Fast Random Search Tree (RRT) algorithm, and Particle Swarm Optimization (PSO) algorithm. These algorithms have poor real-time performance, and threats in complex environments can affect the safety of UAV flight, failing to fully meet the needs of autonomous UAV combat and flight. Therefore, UAVs cannot demonstrate good trajectory planning capabilities.

[0004] In large-scale, complex environments, AI-based UAV trajectory planning algorithms have demonstrated strong environmental adaptability. Deep reinforcement learning is widely used in UAV trajectory planning due to its high adaptability and strong generalization capabilities. Deep reinforcement learning allows an agent to observe the state of its environment. Through the perceptual processing of deep learning, the agent learns the characteristics of the observed environmental state, and then further trains itself through reinforcement learning decision-making to complete the entire deep reinforcement learning structure. Therefore, deep reinforcement learning combines the decision-making capabilities of reinforcement learning with the perceptual capabilities of deep learning, utilizing the powerful representational capabilities of neural networks to solve the perception and decision-making problems of complex systems.

[0005] Autonomous trajectory planning algorithms for UAVs based on deep reinforcement learning mainly include Deep Q-Network (DQN) and Double DQN (DDQN). In traditional Q-learning, when the state and action spaces are discrete and of low dimensionality, a Q-table can be used to store the Q-value of each state and action. However, when the state and action spaces are high-dimensional and continuous, using a Q-table becomes impractical. DQN combines deep learning with Q-learning, using a deep neural network to fit the Q-value, thus solving the representation problem of traditional Q-learning in high-dimensional state and action spaces. It also introduces an experience pool replay technique to address correlation and non-static distribution issues, further improving the learning efficiency of the deep neural network. However, DQN suffers from overestimating the Q-value, resulting in a deviation from the true Q-value and leading to incorrect estimation of action values ​​and outputting a suboptimal policy. To address this, DDQN improves upon DQN by decoupling the selection of the target Q-value action from the calculation of the target Q-value, thereby reducing the overestimation problem and making the estimated Q-value closer to the true value. Therefore, DDQN can learn the state value function more efficiently, solving the problem of overestimation of the value function in DQN.

[0006] Considering the broad application prospects of UAVs and the various threat environments that currently exist in UAV trajectory planning, this invention proposes the DDQN-RTRP (Double Deep Q-Network-based-Real-time Route Plan Algorithm) algorithm based on deep reinforcement learning to address the UAV trajectory planning problem under large-scale, complex, and sudden threats. Compared with the traditional trajectory planning algorithm CPP / CA (Coordinated Path Planning with Collision Avoidance), the algorithm proposed in this invention can provide UAVs with stronger real-time trajectory planning capabilities in large-scale, complex threat environments. Furthermore, this algorithm also demonstrates strong environmental adaptability and generalization ability. Summary of the Invention

[0007] The purpose of this invention is to provide a highly efficient and versatile real-time trajectory planning method for unmanned aerial vehicles (UAVs) to perform missions in environments with large-scale, complex threats and dense obstacles. This method ensures that UAVs avoid obstacles, bypass various complex and sudden threats, and successfully reach the mission location, while simultaneously enabling the UAV swarm to autonomously plan the globally optimal flight path. To achieve this objective, the steps employed in this invention are as follows:

[0008] Step 1: Based on the relevant theoretical knowledge of reinforcement learning, this invention models the real-time trajectory planning problem of UAVs as a Markov decision process. A Markov decision process can usually be represented by a quintuple: (S, A, P, γ, R), where S represents the state space, A represents the action space, P∈[0,1] represents the state transition probability of the agent from the current state s to a state s', γ∈[0,1] is the discount factor, and R represents the reward function;

[0009] Step 2: Establish a drone swarm model: In this invention, only the motion model of drones in a two-dimensional plane is considered, and the initial position of drone i is represented as (xi, yi); This invention updates the position of drones by defining the direction Vi of drones and a fixed motion step size;

[0010] Step 3: Based on the complex threat scenarios of UAV autonomous trajectory planning, a unique non-sparse reward function of this invention is designed to accelerate the convergence speed and learning efficiency of the model. This reward function mainly consists of five parts: the first part is the reward r1 related to the distance from the UAV to the target; the second part is the reward r2 related to the UAV's own state; the third part is the reward r3 related to enemy threats in the environment; the fourth part is the reward r4 related to obstacle collision or being attacked; and the fifth part is the reward r5 for reaching the target range.

[0011] Step 4: Propose a real-time trajectory planning method for UAV swarms in complex environments with enemy threats. This algorithm starts from the real-time trajectory planning task of reconnaissance UAV swarms in complex environments, focusing on the environmental threats that UAVs often encounter, such as enemy threats (e.g., ground missiles, jammers, etc.) and static obstacles (e.g., tall buildings, trees, etc.) and dynamic obstacles (other friendly UAVs). Enemy threats are mainly divided into ground threats (GT) and air threats (AT), and environmental threats are divided into static obstacle threats and dynamic UAV threats. All threats are normalized and modeled. At the same time, using the DDQN-RTRP algorithm, the UAV swarm uses its own onboard sensors to observe the environment. Through continuous learning and environmental exploration, the deep neural network model is updated. The trained algorithm network model can be applied to the real-time trajectory planning task of UAV swarms in different environments.

[0012] The effectiveness of the proposed unmanned swarm cooperative trajectory planning algorithm for complex and sudden threat environments has been verified through simulation experiments. This invention compares the algorithm with the traditional trajectory planning algorithm CPP / CA. CPP / CA adds repulsive forces between agents to the traditional artificial potential field algorithm and designs a new repulsive field function, thus solving the problem of agents getting trapped in local minima. (Appendix) Figure 3 With appendix Figure 4 The following graphs show the success rates of the DDQN-RTRP algorithm proposed in this invention and the traditional path algorithm CCP / CA under different numbers of drones and threats; (Attached) Figure 5 The actual flight trajectory of the UAV is shown in the model trained by the proposed algorithm, with a simulation environment area of ​​1000×1000m2 and 12 obstacles. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the real-time trajectory planning algorithm of the present invention;

[0014] Figure 2 This is a schematic diagram of the neural network structure of the present invention;

[0015] Figure 3 This is a graph showing the success rates of the algorithm of this invention and the CCP / CA algorithm under different numbers of drones.

[0016] Figure 4 This is a graph showing the success rates of the algorithm of this invention and the CCP / CA algorithm under different threat numbers.

[0017] Figure 5 This is the actual flight trajectory diagram of the UAV of the present invention in a simulated environment with an area of ​​1000×1000m2 and 12 environmental obstacles. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings.

[0019] Step 1: Model the UAV autonomous trajectory planning problem as a Markov decision process.

[0020] 1. Definition of state space

[0021] The state space inside the drone is defined as S i It consists of the location information X of drone i. i (x i y i ) and the direction of motion of the drone V i The system consists of several components. Location information can be acquired by sensors such as the drone's GPS, and the drone's direction of movement serves as the action output of the intelligent agent.

[0022] The state space related to obstacles is defined as S0; in order to enable the UAV to avoid the threats of obstacles and other flying objects in a timely manner during flight, this invention uses an omnidirectional laser sensor to detect the environmental information R(x) around the UAV. i y iThe data returned by the omnidirectional laser sensor will be used as the basis for determining whether a collision has occurred with the drone. Specifically:

[0023]

[0024] The state space related to the target is defined as T goal Specifically, it is expressed as:

[0025]

[0026] T goal Let x be the orientation of the target point relative to UAV i, given by the prior information: target point position (x). goal y goal The position information of UAV i is calculated, and thus, the overall state space of the UAV can be represented as:

[0027]

[0028] 2. Definition of Action Space

[0029] The action space is a set of selectable actions. The agent can select the next action from the action space according to the state of the environment. In the real-time trajectory planning algorithm of the present invention, the 32 motion directions that are evenly divided into 360° two-dimensional space are defined as the set of selectable actions of the UAV agent. The next state of the UAV can be obtained by the motion direction and fixed step size.

[0030] 3. State transition probability

[0031] The state transition probability refers to the probability given the current state S. t And Action A t Transition to state S t+1 The probability distribution, i.e.

[0032]

[0033] 4. Discount Factor

[0034] The discount factor γ diminishes the contribution of future rewards to the current state value, and its value is typically between [0, 1]. The more steps required to complete a task, the closer the discount factor γ is to 1. In this invention, the value of the discount factor γ is set to 0.9.

[0035] 5. Reward Function

[0036] The reward function R refers to the reward function given the state S. t Take action A t Afterwards, the reward received by the drone is r = R(S) t A tReinforcement learning algorithms are often highly sensitive to the reward function R, and different reward functions have a significant impact on convergence and learning speed. This invention employs a reward reshaping technique to set a reward unique to this invention. This will be explained in detail in step 3.

[0037] Step 2: Establish the dynamic model of the UAV

[0038] As a rigid body with seven degrees of freedom in three-dimensional space, the motion of a reconnaissance drone can be described in many ways. In this invention, the complex motion model of the drone is not considered; only the two translational degrees of freedom and one rotational degree of freedom in the xoy plane are considered. The kinematic equations of the drone are modeled as follows:

[0039]

[0040] Where, x t i V t i Let x be the position and velocity of the i-th UAV at time t, and C be the flight altitude of the UAV. t i y t i Let v be the coordinates of the UAV along the x-axis and y-axis in the xoy plane, respectively. t i (x), v t i (y) represents the velocity components of the UAV on the x and y axes.

[0041] Step 3: Reshape the reward function

[0042] The reward is the benefit from environmental feedback obtained after the agent updates the drone's state following a response action. The reward function, as the only way for the drone to interact with the reconnaissance environment, serves to convey the target to the agent, and its design is crucial to the algorithm's convergence. The drone trajectory planning problem requires the drone to reach the target area via a safe path consisting of the fewest possible steps. To this end, this invention provides a specific form of reward for the drone through reward reshaping technology, while maintaining policy invariance; specifically, the reward in this invention is divided into five parts.

[0043] The first part is the reward related to the distance the drone is from the target:

[0044]

[0045] Where, Δd goal l represents the difference between the UAV's next state and distance to the target point and its current state and distance to the target point. stepHere, the step size of the drone is used to normalize the reward. As you can see, the obstacle penalty is modeled as a continuous function of distance rather than a discrete penalty value. When the drone approaches the target point, the reward is positive, and vice versa, which encourages the drone to move towards the target point.

[0046] The second part is the reward r2 related to the drone's own state:

[0047]

[0048] Where θ is the current turning angle of the drone, and π is the maximum turning angle of the drone. The reward is also normalized here.

[0049] The third part is the reward r3 related to enemy threats in the environment: r threat =-P T (x i y i That is, the greater the threat probability of the location of drone i, the greater the negative reward the drone receives. Complex environment threat modeling will be explained in detail in step 4.

[0050] The fourth part is the reward r4 related to obstacle collisions or being attacked: if the drone collides with environmental obstacles or other drones or enters the area of ​​least local threat and is attacked by enemy threats, then r4 = -5, otherwise r4 = 0.

[0051] The fifth part is the reward r5 for reaching the target area: if the drone reaches the target area, i.e., d g o al <r goal If the result is positive, then r5 = +5; otherwise, r5 = 0.

[0052] In summary, the global reward function for drone i is defined as follows:

[0053] r=-1+k1·r1+k2·r2+k3·r3+r4+r5 (8)

[0054] Where k1, k2, and k3 are adjustment coefficients for the gains from approaching the target point, turning, and total threat probability, respectively. To prevent the drone from getting stuck in a local optimum, a fixed negative reward of -1 is given to the drone at each step.

[0055] Step 4: Building the Environmental Threat Model and DDQN Algorithm Model

[0056] The environmental threat model proposed in this invention mainly considers that when UAV swarms perform real-time trajectory planning tasks in complex reconnaissance environments, they often encounter environmental threats such as enemy threats (e.g., ground missiles, jammers, etc.) and static obstacles (e.g., tall buildings, trees, etc.) and dynamic obstacles (other friendly UAVs).

[0057] Assume that the coordinates of the i-th (i∈{1,2,...,n}) threat center within the detection range of the drone's sensors are O i =(x T，i y T，i The probability of a drone facing the i-th threat at location (x, y) is:

[0058]

[0059]

[0060] Where, r T r is the distance from the drone to the center of threat i. T，min Let x be the minimum threat radius. The total threat probability of the drone at location (x, y) is:

[0061]

[0062] (1) Enemy threat

[0063] 1) Enemy ground threat

[0064] In aerial combat, ground threats often come from enemy surface-to-air missile battalions, which can launch anti-aircraft missiles to attack our UAVs. Within the attack radius of the enemy's anti-aircraft missiles, their hit rate can be considered to be related to the distance between our UAV and the enemy's anti-aircraft missile launch point. In this invention, the center location of the enemy ground threat is X. GT，j The enemy's ground threat range is Z1(x, y).

[0065] X GT，i =(x GT，i y GT，i ,0) (12)

[0066]

[0067] Where, r GT，min The minimum threat radius of the enemy ground threat is defined as the distance between our reconnaissance drone and the center of the enemy ground threat being less than r. GT，min When it is certain that the enemy ground threat will attack, the threat probability is 1; when the distance between our drone and the center of the enemy ground threat is greater than r... GT，min If so, the probability of enemy ground threat is calculated according to (9).

[0068] 2) Enemy air threat

[0069] In aerial combat, the common aerial threat is enemy jamming aircraft, which can interfere with the communication and navigation signals of friendly UAVs. Therefore, UAVs should avoid these jamming aircraft as much as possible during missions. Similarly, the probability of interference from enemy jamming aircraft is related to the distance between the friendly UAV and the enemy jamming aircraft. In this invention, the center position of the enemy aerial threat is X. AT，j The enemy's air threat range is Z2(x, y).

[0070] X AT，i =(x AT，i y AT，i , z AT，i (14)

[0071]

[0072] Where, r AT，min The minimum threat radius of the enemy's aerial threat is defined as the distance between our drone and the center of the enemy's aerial threat being less than r. AT，min When it is assumed that the enemy's ground threat will definitely attack, the threat probability is 1; when the distance between our drone and the center of the enemy's air threat is greater than r... AT，min If so, the probability of enemy air threat is calculated according to (9).

[0073] (3) Environmental threats

[0074] 1) Static obstacle threat

[0075] When reconnaissance drone swarms are performing missions, their flight altitude is limited due to sensor limitations and their own characteristics. Therefore, the swarms will encounter static obstacles such as tall buildings and trees. In these situations, the reconnaissance drones need to use their onboard sensors to detect the environment in real time and take appropriate avoidance measures. This invention does not consider the complex physical models of static obstacles, but uniformly models them as cylinders, and describes them as follows:

[0076]

[0077] Where, x OB，i y OB，i Let h be the center coordinates of the obstacle in the xoy plane. OB，i r is the height of the cylindrical obstacle. OB，i Let r be the radius of the cross-sectional circle of the cylindrical obstacle in the xoy plane. When calculating the static obstacle threat, r... T，min =r OB，i .

[0078] 2) Threat of Dynamic Unmanned Aerial Vehicles

[0079] Other drones in a drone swarm are dynamic obstacles to the drone itself. If two or more drones are within a certain distance, a collision will occur. Therefore, to prevent flight conflicts and ensure flight safety, obstacle avoidance with other drones must be considered during real-time trajectory planning. Assume the minimum safe distance between drones is r. safe Then the distance r between the drone and other member drones must be r > r safe When calculating the threat probability of static obstacles, r T，min =r safe .

[0080] The method proposed in this invention is mainly an improvement on the DDQN algorithm. In the algorithm, each UAV shares a decision model, which uses the UAV's state information as input and outputs the next action to be performed. The decision model uses a deep Q-network framework, including a target network and an evaluation network. Gaussian noise with a mean of 0 and a variance of 0.1 is added to the outputs of both the target network and the evaluation network. The target network establishes a mapping between the current state and specific actions, enabling DDQN to handle the UAV's action space. To optimize the target network, the evaluation network uses parameter θ... eval The action value function Q(s, a) is approximated, and the evaluation network is strengthened according to the Bellman optimality principle. The target network is then updated to output a better action.

[0081] Based on the characteristic that a smaller TD error in the action value function leads to a more accurate fit of the action value function, let each empirical priority value be:

[0082]

[0083] Where P i =|TD+ε|, where ε is a very small value to prevent the probability of drawing an experience with a TD error of 0 from being non-zero, and an absolute value function is used to prevent the probability from being less than 0.

[0084] The algorithm proposed in this invention uses the technique of experience replay, that is, after the UAV completes each action, the algorithm will record the data (s) obtained from the interaction with the environment. t a t r t s t+1 The data is stored in the experience pool, and the value of different experiences is evaluated to increase the probability of selecting more valuable experiences. High-value experiences are selected first to be input into the neural network to update the weight parameters. When the data in the experience pool reaches a certain amount, N data are randomly selected from the experience pool to update the neural network parameters.

[0085] The network is evaluated using gradient descent training. The loss function is defined as follows:

[0086]

[0087] The network parameters are updated by minimizing the loss function:

[0088]

[0089] The parameters of the target network are updated periodically, meaning that after each training cycle, the target network parameters are updated to the current evaluation network parameters.

[0090] Meanwhile, this invention incorporates a dynamic ε decay strategy during the training process to balance the "exploration-exploitation" relationship in reinforcement learning. Specifically, in the early stages of training, the ε value is relatively large to ensure sufficient exploration of the environment; as training progresses, ε continuously decays to select the optimal strategy; during the training phase, the UAV takes corresponding actions based on the action values ​​output by the target network and the strategies learned from the environment. In this way, the UAV gradually learns to autonomously decide what action to take next, thus achieving autonomous trajectory planning in large-scale, complex threat and dense obstacle environments.

[0091] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. A method for cooperative flight path planning of unmanned swarms in response to sudden threats, comprising the following steps: Step 1: Model the real-time trajectory planning problem of the UAV as a Markov decision process (S, A, P, γ, R); where S represents the state space, A represents the action space, P∈[0,1] represents the state transition probability of the agent from state s to state s', γ∈[0,1] is the discount factor, and R represents the reward function; the state space consists of the UAV internal state space, the obstacle-related state space, and the target-related state space, and the action space is a set of selectable actions consisting of 32 motion directions evenly divided into 360° two-dimensional space, with the discount factor γ taking a value of 0.9; Step 2: Establish a UAV swarm motion model in a two-dimensional plane: The initial position of UAV i is represented as (xi, yi); the position of the UAV is updated by defining the direction Vi of the UAV and a fixed motion step size. The UAV swarm motion model only considers two translational degrees of freedom and one rotational degree of freedom in the xoy plane. Step 3: Design a non-sparse composite reward function. The reward function is constructed using reward reshaping technology. It includes five global reward functions: a reward r1 related to the distance from the UAV to the target, a reward r2 related to the UAV's own state, a reward r3 related to enemy threats in the environment, a reward r4 related to obstacle collision or being hit, and a reward r5 for reaching the target area. Each step has a fixed negative reward of -1. The reward r1 related to the distance from the UAV to the target, the reward r2 related to the UAV's own state, and the reward r3 related to enemy threats in the environment are adjusted by adjustment coefficients k1, k2, and k3, respectively. Step 4: Construct an environmental threat model and combine it with an improved DDQN algorithm to achieve autonomous real-time trajectory planning for UAVs. First, environmental threats are divided into enemy threats and environmental threats. Enemy threats include ground missiles and jammers; environmental threats include static obstacle threats and dynamic obstacle threats. Normalize the model of all threats and calculate the single threat probability and total threat probability of the UAV's location. Then, build a decision model based on the DDQN algorithm. The decision model includes a target network and an evaluation network. Gaussian noise with a mean of 0 and a variance of 0.1 is added to the output of both the target network and the evaluation network. An experience-first replay mechanism is introduced. The evaluation network is trained using gradient descent, and the target network adopts a periodic update strategy. During the training process, a dynamic decay ε-greedy strategy is added to balance the exploration and exploitation relationship. The UAV swarm observes the environment through GPS sensors and uses this algorithm to continuously learn and update the deep neural network model to complete the trajectory planning.

2. The unmanned swarm cooperative trajectory planning method for sudden threats according to claim 1, characterized in that, Modeling the UAV trajectory planning problem as a Markov decision process: (1) Definition of state space: The state space inside the drone is defined as S i It consists of the location information X of drone i. i (x i y i ) and the direction of motion of the drone V i The system consists of several components; the location information can be obtained by the drone's GPS sensor, and the drone's direction of movement serves as the action output of the intelligent agent. The state space related to obstacles is defined as S0; in order to enable the UAV to avoid the threats of obstacles and other flying objects in a timely manner during flight, an omnidirectional laser sensor is used to detect the environmental information R(Xi, yi) around the UAV, and the data returned by the omnidirectional laser sensor is used as the basis for determining whether the UAV has collided; specifically expressed as: The state space related to the target is defined as T goal Specifically, it means: T goal Let x be the orientation of the target point relative to UAV i, given by the prior information: target point position (x). goal y goal The position information of UAV i is calculated, and thus, the overall state space of the UAV can be represented as: (2) Definition of action space: The action space is a set of selectable actions. The agent can select the next action from the action space according to the state of the environment. This method defines the 32 motion directions that are evenly divided into 360° two-dimensional space as the set of selectable actions for the UAV agent. The next state of the UAV can be obtained by the motion direction and fixed step size. (3) State transition probability: The state transition probability refers to the probability of a state transition given the current state space S. t and action space A t In A t When the selected action is 'a', the probability that the drone transitions from state 's' to state 's' is... (4) Discount factor: The discount factor γ decays the contribution of future rewards to the current state value, and its value is usually between [0, 1]. The more steps required to complete a task, the closer the value of the discount factor γ is to 1. In this method, the value of the discount factor γ is set to 0.

9. (5) Reward function: The reward function R refers to the reward function given state s. t Take action a t Afterwards, the reward received by the drone is r = R(s) t a t Reinforcement learning algorithms are often very sensitive to the reward function R, and different reward functions have a great impact on convergence and learning speed. For complex and sudden threat environments, reward reshaping technology is used to set rewards.

3. The unmanned swarm cooperative trajectory planning method for sudden threats according to claim 1, characterized in that, The specific method for constructing a two-dimensional plane drone swarm motion model is as follows: Considering the two translational degrees of freedom and one rotational degree of freedom of the UAVs in the xoy plane, the motion model of the UAV swarm is modeled as follows: Among them, X t i V t i Let x be the position and velocity of the i-th UAV at time t, and C be the flight altitude of the UAV. t i y t i Let v be the coordinates of the UAV along the x-axis and y-axis in the xoy plane, respectively. t i (x), v t i (y) represents the velocity components of the UAV on the x and y axes.

4. The unmanned swarm cooperative trajectory planning method for sudden threats according to claim 1, characterized in that, The specific method for reshaping the reward function is as follows: The reward is the benefit from environmental feedback obtained after the agent updates the drone's state by performing a response action. As the only way for the drone to interact with the reconnaissance environment, the reward function conveys the target to the agent, and its design is crucial to the convergence of the algorithm. The drone trajectory planning problem requires the drone to reach the target area with a safe path consisting of the fewest possible steps. To this end, a specific form of reward is provided for the drone through reward reshaping technology under the premise of policy invariance; specifically, the reward is divided into five parts. The first part is the reward related to the distance the drone is from the target: Where, Δd goal l represents the difference between the UAV's next state and distance to the target point and its current state and distance to the target point. step Here, the step size of the drone is used to normalize the reward. As you can see, the obstacle penalty is modeled as a continuous function of distance rather than a discrete penalty value. When the drone approaches the target point, the reward is positive, and vice versa, which encourages the drone to move towards the target point. The second part is the reward r2 related to the drone's own state: Where θ is the current turning angle of the drone, and π is the maximum turning angle of the drone. The reward is also normalized here. The third part is the reward r3 related to enemy threats in the environment: r3 = -P T (x i y i That is, the greater the threat probability of the location of drone i, the greater the negative reward that drone receives; The fourth part is the reward r4 related to obstacle collisions or being attacked: if the drone collides with environmental obstacles or other drones or enters the minimum enemy threat range and is attacked by the enemy, then r4 = -5; otherwise, r4 = 0. The fifth part is the reward r5 for reaching the target area: if the drone reaches the target area, i.e., d goal <r goal If the value is positive, then r5 = +5; otherwise, r5 = 0. In summary, the global reward function for drone i is defined as follows: r=-1+k1·r1+k2·r2+k3·r3+r4+r5 (8) Where k1, k2, and k3 are adjustment coefficients for the reward r1 related to the distance from the drone to the target, the reward r2 related to the drone's own state, and the reward r3 related to the enemy threat in the environment, respectively. To avoid the drone getting trapped in a local optimum, a fixed negative reward of -1 is given to the drone at each step.

5. The unmanned swarm cooperative trajectory planning method for sudden threats according to claim 1, characterized in that, The specific methods for constructing the environmental threat model and the improved DDQN algorithm are as follows: The environmental threat model is designed to take into account the fact that when UAV swarms perform real-time trajectory planning tasks in complex environments, they often encounter enemy threats and environmental threats. Specifically, enemy threats include ground missiles and jammers; environmental threats include static obstacles such as tall buildings and trees, and dynamic obstacles such as other friendly UAVs. Assume that the coordinates of the i-th (i∈{1,2,...,n}) threat center within the detection range of the drone's sensors are O i =(x T，i y T，i The probability of a drone facing the i-th threat at location (x, y) is: Where, r T r is the distance from the drone to the center of threat i. T,min Given the minimum threat radius, the total threat probability of the drone at location (x, y) is: (1) Enemy threat 1) Enemy ground threat In air combat, the ground threat is often an enemy surface-to-air missile battalion, which can launch anti-aircraft missiles to attack our UAVs. Within the attack radius of the enemy's anti-aircraft missiles, the hit rate can be considered to be related to the distance between our UAV and the enemy's anti-aircraft missile launch point. The center of the enemy ground threat is X. GT，i The enemy's ground threat range is Z1(x, y); X GT，i =(x GT，i ,y GT,i ,0) (12) Where, r GT，min The minimum threat radius of the enemy ground threat is defined as the distance between our reconnaissance drone and the center of the enemy ground threat being less than r. GT,min When it is certain that the enemy ground threat will attack, the threat probability is 1; when the distance between our drone and the center of the enemy ground threat is greater than r... GT，min In this case, the probability of enemy ground threat is calculated according to (9); 2) Enemy air threat In aerial combat, the common aerial threat is enemy jamming aircraft, which can interfere with the communication and navigation signals of friendly UAVs. When UAVs are performing missions, it is crucial to avoid them as much as possible. Similarly, the probability of interference from enemy jamming aircraft is related to the distance between our UAV and the enemy jamming aircraft. The center of the enemy aerial threat is X. AT，i The enemy's air threat range is Z2(x, y); X AT，i =(x AT，i ,y AT，i ,z AT，i ) (14) Where, r AT，min The minimum threat radius of the enemy's aerial threat is defined as the distance between our drone and the center of the enemy's aerial threat being less than r. AT，min When it is assumed that the enemy's ground threat will definitely attack, the threat probability is 1; when the distance between our drone and the center of the enemy's air threat is greater than r... AT，min In this case, the probability of enemy air threat is calculated according to (9); (3) Environmental threats 1) Static obstacle threat When performing missions, drone swarms are limited in flight altitude due to sensor limitations and their own characteristics. Therefore, they encounter static obstacles, including tall buildings and trees. In such cases, reconnaissance drones need to use their onboard sensors to detect the environment in real time and take appropriate avoidance measures. Ignoring the complex physical models of static obstacles, we uniformly model them as cylinders and describe them as follows: Where, x OB,i y oB,i Let h be the center coordinates of the obstacle in the xoy plane. oB,i r is the height of the cylindrical obstacle. OB,i Let r be the radius of the cross-sectional circle of the cylindrical obstacle in the xoy plane. When calculating the static obstacle threat, r... T,min =r oB,i , r is the distance parameter, representing the actual distance between the drone and the obstacle; 2) Threat of Dynamic Unmanned Aerial Vehicles Other drones in a drone swarm are dynamic obstacles to the drone itself. If two or more drones are within a certain distance, a collision will occur. Therefore, to prevent flight conflicts and ensure flight safety, obstacle avoidance with other drones must be considered during real-time trajectory planning. Let's assume the minimum safe distance between drones is r. safe Then the distance r between the drone and other member drones must be r > r safe When calculating the threat probability of static obstacles, r T，min =r safe ; This method primarily improves upon the DDQN algorithm. In this algorithm, each UAV shares a single decision model. The decision model uses the UAV's state information as input and outputs the next action to be performed. The decision model employs a deep Q-network framework, including a target network and an evaluation network. Gaussian noise with a mean of 0 and a variance of 0.1 is added to the outputs of both the target and evaluation networks. The target network establishes a mapping between the current state and specific actions, enabling DDQN to handle the UAV's action space. To optimize the target network, the evaluation network uses parameters θ... eval The action value function Q(s, a) is approximated, and the evaluation network is strengthened according to the Bellman optimality principle. The target network is then updated to output a better action. Based on the characteristic that a smaller TD error in the action value function leads to a more accurate fit of the action value function, let each empirical priority value be: Where P i =|TD+ε|, where ε is a very small value to prevent the probability of the experience with TD error of 0 being non-zero, and to prevent the probability from being less than 0 using an absolute value function; This method uses the technique of experience replay, that is, after the drone completes each action, the algorithm will record the data (s) obtained from the interaction with the environment. t a t r t s t+1 The data is stored in the experience pool, and the value of different experiences is evaluated to increase the probability of selecting more valuable experiences. High-value experiences are selected first to be input into the neural network to update the weight parameters. When the data in the experience pool reaches a certain amount, N data are randomly selected from the experience pool to update the neural network parameters. The network was trained using gradient descent, and the loss function was defined as follows: The network parameters are updated by minimizing the loss function: The parameters of the target network are updated periodically. After each training cycle, the target network parameters are updated to the current evaluation network parameters. At the same time, a dynamic ε decay strategy is added during the training process to balance the "exploration-exploitation" relationship in reinforcement learning. Specifically, in the early stage of training, the ε value is larger to achieve full exploration of the environment; as training progresses, ε will continuously decay to select the optimal strategy; during the training phase, the UAV will learn the strategy and take corresponding actions based on the action values ​​output by the target network.

Images