TD3-based unmanned aerial vehicle cluster obstacle avoidance strategy generation method

Abstract

The invention provides an unmanned aerial vehicle cluster obstacle avoidance strategy generation method based on a double-delay depth deterministic strategy gradient (TD3). According to the method, a Markov decision model is constructed by combining task requirements, environmental conditions and unmanned aerial vehicle performance parameters, the Markov decision model comprises a state space, an action space and a reward function, and the model is trained through a TD3 algorithm. The method comprises the following steps: firstly, determining a target motion direction and density of an unmanned aerial vehicle cluster, and setting a minimum safety distance and an obstacle distance to ensure flight safety; and then, defining a reward function, and guiding the unmanned aerial vehicle to realize effective obstacle avoidance and formation maintenance in a complex environment. Compared with a traditional rule-driven and classical algorithm (such as geometric path planning and an artificial potential field method), the scheme based on the TD3 has higher adaptive capacity and flexibility, the strategy can be rapidly adjusted in a dynamic environment, and local optimum is avoided. Besides, the TD3 algorithm improves the stability and convergence of the strategy through delay strategy updating and experience playback, so that the unmanned aerial vehicle cluster keeps real-time obstacle avoidance and formation stability in a high-density scene. Experimental results show that the method can effectively avoid obstacles, maintain a preset formation and reduce energy consumption in a simulation environment, and the application potential of the method in unmanned aerial vehicle cluster cooperative tasks is proved. The method is suitable for multi-unmanned aerial vehicle cluster tasks in the fields of military, logistics distribution, environment monitoring and the like, and the task execution efficiency and safety can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-UAV autonomous collaborative decision-making and control, and specifically to a method for generating an obstacle avoidance strategy for a UAV cluster based on a Twin Delayed Deep Deterministic policy gradient algorithm (TD3). Background Art

[0002] With the rapid development of drone technology, drone swarms, as a collaborative intelligent system, have shown great application potential. Through the coordinated operation of multiple drones, more efficient air coverage, complex task execution, and multi-target management can be achieved. For example, in the security monitoring of large-scale events, drone swarms can monitor a wide area in real time and detect abnormal situations in a timely manner; in disaster relief, drone swarms can quickly search for survivors and improve rescue efficiency; in environmental monitoring, air quality, forest fires, etc. can be monitored in real time. However, in drone swarm systems, the generation of obstacle avoidance strategies is a crucial issue, especially in dynamic environments, where traditional rule-based or algorithm-based obstacle avoidance solutions may not be able to quickly adapt to complex and changing situations. If the obstacle avoidance strategy is not intelligent enough, it may cause drones to collide, causing equipment damage, or even safety accidents.

[0003] The obstacle avoidance methods of traditional drone swarms usually rely on a combination of rule-driven and classical algorithms. The technical implementation of these methods mainly relies on a series of artificially set behavioral rules and obstacle avoidance decisions based on sensor data. Typical rule-driven methods include setting drones to slow down, turn, rise, and other actions when approaching obstacles. These rules are formulated based on pre-assumptions about the mission environment and lack the ability to adapt to unknown situations. In addition, classical algorithms such as geometry-based path planning algorithms (such as the A algorithm, Dijkstra algorithm) and artificial potential field methods are often used to generate safe paths for drone swarms. The A algorithm finds the optimal path by evaluating the heuristic function of the node, but it needs to be recalculated frequently in a dynamic environment, resulting in a huge amount of calculation. The artificial potential field method simulates gravity and repulsion to navigate the drone, but it is easy to fall into local minima and cannot find the global optimal path. The drone perceives the environment through sensors (such as ultrasonic sensors, lidars, or cameras) and adjusts the path according to these algorithms and preset rules. Although these methods work well in static or simple environments, they show obvious limitations in complex dynamic environments.

[0004] First, the rule-driven approach lacks flexibility. When faced with unforeseen situations (such as sudden moving obstacles or environmental changes), traditional systems may not be able to respond effectively in a timely manner. This is because the preset rules cannot cover all possible scenarios. When situations beyond the scope of the rules occur, the drone may not be able to take appropriate obstacle avoidance actions. In addition, the classic path planning algorithm is usually computationally intensive and it is difficult to ensure efficiency in real-time scenarios where multiple drones operate in coordination. Especially in high-density drone clusters, each drone needs to calculate the optimal path in real time, and the consumption of computing resources and communication delays may result in failure to meet real-time requirements. Especially when the environment or mission objectives are complex and changeable, it is difficult for preset rules and fixed algorithms to dynamically adjust strategies according to actual conditions, resulting in drone clusters easily falling into local optimality or being unable to avoid emerging threats when performing tasks. In contrast, the adaptability and intelligence level of traditional obstacle avoidance methods are low, limiting their application potential in modern complex scenarios.

[0005] The development of drone swarm technology has gone through years of research and practice, especially in the military field, and has made significant progress. The earliest research focused on the coordinated control and communication architecture of the cluster, aiming to imitate the collective behavior patterns of birds and fish in nature. These studies borrowed simple rules from biological groups, such as position alignment, speed matching, and collision avoidance, to achieve collaboration between drones. Modern swarm technology relies on autonomous decision-making algorithms, machine learning, and distributed control systems to enable drones to perform tasks in complex environments. For example, through a distributed communication network, drones can share environmental information and collaborate to complete tasks such as reconnaissance, surveillance, and strike, thereby improving the success rate and efficiency of the mission.

[0006] In recent years, the United States, Israel, China and other countries have made milestone breakthroughs in the military application of drone swarms. Israel deployed drone swarms in actual combat for the first time in 2021 to automatically coordinate and strike multiple targets, and its system is able to complete tasks autonomously without human intervention. The U.S. military is also constantly testing drone clusters, such as testing a scenario in which a single person commanded 45 drones in an exercise in 2021, demonstrating the autonomy and coordination capabilities of drone swarms in complex tasks. These drones are able to autonomously form formations, assign tasks, and collaborate with each other during execution to adapt to rapid changes in the battlefield environment. In addition, China's research shows that the country also has considerable innovation capabilities in swarm drone technology, and has demonstrated the operation of drones launched from warships, conducting reconnaissance and attack missions, reflecting the potential of drone swarms in maritime operations.

[0007] The application of deep reinforcement learning technology in the obstacle avoidance strategy of drone swarms will bring revolutionary changes to multiple industries. In the civilian field, such as logistics distribution, environmental monitoring and agricultural spraying, drone swarms with highly autonomous obstacle avoidance capabilities can greatly improve efficiency and safety. Deep reinforcement learning enables drones to autonomously learn the best path in unknown or dynamically changing environments, reduce human intervention, and reduce operational risks. For example, in logistics distribution, drone swarms can dynamically adjust flight routes based on real-time weather and traffic conditions to ensure that goods are delivered quickly and safely. In addition, this technology can also be applied to urban planning and intelligent traffic management. Through the coordinated operation of aerial drone swarms, the pressure on ground traffic can be alleviated, a three-dimensional transportation system can be realized, and new solutions can be provided for the development of smart cities.

[0008] In the military and security fields, the obstacle avoidance strategy of drone swarms using deep reinforcement learning will greatly enhance the flexibility and success rate of missions. Drone swarms can autonomously collaborate in complex battlefield environments, evade enemy defense systems, and achieve tasks such as intelligence collection, target tracking, and precision strikes. For example, in the face of enemy interference and air defense systems, drone swarms can learn to find the safest and most concealed path to infiltrate the target area. At the same time, this technology can also be used in disaster relief and counter-terrorism operations, perform search and rescue missions in high-risk or unknown environments, and protect personnel safety. Through deep reinforcement learning, drones can identify complex terrain and obstacles and improve the success rate of missions. The obstacle avoidance strategy of drone swarms driven by deep reinforcement learning has broad application prospects, which will promote the development of drone technology in a more intelligent and autonomous direction, and further expand the application of drones in various fields. Summary of the invention

[0009] In view of the shortcomings of the above-mentioned traditional technology, the present invention provides a method for generating a TD3-based drone cluster obstacle avoidance strategy, comprising the following steps:

[0010] Combined with the mission objectives, the constants of the drone cluster are determined, including: (unit vector of the cluster target's motion direction), N (density of the target drone cluster)

[0011] Combined with the actual application environment, determine the constants of the interaction between the drone and the environment, including: d (minimum allowed distance between drones), D o (minimum allowed distance between drone and obstacle), f d (Target formation of drone swarm)

[0012] Combined with the actual performance of the UAV, the constants of the UAV control are determined, including: α max (maximum translational acceleration of the drone), β max(Maximum angular acceleration of the drone)

[0013] Combined with the task type and control preference, determine the constants of the reward function: k1 (energy saving reward constant), k2 (cluster action speed reward constant), k3 (cluster density reward constant), k4 (cluster queue shape reward constant)

[0014] Establish a reward function, use double-delayed deep deterministic policy gradient to train the Markov decision model, and generate a drone cluster obstacle avoidance strategy

[0015] In one embodiment, the Markov decision model is specifically (S, A, R, γ)

[0016] Among them, S is the obstacle avoidance state space of the drone cluster, which is:

[0017] S={X i ,Y i ,Z i ,Φ i ,Θ i ,Ψ i ,Vx i ,Vy i ,Vz i ,ωΦ i ,ωΘ i ,ωΨ i}

[0018] Among them, X i , Y i , Z i are the absolute spatial positions of the i-th UAV in the UAV cluster; Φ i , Θ i , i are the roll angle, pitch angle, and yaw angle of the i-th UAV in the UAV cluster; Vx i , Vy i 、Vz i is the translation speed of the i-th UAV in the UAV cluster on the three-axis projection; ωΦ i 、ωΘ i ,ωΨ i is the three-axis angular velocity of the i-th UAV in the UAV cluster;

[0019] A is the action space of the drone cluster to avoid obstacles, and six continuous control quantities a=[αx i ,αy i ,αz i ,βΦ i ,βΘ i ,βΨ i ]Control the drone to perform actions;

[0020] Among them, αxi ,αy i ,αz i is the translation acceleration of the UAV on the three-axis projection, βΦ i ,βΘ i ,βΨ i is the angular acceleration of the drone;

[0021] R is the reward function for the drone to perform actions;

[0022] γ is the discount rate;

[0023] In one embodiment, the reward function is specifically:

[0024] R=kr+k1r1+k2r2+k3r3+k4r4

[0025] Among them, r is the obstacle avoidance success reward, r1 is the energy saving reward, r2 is the cluster action speed reward, r3 is the cluster density reward, r4 is the cluster queue shape reward, k, k1, k2, k3, k4 are weights;

[0026] The rewards for successful obstacle avoidance are:

[0027] Among them, D ij is the straight-line distance between any two UAVs, D d is the minimum allowed distance between drones, D i is the shortest distance between any UAV and the obstacle, D o is the minimum allowed distance between the drone and the obstacle;

[0028] The energy saving reward, the cluster action speed reward, the cluster coverage reward, and the cluster density reward are

[0029] Among them, V i is the translation speed of the i-th UAV in the UAV cluster, α i is the translation acceleration of the i-th UAV in the cluster; C1 and C2 are weight coefficients used to balance the effects of speed and acceleration to penalize excessive speed or acceleration;

[0030] is the velocity vector of the drone, is the unit vector of the cluster target's motion direction;

[0031] n is the actual drone cluster density, and N is the target drone cluster density;

[0032] The cluster queue shape rewards are: r5=-∑E i

[0033] Among them, E i is the formation error of the i-th UAV in the UAV cluster, defined as:

[0034] Among them, e d 、e θ , is the weight coefficient, satisfying

[0035] in, is the normalized distance between the ith UAV in the UAV cluster and the cluster center of gravity, defined as is the normalized angle between the line connecting the i-th UAV and the center of gravity and the z-axis in the UAV cluster, defined as is the normalized azimuth of the line connecting the i-th UAV in the UAV cluster and the center of gravity on the XY plane, defined as

[0036] in, is the normalized coordinate of the i-th UAV in the UAV cluster in the polar coordinate system with the cluster center of gravity as the origin, defined as:

[0037] is the normalized target relative position of the i-th UAV in the input UAV cluster. The purpose of using this input form is to make the shape description of the target formation independent of the absolute coordinates.

[0038] In one embodiment, the Markov decision model is trained using TD3, specifically:

[0039] Initialize two online critic networks Q o1 , Q o2 and the target critic network Q t1 , Q t2 Used to estimate the value of a state-action pair;

[0040] Initialize online Actor network π o and the target Actor network π t , used to generate actions;

[0041] Create an experience replay buffer to store experience samples sampled during training;

[0042] For each time step t, input the current state St Go to the online Actor network and generate the current action A t =π o (S t ), and add Gaussian noise to the actions to ensure the exploratory nature of the strategy;

[0043] Execute action A t After that, the environment returns to the next state S t+1 , Instant Reward R t , and whether the signal is finished t ;

[0044] The experience sample (S t ,A t ,R t ,S t+1 ,d t ) is stored in the experience replay pool;

[0045] Each time, a batch of N experience data (S t ,A t ,R t ,S t+1 ,d t );

[0046] Using Target Actor Network t Generate target action A′ t+1 =π t (S t+1 ), and add smoothing noise;

[0047] The target state S t+1 and target action A′ t+1 Input target Critic network Q t1 , Q t2 , and get two target Q values:

[0048] Q t1 (S t+1 ,A′ t+1 ),Q t2 (S t+1 ,A′ t+1 )

[0049] Take the minimum target Q value and calculate the final target Q value y t :

[0050] y t =R t +γ(1-d t )min(Q t1 (S t+1 ,A′ t+1 ),Q t2 (St+1 ,A′ t+1 ))

[0051] Minimize the current Q value and the target Q value y t The mean square error between:

[0052]

[0053] Then update the parameters of the online critic network through back propagation;

[0054] Update the online Actor network every two Critic updates;

[0055] The goal of the online Actor network is to maximize the value estimate of the online Critic network for the action generated by the current state:

[0056]

[0057] Optimize the online Actor network through back-propagation so that it can generate better obstacle avoidance strategies under the evaluation of the Critic network;

[0058] The parameters of the target Critic network and the target Actor network are gradually approached to the parameters of the online network through soft updates:

[0059] Q t1 =τQ o1 +(1-τ)Q t1 Q t2 =τQ o2 +(1-τ)Q t2 π t =τπ o +(1-τ)π t

[0060] Where τ is the soft update coefficient, which is usually a very small value;

[0061] After the TD3 network training is completed, the target Actor network π t Meeting with online Actor Network π o They are very close and can be used as the generated obstacle avoidance strategy for drone swarms to guide the obstacle avoidance behavior of drones in complex environments;

[0062] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0064] Figure 1 This is a schematic diagram of a state space model of a drone in an embodiment of the present invention;

[0065] Figure 2 Schematic diagram of the UAV motion space model in an embodiment of the present invention;

[0066] Figure 3 Schematic diagram of the state space model of a drone cluster in an embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the formation parameters of the drone cluster in an embodiment of the present invention; DETAILED DESCRIPTION

[0069] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0070] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but they must be implementable by ordinary technicians in the field. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0071] like Figure 1As shown, the present invention discloses a method for generating a drone cluster obstacle avoidance strategy based on TD3 (TwinDelayed Deep Deterministic Policy Gradient), comprising the following steps:

[0072] Combined with specific mission requirements, determine the target motion direction unit vector of the drone cluster and the target drone cluster density N. For example, assuming that the drone cluster's mission is to fly from starting point A to end point B, and the straight line direction is due east, then The target cluster density N can be set according to mission requirements and environmental conditions. For example, if it is set to 0.1, it means that there are an average of 0.1 drones per cubic meter of space.

[0073] According to the flight environment of the drone, set the minimum allowable distance between drones Dd and the minimum allowable distance between drones and obstacles D o , to ensure flight safety. For example, in an urban environment, setting D d =5 m, D o = 3 meters. The target formation of the drone cluster f d It can be set into a V-shape, circle or other specific formations according to mission requirements.

[0074] According to the physical performance parameters of the UAV, determine the maximum translation acceleration α max and the maximum angular acceleration β max For example, the maximum translational acceleration can be set to α max =2m / s 2 , maximum angular acceleration β max =1rad / s2.

[0075] According to the priority and control preference of the task, the weights k, k1, k2, k3, k4 in the reward function are set. For example, if the task requires energy saving to be prioritized, k = 5.0, k1 = 1.0, k2 = 0.5, k3 = 0.5, k4 = 0.5 can be set.

[0076] A Markov decision model for drone swarms is established, including state space S, action space A, reward function R and discount rate γ.

[0077] Definition of state space S: The state space S contains the position information, attitude information, velocity information and angular velocity information of each drone in the drone cluster:

[0078] S={X i ,Y i ,Z i ,Φ i ,Θ i ,Ψ i ,Vxi ,Vy i ,Vz i ,ωΦ i ,ωΘ i ,ωΨ i}

[0079] Among them, X i , Y i , Z i is the spatial coordinate of the i-th UAV in the UAV cluster; Φ i ,Θ i ,Ψ i Vx is the roll angle, pitch angle, and yaw angle of the i-th UAV in the UAV cluster; i ,Vy i ,Vz i is the translational velocity component of the i-th UAV in the UAV cluster; ωΦ i ,ωΘ i ,ωΨ i is the angular velocity component of the i-th UAV in the UAV cluster.

[0080] Definition of action space A: The action space A consists of six continuous control quantities used to control the movement of the drone:

[0081] a=[αx i ,αy i ,αz i ,βΦ i ,βΘ i ,βΨ i ]

[0082] Among them, αx i ,αy i ,αz i is the translational acceleration of the drone in the x, y, and z axes, βΦ i ,βΘ i ,βΨ i is the corresponding angular acceleration. The design of the action space takes into account the dynamic constraints of the UAV to ensure that the generated control instructions are within the capabilities of the UAV.

[0083] Design of reward function R: The reward function R is designed to guide the drone cluster to learn an effective obstacle avoidance strategy while maintaining formation and energy saving. The specific form of the reward function is:

[0084] R=kr+k1r1+k2r2+k3r3+k4r4

[0085] Among them, r is the reward for successful obstacle avoidance. When the minimum distance between drones is not less than D d , and the minimum distance to obstacles is not less than D oWhen r is 1, it takes 0 otherwise.

[0086] r1 is the energy saving reward, defined as:

[0087]

[0088] Among them, V i is the speed of the i-th UAV, α i is its translational acceleration, C1 and C2 are weight coefficients for balancing the effects of velocity and acceleration.

[0089] r2 is the group action speed bonus, defined as:

[0090]

[0091] This reward encourages drones to move in the direction of the target and improves the efficiency of mission completion.

[0092] r3 is the cluster density reward, defined as:

[0093] r3=-|nN| 2

[0094] Where n is the actual cluster density and N is the target cluster density. This reward encourages the drone cluster to maintain the preset density.

[0095] r4 is the cluster queue shape reward, defined as:

[0096] r4=-∑E i

[0097] Among them, E i The formation error of the i-th drone in the drone cluster is defined as:

[0098]

[0099] Among them, e d 、e θ , is the weight coefficient, satisfying is the normalized polar coordinate of the UAV, are the normalized coordinates of the target formation.

[0100] Setting of discount rate γ: The discount rate γ is used to balance the impact of current rewards and future rewards. It is generally set to 0.99, indicating the importance of future rewards.

[0101] Use TD3 algorithm to train Markov decision model: TD3 algorithm is a reinforcement learning algorithm with good convergence and stability. The training process is as follows:

[0102] Initialize the online critic network Q o1 , Q o2 and the target critic network Q t1 , Q t2 , and the online Actor network π o and the target Actor network π t .

[0103] Create an experience replay pool to store experience samples.

[0104] At each time step t, input the current state S t Go to the online Actor network and generate action A t =π o (S t ), and Gaussian noise is added to increase exploration.

[0105] Execute action A t , the environment returns to the next state S t+1 , Instant Reward R t and termination signal d t .

[0106] The experience sample (S t ,A t ,R t ,S t+1 ,d t ) is stored in the experience replay pool.

[0107] Randomly sample a batch of experience data from the experience replay pool.

[0108] Using Target Actor Network t Generate target action A′ t+1 =π t (S t+1 ), and add smoothing noise.

[0109] Input S t+1 and A′ y+1 To the target Critic network Q t1 , Q t2 , and get the target Q value.

[0110] Take the smaller of the two target Q values ​​and calculate the target value y t :

[0111] y t =R t +γ(1-d t )min(Q t1 (S t+1 ,A′ t+1 ),Q t2(S t+1 ,A′ t+1 ))

[0112] Calculate the loss function of the Critic network:

[0113]

[0114] Update the parameters of the online critic network through back-propagation.

[0115] After every two Critic updates, the online Actor network is updated. The loss function of the Actor network is:

[0116]

[0117] The online Actor network is optimized through back-propagation, enabling it to generate better obstacle avoidance strategies.

[0118] The target network parameters are approximated to the online network parameters by soft updating:

[0119] Q t1 =τQ o1 +(1-τ)Q t1

[0120] Q t2 =τQ o2 +(1-τ)Q t2

[0121] π t =τπ o +(1-τ)π t

[0122] Among them, τ is the soft update coefficient, which is usually set to 0.005.

[0123] Specific implementation case: In order to verify the effectiveness of the method of the present invention, a specific case is used for illustration below.

[0124] Case background: Assume that there is a group of 10 drones that need to cross an area with dense obstacles from starting point A to end point B. The drones need to maintain a specific formation while ensuring safety and saving energy as much as possible.

[0125] Parameter settings:

[0126] Target motion direction unit vector: (east)

[0127] Target drone cluster density: N = 0.1

[0128] Minimum allowed distance between drones: D d =5 meters

[0129] Minimum distance between drone and obstacle: D o =3 meters

[0130] Target formation: V-formation

[0131] Maximum translational acceleration of the drone: α max =2m / s2

[0132] UAV maximum angular acceleration: β max =1rad / s2

[0133] Reward function weights: k = 5.0k1 = 1.0, k2 = 1.0, k3 = 0.5, k4 = 0.5

[0134] The weight coefficients of velocity and acceleration: C1 = 0.5, C2 = 0.5

[0135] Training process:

[0136] Environmental simulation: Construct areas with dense obstacles in the simulation environment, including obstacles of different heights and shapes.

[0137] Network initialization: Initialize the Actor and Critic networks and the experience replay pool required by the TD3 algorithm.

[0138] Parameter settings: learning rate is set to 0.001, batch size is 256, discount rate γ = 0.99, and soft update coefficient τ = 0.005.

[0139] Training cycle: At each time step, all drones input the online Actor network according to the current state to obtain action instructions; add Gaussian noise to increase the exploratory nature of the strategy; execute actions, update the environment state, obtain rewards and the next state; store experience samples in the experience replay pool; and update network parameters;

[0140] Training monitoring: During the training process, monitor the changes in reward values ​​and observe the behavior of the drone cluster.

[0141] Result analysis: After about 100,000 training iterations, the strategy gradually converged; the drone cluster was able to successfully avoid obstacles and maintain a V-shaped formation; the cluster density was maintained near the target density; the drones maintained a safe distance and no collisions occurred; and energy consumption was effectively controlled.

[0142] Conclusion: This example verifies the effectiveness of the method of the present invention. Through reasonable reward function design and TD3 algorithm training, drone clusters can achieve obstacle avoidance and formation maintenance in complex environments.

[0143] The above description is only a preferred embodiment of the present invention, and does not limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the technical concept of the present invention and the content of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating obstacle avoidance strategy for drone clusters based on TD3, characterized in that: The steps include: The obstacle avoidance decision problem of UAV swarm is constructed as a Markov decision model; A reward function is established, and the Markov decision model is trained using a double-delayed deep deterministic policy gradient to generate an obstacle avoidance strategy for a swarm of drones; an obstacle avoidance strategy for a swarm of drones is generated based on the trained Markov decision model.

2. The TD3-based UAV cluster obstacle avoidance strategy generation method according to claim 1 is characterized in that: The Markov decision model is specifically (S, A, R, γ), where: S is the obstacle avoidance state space of the drone cluster, which is: Among them, X i , Y i , Z i are the absolute spatial positions of the i-th drone in the drone cluster; Φ i , Ψ i are the roll angle, pitch angle, and yaw angle of the i-th UAV in the UAV cluster; Vx i , Vy i 、Vz i is the translation speed of the i-th UAV in the UAV cluster on the three-axis projection; ωΦ i , ωΨ i is the three-axis angular velocity of the i-th UAV in the UAV cluster; A is the action space for the drone cluster to avoid obstacles, using six continuous control quantities. Control the drone to perform actions; Among them, αx i ,αy i ,αz i is the translation acceleration of the UAV on the three-axis projection, βΦ i , βΨ i is the angular acceleration of the drone; R is the reward function for the drone to perform actions; γ is the discount rate.

3. The method for generating obstacle avoidance strategy for drone clusters based on TD3 according to claim 2 is characterized in that: The reward function is specifically: R=kr+k1r1+k2r2+k3r3+k4r4 Among them, r is the obstacle avoidance success reward, r1 is the energy saving reward, r2 is the cluster action speed reward, r3 is the cluster density reward, r4 is the cluster queue shape reward, k, k1, k2, k3, k4 are weights; The rewards for successful obstacle avoidance are: Among them, D ij is the straight-line distance between any two UAVs, D d is the minimum allowed distance between drones, D i is the shortest distance between any UAV and the obstacle, D o is the minimum allowed distance between the drone and the obstacle; The energy saving reward, the cluster action speed reward, the cluster coverage reward, and the cluster density reward are Among them, V i is the translation speed of the i-th UAV in the UAV cluster, α i is the translation acceleration of the i-th UAV in the UAV cluster; C1 and C2 are weight coefficients used to balance the effects of speed and acceleration to penalize excessive speed or acceleration; is the velocity vector of the drone, is the unit vector of the cluster target's motion direction; n is the actual drone cluster density, and N is the target drone cluster density; The cluster queue shape rewards are: r4＝-∑E i Among them, E i is the formation error of the i-th UAV in the UAV cluster, defined as: Among them, e d 、e θ , is the weight coefficient, satisfying in, is the normalized distance between the ith UAV in the UAV cluster and the cluster center of gravity, defined as is the normalized angle between the line connecting the i-th UAV and the center of gravity and the z-axis in the UAV cluster, defined as is the normalized azimuth of the line connecting the i-th UAV in the UAV cluster and the center of gravity on the XY plane, defined as in, is the normalized coordinate of the i-th UAV in the UAV cluster in the polar coordinate system with the cluster center of gravity as the origin, defined as: is the normalized target relative position of the i-th UAV in the input UAV cluster. The purpose of using this input form is to make the shape description of the target formation independent of the absolute coordinates.

4. The method for generating a TD3-based drone cluster obstacle avoidance strategy according to claim 1, 2 or 3, characterized in that: The method of using TD3 to train the Markov decision model is specifically as follows: The first step is to initialize the TD3 network: Initialize two online critic networks Q o1 , Q o2 and the target critic network Q t1 , Q t2 Used to estimate the value of a state-action pair; Initialize online Actor network π o and the target Actor network π t , used to generate actions; Create an experience replay buffer to store experience samples sampled during training; The second step is to collect experience samples: For each time step t, input the current state S t Go to the online Actor network and generate the current action A t =π o (S t ), and add Gaussian noise to the actions to ensure the exploratory nature of the strategy; Execute action A t After that, the environment returns to the next state S t+1 , Instant Reward R t , and whether the signal is finished t ; The experience sample (S t ,A t ,R t ,S t+1 ,d t ) is stored in the experience replay pool; The third step is to update the Critic network: Each time, a batch of N experience data (S t ,A t ,R t ,S t+1 ,d t ); Using Target Actor Network t Generate target action A′ t+1 =π t (S t+1 ), and add smoothing noise; The target state S t+1 and target action A′ t+1 Input target Critic network Q t1 , Q t2 , and get two target Q values: Q t1 (S t+1 ,A′ t+1 ),Q t2 (S t+1 ,A′ t+1 ) Take the minimum target Q value and calculate the final target Q value y t : yes t =R t +γ(1-d t )min(Q t1 (S t+1 ,A′ t+1 ),Q t2 (S t+1 ,A′ t+1 )) Minimize the current Q value and the target Q value y t The mean square error between: Then update the parameters of the online critic network through back propagation; Step 4: Update the Actor network: Update the online Actor network every two Critic updates; The goal of the online Actor network is to maximize the value estimate of the online Critic network for the action generated by the current state: Optimize the online Actor network through back-propagation so that it can generate better obstacle avoidance strategies under the evaluation of the Critic network; Step 5: Soft update target network: The parameters of the target Critic network and the target Actor network are gradually approached to the parameters of the online network through soft updates: Q t1 =τQ o1 +(1-τ)Q t1 Q t2 =τQ o2 +(1-τ)Q t2 p t =tp o +(1-τ)π t Where τ is the soft update coefficient, which is usually a very small value; Step 6: Generate the strategy after training: After the TD3 network training is completed, the target Actor network π t Meeting with online Actor Network π o They are very close and can be used as the generated obstacle avoidance strategy for drone swarms to guide the obstacle avoidance behavior of drones in complex environments.

Images