Unmanned aerial vehicle cluster intelligent cooperative obstacle avoidance flight path planning method for multiple targets

Abstract

The invention discloses a multi-target-oriented unmanned aerial vehicle cluster intelligent cooperative obstacle avoidance flight path planning method. The invention belongs to the field of unmanned aerial vehicle control, and mainly solves the problem of multi-unmanned aerial vehicle route planning in a multi-target scene. According to the method, a TD3-PER-based flight path planning method is adopted to train the single-unmanned-aerial-vehicle flight path planning capability, the single-unmanned-aerial-vehicle flight path planning capability is expanded to a multi-unmanned-aerial-vehicle system method, so that the decision-making efficiency and the path optimization capability of each unmanned aerial vehicle in a multi-target task are improved, and a collaborative flight path planning decision-making model is optimized in combination with a segmented modulation strategy, so that the flight path planning efficiency of each unmanned aerial vehicle is improved. The unmanned aerial vehicle has stronger obstacle avoidance and better flight path planning capability, and meanwhile, a set of multi-target non-sparse reward function is designed, so that the unmanned aerial vehicle is better helped to make a behavior decision. According to the method, the multi-unmanned aerial vehicle system is ensured to perform cluster flight on the basis of completing a multi-target task, and efficient obstacle avoidance and intelligent cooperative flight path planning are realized.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned aerial vehicle (UAV) control, and in particular relates to the design of a multi-UAV collaborative trajectory planning method based on deep reinforcement learning. Background Art

[0002] With the rapid development of drone technology, the collaborative operation of drone clusters has gradually been applied in many fields such as urban traffic management, disaster relief, and intelligent logistics, bringing convenience to people's lives and work and providing safety guarantees. The basis for the widespread use of drones lies in the support of trajectory planning technology. Compared with the trajectory planning of a single drone, the collaborative trajectory planning problem of multiple drones is more difficult and complex. Not only do drones need to avoid obstacles in the environment at all times, but they also need to ensure a safe distance between each drone in real time to prevent collisions. Based on deep reinforcement learning, the present invention proposes a multi-task oriented drone cluster intelligent collaborative obstacle avoidance trajectory planning method for the problem of multi-drone trajectory planning in multi-target scenarios. The method proposed in the present invention can use a drone cluster trajectory planning algorithm based on reinforcement learning in a multi-target scenario to perform cluster flight on the basis of completing multi-target tasks, thereby achieving efficient obstacle avoidance and intelligent collaborative trajectory planning. Summary of the Invention

[0003] The purpose of the present invention is to provide an efficient and generalizable trajectory planning method for multiple UAVs to perform various tasks in multi-target scenarios. This method proposes a multi-UAV trajectory planning method based on Twin Delayed Deep Deterministic Policy Gradient with Prioritized Experience Replay (TD3-PER), combined with a segmented modulation strategy to optimize the collaborative trajectory planning decision model, to achieve efficient obstacle avoidance and collaborative trajectory planning for UAVs in complex environments. To achieve this purpose, the steps adopted by the present invention are:

[0004] Step 1: This paper proposes a TD3-PER-based trajectory planning method to train the trajectory planning capabilities of a single UAV and extends it to a multi-UAV system to improve the decision-making efficiency and path optimization capabilities of each UAV in multi-target tasks.

[0005] Step 2: The present invention proposes a phased modulation network structure optimization strategy, which divides the training process of the action strategy network into two stages. The first stage trains two single-target networks, namely the obstacle avoidance network and the target approach network, and the second stage is used to train the global network, so that the UAV has stronger obstacle avoidance and better trajectory planning capabilities.

[0006] Step 3: This paper designs a set of multi-objective non-sparse reward functions, which include goal-driven rewards, obstacle avoidance behavior-guided rewards, and fixed-time penalties, so as to better help drones make behavioral decisions.

[0007] The effectiveness of the multi-target UAV swarm intelligent collaborative obstacle avoidance trajectory planning method proposed in this paper has been verified through simulation experiments. The simulation environment area during the test is 1500×1500 m 2 , the obstacle density is 0.3, and the number of drones is 5. Figure 1 A schematic diagram of the overall framework of the method of the present invention is given; Figure 2 A comparison chart of the average success rate of the algorithms under different target numbers is given, involving TD3-PER, Deep Deterministic Policy Gradient with Prioritized Experience Replay (DDPG-PER), and the original TD3 algorithm; Appendix Figure 3 The UAV trajectory diagram of the method proposed in this invention when the number of targets is 6 is given. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention;

[0009] Figure 2 This is a comparison chart of the average success rate of the algorithm under different target numbers of the method of the present invention;

[0010] Figure 3 This is the UAV track diagram of the method of the present invention when the number of targets is 6; DETAILED DESCRIPTION

[0011] The swarm intelligent collaborative obstacle avoidance trajectory planning method of the present invention is described in further detail below.

[0012] Step 1: Collaborative trajectory planning based on TD3

[0013] To avoid the overestimation problem of the deep deterministic policy gradient algorithm, this paper uses the dual critic network structure of the TD3 algorithm to calculate the Q value and takes the minimum of the two as the final target Q value. The target Q values calculated by the two target critic networks are:

[0014]

[0015] Among them, ω C1′ and ω C2′ They represent the parameters of the two target critic networks respectively, and the final target Q value is:

[0016]

[0017] The update of the two critic network parameters follows the following formula:

[0018]

[0019] Among them, ω Q1 and ω Q2 denote the parameters of the two critic networks respectively; M is the batch size; ψ i It is the importance sampling in priority experience replay.

[0020] The update of action strategy network parameters follows the following formula:

[0021]

[0022] Among them, θμ represents the parameters of the action policy network.

[0023] The updates of the target action policy network and the target critic network are delayed and adopt a "soft update" approach:

[0024]

[0025] where τ is the network update parameter and τ∈[0 , 1].

[0026] At the same time, in order to break the correlation between samples and fully mine and utilize experience samples, the experience replay mechanism is introduced into the TD3 algorithm, and the TD3-PER algorithm is proposed. The priority of sample i is calculated from the TD error, that is:

[0027]

[0028] where R i represents the reward value in the current experience, δ i is the TD error, ρ is a constant greater than 0, ensuring that the sampling probability of the sample is not 0, and λ is the reward value adjustment coefficient. Therefore, the sampling probability of sample i is as follows:

[0029]

[0030] Where N is the total number of samples, Indicates the priority of the i-th sample, and α∈[0,1] is a tuning parameter. In order to reduce the error caused by the change of sample priority due to the addition of the priority experience replay mechanism, importance sampling is introduced to stabilize the training process:

[0031]

[0032] Among them, M is the sample batch size, β is the influencing parameter, which determines the influence of the corresponding sample on the convergence result. In actual training, ψ i δ i Will replace the original time difference error δ i , used to update the neural network parameters.

[0033] Step 2: Modulate the network structure optimization strategy in stages

[0034] In order to enable the UAV to have stronger obstacle avoidance and better trajectory planning capabilities, the present invention adopts a segmented modulation method and divides the training process of the action strategy network into two stages. The first stage trains two single-target networks, namely the obstacle avoidance network and the target approach network, and the second stage is used to train the global network.

[0035] In the first stage, the agent first extracts the obstacle-related state S from the global state space. o ={S e , S n} and target-related state S g , and input them into two independent sub-action strategy networks for training obstacle avoidance and target approach capabilities.

[0036] On the one hand, the sub-action strategy network 1 converts S o As input, the two-layer neural network outputs the obstacle avoidance related action A o On the other hand, the sub-action strategy network 2 converts S g As input, the two-layer neural network outputs the target approximation related action A g .

[0037] In the second stage, the agent combines the global state space S with the intermediate output action A o and A g As the input of the global network, it makes the final decision output.

[0038] Among them, the global state space S={S u , S e , S n , S g}. S u =[θ t , v t , w t ], represents the state of the drone itself, including the forward direction θ t , flight speed v t and the flight altitude change w t ;S e ={d t 1，j , d t 2,j , dt 3,j |j∈[1, 12]}, represents the state of surrounding obstacles, including the distances of all surrounding obstacles detected by the three-layer lidar; Indicates the state of the nearby drones, consisting of the distances to the two nearest neighboring drones; Represents the target mission point state, which is composed of the relative distance and angle between the current position and the next target mission point; the UAV’s action space A=[a t ,φ t ,k t ], where a t ∈[-1, 1] is used to control the speed change of the drone, φ t ∈[-1, 1] is used to control the steering angle of the drone, k t ∈[-1, 1] is used to control the flight altitude of the UAV, A o and A g is the intermediate output action, which has the same dimension as the drone action space.

[0039] Step 3: Design a multi-objective non-sparse reward function

[0040] In order to better help drones make behavioral decisions, this paper designs a set of multi-objective non-sparse reward functions for multi-drone systems. For each drone i, its reward function consists of three parts, as follows. The first part is the reward for approaching the target, i.e.

[0041]

[0042] Where e is the natural logarithm constant; D0 is the maximum distance threshold between the drone and the target; is the distance between the UAV and the target at time t; is the distance between the UAV and the target at time t-1; when the distance between the UAV and the target exceeds D0, it will receive a fixed penalty.

[0043] The second part is the reward for avoiding obstacles, i.e.

[0044]

[0045] Among them, D r is the shortest distance threshold between the drone and the obstacle; d min is the minimum distance between the drone and nearby obstacles; ρ is a penalty constant. When the drone is too close to the obstacle and is less than the distance threshold D r , there will be a relatively large penalty, prompting no one to stay away from obstacles.

[0046] The third part is a fixed time step penalty, which is used to encourage the drone to reach the target as quickly as possible.

[0047] r3=-3. (12)

[0048] Therefore, the total reward function of drone i is:

[0049] R=r1+r2+r3. (13)

[0050] By setting up the above three rewards, the drone can ensure efficient obstacle avoidance and intelligent collaborative trajectory planning in multi-target scenarios.

[0051] The contents not described in detail in the present application belong to the prior art known to professional and technical personnel in this field.

Claims

1. A multi-target UAV swarm intelligent collaborative trajectory planning method, the steps adopted are: Step 1: This paper proposes a trajectory planning method based on TD3-PER (Prioritized Experience Replay) to train the trajectory planning capabilities of a single UAV and extends it to a multi-UAV system to improve the decision-making efficiency and path optimization capabilities of each UAV in multi-target tasks. Step 2: The present invention proposes a phased modulation network structure optimization strategy, which divides the training process of the action strategy network into two stages. The first stage trains two single-target networks, namely the obstacle avoidance network and the target approach network, and the second stage is used to train the global network, so that the UAV has stronger obstacle avoidance and better trajectory planning capabilities. Step 3: This paper designs a set of multi-objective non-sparse reward functions, which include goal-driven rewards, obstacle avoidance behavior-guided rewards, and fixed-time penalties, so as to better help drones make behavioral decisions.

2. The method according to claim 1, characterized in that The method of collaborative trajectory planning based on TD3 is: To avoid the overestimation problem of the deep deterministic policy gradient algorithm, this paper uses the dual critic network structure of the TD3 algorithm to calculate the Q value and takes the minimum of the two as the final target Q value. The target Q values calculated by the two target critic networks are: Among them, ω C1 and ω C2 They represent the parameters of the two target critic networks respectively, and the final target Q value is: The update of the two critic network parameters follows the following formula: Among them, ω Q1 and ω Q2 denote the parameters of the two critic networks respectively; M is the batch size; ψ i It is the importance sampling in priority experience replay. The update of action strategy network parameters follows the following formula: Among them, θ μ Represents the parameters of the action policy network. The updates of the target action policy network and the target critic network are delayed and adopt a "soft update" approach: where τ is the network update parameter and τ∈[0,1]. At the same time, in order to break the correlation between samples and fully mine and utilize experience samples, the experience replay mechanism is introduced into the TD3 algorithm, and the TD3-PER algorithm is proposed. The priority of sample i is calculated from the TD error, that is: where R i represents the reward value in the current experience, δ i is the TD error, ρ is a constant greater than 0, ensuring that the sampling probability of the sample is not 0, and λ is the reward value adjustment coefficient. Therefore, the sampling probability of sample i is as follows: Where N is the total number of samples, Indicates the priority of the i-th sample, and α∈[0,1] is a tuning parameter. In order to reduce the error caused by the change of sample priority due to the addition of the priority experience replay mechanism, importance sampling is introduced to stabilize the training process: Among them, M is the sample batch size, β is the influencing parameter, which determines the influence of the corresponding sample on the convergence result. In actual training, ψ i δ i Will replace the original time difference error δ i , used to update the neural network parameters.

3. The method according to claim 1, characterized in that The optimization strategy for the phased modulation network structure is: In order to enable the UAV to have stronger obstacle avoidance and better trajectory planning capabilities, the present invention adopts a segmented modulation method and divides the training process of the action strategy network into two stages. The first stage trains two single-target networks, namely the obstacle avoidance network and the target approach network, and the second stage is used to train the global network. In the first stage, the agent first extracts the obstacle-related state S from the global state space. o ={S e , S n } and target-related state S g , and input them into two independent sub-action strategy networks for training obstacle avoidance and target approach capabilities. On the one hand, the sub-action strategy network 1 converts S o As input, the two-layer neural network outputs the obstacle avoidance related action A o On the other hand, the sub-action strategy network 2 converts S g As input, the two-layer neural network outputs the target approximation related action A g . In the second stage, the agent combines the global state space S with the intermediate output action A o and A g As the input of the global network, it makes the final decision output. Among them, the global state space S={S u , S e , S n , S g }. S u =[θ t ,v t , w t ], represents the state of the drone itself, including the forward direction θ t , flight speed v t and the flight altitude change w t ;S e ={d t 1，j , d t 2,j , d t 3，j |j∈[1, 12]}, represents the state of surrounding obstacles, including the distances of all surrounding obstacles detected by the three-layer lidar; Indicates the state of the nearby drones, consisting of the distances to the two nearest neighboring drones; Represents the target mission point state, which is composed of the relative distance and angle between the current position and the next target mission point; the UAV’s action space A=[a t ,φ t , k t ], where a t ∈[-1, 1] is used to control the speed change of the drone, φ t ∈[-1, 1] is used to control the steering angle of the drone, k t ∈[-1, 1] is used to control the flight altitude of the UAV, A o and A g is the intermediate output action, which has the same dimension as the drone action space.

4. The method according to claim 1, characterized in that The multi-objective non-sparse reward function is: In order to better help drones make behavioral decisions, this paper designs a set of multi-objective non-sparse reward functions for multi-drone systems. For each drone i, its reward function consists of three parts, as follows. The first part is the reward for approaching the target, i.e. Where e is the natural logarithm constant; D0 is the maximum distance threshold between the drone and the target; is the distance between the UAV and the target at time t; is the distance between the UAV and the target at time t-1; when the distance between the UAV and the target exceeds D0, it will receive a fixed penalty. The second part is the reward for avoiding obstacles, i.e. Among them, D r is the shortest distance threshold between the drone and the obstacle; d min is the minimum distance between the drone and nearby obstacles; ρ is a penalty constant. When the drone is too close to the obstacle and is less than the distance threshold D r , there will be a relatively large penalty, prompting no one to stay away from obstacles. The third part is a fixed time step penalty, which is used to encourage the drone to reach the target as quickly as possible. r3＝-3. (12) Therefore, the total reward function of drone i is: R＝r1+r2+r3. (13) By setting up the above three rewards, the drone can ensure efficient obstacle avoidance and intelligent collaborative trajectory planning in multi-target scenarios.

Images