Layered cooperative multi-unmanned aerial vehicle formation route planning method

Abstract

The invention designs a set of leader-follower-based dynamic formation model. The model is composed of following gravitational force and anti-collision repulsive force and is used for constructing a customized reward function to assist in adjusting the formation of the follower unmanned aerial vehicle. A set of dynamic formation reward scheme is designed on the basis of a multi-unmanned aerial vehicle system, two sets of reward functions are designed for a leader and a follower, and the collaborative formation level of the leader and the follower is effectively improved. On the basis of a multi-agent double-delay depth deterministic strategy gradient method (MATD3), the invention provides a multi-agent double-delay depth deterministic strategy gradient method (Leader-Follower MATD3, LFMATD3) which is combined with a navigator and follower model. The ability of single unmanned aerial vehicle flight path planning is trained by adopting a double-delay depth deterministic strategy gradient method (TD3) and is expanded to a method of a multi-unmanned aerial vehicle system, so that the behavior decision-making ability of the unmanned aerial vehicle is effectively improved. According to the method, it is ensured that the multi-unmanned-aerial-vehicle system achieves real-time dynamic allocation of tasks, meanwhile, the higher flight path planning capacity is obtained, and the effectiveness and robustness of the method are proved through simulation experiments.

Description

Technical Field

[0001] The present invention belongs to the field of UAV control, and in particular relates to the design of a multi-UAV formation collaborative trajectory planning method based on deep reinforcement learning. Background Art

[0002] Unmanned aerial vehicle (UAV) technology has rapidly advanced in recent years, and multi-UAV trajectory planning technology has shown great potential in a variety of fields, including agricultural seeding, environmental monitoring, and reconnaissance and early warning. As the variety and difficulty of missions increase, the demand for collaborative multi-UAV formations also increases. Therefore, improving mission efficiency and UAV coordination is a key research goal. However, in actual flight, trajectory planning for a single UAV often faces numerous challenges, such as real-time obstacle avoidance, such as buildings and trees, and the need to rapidly adapt to changing environments, such as dynamic ones, to ensure flight safety and mitigate risk. Multi-UAV formations present even greater challenges, requiring not only obstacle avoidance but also appropriate spacing and angles between drones, as well as the ability to instantly reconfigure or change formations. Therefore, while each drone must have an accurate and safe route, multi-UAV formations must ensure they can handle complex situations and emergencies, while optimizing inter-UAV flight connectivity.

[0003] Therefore, this paper proposes a hierarchical collaborative multi-UAV formation trajectory planning method based on deep reinforcement learning. This method leverages a reward function based on an artificial potential field to enable UAVs to flexibly adapt to complex environments, maintain formation, and improve system scalability. Summary of the Invention

[0004] The purpose of this invention is to enable multiple drones to perform various tasks in formation in dense obstacle environments, providing an efficient and highly generalizable trajectory planning method. Specifically, this method uses a reward function with artificial potential forces to train followers, introduces a leader-follower model to maintain drone formation, and uses a multi-agent dual-delay deep deterministic policy gradient method with prioritized experience replay for multi-drone trajectory planning. To achieve this goal, the present invention employs the following steps:

[0005] Step 1: We design a leader-follower dynamic formation model. This model consists of a following force and an anti-collision force, which are used to construct a customized reward function to assist in adjusting the formation of follower drones.

[0006] Step 2: The present invention models the multi-UAV collaborative trajectory planning problem as a partially observable Markov decision process (POMDP), because the UAVs in this mission environment have incomplete environmental information.

[0007] Step 3: This paper designs a dynamic formation reward scheme based on a multi-UAV system. Two sets of reward functions are designed for the leader and followers, including rewards for approaching the target, obstacle avoidance, collision avoidance, and maintaining communication, to encourage the leader and followers to achieve a collaborative formation.

[0008] Step 4: Based on the Multi-Agent Double-Delay Deep Deterministic Policy Gradient Method (MATD3), a Multi-Agent Double-Delay Deep Deterministic Policy Gradient Method (LFMATD3) combined with a leader-follower model is proposed. This method uses the double-delay deep deterministic policy gradient method (TD3) to train the trajectory planning capabilities of a single UAV and extends it to a multi-UAV system.

[0009] The effectiveness of the hierarchical cooperative multi-UAV formation trajectory planning method proposed in this invention has been verified through simulation experiments. Figure 1 This is the overall logical framework diagram, Figure 2 and attached Figure 3 They are formation effect diagram and convergence performance curve diagram, attached Figure 4 and attached Figure 5 The success rate comparison chart and formation duration comparison chart of the proposed method and the baseline method (MATD3) under different obstacle densities are given respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Attachment Figure 1 It is a schematic diagram of the overall logical framework of the method of the present invention;

[0011] Attachment Figure 2 It is a formation effect diagram of the method of the present invention;

[0012] Attachment Figure 3 is a graph showing the convergence performance of the method of the present invention;

[0013] Attachment Figure 4 This is a comparison chart of the success rates of the method of the present invention under different obstacle densities;

[0014] Attachment Figure 5 This is a comparison chart of formation duration under different obstacle densities using the method of the present invention; Specific implementation methods

[0015] The present invention is described in further detail below with reference to the accompanying drawings and examples.

[0016] Step 1: Build a leader-follower-based dynamic formation model (1)

[0018] The function following gravity is:

[0019]

[0020] Among them, p j is the current position of follower drone j, p leader is the leader drone position, d(p j , p leader ) is the relative distance between follower j and the leader, D0 is the distance threshold, and ξ is the gravitational coefficient. To follow gravity, the direction is pointed from the follower drone to the leader drone. (2)

[0022] Anti-collision repulsion function:

[0023]

[0024] Where ε is the repulsion coefficient, D1 is the safety threshold between machines, d(p j , p k ) is the relative distance between follower j and follower k, so the total anti-collision repulsive force on follower j is:

[0025]

[0026] Among them, N is the other follower drones other than follower drone j, and its total repulsive force value is

[0027] Step 2: Model the multi-UAV collaborative trajectory planning problem as a partially observable Markov decision process

[0028] (1) State space

[0029] In the leader-follower model, the present invention defines the state space S of the leader UAV as leader for:

[0030] S leader ={s self , s obstacles , s goal}. (4)

[0031] where s self Status information for the leader drone, including v t ,θ t ;S obstacles To obtain obstacle status information, it includes 16 laser radars in different directions to obtain obstacle information around the drone. s goal is the target-related state space, including the straight-line distance d between the leader UAV and the target 17 The angle between the leader drone's flight direction and the target direction

[0032] The state space S of follower drone j j for:

[0033] S j ={s self , s obstacles , S l , s other}. (5)

[0034] Among them, s self is the status information of the follower UAV j, including v t ,θ t ;S obstacles To obtain obstacle status information, it includes 16 laser radars in different directions to obtain obstacle information around the drone s l is the state space associated with the leader, including the horizontal straight-line distance between follower UAV j and the leader UAV and the angle between the flight direction of the follower UAV j and the direction of the leader UAV S other is the distance between follower j and other followers in the vicinity, including the distance between follower j and the two adjacent follower drones

[0035] (2) Action space:

[0036] The action space is the set of actions that a drone can take. This paper represents the action space of a single drone as:

[0037] A i =[a i ,φ i (6)

[0038] a i and φ i They are the increase and decrease of the speed and heading of the drone i respectively.

[0039] Therefore, the action space of all drones is:

[0040] A={A1…A N}. (7)

[0041] Step 3: Set up a dynamic formation reward scheme based on a multi-UAV system

[0042] To effectively motivate drones to collaborate and guide them to explore deeper, this paper designs unique reward functions for different roles. For leader drone i, its reward function consists of three parts. The first part is the reward for approaching the target:

[0043]

[0044] in is the distance between UAV i and the target at time t; is the straight-line distance between UAV i and the target at time t-1.

[0045] The second part is the obstacle avoidance reward function:

[0046]

[0047] in The drone is equipped with 16 laser radars in different directions to obtain the distance of obstacles around the drone; is the minimum of these distances; D obstacle is the distance threshold from drone i to the obstacle; otherwise, it is the case that the obtained distance is greater than or equal to the distance threshold.

[0048] The third part is the i-time step penalty of the leader drone to ensure that the drones can complete the task as quickly as possible:

[0049]

[0050] Therefore, the total reward of leader drone i is:

[0051]

[0052] For the follower drone j, its reward function consists of four parts. The first part is the angle constraint reward function:

[0053]

[0054] Among them, α is the angle weight coefficient; θ j is the angle between the follower UAV j and the connecting lines of other adjacent UAVs; θ0 is the angle that the follower UAV j is expected to maintain with its adjacent UAVs.

[0055] The second part is the distance constraint reward function:

[0056]

[0057] In this formation, the collision distance threshold D c It can ensure that followers can effectively avoid collisions and obstacles; is the distance between follower UAV j and its two neighboring UAVs; is the minimum of these distances.

[0058] The third part is the following constraint reward function:

[0059]

[0060] Among them, d 17，t is the distance between the follower UAV and the leader at time t; d 17,t-1 is the distance between the follower and the leader at time t-1; d l is the distance from follower UAV j to leader UAV i; e is the base of natural logarithm; otherwise, d l Not in the range [5, 25].

[0061] The fourth part is the potential field force reward function:

[0062]

[0063] in, is the following gravity reward of follower drone j at time t-1; is the following gravity reward of follower drone j at time t, is the anti-collision repulsion reward of follower drone j at time t.

[0064] Therefore, the total reward of follower drone j is:

[0065]

[0066] Step 4: Multi-agent double-delayed deep deterministic policy gradient method combined with leader-follower model

[0067] Based on the TD3 training capability of each UAV trajectory planning, it is extended to a multi-UAV system with a leader and a follower. In the LFMATD3 algorithm proposed in this paper, each UAV contains six networks: the critic network Q1 (s, a1, ..., a N |θ Q1 ) and Q2(s, a1, ..., a N |θ Q2 ), action network π(o|θ π ), and the corresponding target network Q′1(s′, a′1, ..., a′ N |θ′ Q1 ), Q′2(s′,a′1,...,a′ N |θ′ Q2 ),π′(o′|θ′ π ), where θ π is the weight of the action neural network, θ Qis the critic network weight, evaluating the value of the state-action pair, and o is the observation value. The two critic networks are used to evaluate the value function of a given state and action, reducing the overestimation of Q values. The action network provides action decisions by observing the surrounding environment.

[0068] In the training phase, a batch of samples are first sampled from the priority experience replay buffer and the network parameters of all drones are updated using these samples. In this process, drone i will update the network parameters of all drones according to its observation value o at the current time step. i，t To determine the action, and output the action a i，t Feedback to the environment to obtain the reward value r of the current time step i and the observation value o at the next time step i,t+1 , and save it in the priority experience replay pool, and repeat the training until the end. At the same time, in order to simulate the impact of uncertain factors in the environment on the action, the present invention introduces Ornstein-Uhlenbeck noise in the output action, that is, where μ1 represents the mean, θ1 is the regression coefficient, and σ1 is the diffusion factor.

[0069] a i，t =π i (o i，t |θ i，t )+OU(μ1,θ1,σ1). (17)

[0070] Where μ1 represents the mean, θ1 is the regression coefficient, and σ1 is the diffusion factor.

[0071] The action network updates its parameters through policy gradient ascent:

[0072]

[0073] where θ Q1 are the parameters of the critic network, Q1 is the output of the critic network, is the gradient of the Q function with respect to action a under state s and action a1, In state o, action a is about the action network parameter θ μ The gradient of , M represents the number of samples used to update the network.

[0074] Update the critic network by minimizing the Bellman error:

[0075]

[0076] where y is the target value found by the two target critic networks; θ Q1 ,θ Q2 Denote the parameters of the two critic networks respectively; Q i，1 , Q i,2Represent the outputs of the two critic networks respectively;

[0077] L represents the minimum error between the output Q value and the target value.

[0078] At the same time, each target network adopts the soft update method:

[0079]

[0080] Among them, θ π ,θ′ π are the parameters of the action network and the target action network, θ Q1 ,θ′ Q1 are the parameters of critic network 1 and target critic network 1, θ Q2 ,θ′ Q2 are the parameters of critic network 2 and target critic network 2, respectively.

[0081] To improve the utilization of MATD3 training samples and the robustness of the algorithm, this paper incorporates Prioritized Experience Replay (PER) into the MATD3 algorithm. The proposed LFMATD3_PER algorithm updates sample priorities by calculating the TD error to avoid sampling probabilities of zero, and uses importance sampling weights to calculate the weighted loss function.

[0082] The contents not described in detail in the present application belong to the prior art known to professional and technical personnel in this field.

Claims

1. A hierarchical cooperative multi-UAV formation trajectory planning method. The steps used are: Step 1: We design a leader-follower dynamic formation model. This model consists of a following force and an anti-collision force, which are used to construct a customized reward function to assist in adjusting the formation of follower drones. Step 2: The present invention models the multi-UAV collaborative trajectory planning problem as a partially observable Markov decision process (POMDP), because the UAVs in this mission environment have incomplete environmental information. Step 3: This paper designs a dynamic formation reward scheme based on a multi-UAV system. Two sets of reward functions are designed for the leader and followers, including rewards for approaching the target, obstacle avoidance, collision avoidance, and maintaining communication, to encourage the leader and followers to achieve a collaborative formation. Step 4: Based on the Multi-Agent Double-Delay Deep Deterministic Policy Gradient Method (MATD3), a Multi-Agent Double-Delay Deep Deterministic Policy Gradient Method (LFMATD3) combined with a leader-follower model is proposed. This method uses the double-delay deep deterministic policy gradient method (TD3) to train the trajectory planning capabilities of a single UAV and extends it to a multi-UAV system.

2. The method according to claim 1, characterized in that The method for constructing a dynamic formation model based on leader-follower is: (1) The function following gravity is: Among them, p j is the current position of follower drone j, p leader is the leader drone position, d(p j , p leader ) is the relative distance between follower j and the leader, D0 is the distance threshold, and ξ is the gravitational coefficient. To follow gravity, the direction is pointed from the follower drone to the leader drone. (2) Anti-collision repulsion function: Where ε is the repulsion coefficient, D1 is the safety threshold between machines, d(p j , p k ) is the relative distance between follower j and follower k, so the total anti-collision repulsive force on follower j is: Among them, N is the other follower drones other than follower drone j, and its total repulsive force value is 3. The method according to claim 1, characterized in that The method of modeling the multi-UAV collaborative trajectory planning problem as a partially observable Markov decision process is as follows: (1) State space In the leader-follower model, the present invention defines the state space S of the leader UAV as leader for: S leader ={s self ,s obstacles ,s goal }. (4) where s self Status information for the leader drone, including v t ,θ t ;S obstacles To obtain obstacle status information, it includes 16 laser radars in different directions to obtain obstacle information around the drone. s goal is the target-related state space, including the straight-line distance d between the leader UAV and the target 17 The angle between the leader drone's flight direction and the target direction The state space S of follower drone j j for: S j ={s self ,s obstacles ,s l ,s other }. (5) Among them, s self is the status information of the follower UAV j, including v t ,θ t ;S obstacles To obtain obstacle status information, it includes 16 laser radars in different directions to obtain obstacle information around the drone s l is the state space associated with the leader, including the horizontal straight-line distance between follower UAV j and the leader UAV and the angle between the flight direction of the follower UAV j and the direction of the leader UAV s other is the distance between follower j and other followers in the vicinity, including the distance between follower j and the two adjacent follower drones (2) Action space: The action space is the set of actions that a drone can take. This paper represents the action space of a single drone as: A i =[a i ,f i ]. (6) a i and φ i They are the increase and decrease of the speed and heading of the drone i respectively. Therefore, the action space of all drones is: A={A1…A N }. (7) 4. The method according to claim 1, characterized in that The dynamic formation reward scheme based on the multi-UAV system is: To effectively motivate drones to collaborate and guide them to explore deeper, this paper designs unique reward functions for different roles. For leader drone i, its reward function consists of three parts. The first part is the reward for approaching the target: in is the distance between UAV i and the target at time t; is the straight-line distance between UAV i and the target at time t-1. The second part is the obstacle avoidance reward function: in The drone is equipped with 16 laser radars in different directions to obtain the distance of obstacles around the drone; is the minimum of these distances; D obstacle is the distance threshold from drone i to the obstacle; otherwise, it is the case that the obtained distance is greater than or equal to the distance threshold. The third part is the i-time step penalty of the leader drone to ensure that the drones can complete the task as quickly as possible: Therefore, the total reward of leader drone i is: For the follower drone j, its reward function consists of four parts. The first part is the angle constraint reward function: Among them, α is the angle weight coefficient; θ j is the angle between the follower UAV j and the connecting lines of other adjacent UAVs; θ0 is the angle that the follower UAV j is expected to maintain with its adjacent UAVs. The second part is the distance constraint reward function: In this formation, the collision distance threshold D c It can ensure that followers can effectively avoid collisions and obstacles; is the distance between follower UAV j and its two neighboring UAVs; is the minimum of these distances. The third part is the following constraint reward function: Among them, d 17，t is the distance between the follower UAV and the leader at time t; d 17,t-1 is the distance between the follower and the leader at time t-1; d l is the distance from follower UAV j to leader UAV i; e is the base of natural logarithm; otherwise, d l Not in the range [5, 25]. The fourth part is the potential field force reward function: in, is the following gravity reward of follower drone j at time t-1; is the following gravity reward of follower drone j at time t, is the anti-collision repulsion reward of follower drone j at time t. Therefore, the total reward of follower drone j is:

5. The method according to claim 1, characterized in that The multi-agent double-delay deep deterministic policy gradient method combined with the leader-follower model is: Based on the TD3 training capability of each UAV trajectory planning, it is extended to a multi-UAV system with a leader and a follower. In the LFMATD3 algorithm proposed in this paper, each UAV contains six networks: the critic network Q1 (s, a1, ..., a N |θ Q1 ) and Q2(s, a1, ..., a N |θ Q2 ), action network π(o|θ π ), and the corresponding target network Q′1(s′, a′1, ..., a′ N |θ′ Q1 ), Q′2(s′, a′1,..., a′ N |θ′ Q2 ),π′(o|θ′ π ), where θ π is the weight of the action neural network, θ Q is the critic network weight, evaluating the value of the state-action pair, and o is the observation value. The two critic networks are used to evaluate the value function of a given state and action, reducing the overestimation of Q values. The action network provides action decisions by observing the surrounding environment. In the training phase, a batch of samples are first sampled from the priority experience replay buffer and the network parameters of all drones are updated using these samples. In this process, drone i will update the network parameters of all drones according to its observation value o at the current time step. i，t To determine the action, and output the action a i,t Feedback to the environment to obtain the reward value r of the current time step i and the observation value o at the next time step i,t+1 , and save it in the priority experience replay pool, and repeat the training until the end. At the same time, in order to simulate the impact of uncertain factors in the environment on the action, the present invention introduces Ornstein-Uhlenbeck noise in the output action, that is, where μ1 represents the mean, θ1 is the regression coefficient, and σ1 is the diffusion factor. a i,t =π i (the i,t |θ i，t )+OU(μ1,θ1,σ1). (17) Where μ1 represents the mean, θ1 is the regression coefficient, and σ1 is the diffusion factor. The action network updates its parameters through policy gradient ascent: where θ Q1 are the parameters of the critic network, Q1 is the output of the critic network, is the gradient of the Q function with respect to action a under state s and action a1, In state o, action a is about the action network parameter θ μ The gradient of , M represents the number of samples used to update the network. Update the critic network by minimizing the Bellman error: where y is the target value found by the two target critic networks; θ Q1 ,θ Q2 Denote the parameters of the two critic networks respectively; Q i，1 , Q i，2 Represent the outputs of the two critic networks respectively; L represents the value that minimizes the error between the output Q value and the target value. At the same time, each target network adopts the soft update method: Among them, θ π ,θ′ π are the parameters of the action network and target action network, θ Q1 ,θ′ Q1 are the parameters of critic network 1 and target critic network 1, θ Q2 ,θ′ Q2 are the parameters of critic network 2 and target critic network 2, respectively. To improve the utilization of MATD3 training samples and the robustness of the algorithm, this paper incorporates Prioritized Experience Replay (PER) into the MATD3 algorithm. The proposed LFMATD3_PER algorithm updates sample priorities by calculating the TD error to avoid sampling probabilities reaching 0, and uses importance sampling weights to calculate the weighted loss function.

Images