Formation path planning method fusing experience sharing and balance award Actor-Critic network

By adopting the Actor-Critic network method that integrates experience sharing and balanced rewards in UAV formation path planning, the efficiency and convergence speed of UAV formation path planning in complex environments is solved, and avoiding between UAVs is achieved The requirements for collision and formation ensure the safe and efficient completion of formation tasks.

CN118915772APending Publication Date: 2024-11-08NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411087223.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-08

Smart Images

  • Figure CN118915772A_ABST
    Figure CN118915772A_ABST
Patent Text Reader

Abstract

The invention discloses a formation path planning method fusing experience sharing and a balance reward Actor-Critic network. The method comprises the following steps: firstly, constructing a task scene of unmanned aerial vehicle formation path planning; secondly, by setting formation and path planning balance rewards, solving balanced formation rewards and path rewards according to sparsity and information criteria; according to an experience sharing strategy, each unmanned aerial vehicle can acquire learning experience of other unmanned aerial vehicles while learning and exploring; and finally, after the whole model is trained, unmanned aerial vehicle formation path planning is realized in a dynamic environment. Based on a multi-agent actor-commentator algorithm basic framework, formation and path planning rewards are set by using balance rewards, an experience sharing strategy is proposed, the problem that the convergence speed of a formation path planning method is relatively low is effectively solved, and efficient path planning of the formation unmanned aerial vehicle can be ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of UAV cluster path planning, and specifically is a formation path planning method integrating experience sharing and balanced reward Actor-Critic network. Technical Background

[0002] In recent years, with the rapid development of artificial intelligence technology, reinforcement learning has shown great potential in the field of drones. Compared with a single drone, drone formations can better meet the growing practical needs. As the flight environment of drones becomes more complex and the demand for flight missions increases, how to balance the formation requirements, while considering avoiding collisions with obstacles, to achieve collision avoidance between drones and the requirements of the overall formation, and to efficiently complete the formation path planning task, has brought new challenges to the existing drone formation path planning methods. It has become a research hotspot in recent years and has received widespread attention from researchers.

[0003] In view of the formation path planning problem, Yan C. et al. applied Q-learning to UAV path planning, but only for scenes with specific obstacles and did not consider the formation problem. Li B. et al. based on the DDPG algorithm, achieved good results in UAV formation path planning, but the model training time was long and the model would be difficult to converge as the complexity of the environment increased. Yu C. et al. studied the multi-agent proximal policy optimization algorithm and verified the effect of the algorithm in the multi-agent path planning problem by using the method of centralized value function. Wang L et al. used reinforcement learning to estimate the value of the environmental state and completed the path planning of multiple separated agents, but did not consider the formation problem between agents. Chen YF et al. used the multi-agent deep deterministic policy gradient method to independently manage the trajectory of each UAV and realized simple path planning for multiple UAVs. However, since its optimization goal is to improve the fairness of competition between UAVs and reduce overall energy consumption, it does not consider the constraints of the path on the conditions such as obstacle avoidance and energy consumption of UAVs. How to combine the characteristics of UAV formation path planning, further optimize the reinforcement learning algorithm, and realize safe and efficient UAV formation path planning tasks still needs further research. Summary of the invention

[0004] The purpose of the present invention is to provide a formation path planning method integrating experience sharing and balanced reward Actor-Critic network, which can well achieve the requirements of collision avoidance and overall formation between UAVs while considering avoiding collision with obstacles, and has the advantage of fast convergence speed.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The formation path planning method that integrates experience sharing and balanced reward Actor-Critic network includes:

[0007] Step 1 constructs the mission scenario for UAV formation path planning, including the number of UAVs, formation, own position, starting position, end position and obstacle information.

[0008] Step 2: Use the balanced reward setting formula to solve the formation reward setting value and the reward setting value of the drone when it reaches the end point The specific method is:

[0009] (2.1) First, define the sparsity evaluation model and informativeness evaluation model of rewards, as follows:

[0010] Reward sparsity evaluation model It is expressed as:

[0011]

[0012] In the formula, is the support set of the reward, where s is the local state and S is the global state. is the agent reward, a is the local action, For joint action.

[0013] Informative Evaluation Model of Rewards It is expressed as:

[0014]

[0015] Where l(·) is a monotone non-decreasing concave function, To satisfy The optimal difference, is the optimal strategy, h is the number of rounds, is the total number of rounds.

[0016] (2.2) Using the reward information evaluation model (2) to find the optimal solution Specifically:

[0017]

[0018] In the formula, As sub-goal, The overall goal is to meet

[0019] (2.3) Find the optimal solution for equation (3), and we get

[0020] (2.4) The equilibrium reward setting formula is defined as:

[0021]

[0022] In the formula, D is prior knowledge, λ≥0;

[0023] (2.5) Use the forward stepwise selection method to solve the equilibrium reward setting formula (4) and obtain the formation reward setting value and the reward setting value of the drone when it reaches the end point

[0024] Step 3: Establish the Actor network and Critic network of each drone, and obtain the strategy loss function L(θ i ) and the loss function L(φ i ), learn to update network parameters and obtain higher learning efficiency.

[0025] Step 4: Set the state space S, action space A, and obstacle avoidance reward r for the UAV formation path planning obs and aircraft collision avoidance reward uav , the equilibrium reward solved by step 2 Get formation reward r frmation and path planning reward r path , calculate the total reward function R of the UAV formation path planning.

[0026] Step 5 trains the UAV formation path planning model of MAAC with experience sharing combined with balanced rewards.

[0027] Step 6 applies the trained experience sharing combined with balanced reward MAAC algorithm to the actual flight scenario to obtain the formation path planning route.

[0028] The present invention has the following advantages:

[0029] 1. Based on the Actor-Critic algorithm, the present invention proposes an experience sharing optimization strategy to achieve more efficient learning by sharing experience among agents, thus solving the problem of low learning efficiency caused by only using the experience of each agent's own sampling trajectory to update the loss function of the value network in the traditional reinforcement learning algorithm.

[0030] 2. The present invention uses the sparsity and informativeness evaluation model of rewards in the reward setting of formation path planning to set a balanced formation path planning reward, and applies it in the experience sharing MAAC algorithm to achieve balanced learning of drone formation and path planning, which is conducive to guiding the learning of the optimal path.

[0031] Description of the accompanying tables and graphs

[0032] Figure 1 Flow chart of the method of the present invention.

[0033] Figure 2 Comparison of the total rewards obtained by the method proposed in this invention and the MADDPG algorithm training.

[0034] Figure 3 It is the cumulative reward of each drone.

[0035] Figure 4 This is the path planning curve for the UAV formation under the method of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is described in detail with reference to the accompanying drawings.

[0037] The formation path planning method of the present invention integrating experience sharing and balanced reward Actor-Critic network specifically comprises the following steps:

[0038] Step 1: Construct the mission scenario of UAV formation path planning, including the number of UAVs, formation, own position (x, y, z), starting position (x siarl ,y start , z start ), end position (x end ,y end , z end ) and obstacle information obs.

[0039] Step 2: Use the balanced reward setting formula to solve the formation reward setting value and path planning reward setting value The specific method is:

[0040] (2.1) First, define the sparsity evaluation model and informativeness evaluation model of rewards, as follows:

[0041] Reward Sparsity Evaluation Model It is expressed as:

[0042]

[0043] In the formula, is the support set of the reward, where s is the local state and S is the global state. is the agent reward, a is the local action, For joint action.

[0044] Informative Evaluation Model of Rewards It is expressed as:

[0045]

[0046] Where l(·) is a monotone non-decreasing concave function, To satisfy The optimal difference, is the optimal strategy, h is the number of rounds, is the total number of rounds.

[0047] (2.2) Using the reward information evaluation model (6) to find the optimal solution Specifically:

[0048]

[0049] In the formula, As sub-goal, For the overall goal, meet

[0050] (2.3) Find the optimal solution for equation (7), and we get

[0051] (2.4) The equilibrium reward setting formula is defined as:

[0052]

[0053] In the formula, D is prior knowledge, λ≥0;

[0054] (2.5) The forward stepwise selection method is used to solve the equilibrium reward setting formula (8) to obtain the formation reward setting value and the reward setting value of the drone when it reaches the end point

[0055] Step 3: Establish the Actor network and Critic network of each drone, and use the strategy loss function L(θ i ) and the loss function L(φ i ) Learn to update network parameters and realize interactive learning between multiple agents and the environment, which specifically includes the following steps:

[0056] (3.1) Set the Actor network and Critic network structure of each drone agent, both of which are double-layer convolutional structures.

[0057] (3.2) Define the multi-agent Markov decision tuple for:

[0058]

[0059] In the formula, is the total number of agents, is the corresponding global state space, is the observation space corresponding to agent i, is the action space of agent i, is the state transfer function, is the reward function corresponding to agent i.

[0060] (3.3) Assume that the strategy for other agents other than itself is β, calculate the loss function of the gradient optimization of strategy β Specifically:

[0061]

[0062] In the formula, φ={φ1, φ2,...φ N} is the Actor network parameter of N agents, a t is the action of the agent at time t, o t is the observed state of the agent at time t, π={π1,π2,...π N} is a set of strategies with parameter φ, r t is the reward of the agent’s Actor network at time t, γ t is the learning rate of the agent at time t, V is the value of the agent's Critic network to the corresponding generation, o t+1 is the observed state of the agent at time t+1, and θ is the parameter of the agent’s Critic network.

[0063] (3.4) Through the experience sharing strategy between agents, the loss function L(φ i ) and the loss function L(θ of the Critic network i ), specifically:

[0064]

[0065]

[0066] Where K represents the Kth agent, t represents the time t, and θ t are the parameters of the Critic network of agent i, Satisfy the following formula:

[0067]

[0068] In the formula, is the reward of agent K at time t, γ is the discount factor, is the observed state of agent K at time t+1.

[0069] Equations (11) and (12) reflect the experience sharing strategy, that is, in the learning and exploration process, agent i derives and trains the strategy network and value network by the experience collected from agent K, thereby achieving more efficient learning.

[0070] Step 4: Set the state space S, action space A, and obstacle avoidance reward r for the UAV formation path planning obs and aircraft collision avoidance reward uav , the equilibrium reward solved by step 2 Get formation reward r formation and path planning reward r path , calculate the total reward function R of the UAV formation path planning, the steps are as follows:

[0071] (4.1) Set the state space S of the UAV formation path planning, which is composed of the global state s g and the local state s l The composition is as follows:

[0072] Global States g Including the drone's own position (x, y, z), the starting point position (x start ,y sart , z start )、End point position (x end ,y end , z end ), the specific expression is:

[0073] s g ={(x, y, z), (x start ,y start , z start ),(x end ,y end , z end )} (13)

[0074] Local state s l Including obstacle information obs and distance information between other drones (d0, ...d N-1 ), the specific expression is:

[0075] s l ={(d0, ...d N-1 ),obs} (14)

[0076] The state space S of the UAV formation path planning is expressed as:

[0077] S={s l ,s g} (15)

[0078] (4.2) Calculate the UAV's action space A, which is composed of the UAV's six-degree-of-freedom motion information of up, down, east, south, west, and north according to the state and the UAV speed information, specifically:

[0079] A={(up,north,east,south,west,down),v} (16)

[0080] (4.3) Calculate the obstacle avoidance reward r obs , the specific form is:

[0081]

[0082] Where, d i is the distance between UAV i and the obstacle, d safe is the set obstacle safety distance, and γ1 is the proportional coefficient.

[0083] (4.4) Calculate the reward r for the drone to avoid collision uav , the specific form is:

[0084]

[0085] Where, d ij is the distance between UAV i and UAV j, d uav_safe is the set safety distance between machines, and γ2 is the proportional coefficient.

[0086] (4.5) The balance reward obtained from step 2 Calculate the formation reward r of the current formation formation and the path planning reward r for reaching the destination according to the UAV formation path planning task path , the specific process is:

[0087] r formatton The specific expression is:

[0088]

[0089] In the formula, is the formation reward setting value, d ij is the distance between UAV, and UAV j, and D is the distance between UAVs under the set formation.

[0090] r path The specific expression is:

[0091]

[0092] In the formula, The reward for the drone when it reaches the end point. is the information reward given by the drone based on the distance from the end point, γ3 is the proportional coefficient, is the penalty for failing to reach the end point within the set number of iterations.

[0093] (4.6) Calculate the total reward function R of the UAV formation path planning, including the formation reward r of the current formation formation , the path planning reward r for reaching the destination according to the UAV formation path planning task path Two main mission rewards, rewards for drones avoiding obstacles obs Reward for avoiding collisions with drones uav Two intrinsic requirement task rewards, specifically:

[0094] R=r formation +r path +r obs +r uav (twenty one)

[0095] Step 5: Using the total reward R of the UAV formation path planning in formula (21), the MAAC network with experience sharing combined with balanced rewards is trained for the UAV formation path planning. The specific steps are as follows:

[0096] (5.1) Set the number of drones, initialize the MAAC network, and set the formation reward R formation and path search reward R path The initial reward values ​​are shown in the following table.

[0097] Main parameter settings for MAAC network training

[0098]

[0099] (5.2) For each episode, initialize the state space of the UAV formation path planning

[0100] (5.3) At time k, use the formation reward setting value solved in step 2 and the reward setting value of the drone formation when it reaches the end point Use step 3 to calculate the loss function L(θ i ) and the loss function L(φ i ), update the Actor network weight φ of agent i i , the weight θ of the Critic network i , use step 4 to solve the total reward R of the UAV formation path planning.

[0101] (5.4) According to the reward value R of the UAV formation path planning, the corresponding action is performed according to the state of the UAV to obtain the corresponding path value.

[0102] (5.5) If all episodes have not been trained, jump to step (5.2); if all episodes have been trained, the network training is completed.

[0103] Step 6 applies the trained experience sharing combined with balanced reward MAAC network to the actual flight scenario to obtain the path planning route of the UAV formation, specifically:

[0104] (6.1) The UAV formation obtains the location of the target, the location of the obstacle, and the current motion state of the UAV;

[0105] (6.2) The current motion state of the drone is used to obtain the predicted action of the drone using the Actor network.

[0106] (6.3) According to the current motion state and predicted action of the UAV, the Critor network is used to evaluate the current state of the UAV and obtain the current state reward value of the UAV.

[0107] (6.4) Calculate the corresponding optimal path based on the current state reward value of the drone.

[0108] (6.5) If the UAV does not reach the target location, jump to step (6.2); if it reaches the target location, the path planning ends.

[0109] In order to verify the feasibility and effectiveness of the method, the present invention is further described in detail below in conjunction with a simulation example.

[0110] On the Ubuntu operating system, the Pytorch framework was used to build a model for simulation experiments. The simulation environment uses a 500m×500m×25m three-dimensional space, with 4 drones, and the 4 drones are required to plan their paths in an equilateral quadrilateral formation. The front of the formation is drone UAV1, the two sides are UAV2 and UAV3, and the rear is drone UAV4. In this area, the starting position of the drone is (450m, 5m, 5m), which is marked with a yellow dot in the figure, and the target position is (5m, 450m, 10m), which is marked with a red star in the figure. A certain number of static regular building obstacles are randomly generated, and two dynamic obstacles are located above the building near the starting point and near the end point.

[0111] Figure 2 The total rewards obtained by the method proposed in the present invention and the MADDPG algorithm are compared. The upper curve in the figure is the total reward obtained by the method proposed in the present invention, and the lower curve is the total reward obtained by the MADDPG algorithm. It can be seen that the method proposed in the present invention converges faster than the MADDPG algorithm and can obtain greater rewards under the same environment settings, which shows the effectiveness of the experience sharing optimization strategy proposed in the present invention.

[0112] Figure 3is the cumulative reward of each drone. It can be seen from the figure that UAV1 at the front of the formation can complete convergence within a certain number of iterations, while UAV2, UAV3 at the sides and rear of the formation and UAV4 at the rear end can apply the learning experience of the leading drone, so the reward fluctuation is small and convergence is completed in a faster time, which shows the effectiveness of the experience sharing mechanism in the method proposed in the present invention.

[0113] Figure 4 It is the path planning curve of the UAV formation under the method of the present invention. As can be seen from the figure, the dynamic obstacle area near the initial position has little effect on the final UAV formation and planned path because it is at a higher position. The four UAVs quickly completed the set equilateral quadrilateral formation assembly and flew towards the end point. Near the end point, the presence of dynamic obstacles forced the UAV formation to choose a new path to avoid the area. While ensuring the requirements of collision avoidance and obstacle avoidance, each UAV successfully maintained the formation and arrived at the end point, indicating the feasibility of the UAV formation path planning method under the algorithm of the present invention.

Claims

1. A formation path planning method integrating experience sharing and balanced reward Actor-Critic network, characterized by: The method comprises the following steps: Step 1 constructs the mission scenario for UAV formation path planning, including the number of UAVs, formation, own position, starting position, end position and obstacle information. Step 2: Use the balanced reward setting formula to solve the formation reward setting value and the reward setting value of the drone when it reaches the end point The specific method is: (2.1) First, define the sparsity evaluation model and informativeness evaluation model of rewards, as follows: Reward Sparsity Evaluation Model It is expressed as: In the formula, is the support set of the reward, where s is the local state and S is the global state. is the agent reward, a is the local action, For joint action. Informative Evaluation Model of Rewards It is expressed as: In the formula, is a monotone non-decreasing concave function, To satisfy The optimal difference, is the optimal strategy, h is the number of rounds, is the total number of rounds. (2.2) Using the reward information evaluation model (2) to find the optimal solution Specifically: In the formula, As sub-goal, The overall goal is to meet (2.3) Find the optimal solution for equation (3), and we get (2.4) Define the equilibrium reward setting formula as: In the formula, D is prior knowledge, λ≥0; (2.5) Use the forward stepwise selection method to solve the equilibrium reward setting formula (4) and obtain the formation reward setting value and the reward setting value of the drone when it reaches the end point Step 3: Establish the Actor network and Critic network of each drone, and obtain the strategy loss function L(θ i ) and the loss function L(φ i ), learn to update network parameters and obtain higher learning efficiency. Step 4: Set the state space S, action space A, and obstacle avoidance reward r for the UAV formation path planning obs and aircraft collision avoidance reward nav , the equilibrium reward solved in step 2 Get formation reward r formation and path planning reward r path , calculate the total reward function R of the UAV formation path planning. Step 5 trains the UAV formation path planning model of MAAC with experience sharing combined with balanced rewards. Step 6 applies the trained experience sharing combined with balanced reward MAAC algorithm to the actual flight scenario to obtain the formation path planning route.

2. A formation path planning method integrating experience sharing and balanced reward Actor-Critic network, characterized by: In step 3, the Actor network and Critic network of each drone are established, and the strategy loss function L(θ i ) and the loss function L(φ i ), learning and updating network parameters to obtain higher learning efficiency, specifically including the following steps: (2.1) Define the multi-agent Markov decision tuple for: In the formula, is the total number of agents, is the corresponding global state space, is the observation space corresponding to agent i, is the action space of agent i, is the state transfer function, is the reward function corresponding to agent i. (2.2) Assume that the strategy for other agents other than itself is β, calculate the loss function of the gradient optimization of strategy β Specifically: In the formula, φ={φ1, φ2,...φ N } is the Actor network parameter of N agents, a t is the action of the agent at time t, o t is the observed state of the agent at time t, π={π1,π2,...π N } is a set of strategies with parameter φ, r t is the reward of the agent’s Actor network at time t, γ t is the learning rate of the agent at time t, V is the value of the agent's Critic network to the corresponding generation, o t+1 is the observed state of the agent at time t+1, and θ is the parameter of the agent’s Critic network. (2.3) Through the experience sharing strategy between agents, the loss function L(φ i ) and the loss function L(θ of the Critic network i ), specifically: Where K represents the Kth agent, t represents the time t, and θ i are the parameters of the Critic network of agent i, Satisfy the following formula: In the formula, is the reward of agent K at time t, γ is the discount factor, is the observed state of agent K at time t+1. Equations (7) and (8) reflect the experience sharing strategy, that is, in the learning and exploration process, agent i derives and trains the strategy network and value network by using the experience collected from agent K, thereby achieving more efficient learning.

3. A formation path planning method integrating experience sharing and balanced reward Actor-Critic network, characterized in that: In step 4, the state space S, action space A, and obstacle avoidance reward r of the drone formation path planning are set ob , and collision avoidance rewards uav , the equilibrium reward solved in step 2 Get formation reward r formation , path planning reward r path , calculate the total reward function R of the UAV formation path planning, the steps are as follows: (3.1) Set the state space S of the UAV formation path planning, which is composed of the global state s g and the local state s l The composition is as follows: Global States g Including the drone's own position (x, y, z), the starting point position (x start ,y start , z start )、End point position (x end ,y end , z end ), the specific expression is: s g ={(x,y,z),(x start ,y start ,z start ),(x end ,y end ,z end )} (9) Local state s l Including obstacle information obs and distance information between other drones (d0, ...d N-1 ), the specific expression is: s l ={(d0,...d N-1 ),obs} (10) The state space S of the UAV formation path planning is expressed as: S={s l ,s g } (11) (3.2) Calculate the UAV's action space A, which is composed of the UAV's six-degree-of-freedom motion information of up, down, east, south, west, and north according to its state and the UAV's speed information, specifically: A={(up,north,east,south,west,down),v} (12) (3.3) Calculate the obstacle avoidance reward r obs , the specific form is: Where, d i is the distance between UAV i and the obstacle, d safe is the set obstacle safety distance, and γ1 is the proportional coefficient. (3.4) Calculate the reward r for the drone to avoid collision uav , the specific form is: Where, d ij is the distance between UAV i and UAV j, d uav_safe is the set safety distance between machines, and γ2 is the proportional coefficient. (3.5) The balance reward obtained from step 2 Calculate the formation reward r of the current formation formation and the path planning reward r for reaching the destination according to the UAV formation path planning task path , the specific process is: r formation The specific expression is: In the formula, Set the value of the formation reward, d ij is the distance between UAV i and UAV j, and D is the distance between UAVs in the set formation. r path The specific expression is: In the formula, Set the reward value for the drone when it reaches the end point. is the information reward given by the drone based on the distance from the end point, γ3 is the proportional coefficient, is the penalty for failing to reach the end point within the set number of iterations. (3.6) Calculate the total reward function R of the UAV formation path planning, including the formation reward r of the current formation formation , the path planning reward r for reaching the destination according to the UAV formation path planning task path Two main mission rewards, rewards for drones avoiding obstacles obs Reward for avoiding collisions with drones uav Two intrinsic requirement task rewards, specifically: R=r formation +r path +r obs +r uav (17)。

Citation Information

Cited By

  • Multi-mobile carrier order operation control method

    CN119476060A

  • Real-time obstacle avoidance unmanned aerial vehicle flight path planning method

    CN119937626A

  • TD-MATD3-based offshore unmanned ship cluster path planning method

    CN120871870A

  • Distribution network field survey path planning method based on near-end strategy optimization PPO

    CN121455152A