Unmanned aerial vehicle cluster task planning algorithm based on hierarchical multi-agent deep reinforcement learning and evaluation method thereof

Abstract

The invention discloses an unmanned aerial vehicle cluster task planning algorithm based on hierarchical multi-agent deep reinforcement learning and an evaluation method thereof, belongs to the technical field of unmanned aerial vehicles, and solves the problems of low online solving efficiency, large calculation amount, poor stability and high efficiency generally existing in a conventional unmanned aerial vehicle cluster cooperative task planning problem solving algorithm. And the method is difficult to adapt to complex dynamic scenes. The method comprises the following steps: firstly, respectively establishing a Markov decision process model of a cluster task allocation problem part and a Markov decision process model of a track planning problem part; secondly, a layered deep reinforcement learning framework based on multi-agent deep reinforcement learning is provided, and synchronous solving of task allocation and route planning association coupling problems is achieved through coupled training and a layered solving mode; in addition, an unmanned aerial vehicle cluster combat simulation environment is established based on UE4 and Airsim, evaluation indexes are designed, and the result shows that the method has good performance in cluster task planning, and the intelligent and actual combat level of cluster combat can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to an unmanned aerial vehicle cluster task planning algorithm and an evaluation method thereof based on hierarchical multi-agent deep reinforcement learning. Background Art

[0002] Unmanned aerial vehicle systems are widely used in military and civilian fields. They have the advantages of low cost, flexibility, and easy deployment. They are currently widely used in battlefield reconnaissance, enemy strikes, self-search and rescue, and other scenarios. Due to the limited endurance and equipment carried by a single drone, it is unable to complete complex and large-scale mission requirements. A drone swarm system composed of multiple drones can complete more complex tasks through autonomous decision-making and control of drone swarms. Drone swarm mission planning technology is one of the core technologies for realizing autonomous decision-making and control of drone swarms. The purpose of mission planning technology is to assign a set of optimal target points to each drone to maximize the overall mission efficiency and enable the drone to avoid obstacles, threat areas, and no-fly zones in the environment while reaching the target point.

[0003] At present, most methods separate the task allocation and trajectory planning in the task planning technology, ignoring the coupling effect of the actual task allocation and the flight trajectory. Such methods often model the task allocation and trajectory planning problems separately, convert the task planning problem into a multi-objective optimization problem, and then use intelligent optimization algorithms such as ant colony algorithm and genetic algorithm to solve it. Such methods require a lot of computing time and resources and cannot meet the real-time and dynamic requirements.

[0004] With the development of artificial intelligence, the use of deep reinforcement learning to solve UAV mission planning problems has shown its superiority. Most methods based on deep reinforcement learning either need to obtain global information in the task allocation stage and require a centralized ground station to issue task allocation instructions, or simplify the task planning problem to meet the convergence of the algorithm. The generalization and robustness of their task planning models are poor. Summary of the invention

[0005] The present invention provides a UAV swarm task planning algorithm based on hierarchical multi-agent deep reinforcement learning and an evaluation method thereof, designs a distributed UAV swarm task planning algorithm, considers the coupling of task allocation and trajectory planning problems at the same time, establishes a UAV swarm collaborative task planning simulation system, and designs a cluster task planning algorithm performance evaluation method based on the simulation system.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions:

[0007] A drone swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning includes the following steps:

[0008] Step 1: According to the task requirements, for multi-UAV multi-target scenarios, the Markov decision process models of UAV swarm task allocation and trajectory planning are established respectively;

[0009] Step 2: Based on the Markov decision process model of UAV cluster task allocation and trajectory planning established in step 1, a QMIX-based task allocation algorithm is designed for the UAV cluster task allocation problem, and a MADDPG-based trajectory planning algorithm is designed for the UAV cluster trajectory planning problem, using a centralized training and distributed execution approach; the QMIX-based task allocation algorithm contains two networks, namely the Q network and the Mix network, where the input of the Q network is the observation of the task allocation of the UAV cluster task Output is the task number a selected at the next moment T ={i}, the network contains two hidden layers with a dimension of 200, and uses the relu activation function. The input of the Mix network is the joint observation of all drones and the tasks selected by all drones. The output is the weighted sum of the Q network outputs of all drones. The network contains two hidden layers with a dimension of 200, and uses the relu activation function. The trajectory planning algorithm based on MADDPG contains an Actor network and a Critic network, where the input of the Actor network is the observation of the drone cluster trajectory planning. The output is the speed control command of the UAV. The input of the Critic network is the joint observation of all UAVs and the speed control command of all UAVs. The output is the evaluation of the current action under the current state.

[0010] Step 3: Based on the task allocation and trajectory planning algorithms designed in step 2, a hierarchical deep reinforcement learning framework is designed. On the basis of considering the coupling of the two sub-problems of task allocation and trajectory planning, the task allocation and trajectory planning problems are solved by synchronous training. The hierarchical deep reinforcement learning framework is divided into a top layer and a bottom layer. The top layer is the task allocation layer, which assigns an optimal target point to each drone and outputs the task selected by the drone at the next moment. The bottom layer is the trajectory planning layer, which provides collision-free speed instructions for each drone and outputs the drone speed control instructions according to the task selected by the top-level task allocation layer, so that it can reach the target point smoothly. During training, the output result of the top-level task allocation affects the input of the bottom-level trajectory planning, and the output of the bottom-level trajectory planning affects the reward of the top-level task allocation, realizing an interactive training method. By maintaining the coupling relationship between task allocation and trajectory planning during training, the relationship between task allocation and trajectory planning is implicitly learned during the training of each model.

[0011] In the above steps, the Markov decision process model of the task allocation part in step 1 includes the observation space of the UAV cluster task allocation in Including the current position of the drone, attack capability, the position of all mission points, attack requirements, and the action space a of the drone cluster task allocation T ={i}, i represents the task number selected by the drone at the next moment, and the reward function of the drone cluster is r T , including two parts: trajectory cost and attack penalty; the trajectory planning part Markov decision process model, including the observation space of UAV cluster trajectory planning It contains the location information of the target point, the location information of the drone itself, the speed information of the drone itself, and the external sensor information; the action space a of the drone cluster trajectory planning P = {v x ,v y}, contains the speed instructions of the drone in the x and y directions at the next moment; the reward function r of the drone cluster trajectory planning P , including transfer rewards, success rewards and collision penalties.

[0012] Another aspect of the present invention further provides an evaluation method for the above-mentioned UAV cluster mission planning algorithm, comprising the following steps:

[0013] Step 1: Based on UE4, a simulation environment for drone swarm mission planning is established. The simulation environment includes a three-dimensional environment with static obstacles. Airsim is used as the drone control interface to interact with the simulation environment, control drone movement, and obtain drone information.

[0014] Step 2: Design evaluation indicators to quantitatively evaluate the performance of the task allocation and trajectory planning algorithms designed in step 2. The evaluation indicators include average reward, success rate, collision rate, trajectory efficiency, motion smoothness, and trajectory curvature.

[0015] Beneficial effects: The present invention provides a UAV cluster task planning algorithm based on hierarchical multi-agent deep reinforcement learning and an evaluation method thereof. When modeling the task planning problem, the task planning is decomposed into two sub-problems of task allocation and track planning. When solving the task planning problem, a joint solution framework is designed to solve the task allocation and track planning problems at the same time. During training, the output result of the top-level task allocation affects the input of the bottom-level track planning, and the output of the bottom-level track planning affects the reward of the top-level task allocation, so as to realize an interactive training method. By maintaining the coupling relationship between task allocation and track planning during training, the relationship between task allocation and track planning is implicitly learned in the process of training each model, thereby solving the coupling problem that the existing task planning algorithm ignores task allocation and track planning. The present invention implements the training of task allocation and track planning algorithms based on a multi-agent deep reinforcement learning algorithm, so that the UAV can select a suitable task and reach the task area without obtaining global information, and complete the task planning of the UAV cluster by relying on local information in an unknown environment, thereby improving the autonomous capability of the UAV. The present invention also builds a UAV cluster combat simulation environment based on UE4 and Airsim, and designs evaluation indicators, so as to comprehensively, reasonably and quantitatively evaluate the effect of the task planning algorithm in a physical simulation environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a schematic diagram of a drone equipped with 7 distance sensors in an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the network architecture of the QMIX and MADDPG algorithms in an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a hierarchical deep reinforcement learning framework in an embodiment of the present invention;

[0019] Figure 4 It is the average reward of MADDPG, MATD3, and MAPPO algorithms during training;

[0020] Figure 5 This is a schematic diagram of the task planning of 5 drones and 5 target points in a simulation environment according to the MAPPO algorithm in an embodiment of the present invention;

[0021] Figure 6 This is a schematic diagram of the mission planning of 5 drones and 5 target points in a simulation environment according to the MADDPG algorithm in an embodiment of the present invention;

[0022] Figure 7 It is a schematic diagram of task planning of a task planning algorithm for 10 drones and 10 target points in a simulation environment in an embodiment of the present invention;

[0023] Figure 8Schematic diagram of deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] A method for UAV swarm task planning based on hierarchical multi-agent deep reinforcement learning includes the following steps:

[0026] 1. Multi-agent Deep Reinforcement Learning Model

[0027] According to the mission requirements, for multi-UAV multi-target scenarios, Markov decision process models for UAV swarm task allocation and trajectory planning are established respectively;

[0028] (1) Top-level task allocation model

[0029] 1) State and observation space

[0030] The state contains global information, and the observation is the local information observed by each drone. In the top-level task allocation model, the observation of each drone includes the current position of the drone, attack capability, and the position and attack requirements of all mission points, which are defined as: The observations of all drones constitute the global state, which is defined as:

[0031] 2) Action Space

[0032] The action of the drone is defined as the mission number selected at each moment, a T ={i}, i represents the mission number selected by the drone at the next moment.

[0033] 3) Reward Function

[0034] In the task allocation process, all drones share a global reward. The reward function includes two parts: track cost and attack penalty. The track cost is shown as follows:

[0035] r d =-(d1+d2+...d i ...+d N )

[0036] Where: d i represents the actual trajectory length of UAV i to reach its selected target point;

[0037] The attack penalty is as follows:

[0038] r c = -λ×η

[0039] Where: η is the mismatch between the UAV attack capability and the mission attack requirement, and λ is a constant.

[0040] Therefore, the global reward is as follows:

[0041] r T =δ1r d +δ2r c

[0042] Where: δ1, δ2 are constants;

[0043] (2) Underlying trajectory planning model

[0044] 1) State and observation space

[0045] In the process of trajectory planning, the observation space contains the position information, speed information, target point position information and external sensor information of each UAV, which is defined as: where g i is the polar coordinate information of the target point corresponding to the drone relative to the drone, v x , v y is the speed information of the drone in the x and y directions at the current moment, For external sensor information, such as Figure 1 As shown in the figure, the drone is equipped with 7 distance sensors, which return the distance information from the 7 drones to the obstacles. The observations of all drones constitute the global state.

[0046] 2) Action Space

[0047] To simplify the training of the model, it is assumed that the drone flies at a fixed altitude. The action space contains the velocity components of the drone in the x and y directions, a P = {v x ,v y};

[0048] 3) Reward Function

[0049] The reward function includes transfer reward, success reward and collision penalty. The transfer reward is to avoid sparse rewards, so that the drone has a tiny reward at each time step to guide the drone to reach the target point. The transfer reward is as follows:

[0050] r1=α(d t-1 -d t )

[0051] Where: d t is the distance from the drone to the target point at time t, α is a constant;

[0052] The success reward represents the incentive for the drone to reach the target point, as shown in the following formula:

[0053]

[0054] Where: r arrival is a positive number, p i is the current coordinate of the drone, g i The coordinates of the mission point selected for the drone;

[0055] The collision penalty is used to guide the drones to avoid obstacles and avoid collisions with each other, as shown in the following formula:

[0056]

[0057] Where: R is the radius of the drone, The shortest distance to obstacles;

[0058] The total reward is as follows:

[0059] r P =r1+r2+r3

[0060] 2. Design of task allocation algorithm based on QMIX and trajectory planning algorithm based on MADDPG

[0061] The overall architecture of the network structure is as follows Figure 2 As shown in the figure, each drone contains a task allocation network and a trajectory planning network. The input of the task allocation network is the current drone's position, attack capability, and the positions and attack requirements of all tasks. After a series of linear layers, the output is the value of the current drone's selected task. Then, based on the greedy principle, the task with the highest value is selected as the task selection at the next moment. The input of the trajectory planning network is the location information of the target point, the drone's motion information, and the external sensor information. After being processed by the linear layer, the output is the speed instruction at the next moment.

[0062] The hidden layer of the task allocation network is a fully connected network with two dimensions of 200, and the hidden layer of the trajectory planning network is a fully connected network with two dimensions of 64. The network uses the relu activation function. It is worth noting that the task allocation network is solved using the QMIX algorithm. In addition to the task allocation network, a hybrid network needs to be defined during training to weighted sum the Q values ​​output by all task selection networks. The input of this network is all Q values ​​and the global state, and the output is the global value after the weighted sum of the Q values. The MADDPG algorithm is used to solve the trajectory planning, and the algorithm training is implemented based on the Actor-Critic framework. Therefore, a centralized Critic network is required during the training process. Its structure is similar to that of the Actor network. The input is all the information of all drones, and the output is the evaluation of the current drone action considering the global information and the actions of other drones.

[0063] 3. Hierarchical Deep Reinforcement Learning Framework Design

[0064] The task allocation model is trained using the QMIX algorithm. The QMIX algorithm is an extension of the DQN algorithm and is a multi-agent reinforcement learning algorithm based on value decomposition. The value decomposition method allows drones to access global information during training, thereby training from a global perspective. The idea of ​​the QMIX algorithm is to use a neural network f to approximate the global value Q total (s,u), satisfying the following conditions:

[0065]

[0066] In the formula, Q i is the value function of each drone;

[0067] This condition makes Q total and Q i The relationship between is monotonic, thus ensuring the following condition holds:

[0068]

[0069] The meaning of the above formula is to ensure that Q total (s,u) arg max joint action u With each Q i The arg max obtained [u1,u2,...,u N ] is the same, so that the local optimal action selected by each drone is exactly part of the global optimal action, Q total The update adopts a similar way to DQN, which learns the target global action value function corresponding to the optimal strategy by minimizing the loss function. The global action value function is shown as follows:

[0070]

[0071] Where L(θ) is the loss function of the hybrid network, Q * (s,a|θ) is the global value at the current moment, Q * (s′, a′) is the global value at the next moment, r is the reward at the current moment, s, s′, a, a′ are the state and action at the current moment and the next moment respectively, θ is the network parameter, and γ is the discount factor.

[0072] The training of the trajectory planning model uses the MADDPG algorithm, which is an extension of the deterministic policy gradient algorithm (DDPG). Its gradient can be written as follows:

[0073]

[0074] In the formula, is the policy gradient of the i-th drone, μ i (a i |o i ) is the Actor network of the i-th drone, is the global critic network, x is the global state, a N For the action of the Nth drone, update the Actor network by maximizing the policy gradient.

[0075] The centralized critic network is updated using the following formula:

[0076]

[0077] Where: is the output of the Critic network at the current moment, is the output of the Critic network at the next moment, x, x′ is the global state at the current moment and the next moment, a N ,a′ N is the action of the Nth drone at the current moment and the next moment.

[0078] like Figure 3 As shown in the figure, during the training process, the two algorithms are trained synchronously and influence each other. The input of the underlying trajectory planning network depends on the output of the top-level task allocation network. At the same time, the top-level task allocation network does not interact with the environment directly, but interacts with the environment through the output of the underlying trajectory planning network. The state and reward of the top-level network depend on the output of the underlying network. By maintaining the coupling relationship between task allocation and trajectory planning during training, the relationship between the two processes is implicitly learned on the basis of updating the two network models, thereby improving the efficiency of the overall task planning algorithm. The detailed algorithm flow is as follows.

[0079]

[0080]

[0081] The evaluation of the above-mentioned UAV swarm mission planning algorithm includes the following steps:

[0082] 1. Establish a simulation environment for drone swarm mission planning based on UE4

[0083] In order to compare and verify the effect of the task planning algorithm designed above in multi-UAV swarm task planning, the simulation environment adopted is the AirSim UAV simulation simulator developed by Microsoft. The AirSim UAV simulation simulator developed based on Unreal Engine has a close-to-real quadcopter dynamics model and rich API interfaces, which is very suitable for simulation experiments of UAV swarms.

[0084] The simulation and reinforcement learning model training are developed in Python 3.9.19 and PyTorch 1.13.0. The specific hardware configuration of the training environment is shown in the following table:

[0085]

[0086] A total of about 5,000 episodes were trained, and the algorithm took about 12 hours to converge. The length, width and height of the environment were set to 60m×60m×50m, the obstacles were several cylinders or cuboids of different sizes, the target point was a sphere with a radius of 0.15m, the radius of the drone was 1m, the speed was set to (-2, 2)m / s, the drone's flight altitude was fixed at 3 meters, and the control frequency was set to 5HZ. The parameter settings during the network and training process are shown in the table below:

[0087]

[0088]

[0089] 2. Design evaluation indicators to quantitatively evaluate algorithm performance

[0090] (1) Evaluation index design

[0091] In order to evaluate the quality of task planning, a set of quantitative evaluation indicators was designed and 100 experiments were conducted to evaluate the effectiveness of the algorithm from multiple perspectives, including security, effectiveness, and robustness:

[0092] 1) Average reward: The average reward is the average of the rewards of all drones at each time step in the process of completing the task. A higher average reward means a higher overall mission effectiveness.

[0093] 2) Success rate: the percentage of drones that reach the target location within the specified time without colliding with other drones;

[0094] 3) Collision rate: the percentage of drones that failed to complete their missions due to collisions with obstacles;

[0095] 4) Trajectory efficiency: the ratio of the sum of the straight-line distances between the starting point and the target point of all drones to the sum of the actual trajectory lengths of all drones;

[0096] 5) Motion smoothness: Motion smoothness reflects the change in the speed of the drone. The smaller the motion smoothness, the smoother the drone's motion. The calculation formula is as follows:

[0097]

[0098] Where: v i represents the speed of the UAV, N represents the number of UAVs, and T is the total time step;

[0099] 6) Track curvature: The curvature reflects the turning point of the track. The smaller the curvature, the smoother the track. The calculation formula is as follows:

[0100]

[0101] Where: x , y is the coordinate of the drone, N is the number of drones, and T is the total time step

[0102] (2) Algorithm performance evaluation

[0103] 1) Convergence analysis

[0104] The changing trends of the average rewards of various algorithms during training are as follows: Figure 4 As shown in the figure, it can be seen that compared with MATD3 and MAPPO algorithms, MADDPG algorithm has a faster convergence speed.

[0105] 2) Algorithm performance analysis

[0106] The following table shows the evaluation indicators of MADDPG and MAPPO algorithms when the number of drones is 5.

[0107]

[0108] As can be seen from the table, compared with the MAPPO algorithm, the MADDPG algorithm has a higher average reward and a higher success rate.

[0109] Figure 5 and Figure 6This is a schematic diagram of task planning for the MADDPG and MAPPO algorithms in a simulation environment. It can be seen intuitively from the figure that in the path planning process, the MADDPG algorithm is farther away from obstacles and has a better obstacle avoidance effect than the MAPPO algorithm.

[0110] The following table compares the centralized task allocation algorithm and the QMIX task allocation algorithm.

[0111]

[0112] Compared with the centralized Hungarian algorithm, the top-level task allocation algorithm obtained by QMIX training has an overall efficiency similar to that of the centralized task allocation algorithm. However, each drone in the QMIX task allocation algorithm does not need to know the information of all drones. It is a distributed task allocation algorithm. When the algorithm performance is not much different, its scalability and robustness are better than the centralized task allocation algorithm.

[0113] 3) Generalization analysis

[0114] In the simulation environment of the urban scene, the environment size is 400m×400m×50m, and the mission planning algorithm of 10 drones and 10 target points is tested. The simulation results are as follows: Figure 7 As shown in Figure 3, the experimental results show that the task planning algorithm proposed in this paper can complete the task planning of 10 drones and 10 target points.

[0115] The above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. For ordinary technicians in this technical field, within the scope of their knowledge and without departing from the principle of the present invention, several improvements can be made to the present invention, and these improvements are also regarded as the protection scope of the present invention.

Claims

1. A UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning, characterized in that: The following steps are involved: Step 1: According to the task requirements, for multi-UAV multi-target scenarios, the Markov decision process models of UAV swarm task allocation and trajectory planning are established respectively; Step 2: Based on the Markov decision process model of UAV cluster task allocation and trajectory planning established in step 1, a QMIX-based task allocation algorithm is designed for the UAV cluster task allocation problem, and a MADDPG-based trajectory planning algorithm is designed for the UAV cluster trajectory planning problem, using a centralized training and distributed execution approach; Step 3: Based on the task allocation and trajectory planning algorithms designed in step 2, a hierarchical deep reinforcement learning framework is designed. Considering the coupling of the two sub-problems of task allocation and trajectory planning, the task allocation and trajectory planning problems are solved through simultaneous training.

2. The UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 1 is characterized in that: The Markov decision process models of the UAV cluster task allocation and trajectory planning parts include the observation space, action space, and reward function of the UAV cluster task allocation.

3. The UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 2 is characterized in that: The observation space in the Markov decision process model of the task allocation part includes the current position of the drone, the attack capability, the position of all task points, and the attack requirements; the action of the drone is defined as the task number selected at each moment; the reward function r T It includes two parts: track cost and attack penalty, as shown in the following formula: r T =δ1r d +δ2r c Where: δ1, δ2 are constants, r d is the track cost, r c Punishment for attack; The observation space in the Markov decision process model of the trajectory planning part includes the position information, speed information, position information of the target point and information of the external sensor of the UAV; The action space contains the speed instructions of the drone in the x and y directions at the next moment; the reward function r P It includes transfer reward, success reward and collision penalty, as shown in the following formula: r P ＝r1+r2+r3 Where: r1 is the transfer reward, r2 is the success reward, and r3 is the collision penalty.

4. The UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 1 is characterized in that: The task allocation algorithm based on QMIX includes two networks, namely Q network and Mix network. The input of Q network is the observation of task allocation of UAV cluster, and the output is the task sequence number selected at the next moment. The network contains two hidden layers with a dimension of 200 and uses relu activation function. The input of Mix network is the joint observation of all UAVs and the tasks selected by all UAVs. The output is the weighted sum of the outputs of Q network of all UAVs. The network contains two hidden layers with a dimension of 200 and uses relu activation function. The training adopts centralized training and distributed execution. The specific training process is as follows: Input the local state of each drone at the current moment and the next moment into the Q network to obtain the value of the action at the current moment and the next moment respectively; The output and global state of all Q networks at the current moment and the next moment are input to the Mix network. The Mix output is the prediction of the global value at the current moment and the prediction of the global value at the next moment respectively. Based on the TD error, the global return at the current moment is calculated using the reward and the prediction of the global value at the next moment; The mean square error between the global return at the current moment and the prediction of the global value at the current moment is calculated as the loss function, and the Q network and Mix network are updated by minimizing the loss function.

5. The UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 1 is characterized in that: The trajectory planning algorithm based on MADDPG includes an Actor network and a Critic network. The input of the Actor network is the observation of the UAV cluster trajectory planning, and the output is the speed control command of the UAV. The input of the Critic network is the joint observation of all UAVs and the speed control command of all UAVs, and the output is the evaluation of the current action under the current state. The training adopts the centralized training and distributed execution mode. The specific training process is as follows: Input the global state of the next moment into the Critic network, and output the value of the next moment; According to the value of the next moment and the reward of the current moment, based on the TD error, calculate the return of the current moment; The global state at the current moment is input into the Critic network, and the output is the value at the current moment as the predicted global value; Calculate the mean square error between the global return and the predicted global return as the loss function, and update the Critic network by minimizing the loss function; The output of the negative Critic network is used as the loss function of the Actor network, and the Actor network is updated by minimizing the loss function.

6. The UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 1 is characterized in that: The hierarchical deep reinforcement learning framework is divided into a top-level task allocation layer and a bottom-level trajectory planning layer. The top-level task allocation layer allocates an optimal target point to each UAV and outputs the task selected by the UAV at the next moment; the bottom-level trajectory planning layer provides each UAV with a collision-free speed instruction, and outputs the UAV speed control instruction according to the task selected by the top-level task allocation layer, so that it can reach the target point smoothly; during the training process, the QMIX-based task allocation algorithm and the MADDPG-based trajectory planning algorithm are trained synchronously and influence each other; the input of the bottom-level trajectory planning network depends on the output of the top-level task allocation network, and the top-level task allocation network does not interact with the environment directly, but interacts with the environment through the output of the bottom-level trajectory planning network. The state and reward of the top-level network depend on the output of the bottom-level network. By maintaining the coupling relationship between task allocation and trajectory planning during training, the relationship between the two processes is implicitly learned on the basis of updating the two network models.

7. The UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 6 is characterized in that: The task allocation algorithm based on QMIX makes the local optimal action selected by each drone just a part of the global optimal action. By minimizing the loss function, the target global action value function corresponding to the optimal strategy is learned. The global action value function is shown as follows: Where: L(θ) is the loss function of the hybrid network, Q * (s,a|θ) is the global value at the current moment, Q * (s′,a′) is the global value at the next moment, r is the reward at the current moment, s, s′, a, a′ are the state and action at the current moment and the next moment respectively, θ is the network parameter, and γ is the discount factor.

8. The evaluation method of the UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to any one of claims 1 to 7 is characterized in that: The following steps are involved: Step 1: Based on UE4, a simulation environment for drone swarm mission planning is established. The simulation environment includes a three-dimensional environment with static obstacles. Airsim is used as the drone control interface to interact with the simulation environment, control drone movement, and obtain drone information. Step 2: Design evaluation indicators to quantitatively evaluate the performance of the designed task allocation and trajectory planning algorithms.

9. The evaluation method of the UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 8 is characterized in that: The evaluation indicators include average reward, success rate, collision rate, trajectory efficiency, motion smoothness and trajectory curvature.

10. The evaluation method of the UAV swarm mission planning algorithm based on hierarchical multi-agent deep reinforcement learning according to claim 9 is characterized in that: The calculation formula of the motion smoothness is: In the formula, v i represents the speed of the UAV, N represents the number of UAVs, and T is the total time step; The calculation formula of the trajectory curvature is: Where: x , y is the coordinate of the UAV, N is the number of UAVs, and T is the total time step.

Images