A layered reward-based MADDPG multi-unmanned aerial vehicle cooperative obstacle avoidance method

Abstract

The present application belongs to the technical field of multi-aircraft cooperative control, and particularly relates to a MADDPG multi-unmanned aircraft cooperative obstacle avoidance method based on hierarchical rewards, comprising the following steps: S10: a multi-unmanned aircraft interacts with an environment to obtain an initial state space, an action space and reward information thereof; S20: network parameters are initialized, and the obtained state, action and reward data are stored in an experience pool; S30: a hierarchical reward MADDPG algorithm is designed to train according to the experience pool sampling data, and generate a new action; S40: the unmanned aircraft interacts with the environment according to the new action, updates the state and the reward, and stores the new state, action and reward in the experience pool; and S50: the algorithm model of the hierarchical reward MADDPG algorithm after stable training is deployed to the unmanned aircraft, and multi-unmanned aircraft cooperative obstacle avoidance is realized. Compared with the traditional multi-unmanned aircraft cooperative obstacle avoidance method, the present application designs a reward function based on a hierarchical idea, and effectively solves the problems of slow convergence speed caused by sparse rewards and poor obstacle avoidance effect caused by single rewards of the MADDPG algorithm.

Description

Technical Field

[0001] This solution belongs to the field of multi-aircraft cooperative control technology, specifically involving a multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG). Background Technology

[0002] In today's world, drone technology is developing rapidly, and its applications are becoming increasingly widespread. Especially in complex environments such as jungles, mountains, and cities, the autonomous navigation and path planning capabilities of multiple drones are of paramount importance. In such environments, drones need to possess a high degree of autonomy and flexibility to avoid obstacles while maintaining efficient flight paths during mission execution.

[0003] The MADDPG algorithm is a multi-agent deep reinforcement learning algorithm based on a centralized training and distributed execution framework, suitable for multi-agent environments with continuous action spaces. Centralized training means that during training, each agent can access global information (including the states and actions of other agents) to better learn the policy. Distributed execution means that during the execution phase, each agent relies only on its own local observations to complete the decision-making process. In the MADDPG algorithm, an independent Actor-Critic structure is designed for each agent.

[0004] The Actor network outputs the agent's policy, i.e., the corresponding action. The Critic network evaluates the quality of the agent's current policy, specifically by measuring the action-value function (Q-value). During training, the Critic network takes the global state and all agents' actions as input, aiming to minimize the Bellman error, i.e., minimizing the loss function. The Actor network takes the agents' local observations as input, and its optimization objective is to maximize the Q-value of the Critic network's output through policy updates, thereby generating and outputting the action policies of each agent. These policies directly affect the overall performance of the system.

[0005] The MADDPG algorithm has the following advantages: (1) it can solve the non-stationarity problem in multi-agent deep reinforcement learning environments; (2) it can be applied to multi-scenario tasks; (3) the experience replay mechanism avoids local optima; and (4) it designs a stable training process for the target network. Based on the above advantages, the MADDPG algorithm is more suitable for solving the problem of collaborative obstacle avoidance control for multi-UAV material delivery. Therefore, the MADDPG algorithm is selected as the basic algorithm.

[0006] However, this algorithm still has some shortcomings. The MADDPG algorithm typically uses simple reward functions, such as considering only the target distance (e.g., Euclidean distance) as the sole reward signal or using only binary collision penalties for obstacle avoidance reward design. It lacks fine-grained evaluation of obstacle avoidance behavior, neglecting factors such as aggregation effects and energy consumption. While this simple reward function design is easy to implement, it cannot provide sufficient guidance information. In complex tasks, it may fail to effectively guide the agent to learn complex behaviors, or even cause the agent to fall into suboptimal strategies or fail to complete the task. Falling into suboptimal strategies is because simple reward functions may cause the agent to overemphasize short-term rewards while ignoring long-term goals. Failure to complete the task is because simple rewards are difficult to guide complex behaviors. Complex tasks usually need to be completed in stages, and simple reward functions can only provide feedback after the task is completed, lacking guidance for the intermediate process. This ultimately leads to problems such as slow convergence speed due to sparse rewards and poor obstacle avoidance performance due to a single reward in multi-UAV material delivery tasks in complex environments. Summary of the Invention

[0007] The purpose of this solution is to provide a multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG) to address the problems of slow convergence speed and poor obstacle avoidance performance when the MADDPG algorithm is applied to multi-UAV cooperative obstacle avoidance tasks.

[0008] To achieve the above objectives, this solution provides a multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG), comprising the following steps:

[0009] S10: Multiple drones interact with the environment to obtain their initial state space, action space, and reward information;

[0010] S20: Initialize network parameters and store the acquired status, action, and reward data into the experience pool;

[0011] S30: The designed hierarchical reward MADDPG algorithm is trained based on the sampled data from the experience pool to generate new actions;

[0012] S40: The drone interacts with the environment based on new actions, updates its status and rewards, and stores the new status, actions, and rewards in the experience pool;

[0013] S50: Deploy the network model trained by the hierarchical reward MADDPG algorithm on drones to achieve multi-drone collaborative obstacle avoidance.

[0014] The overall reward in step S40 includes the top-level reward r. u and underlying reward r d The specific design is as follows:

[0015] r t =ru +r d ;

[0016] Top-level reward r u Considering global step-by-step guidance and combining global state information, the final delivery target of the drone is broken down into multiple phased local targets, gradually guiding the drones towards the final delivery target to ensure rapid overall convergence of multiple drones; at the top-level reward r u Under guidance, the underlying reward r d Based on the characteristics of obstacles in the mission environment, and taking into account factors such as internal collision avoidance, external obstacle avoidance, boundary constraints, aggregation effect, and overall energy consumption, specific planning and design of rewards for individual drones are carried out.

[0017] Furthermore, the top-level reward includes a local target point attraction reward and a penalty for being stuck at a local target point, specifically designed as follows:

[0018] r u =r point +r p ;

[0019] Local target point traction reward r point Designed as follows:

[0020]

[0021] Where k1 is the adjustment coefficient, 0 <k1<1;P i Let P be the current position of drone i, where i = 1, 2, ..., N; subl Let be the l-th local target point, l∈L; ε∈(0,0.01);

[0022] Penalty for being stuck at a local target point p Designed as follows:

[0023]

[0024] Where k2 is the adjustment coefficient, 0 <k2<1; and Let be the current state of UAV i at time t and its position at the previous time, respectively; σ is a set local oscillation distance, σ∈(2,10).

[0025] Furthermore, the underlying reward r d Including internal obstacle avoidance reward r in External obstacle avoidance reward r out Boundary reward r b Aggregation reward r g and overall energy consumption bonus r e The specific design is as follows:

[0026] r d =r in +r out +r b +r g +r e ;

[0027] Internal obstacle avoidance reward r in Designed as follows:

[0028]

[0029] Where, d safi Let d be the minimum safe flight distance for drone i. safj Let be the minimum safe flight distance for drone i. When the distance d between any drone i and drone i is... ij Less than the minimum safe flight distance d between the two drones sfi +d safi For i,j∈N and i≠i, a reward r1 is used to punish this wrong behavior;

[0030] Furthermore, external obstacle avoidance rewards r out Designed as follows:

[0031]

[0032] Where θ is the deflection angle of the UAV along the edge; The target vector is the direction in which the UAV is pointing towards the local target point; The normal vector of the obstacle; θ is the angle between the target vector and the normal vector; η is the angle formed by the intersection of the target vector and the normal vector (0≤η≤90°); i The yaw angle of the drone. Drone and obstacle o. m The distance between them is m = 1, 2, ..., M; when Approaching the edge reward trigger distance d ω When the obstacle avoidance bonus is triggered, r is obtained. θ Guide the drone to move along a deflection angle θ to avoid obstacles; when Less than the minimum safe distance d of drone i safi If a collision occurs, a reward of r2 is applied to penalize the drone's actions in the current state.

[0033] Furthermore, the boundary reward r b Designed as follows:

[0034]

[0035] When the distance d between the drone i and the airspace boundary i,bLess than the minimum safe flight space distance of the boundary When the drone's flight position exceeds the airspace boundary, a penalty is imposed using a reward r3.

[0036] Furthermore, the aggregation effect reward r g Designed as follows:

[0037]

[0038] Where, p v p represents the center point of each drone's current location; i Let i be the position of drone i. The smaller the sum of the distances of all drones from the center point, the greater the reward, thus guiding the drones to maintain a certain clustering effect;

[0039] Furthermore, during the flight of multiple drones, energy consumption must be considered. When drones are performing material delivery missions, the shorter the sum of the total flight path distances of all drones when they reach the delivery target area, the more energy is saved, and the greater the overall energy consumption bonus r. e Designed as:

[0040]

[0041] In the formula, d i Let d be the sum of the distances of the trajectories of UAV i; ∑d is the sum of the distances of the trajectories of all UAVs. i The smaller the value, the greater the reward, guiding the drone to explore the optimal movement path and conserve energy.

[0042] (7) Overall energy consumption bonus r e Designed as follows:

[0043]

[0044] Where, d i Let d be the sum of the distances of the trajectories of UAV i; ∑d is the sum of the distances of the trajectories of all UAVs. i The smaller the value, the greater the reward, guiding the drones to explore the optimal flight path and conserve energy. When multiple drones are in flight, energy consumption must be considered. For drones performing material delivery missions, the shorter the sum of the total flight path distances of all drones when reaching the delivery target area, the more energy is saved.

[0045] Compared with traditional UAV cooperative obstacle avoidance methods, the key to this method lies in the design of a reward function based on a hierarchical approach. By designing a hierarchical reward function, this method solves the problems of slow convergence speed due to sparse rewards and poor obstacle avoidance performance due to monotonous rewards in the MADDPG algorithm for multi-UAV cooperative obstacle avoidance material delivery tasks. Attached Figure Description

[0046] Figure 1This is a flowchart of a multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG) according to the present invention.

[0047] Figure 2 This is a diagram illustrating the algorithm structure of a multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG) according to the present invention.

[0048] Figure 3 This is a schematic diagram of a three-drone swarm supply delivery mission in a jungle warfare environment, as described in this embodiment of the invention.

[0049] Figure 4 This is a cumulative reward curve of the MADDPG algorithm for 3 UAV sorties in a jungle warfare environment, as shown in this embodiment of the invention.

[0050] Figure 5 This is a cumulative reward curve for three UAV sorties in a jungle warfare environment, as shown in this embodiment of the invention. Detailed Implementation

[0051] The basic process and algorithm structure are as follows: Figure 1 Appendix Figure 2 As shown:

[0052] This embodiment discloses a multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG), which specifically includes the following steps:

[0053] S10: Multiple drones interact with the environment to obtain their initial state space, action space, and reward information;

[0054] S20: Initialize network parameters and store the acquired status, action, and reward data into the experience pool;

[0055] S30: The designed hierarchical reward MADDPG algorithm is trained based on the sampled data from the experience pool to generate new actions;

[0056] S40: The drone interacts with the environment based on new actions, updates its status and rewards, and stores the new status, actions, and rewards in the experience pool;

[0057] S50: Deploy the network model trained by the hierarchical reward MADDPG algorithm on drones to achieve multi-drone collaborative obstacle avoidance.

[0058] The drone i obtains the environmental state s based on the environmental information at the current time t. t,i Action a is generated through the Actor-Critic network architecture. t,i And execute, the drone obtains a reward r t And enter the next state s t+1 , producing E = (S t A t ,Rt ,S t+1 ) is a set of training data, where S t Let A be the state of the UAV at time t. t For drone actions, R t For drone reward revenue, S t+1 The next state of the drone;

[0059] State-space data refers to the set of all possible states that a drone can perceive in its environment, and it is the key basis for the algorithm to make decisions. Considering the characteristics of multi-drone material delivery missions in complex obstacle environments, the drone's state-space data S mainly includes the drone's own information s. i Friendly information Obstacle Information (OBS) i , is represented as:

[0060]

[0061] Drone i Self Information s i Including the drone's position coordinates (x i ,y i ), velocity (v) along the x-axis and y-axis ix ,v iy ) and the relative position of the drone and the target point [x i,tar ,y i,tar ], represented as:

[0062] s i =[x i ,y i ,v ix ,v iy ,x i,tar ,y i,tar ];

[0063] Friendly information Includes the relative positions (x, y) of all drones within the cluster except drone i to drone i. i,j ,y i,j ), where i,j∈N and i≠j, and N is the number of drones in the cluster, expressed as:

[0064]

[0065] Obstacle Information (OBS) i Includes drones and obstacles. m relative position and obstacle speed And constraint information inf, where m = 1, 2, ..., M, M is the number of obstacles, expressed as:

[0066]

[0067] The constraint information (inf) includes whether the drone avoids collisions with friendly drones (inf1), whether it avoids collisions with obstacles (inf2), and whether it flies out of the environmental boundary (inf3) during the material delivery and assembly process, and is represented as follows:

[0068] inf = [inf1, inf2, inf3];

[0069] The action space A encompasses all possible actions a drone might perform during a delivery mission. It generally adheres to the principles of simplicity and efficiency and is a crucial factor in drone strategy generation. The drone's action space is designed with velocity v along the x-axis as its coordinates. ix Velocity v along the y-axis iy The action space A is represented as:

[0070] A = [v ix ,v iy ];

[0071] Design an overall reward function; the overall reward includes a top-level reward r. u and underlying reward r d The specific design is as follows:

[0072] r t =r u +r d ;

[0073] From a global perspective, the top-level reward r u By breaking down the final goal into multiple phased local objectives, multiple UAVs are guided to gradually approach the final goal, with a top-level reward r. u Under guidance, the underlying reward r d Based on the characteristics of obstacles in the mission environment, and on the basis of the drone completing its own material delivery mission, a specific plan and design is made for the reward of the drone.

[0074] The top-level rewards include rewards for being drawn to local objectives and penalties for being stuck at local objectives. The specific design is as follows:

[0075] r u =r point +r p ;

[0076] (1) Local target point traction reward r point Designed as follows:

[0077]

[0078] Where k1 is the adjustment coefficient, 0 <k1<1;P i Let P be the current position of drone i, where i = 1, 2, ..., N;subl Let be the l-th local target point, l∈L; ε∈(0,0.01);

[0079] (2) Penalty for being stuck at a local target point p Designed as follows:

[0080]

[0081] Where k2 is the adjustment coefficient, 0 <k2<1; and Let be the current state of UAV i at time t and its position at the previous time, respectively; σ is a set local oscillation distance, σ∈(2,10).

[0082] Furthermore, the underlying reward r d Including internal obstacle avoidance reward r in External obstacle avoidance reward r out Boundary reward r b Aggregation reward r g and overall energy consumption bonus r e The specific design is as follows:

[0083] r d =r in +r out +r b +r g +r e ;

[0084] (3) Internal obstacle avoidance reward r in Designed as follows:

[0085]

[0086] Where, d safi Let d be the minimum safe flight distance for drone i. safj Let d be the minimum safe flight distance for drone j. When the distance d between any drone i and drone j is... ij Less than the minimum safe flight distance d between the two drones safi +d safj For i,j∈N and i≠j, a reward r1 is used to punish this wrong behavior;

[0087] (4) External obstacle avoidance reward r out Designed as follows:

[0088]

[0089] Where θ is the deflection angle of the UAV along the edge; The target vector is the direction in which the UAV is pointing towards the local target point; The normal vector of the obstacle; θ is the angle between the target vector and the normal vector; η is the angle formed by the intersection of the target vector and the normal vector (0≤η≤90°); i The yaw angle of the drone. Drone and obstacle o. m The distance between them is m = 1, 2, ..., M; when Approaching the edge reward trigger distance d ω When the obstacle avoidance bonus is triggered, r is obtained. θ Guide the drone to move along a deflection angle θ to avoid obstacles; when Less than the minimum safe distance d of drone i safi If a collision occurs, a reward of r2 is applied to penalize the drone's actions in the current state.

[0090] (5) Boundary reward r b Designed as follows:

[0091]

[0092] When the distance d between the drone i and the airspace boundary i,b Less than the minimum safe flight space distance of the boundary When the drone's flight position exceeds the airspace boundary, a penalty is imposed using a reward r3.

[0093] (6) Gathering effect reward r g Designed as follows:

[0094]

[0095] Where, p v p represents the center point of each drone's current location; i Let i be the position of drone i. The smaller the sum of the distances of all drones from the center point, the greater the reward, thus guiding the drones to maintain a certain clustering effect;

[0096] Furthermore, during the flight of multiple drones, energy consumption must be considered. When drones are performing material delivery missions, the shorter the sum of the total flight path distances of all drones when they reach the delivery target area, the more energy is saved, and the greater the overall energy consumption bonus r. e Designed as:

[0097]

[0098] In the formula, d i Let d be the sum of the distances of the trajectories of UAV i; ∑d is the sum of the distances of the trajectories of all UAVs. i The smaller the value, the greater the reward, guiding the drone to explore the optimal movement path and conserve energy.

[0099] (7) Overall energy consumption bonus re Designed as follows:

[0100]

[0101] Where, d i Let d be the sum of the distances of the trajectories of UAV i; ∑d is the sum of the distances of the trajectories of all UAVs. i The smaller the value, the greater the reward, guiding the drones to explore the optimal flight path and conserve energy. When multiple drones are in flight, energy consumption must be considered. For drones performing material delivery missions, the shorter the sum of the total flight path distances of all drones when reaching the delivery target area, the more energy is saved.

[0102] The training steps include:

[0103] Step S301, calculate the objective function

[0104]

[0105] In the formula, Let be the action reward value obtained by agent i at time t through the h-th set of experience data, i.e., the total reward; The target Critic network outputs the action value at time t+1 using the h-th set of empirical data; Here, γ represents the target Critic network parameters, and γ is the discount factor.

[0106] Step S302, calculate the loss function

[0107]

[0108] In the formula, The action value output by the Critic network based on the h-th set of empirical data. These are the parameters for the Critic network.

[0109] Step S303: Update Critic network parameters

[0110] Updating Critic network parameters using stochastic gradient descent Represented as:

[0111]

[0112] In the formula, Let be the parameters of the Critic network for drone i at time t+1; α is the learning rate of the Critic network. Indicates gradient calculation; step S304, update Actor network parameters.

[0113] Output using the Critic network Value calculation of Actor network gradient Update the Actor network parameters using gradient ascent. Represented as:

[0114]

[0115] In the formula, Let i be the network parameters of the Actor network at time t; Let be the network parameters of the Actor at time t+1; β is the learning rate parameter of the Actor network; This indicates gradient calculation.

[0116] Step S305: Update the network parameters of the target Actor and Critic.

[0117]

[0118] In the formula, τ is a constant between 0 and 1, typically taken as 0.1 or 0.01;

[0119] When the average reward value per game falls below a certain threshold, it indicates that the algorithm training has reached a convergence state, at which point the agent's Actor network... and Critic network parameters To achieve stability.

[0120] To verify the effectiveness of this method, a collaborative obstacle avoidance experiment for material delivery in a jungle environment was conducted.

[0121] For material delivery missions in jungle warfare environments, taking three UAVs as an example, this paper compares and analyzes the convergence speed and obstacle avoidance performance of the MADDPG method and the proposed method.

[0122] (1) Experimental Description

[0123] Jungle warfare environments are characterized by dense trees, vegetation cover, and rich biodiversity including birds and insects. When multiple UAVs perform supply delivery missions in jungle environments, high-speed dynamic obstacles, such as birds, are a primary consideration. In a simulated jungle warfare environment, combat personnel's uniforms are soaked after swimming across a river, and to avoid detection, they cannot use open flames to dry their clothing. UAVs are now being used for supply delivery support, and the main obstacle encountered en route is dynamic birds.

[0124] Design of a 3-drone swarm for supply delivery mission in a jungle warfare environment, as follows: Figure 3As shown. The mission environment is a rectangular battlefield airspace with a depth of 10km (length) and a width of 6km (width). Within this airspace are two low-speed and two high-speed bird obstacles. Red circles represent low-speed dynamic bird obstacles, such as small birds like sparrows and hummingbirds, while black circles represent high-speed dynamic bird obstacles, such as medium-sized birds like pigeons and crows. The dynamic birds move randomly within the airspace by adding random directional perturbations. The maximum speed of low-speed birds does not exceed the maximum speed of the UAV, and the maximum speed of high-speed birds does not exceed 1.5 times the maximum speed of the UAV. Three UAVs, carrying approximately 60kg of Type 21 training uniforms and other supplies, randomly depart from their initial standby position on the left side of the map, coordinate obstacle avoidance, and reach the delivery target area on the right side to resupply the equipment.

[0125] In the experiment, the drone was 1.5m in size, the low-speed bird was 0.2m in size, and the high-speed bird was 0.25m in size. In a rectangular battlefield airspace that was 10km long and 6km wide, their size was too small to be displayed. Therefore, the drone and the bird were magnified by 100 times and 750 times respectively for display.

[0126] The conditions for a successful drone swarm delivery mission are:

[0127] 1) All drones arrive at the delivery target area and assemble, meeting r tar ≥d i,tar ;

[0128] 2) No collisions occurred between drones within the cluster during the journey, satisfying d. i,j ≥d safi +d safj ;

[0129] 3) No collisions occurred between the drone and obstacles, meeting the requirements.

[0130] 4) No drones flew out of the airspace boundary, meeting the requirements.

[0131] Based on data on the size and flight speed of common birds in jungles, the specific environmental parameter settings for multi-UAV swarm material delivery missions in jungle warfare environments are shown in Table 1.

[0132] Table 1 Experimental parameters in jungle warfare environment

[0133]

[0134] (2) Algorithm convergence speed analysis

[0135] For material delivery missions in jungle warfare environments, taking three drones as an example, we compare and analyze the convergence speed of the MADDPG algorithm and our proposed method.

[0136] The formula for calculating the convergence rate improvement rate δ is:

[0137]

[0138] e1 is the number of training sessions when this method converges, and e2 is the number of training sessions when the MADDPG algorithm converges.

[0139] The obstacle avoidance success rate η is

[0140]

[0141] In the formula, c is the number of tests and evaluations of the algorithm, c1 is the number of tests where internal collisions between drones caused mission failure, c2 is the number of tests where drones collided with obstacles caused mission failure, and c3 is the number of tests where drones flew out of the airspace boundary caused mission failure.

[0142] Figure 4 and Figure 5 The images show the cumulative reward curves for the MADDPG algorithm and the proposed method for three drones in a jungle warfare environment.

[0143] In the graph, the horizontal axis represents the number of training sessions, and the vertical axis represents the average cumulative reward value of the three drones in each training session. For the MADDPG algorithm, the cumulative reward curve tends to stabilize after the 70th training session, indicating that the algorithm has essentially converged, with an average cumulative reward value of 2200. For this method, the cumulative reward curve begins to stabilize after the 58th training session, indicating that the algorithm has entered a convergent state, with an average cumulative reward value of 3800. It is evident that this method converges faster and also achieves a higher average reward value after convergence.

[0144] Table 2 records the number of training sessions required for convergence of this method in each experiment and the improvement in convergence speed compared with the MADDPG algorithm.

[0145] Table 2. Analysis of convergence speed of algorithms for three UAVs in jungle warfare environment.

[0146]

[0147] Experimental results show that our method achieves an average convergence speed improvement of 16.2% compared to the MADDPG algorithm. It exhibits faster convergence characteristics compared to the MADDPG algorithm. Our method designs a reward function based on a hierarchical approach. The top-level reward considers global step-by-step guidance, decomposing the final delivery target into multiple staged local targets, gradually guiding the drone towards the delivery target area, thereby accelerating the overall rapid convergence of the drone.

[0148] (3) Analysis of the obstacle avoidance effect of the algorithm

[0149] The models trained using the MADDPG algorithm and our proposed method were evaluated in the same environment. 100 evaluation rounds were selected, and the collision occurrences during these 100 rounds are recorded in Table 3. The obstacle avoidance success rate was calculated. Our proposed method showed an average 9% improvement in obstacle avoidance success rate compared to the MADDPG algorithm, indicating that the HR-MADDPG algorithm has stronger environmental adaptability in multi-UAV material delivery tasks. The MADDPG algorithm's reward design is relatively simple, only considering distance-based target guidance and simple binary collision penalties. When UAVs encounter high-speed dynamic obstacles, they are not timely in avoiding them, resulting in mediocre obstacle avoidance performance. Our proposed method uses a hierarchical reward function design. The bottom-level reward incorporates edge-based obstacle avoidance, more comprehensively considering the relationship between dynamic obstacles and the UAV itself, thus making dynamic obstacle avoidance more flexible.

[0150] Table 3 Obstacle Avoidance Success Rate of 3 Drones in Jungle Warfare Environment

[0151]

[0152] Finally, pseudocode of the algorithm is provided for reference by those skilled in the art. It does not affect the implementation of this solution. The specific code needs to be adapted to the actual programming software. The specific pseudocode is as follows.

[0153]

[0154]

[0155] The above descriptions are merely embodiments of the present invention, and common knowledge regarding specific structures and characteristics is not elaborated upon here. It should be noted that those skilled in the art can make various modifications and improvements without departing from the structure of the present invention, and these should also be considered within the scope of protection of the present invention. These modifications and improvements will not affect the effectiveness of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A multi-UAV cooperative obstacle avoidance method based on hierarchical rewards (MADDPG), comprising the following steps: S10: Multiple drones interact with the environment to obtain their initial state space, action space, and reward information; S20: Initialize network parameters and store the acquired status, action, and reward data into the experience pool; S30: The designed hierarchical reward MADDPG algorithm is trained based on the sampled data from the experience pool to generate new actions; S40: The drone interacts with the environment based on new actions, updates its status and rewards, and stores the new status, actions, and rewards in the experience pool; S50: Deploy the algorithm model after the hierarchical reward MADDPG algorithm has been trained and converged to a stable state on UAVs to achieve multi-UAV collaborative obstacle avoidance; Its features are: The update of the state and reward in step S40 is based on the hierarchical design of the overall reward function, which is based on the hierarchical idea of ​​"global step-by-step guidance and individual dynamic programming". The overall reward function includes the top-level reward r. u and underlying reward r d The specific design is as follows: r t ＝r u +r d ； Top-level reward r u Considering global step-by-step guidance and combining global state information, the final delivery target of the drone is broken down into multiple phased local targets, gradually guiding the drones towards the final delivery target to ensure rapid overall convergence of multiple drones; at the top-level reward r u Under guidance, the underlying reward r d Based on the characteristics of obstacles in the mission environment, and taking into account factors such as internal collision avoidance, external obstacle avoidance, boundary constraints, aggregation effect, and overall energy consumption, specific planning and design of rewards for individual drones are carried out.

2. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 1, characterized in that: The top-level reward r u Including local target point traction reward r point and the penalty for being stuck at a local target point p The specific design is as follows: r u ＝r point +r p ； Local target point traction reward r point Designed as follows: Where k1 is the adjustment coefficient, 0 <k1<1;P i Let P be the current position of drone i, where i = 1, 2, ..., N; subl Let be the l-th local target point, l∈L; ε∈(0,0.01); Wandering punishment r p Designed as follows: In the formula, k2 is the adjustment coefficient, 0 <k2<1; and Let be the current state of UAV i at time t and its position at the previous time, respectively; σ is a set local oscillation distance, σ∈(2,10).

3. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 1, characterized in that: The underlying reward r d Including internal obstacle avoidance reward r in External obstacle avoidance reward r out Boundary rewards, aggregation b Collect rewards r g and overall energy consumption bonus r e The specific design is as follows: r d ＝r in +r out +r b +r g +r e。 4. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 3, characterized in that: The internal obstacle avoidance reward r in Designed as follows: Where, d safi Let d be the minimum safe flight distance for drone i. safj Let d be the minimum safe flight distance for drone j; when the distance d between any drone i and drone j is... ij Less than the minimum safe flight distance d between the two drones safi +d safj For i,j∈N and i≠j, a reward r1 is used to punish this wrong behavior.

5. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 3, characterized in that: The external obstacle avoidance reward r out Designed as follows: Where θ is the deflection angle of the UAV along the edge; The target vector is the direction in which the UAV is pointing towards the local target point; The normal vector of the obstacle; θ is the angle between the target vector and the normal vector; η is the angle formed by the intersection of the target vector and the normal vector (0≤η≤90°); i Yaw angle of the drone; drone and obstacle o m The distance between them is when Approaching the edge reward trigger distance d ω When the obstacle avoidance bonus is triggered, r is obtained. θ Guide the drone to move along a deflection angle θ to avoid obstacles; when Less than the minimum safe distance d of drone i safi If a collision occurs, a reward r2 is applied to penalize the drone's actions in the current state.

6. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 3, characterized in that: The boundary reward r b Designed as follows: When the distance d between the drone i and the airspace boundary i,b Less than the minimum safe flight space distance of the boundary When the drone's flight position exceeds the airspace boundary, a penalty is imposed using a reward r3.

7. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 3, characterized in that: The aggregation effect reward r g Designed as follows: Where, p v p represents the center point of each drone's current location; i Let i be the position of drone i. The smaller the sum of the distances of all drones from the center point, the greater the reward, thus guiding the drones to maintain a clustered effect.

8. The MADDPG multi-UAV cooperative obstacle avoidance method based on hierarchical rewards as described in claim 3, characterized in that: The overall energy consumption reward r e Designed as follows: Where, d i Let d be the sum of the distances of the trajectories of UAV i; ∑d is the sum of the distances of the trajectories of all UAVs. i The smaller the value, the greater the reward, guiding the drone to explore the optimal movement path and conserve energy.

Images