Multi-unmanned aerial vehicle cooperative hunting method based on obstacle contribution angle

Abstract

The invention specifically discloses a multi-unmanned aerial vehicle cooperative hunting method based on an obstacle contribution angle, and relates to the technical field of multi-agent cooperative control and path planning. The method comprises the following steps: constructing a surrounding scene comprising N chasing unmanned aerial vehicles, an escape unmanned aerial vehicle and a static obstacle; constructing a traditional decision logic of an escape unmanned aerial vehicle and a decision logic of a chasing unmanned aerial vehicle based on an MADDPG algorithm based on the surrounding scene; based on the constructed MADDPG decision logic, designing a multi-unmanned aerial vehicle encircling strategy reward function based on course learning; and combining the reward function with the MADDPG decision logic to obtain a multi-unmanned aerial vehicle cooperative hunting strategy based on the MADDPG. According to the method, the MADDPG algorithm is adopted to design the hunting strategy, the hunting process is divided into four states of searching, surrounding, transferring and capturing, the obstacles are fully utilized to construct the reward function of each state, and the hunting success rate is improved.

Description

Technical Field

[0001] This invention relates to the field of multi-agent cooperative control and path planning technology, and in particular to a multi-UAV cooperative encirclement method based on obstacle contribution angle. Background Technology

[0002] In numerous fields such as military reconnaissance, security monitoring, emergency rescue, and airspace control, multi-UAV collaborative encirclement missions have demonstrated extremely high application value and practical significance. This mission can be modeled as a typical pursuit-escape game model. In this model, N pursuing UAVs need to cooperate efficiently to gradually pursue and encircle a single escaping UAV, while the escaping UAV will actively adopt various maneuver strategies to try to escape the encirclement. The entire process involves several core technical aspects such as path planning, dynamic obstacle avoidance, collaborative decision-making, and strategy optimization for the pursuing UAVs.

[0003] Currently, research on multi-UAV collaborative encirclement and capture mainly focuses on two major directions: differential game theory and cooperative control theory, providing fundamental theoretical support for the technical implementation of encirclement and capture tasks. However, in practical application scenarios, complex environmental conditions (such as the presence of static obstacles) and the dynamic characteristics of escaping UAVs pose many insurmountable key challenges to existing technical solutions, severely limiting their reliable deployment and efficient application in real-world scenarios.

[0004] On the one hand, in complex encirclement scenarios with static obstacles, existing technologies generally treat obstacles as simply passive avoidance targets. This approach forces pursuit drones to simultaneously optimize both the encirclement objective and obstacle avoidance: they must narrow the encirclement of the escaping drone through coordinated movement while avoiding collisions with obstacles, scene edges, and other pursuit drones. This conflict between the two objectives not only significantly increases the difficulty of strategy decision-making for pursuit drones but also causes the spatial complexity of strategy learning to increase exponentially. In actual decision-making, pursuit drones often struggle to balance encirclement efficiency and obstacle avoidance safety, directly leading to a significant decrease in the success rate of encirclement missions.

[0005] On the other hand, insufficient dynamic adaptability to high-speed maneuvering escape drones is another core challenge faced by existing technologies. While existing solutions set speed boundaries between the pursuing and escaping drones (typically setting the escape drone's speed boundary to be greater than the pursuing drone's), existing methods struggle to quickly adjust the encirclement strategy to adapt to the dynamic changes when the escape drone's actual speed exceeds expectations or it suddenly changes direction. The high-speed movement or sudden maneuvers of the escape drone can easily cause the already formed encirclement to collapse. During the process of reorganizing the encirclement formation, the pursuing drone must simultaneously cope with the dual pressures of dynamic obstacle avoidance and path replanning. The decision-making response speed, strategy adaptability, and coordination efficiency of existing methods cannot meet the real-time and stability requirements of such scenarios, making it difficult to rebuild a complete encirclement and ultimately leading to the failure of the encirclement mission.

[0006] In summary, existing multi-drone collaborative capture technologies have significant shortcomings in terms of obstacle scene handling and dynamic adaptability to high-speed maneuvering escape drones, making it difficult to efficiently meet the capture needs in complex dynamic scenarios. Summary of the Invention

[0007] The purpose of this invention is to propose a multi-UAV cooperative encirclement method based on obstacle contribution angle. By constructing an encirclement scenario with static obstacles, designing the traditional decision-making logic of the escaping UAV and the MADDPG decision-making logic of the pursuing UAV, and integrating a reward function based on curriculum learning that includes collision avoidance, distance, and phased cooperative encirclement, this method achieves strategy guidance under multi-objective optimization, reduces the complexity of strategy learning, improves adaptability to complex scenarios and dynamic escaping targets, and efficiently completes multi-UAV cooperative encirclement tasks.

[0008] To achieve the above objectives, this invention proposes a multi-UAV cooperative encirclement method based on obstacle contribution angle, the steps of which are as follows: Step S1: Construct a drone encirclement scenario, the encirclement scenario including N Specific information about a pursuit drone, an escape drone, and a static obstacle; Step S2: Based on the drone encirclement scenario, establish the traditional decision-making logic for escaping drones; Step S3: Based on the drone encirclement scenario, build a multi-agent deep deterministic policy gradient (MADDPG) decision logic for chasing drones. Step S4: Based on the MADDPG decision logic, design a reward function for a multi-UAV encirclement strategy based on course learning to achieve strategy guidance under multi-objective optimization conditions. Step S5: Integrate the reward function with the MADDPG decision logic to generate a multi-drone collaborative capture strategy based on MADDPG.

[0009] Preferably, in step S1, the drone encirclement and capture scenario is constructed, and the steps are as follows: Step S11: Construct a finite closed region in a two-dimensional Cartesian coordinate system, and define the obstacle D using a circle with center O and radius... The circle is represented by the following formula: ; in, The area covered by the obstacle, ( , () represents the coordinates of the center of the obstacle circle. Let D be the radius of the obstacle. Step S12: Construct the kinematic characteristics of the UAV, using the following formula: ; in, To track down drones The plane coordinate position, To track down drones speed magnitude, To track down drones velocity direction and x The angle along the positive direction of the axis. To track down drones The acceleration; Step S13: Set up the pursuit drone P The velocity boundary is Escape drone E The velocity boundary is ,and .

[0010] Preferably, in step S2, the traditional escape decision-making mechanism for the escaping drone is selected based on a potential field model of repulsive force, and each pursuing drone... A repulsive force is generated in the displacement vector detection of the escaping drone. The strength of this force is inversely proportional to the square of the relative distance. Environmental edges and obstacles generate a repulsive potential field through the nearest point of action, thus realizing a dynamic avoidance mechanism. The formula for synthesizing the motion vector of the escaping drone is as follows: ; in, For the motion vector of the escaping drone, The coordinates of the escaping drone. To track down drones coordinates Let D be the coordinates of the obstacle. The coordinates of the closest point to the escaping drone at the edge of the scene. , , These represent the repulsive force coefficients of the pursuing drone, obstacles, and boundaries on the escaping drone, respectively. , , The first i The distance between the pursuit drone, obstacles, boundaries, and the escaping drone. i , k It is an integer.

[0011] Preferably, in step S3, the specific steps are as follows: Step S31: Initialize the network parameters of the policy network Actor, the value network Critic, and the training parameters of the MADDPG algorithm; Step S32: Copy the Actor and Critic network parameters to the Target Actor network and the Target Critic network; Step S33: Randomly initialize the initial observations of the pursuit drone, and reset the rewards for both the pursuit and pursuit sides to 0; Step S34: The Actor networks of both the pursuing and fugitive drones continuously interact with the environment, storing and retrieving data. ) Add to experience pool M until it is full; among which, This represents the global state of the environment at time step m. The instant reward for tracking down the drone at time step m. The instant reward for escaping the drone at time step m. The actions that all drones jointly execute at time step m+1 according to their respective strategies. For the environment to receive joint actions Then, transition to the next global state; Step S35: Compare whether the maximum number of simulation steps for a round has been reached.

[0012] Preferably, in step S35, the comparison is made to determine whether the maximum number of simulation steps for a round has been reached. The specific steps are as follows: Step S351: If the maximum number of simulation steps is reached, take K data points from the experience pool M, update the parameters of the Actor and Critic networks, and calculate the loss function of the Critic network. And update the Critic network parameters, as shown in the following formula: ; in, For the target Q value, Let be the global state of the environment when the t-th data is received. When the t-th data is... i The actions performed by the drone The parameters to be optimized for the current Critic network are as follows: For the current Critic network, t To update the network parameters t One data point, ; Calculate the policy gradient of the Actor network And update the Actor network parameters, as shown in the following formula: ; in, For the first i Actor network parameters for a drone Operator for finding gradient For the local observation of the i-th UAV at the t-th data point, Let be the parameters to be optimized for the Actor network of the i-th drone. For the action of the i-th drone Operators for finding the gradient; The update formula for the Actor network parameters of both the pursuing and fugitive parties is as follows: ; in, The learning rate of the Actor network is a pre-set positive number that controls the step size of parameter updates; Then return to step S34; Step S352: If the maximum number of simulation steps in a round is reached, compare whether the maximum number of training rounds has been reached: Step S3521: If the maximum number of training rounds is reached and the reward function converges, end the training; if the maximum number of training rounds is reached but the reward function does not converge, return to step S32. Step S3522: If the maximum number of training rounds has not been reached, return to step S32.

[0013] Preferably, in step S4, the specific steps are as follows: Step S41: Design collision avoidance rewards The details are as follows: The purpose of the collision avoidance reward is to guide the pursuit drone to avoid collisions with obstacles, scene edges, and other pursuit drones, as shown in the following formula: ; in, For the safety threshold radius, For positive integers, To track down drones To the obstacle D The minimum distance, To track down drones The minimum distance to the edge of the scene. To track down drones and tracking drones The distance between them; Step S42, Design Distance Reward The details are as follows: The purpose of distance rewards is to guide the pursuit drone to reduce the distance between itself and the escaping drone, thus improving the pursuit drone's performance. The distance reward is set as follows: ; in, It is a positive coefficient. To track the vector pointing from the location of the drone to the location of the escaping drone, To track down drones velocity vector The maximum speed required to track the drone.

[0014] Step S43: The design of a multi-drone collaborative encirclement reward is divided into four stages: search, encirclement, transfer, and capture.

[0015] Preferably, in step S43, a reward system for multi-drone collaborative capture is designed, with the specific steps as follows: Step S431, Search Phase: The escaping drone is located outside the closed envelope formed by the pursuing drone and the obstacle. The determination criteria are as follows: ; in, For the escape drone E, the first i The pursuit drone and the first i +1 area of ​​the largest convex region formed by the pursuit drones Let O be the area of ​​the largest convex region formed by the escape drone E and the Nth pursuing drone, with center O as the center. Let O be the area of ​​the largest convex region formed by the escape drone E and the first pursuing drone with center O as the center. Let O be the area of ​​the largest convex region formed by N pursuing drones centered at point O. At this point, the pursuit game is in the search phase, and the reward function for the search phase is as follows: ; in, Rewards for capturing drones during the search phase; Step S432, Encirclement Phase: The escaping drone is located inside the closed envelope formed by the pursuing drone and the obstacles. The drones are numbered from 1 to N in a counter-clockwise order. i When = N, let i+1 = 1; when i When = 1, let i -1 = N; Based on the relative positions of each drone and the obstacles, there are three cases to determine whether to maintain the encirclement phase, and the corresponding reward function is used to guide the pursuing drone to start shrinking the encirclement. Step S433, the transfer phase is the transition phase from the encirclement phase to the capture phase. The pursuing drones continuously shrink the encirclement while adjusting the distance between each other. Step S434: During the capture phase, the Apollonius circles formed by the pursuing drone, the escaping drone, and the obstacles intersect or are tangent to each other, and the determination criteria are as follows: ; The reward function for the capture phase is designed as follows: ; in, For collective positioning angles, As a reward for capturing drones during the capture phase, It is a positive coefficient.

[0016] Preferably, in step S432, based on the relative positions of each drone and the obstacle, three cases are identified to determine whether the encirclement phase is maintained, and a corresponding reward function is used to guide the pursuing drone to begin narrowing the encirclement. Scenario 1: For those that satisfy pursuit drones That is, to hunt down drones If there are no obstacles on either side, then: ; At this point, the area is in an encirclement phase; among which... The maximum speed ratio between the escape drone and the pursuit drone. To track down drones and tracking drones The distance between them To track down drones The distance to the escaping drone; the reward function during the encirclement phase is as follows: ; in, Rewards for capturing drones during the encirclement phase. For the first j The distance between the pursuing drone and the escaping drone; Scenario 2: For those that satisfy pursuit drones That is, to hunt down drones If the right side is an obstacle and the left side is another pursuit drone, then: ; in, To track down drones The center of the Apollonius circle formed by the escape drone E and the escape drone E To the center of the obstacle distance, To track down drones and tracking drones The distance between them To track down drones Distance from the escaping drone To track down drones The radius of the Apollonius circle formed by the escaping drone E; This condition indicates the pursuit of drones. It forms an encirclement around the drone on its left and defends the gap between the drone and the obstacle on its right. The reward function is as follows: ; Scenario 3: For those that meet the requirements pursuit drones That is, to hunt down drones To the right are other pursuit drones and to the left are obstacles. If the following conditions are met: ; That is, to hunt down drones The drone forms an encirclement around the pursuing drone on the right, while defending the gap between the drone and the obstacle on the left. The reward function is as follows: .

[0017] Preferably, in step S433, the Apollonius circle occupancy angle and contribution angle are introduced during the transfer stage, and the specific calculations are as follows: Positioning Angle: Defining the escape drone E and the pursuit drone Apollonius Circle The included angle formed by the lines connecting the tangent points is the angle for tracking drones. The occupancy angle is denoted as The formula is as follows: ; Collective positioning angle: The union of the positioning angles of the pursuit drones participating in the encirclement is the collective positioning angle, denoted as... The formula is as follows: ; When collective positioning angle At that time, a closed encirclement was formed; Contributors to the Capture: Draw a ray along any radial direction of the escaping drone E. The owner or obstacle of the first Apollonius circle intersecting the ray is defined as the contributor to the capture along that ray direction, denoted as... ; Contribution Angle: The range of angles actually generated by a single UAV's Apollonius circle or obstacles during the encirclement is defined as the contribution angle. ; Regarding the direction of motion of the escaping drone E Perform k samplings, then categorize and accumulate the contribution angles of each drone or obstacle according to the contributors in these k directions to obtain the contribution angle of each drone or obstacle. : ; in, Indicates the first i Contributors to the encirclement and capture in all directions, For the sampling flag function, when hour, Otherwise, it is 0; ; Contribution Corner This represents the capture contribution of each pursuing drone and obstacle in the escape direction of the escaping drone, using the capture contribution angle of each pursuing drone or obstacle to the collective occupancy angle. The calculation is performed using the following formula: ; in, To track down drones Contribution corner, Contribution angle for the escape drone E; Reward function during the transition phase and collective position angle Positive correlation, and with the pursuit drone continuously narrowing the encirclement, the reward function during the transfer phase is as follows: ; in, Rewards for capturing drones during the transfer phase. It is a positive coefficient.

[0018] Therefore, this invention proposes a multi-UAV cooperative encirclement method based on obstacle contribution angle, which has the following advantages: This invention provides a multi-UAV cooperative encirclement method based on obstacle contribution angle. For multi-UAV cooperative encirclement tasks in obstacle scenarios, the method selects a multi-agent deep reinforcement learning algorithm based on MADDPG to design the encirclement strategy. Combined with course learning, the encirclement process is divided into four stages: search, surround, transfer and capture. Reward functions are constructed for each stage to improve the success rate of encirclement and reduce the time required for successful encirclement.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is a flowchart of a multi-UAV cooperative encirclement method based on obstacle contribution angle according to the present invention; Figure 2 This is a structural diagram of the multi-agent deep deterministic policy gradient (MADDPG) framework in this invention; Figure 3 This is a schematic diagram of the search phase provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the encirclement stage provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the occupancy angle provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. The described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0023] Example like Figure 1 As shown, this invention provides a multi-UAV cooperative encirclement method based on obstacle contribution angle, the steps of which are as follows: Step S1: Construct a drone encirclement scenario, the encirclement scenario including N Specific information about a pursuit drone, an escape drone, and a static obstacle; the steps for constructing the drone encirclement scenario are as follows: Step S11: Construct a finite closed region in a two-dimensional Cartesian coordinate system, and define the obstacle D using a circle with center O and radius... The circle is represented by the following formula: ; in, The area covered by the obstacle, ( , () represents the coordinates of the center of the obstacle circle. Let D be the radius of the obstacle. Step S12: Construct the kinematic characteristics of the UAV, using the following formula: ; in, To track down drones The plane coordinate position, To track down drones speed magnitude, To track down drones velocity direction and x The angle along the positive direction of the axis. To track down drones The acceleration; Step S13: Set up the pursuit drone P The velocity boundary is Escape drone E The velocity boundary is ,and .

[0024] Step S2: Based on the drone encirclement scenario, establish the traditional decision-making logic for escaping drones; Traditional escape decision-making mechanisms for escaping drones are based on a potential field model using repulsive forces, with each pursuit drone... A repulsive force is generated in the displacement vector detection of the escaping drone. The strength of this force is inversely proportional to the square of the relative distance. Environmental edges and obstacles generate a repulsive potential field through the nearest point of action, thus realizing a dynamic avoidance mechanism. The formula for synthesizing the motion vector of the escaping drone is as follows: ; in, For the motion vector of the escaping drone, The coordinates of the escaping drone. To track down drones coordinates Let D be the coordinates of the obstacle. The coordinates of the closest point to the escaping drone at the edge of the scene. , , These represent the repulsive force coefficients of the pursuing drone, obstacles, and boundaries on the escaping drone, respectively. , , The first i The distance between the pursuit drone, obstacles, boundaries, and the escaping drone. i , k It is an integer.

[0025] Step S3: Based on the drone encirclement scenario, build a multi-agent deep deterministic policy gradient (MADDPG) decision logic for chasing drones, such as... Figure 2 As shown, the specific steps are as follows: Step S31: Initialize the network parameters of the policy network Actor, the value network Critic, and the training parameters of the MADDPG algorithm; Step S32: Copy the Actor and Critic network parameters to the Target Actor network and the Target Critic network; Step S33: Randomly initialize the initial observations of the pursuit drone, and reset the rewards for both the pursuit and pursuit sides to 0; Step S34: The Actor networks of both the pursuing and fugitive drones continuously interact with the environment, storing and retrieving data. ) Add to experience pool M until it is full; among which, This represents the global state of the environment at time step m. The instant reward for tracking down the drone at time step m. The instant reward for escaping the drone at time step m. The actions that all drones jointly execute at time step m+1 according to their respective strategies. For the environment to receive joint actions Then, transition to the next global state; Step S35: Compare whether the maximum number of simulation steps for a round has been reached. The specific steps are as follows: Step S351: If the maximum number of simulation steps is reached, take K data points from the experience pool M, update the parameters of the Actor and Critic networks, and calculate the loss function of the Critic network. And update the Critic network parameters, as shown in the following formula: ; in, For the target Q value, Let be the global state of the environment when the t-th data is received. When the t-th data is... i The actions performed by the drone The parameters to be optimized for the current Critic network are as follows: For the current Critic network, t To update the network parameters t One data point, ; Calculate the policy gradient of the Actor network And update the Actor network parameters, as shown in the following formula: ; in, For the first iActor network parameters for a drone Operator for finding gradient For the local observation of the i-th UAV at the t-th data point, Let be the parameters to be optimized for the Actor network of the i-th drone. For the action of the i-th drone Operators for finding the gradient; The update formula for the Actor network parameters of both the pursuing and fugitive parties is as follows: ; in, The learning rate of the Actor network is a pre-set positive number that controls the step size of parameter updates; Then return to step S34; Step S352: If the maximum number of simulation steps in a round is reached, compare whether the maximum number of training rounds has been reached: Step S3521: If the maximum number of training rounds is reached and the reward function converges, end the training; if the maximum number of training rounds is reached but the reward function does not converge, return to step S32. Step S3522: If the maximum number of training rounds has not been reached, return to step S32.

[0026] Step S4: Based on the MADDPG decision logic, design a reward function for a multi-UAV encirclement strategy based on course learning to achieve strategy guidance under multi-objective optimization conditions. The specific steps are as follows: Step S41: Design collision avoidance rewards The details are as follows: The purpose of the collision avoidance reward is to guide the pursuit drone to avoid collisions with obstacles, scene edges, and other pursuit drones, as shown in the following formula: ; in, For the safety threshold radius, For positive integers, To track down drones To the obstacle D The minimum distance, To track down drones The minimum distance to the edge of the scene. To track down drones and tracking drones The distance between them; Step S42, Design Distance Reward The details are as follows: The purpose of distance rewards is to guide the pursuit drone to reduce the distance between itself and the escaping drone, thus improving the pursuit drone's performance. The distance reward is set as follows: ; in, It is a positive coefficient. To track the vector pointing from the location of the drone to the location of the escaping drone, To track down drones velocity vector The maximum speed required to track the drone.

[0027] Step S43: The multi-drone collaborative capture reward design is divided into four stages: search, encirclement, transfer, and capture, as detailed below: Step S431, as Figure 3 As shown, during the search phase, the escaping drone is located outside the closed envelope formed by the pursuing drone and the obstacle. The determination criteria are as follows: ; in, For the escape drone E, the first i The pursuit drone and the first i +1 area of ​​the largest convex region formed by the pursuit drones Let O be the area of ​​the largest convex region formed by the escape drone E and the Nth pursuing drone, with center O as the center. Let O be the area of ​​the largest convex region formed by the escape drone E and the first pursuing drone with center O as the center. Let O be the area of ​​the largest convex region formed by N pursuing drones centered at point O. At this point, the pursuit game is in the search phase, and the reward function for the search phase is as follows: ; in, Rewards for capturing drones during the search phase; Step S432, as Figure 4 As shown, during the encirclement phase, the escaping drone is located inside the closed envelope formed by the pursuing drone and the obstacles. The drones are numbered from 1 to N in a counter-clockwise order. i When = N, let i +1 = 1; when i When = 1, let i -1 = N; Based on the relative positions of each drone and the obstacles, three cases are considered to determine whether the encirclement phase should continue, and a corresponding reward function is used to guide the pursuing drones to begin narrowing the encirclement, as detailed below: Scenario 1: For those that satisfy pursuit drones That is, to hunt down drones If there are no obstacles on either side, then: ; At this point, the area is in an encirclement phase; among which... The maximum speed ratio between the escape drone and the pursuit drone. To track down drones and tracking drones The distance between them To track down drones The distance to the escaping drone; the reward function during the encirclement phase is as follows: ; in, Rewards for capturing drones during the encirclement phase. For the first j The distance between the pursuing drone and the escaping drone; Scenario 2: For those that satisfy pursuit drones That is, to hunt down drones If the right side is an obstacle and the left side is another pursuit drone, then: ; in, To track down drones The center of the Apollonius circle formed by the escape drone E and the escape drone E To the center of the obstacle distance, To track down drones and tracking drones The distance between them To track down drones Distance from the escaping drone To track down drones The radius of the Apollonius circle formed by the escaping drone E; This condition indicates the pursuit of drones. It forms an encirclement around the drone on its left and defends the gap between the drone and the obstacle on its right. The reward function is as follows: ; Scenario 3: For those that meet the requirements pursuit drones That is, to hunt down drones To the right are other pursuit drones and to the left are obstacles. If the following conditions are met: ; That is, to hunt down drones The drone forms an encirclement around the pursuing drone on the right, while defending the gap between the drone and the obstacle on the left. The reward function is as follows: .

[0028] Step S433, the transfer phase, is a transitional phase from the encirclement phase to the capture phase. The pursuing drones continuously shrink the encirclement while adjusting the distance between them. Simultaneously, the Apollonius circle occupancy angle and contribution angle are introduced during the transfer phase, calculated as follows: like Figure 5 As shown, the escape drone E and the pursuit drone Apollonius Circle The included angle formed by the lines connecting the tangent points is the angle for tracking drones. The occupancy angle is denoted as The formula is as follows: ; Collective positioning angle: The union of the positioning angles of the pursuit drones participating in the encirclement is the collective positioning angle, denoted as... The formula is as follows: ; When collective positioning angle At that time, a closed encirclement was formed; Contributors to the Capture: Draw a ray along any radial direction of the escaping drone E. The owner or obstacle of the first Apollonius circle intersecting the ray is defined as the contributor to the capture along that ray direction, denoted as... ; Contribution Angle: The range of angles actually generated by a single UAV's Apollonius circle or obstacles during the encirclement is defined as the contribution angle. ; Regarding the direction of motion of the escaping drone E Perform k samplings, then categorize and accumulate the contribution angles of each drone or obstacle according to the contributors in these k directions to obtain the contribution angle of each drone or obstacle. : ; in, Indicates the first i Contributors to the encirclement and capture in all directions, For the sampling flag function, when hour, Otherwise, it is 0; ; Contribution Corner This represents the capture contribution of each pursuing drone and obstacle in the escape direction of the escaping drone, using the capture contribution angle of each pursuing drone or obstacle to the collective occupancy angle. The calculation is performed using the following formula: ; in, To track down drones Contribution corner, Contribution angle for the escape drone E; Reward function during the transition phase and collective position angle Positive correlation, and with the pursuit drone continuously narrowing the encirclement, the reward function during the transfer phase is as follows: ; in, Rewards for capturing drones during the transfer phase. It is a positive coefficient.

[0029] Step S434: During the capture phase, the Apollonius circles formed by the pursuing drone, the escaping drone, and the obstacles intersect or are tangent to each other, and the determination criteria are as follows: ; The reward function for the capture phase is designed as follows: ; in, For collective positioning angles, As a reward for capturing drones during the capture phase, It is a positive coefficient.

[0030] Step S5: Integrate the reward function with the MADDPG decision logic to generate a multi-drone collaborative capture strategy based on MADDPG.

[0031] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0032] Therefore, this invention provides a multi-UAV collaborative encirclement method based on obstacle contribution angle. By using the MADDPG multi-agent deep reinforcement learning algorithm to design the encirclement strategy, and combining course learning to divide the encirclement process into four stages: search, surround, transfer, and capture, and construct corresponding reward functions for each stage, it can effectively improve the success rate of multi-UAV collaborative encirclement in obstacle scenarios, while reducing the time required for successful encirclement.

[0033] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multi-UAV cooperative hunting method based on obstacle contribution angle, characterized in that, The specific steps are as follows: Step S1, constructing a UAV encirclement scene, the encirclement scene comprising N specific information of one pursuit UAV, one escape UAV and one static obstacle; Step S2, based on the scene of surrounding the unmanned aerial vehicle, the traditional decision logic of escaping unmanned aerial vehicle is built; Step S3, based on the scene of surrounding the unmanned aerial vehicle, the multi-agent deep deterministic policy gradient MADDPG decision logic for chasing the unmanned aerial vehicle is built; Step S4, based on the MADDPG decision logic, the multi-unmanned aerial vehicle surrounding strategy reward function based on curriculum learning is designed to realize the strategy guidance under the condition of multi-objective optimization; Step S5, the reward function is combined with the MADDPG decision logic to generate the multi-unmanned aerial vehicle cooperative surrounding strategy based on MADDPG.

2. The multi-UAV cooperative hunting method based on obstacle contribution angle according to claim 1, characterized in that, In step S1, the unmanned aerial vehicle surrounding scene is constructed, and the steps are as follows: Step S11, construct a finite closed region in a two-dimensional Cartesian coordinate system, and represent the obstacle D as a circle with center O and radius , as follows: ； wherein, is the covered area of the obstacle, , is the center coordinate of the obstacle, is the radius of the obstacle D; Step S12, the kinematics of the unmanned aerial vehicle is constructed, and the formula is as follows: ； in, To hunt down drones The plane coordinate position, To hunt down drones speed magnitude, To track down drones velocity direction and x The angle along the positive direction of the axis. To hunt down drones The acceleration; Step S13, setting the speed boundary of the pursuit drone P is , the speed boundary of the escape drone E is , and .

3. The multi-UAV cooperative hunting method based on obstacle contribution angle of claim 1, wherein, In step S2, the traditional escape decision mechanism of the escape UAV selects a repulsive force potential field model, each pursuit UAV The repulsive effect is generated on the displacement vector of the escape UAV, the action strength is inversely proportional to the square of the relative distance, the environmental edge and the obstacle generate a repulsive potential field through the nearest action point to realize a dynamic avoidance mechanism, and the motion vector synthesis formula of the escape UAV is as follows: ； wherein, is the motion velocity vector of the escape drone, is the coordinate of the escape drone, is the coordinate of the pursuit drone, is the coordinate of the obstacle D, is the coordinate of the border closest to the escape drone, , , , are repulsion force coefficients of the pursuit drone, the obstacle, and the border to the escape drone, respectively, , , are distances from the escape drone to the first i pursuit drone, the obstacle, and the border, respectively, i , k is an integer.

4. The multi-UAV cooperative hunting method based on obstacle contribution angle according to claim 1, characterized in that, In step S3, the specific steps are as follows: Step S31, the network parameters of the strategy network Actor and the value network Critic and the training parameters of the MADDPG algorithm are initialized; Step S32, the Actor and Critic network parameters are copied to the target actor network Target Actor and the target network Target Critic; Step S33, the initial observation of the pursuer and the evader is randomly initialized, and the rewards of both sides are set to 0; Step S34, the Actor networks of the pursuer and the evader drone respectively interact with the environment continuously, access data ) into the experience pool M until it is filled; wherein, is the global state of the environment at time step m, is the immediate reward obtained by the pursuer drone at time step m, is the immediate reward obtained by the evader drone at time step m, is the joint action performed by all drones at time step m+1 according to their respective policies, is the next global state of the environment after receiving the joint action . Step S35, whether the maximum simulation step number of the round is reached is compared.

5. The multi-UAV cooperative hunting method based on obstacle contribution angle according to claim 4, characterized in that, In step S35, whether the maximum simulation step number of the round is reached is compared, and the specific steps are as follows: Step S351, if the maximum number of simulation steps per round is reached, K data are taken from the experience pool M, the Actor and Critic network parameters are updated, and the Critic network loss function is calculated and the Critic network parameters are updated, as follows: ； in, For the target Q value, Let t represent the global state of the environment when the t-th data is received. When the t-th data is... i The actions performed by the drone The parameters to be optimized for the current Critic network are as follows: For the current Critic network, t To update the network parameters t One data point, ; Computing actor network policy gradients and updating the actor network parameters as follows: ； wherein, is an actor network parameter for the i-th drone i is an operator that computes the gradient of, is a local observation of the i-th drone at the t-th data time, is an optimization parameter of the actor network of the i-th drone, is an action for the i-th drone is an operator that computes the gradient of.​ The update formula of the Actor network parameters of the pursuer and the evader is: ； wherein, is the learning rate for the Actor network; Then return to step S34; Step S352, if the maximum simulation step number of the round is reached, whether the maximum training round number is reached is compared: Step S3521, if the maximum training round number is reached and the reward function converges, the training is ended; if the maximum training round number is reached and the reward function does not converge, return to step S32; Step S3522, if the maximum training round number is not reached, return to step S32.

6. The multi-UAV cooperative hunting method based on obstacle contribution angle according to claim 1, characterized in that, In step S4, the specific steps are as follows: Step S41, design anti-collision reward The specific process is as follows: The role of the anti-collision reward is to guide the pursuer to avoid collision with obstacles, scene edges and other pursuers, and the formula is as follows: ； wherein is a safety threshold radius, is a normal number, is a pursuit drone to an obstacle, D is a minimum distance, is a pursuit drone to a scene edge, is a pursuit drone and a pursuit drone distance between; Step S42, design distance reward as follows: The role of the distance reward is to guide the chasing drone to reduce the distance to the escaping drone. The chasing drone is provided with a distance reward as follows: ； in, It is a positive coefficient. To track the vector pointing from the location of the drone to the location of the escaping drone, To hunt down drones velocity vector To capture the drone at its maximum speed; Step S43, the multi-unmanned aerial vehicle cooperative surrounding reward is designed into four stages, namely search, surround, transfer and capture.

7. The multi-UAV cooperative hunting method based on obstacle contribution angle according to claim 6, characterized in that, In step S43, the multi-unmanned aerial vehicle cooperative surrounding reward is designed, and the specific steps are as follows: Step S431, in the search stage, the evader is located outside the closed envelope formed by the pursuer and the obstacle, and the judgment criteria are as follows: ； wherein, Aeis an area of a largest convex region formed by the escape drone E, the first i pursuit drone, and the Nth i pursuit drone, Aeis an area of a largest convex region formed by the escape drone E, the first pursuit drone, and the Nth pursuit drone with the center O, At this time, the pursuit game is in the search stage, and the reward function of the search stage is as follows: ； wherein, a reward for pursuing the drone for the search phase; Step S432, surrounding stage, the escape unmanned aerial vehicle is located inside the closed envelope formed by the pursuit unmanned aerial vehicle and the obstacle, the unmanned aerial vehicles are numbered from 1 to N in counterclockwise order, when i =N, let i +1=1; when i =1, let i -1=N; according to the relative position of each unmanned aerial vehicle and the obstacle, three cases are divided, whether to maintain in the surrounding stage, and the corresponding reward function is adopted to guide the pursuit unmanned aerial vehicle to start to reduce the surrounding ring; Step S433, the transfer stage is a transition stage from the surround stage to the capture stage, and the pursuer continuously shrinks the surrounding circle while adjusting the distance between them; Step S434, in the capture stage, the pursuer and the evader and the obstacle form an Apollonian circle that intersects or is tangent to each other, and the judgment criteria are as follows: ； The reward function of the capture stage is designed as follows: ； wherein, is the collective occupancy angle, is the reward for capturing the drone in the pursuit phase, is a positive coefficient.

8. The multi-UAV cooperative hunting method based on obstacle contribution angle according to claim 7, characterized in that, In step S432, according to the relative position of each unmanned aerial vehicle and the obstacle, three cases are divided, whether to maintain in the surrounding stage is judged, and the corresponding reward function is adopted to guide the pursuer to start shrinking the surrounding circle; Case 1: For the pursuit drone that meets , i.e. the pursuit drone is not an obstacle on both sides, if it meets: ； This is the encircle phase; where, The maximum speed ratio of the escape drone and the chase drone, The distance between the escape drone and the chase drone The distance between the escape drone and the chase drone The distance between the escape drone and the chase drone The distance between the escape drone and the chase drone The distance between the escape drone and the chase drone; the reward function in the encircle phase is as follows: ； wherein, a reward for surrounding phase pursuit drone, a distance between the j th pursuit drone and the escape drone; Case two: for the pursuit drone that meets the right side of the pursuit drone is an obstacle and the left side is other pursuit drones, if it meets: ； wherein, pursuit drone the center of the Apollonian circle formed by the pursuit drone to the center of the obstacle circle the distance, pursuit drone and the pursuit drone the distance between, pursuit drone the distance of the pursuit drone pursuit drone the radius of the Apollonian circle formed by the pursuit drone E; This condition represents the pursuit of the drone The reward function is as follows: ； Scenario 3: For those that meet the requirements pursuit drones That is, to hunt down drones To the right are other pursuit drones and to the left are obstacles. If the following conditions are met: ； Chase drone Surrounding the right chase drone and defending the gap between the left obstacle, the reward function is as follows: 。 9. The multi-UAV cooperative hunting method based on obstacle contribution angle of claim 7, wherein, In step S433, the Apollonius circle occupation angle and the contribution angle are introduced in the transfer stage, and the specific calculation is as follows: Positioning Angle: Defining the escape drone E and the pursuit drone Apollonius Circle The included angle formed by the lines connecting the tangent points is the angle for tracking drones. The occupancy angle is denoted as The formula is as follows: ； The collective occupation angle is the union of the occupation angles of the pursuit drones participating in the hunting, i.e., the collective occupation angle is denoted as , and the formula is as follows: ； When the collective occupies an angle of 360 degrees, a closed enclosure is formed; Cornering contributor: along any radial direction of the escaping drone E, do a ray, the first owner of the Apollonian circle intersecting the ray or the obstacle defines the cornering contributor in the ray direction, denoted as ; contribution angle: the range of angles that an individual drone's Apollonian circle or obstacle actually makes as a cornering contributor in a cornering is defined as the contribution angle ; to the direction of motion of the escaping drone E k times, and then the k directions are classified and accumulated according to the cornering contributor to obtain the contribution angle of each drone or obstacle : ； in, Indicates the first i Contributors to the encirclement and capture in all directions, For the sampling flag function, when hour, Otherwise, it is 0; ； contribution angle representing the contribution of each pursuit drone and obstacle to the containment of the escape drone in the escape direction, the collective containment angle is calculated using the containment contribution angle of each pursuit drone or obstacle The calculation is performed as follows: ； wherein to chase the drone the contribution angle of the drone, the contribution angle of the escape drone E; Transition phase reward function and collective bearing angle The positive correlation, while chasing the drone constantly narrowing the circle, the transition phase reward function is as follows: ； wherein, a reward for the pursuit of the drone for the transfer phase, is a positive coefficient.

Images