A Multi-UAV Cooperative Encirclement Credit Allocation Method Based on Geometric Counterfactual Baselines
By constructing a geometric counterfactual baseline and a differentiated reward function, the problems of high credit allocation complexity and policy degradation in multi-UAV cooperative encirclement are solved. Individual-level credit allocation with low computational overhead and high training stability is achieved, thereby improving the synergy and efficiency of UAV cooperative encirclement.
Patent Information
- Application Number
- CN202611275115.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
In existing multi-drone cooperative capture methods, credit allocation suffers from high computational complexity, high training costs, and credit allocation failure, making it difficult to accurately distinguish the contribution of individual drones and leading to the degradation of cooperative strategies.
The credit allocation method based on geometric counterfactual baseline constructs the geometric centroid of the hunter drone as the counterfactual baseline, calculates the marginal contribution of a single hunter drone to the dispersion of the encirclement angle, constructs a differentiated reward function, and adopts a reinforcement learning strategy to optimize the collaborative encirclement strategy.
It achieves individual-level differentiated credit allocation with low computational overhead and high training stability, improves the coordination and task execution efficiency of multi-UAV collaborative capture, and alleviates the problems of credit allocation failure and collaborative strategy degradation.
Smart Images

Figure CN123114031A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of autonomous control and intelligent decision-making of unmanned aerial vehicle (UAV) swarms, and specifically relates to a multi-UAV collaborative encirclement and capture credit allocation method based on geometric counterfactual baselines. Background Technology
[0002] Multi-UAV cooperative encirclement refers to multiple UAVs independently performing calculations in perception, decision-making, planning, and control, and then collaboratively approaching, surrounding, and capturing a target UAV. Multi-UAV cooperative encirclement has significant application value in scenarios such as low-altitude security, target interception, key area protection, and unmanned system warfare. The successful completion of an encirclement mission largely depends on whether the multiple UAVs can form a reasonable spatial encirclement configuration and accurately distinguish the actual contributions of different UAVs to the overall encirclement effect during training.
[0003] In existing research on multi-drone cooperative capture methods, reinforcement learning methods have been widely used in drone swarm cooperative decision-making tasks due to their ability to autonomously learn control strategies through agent interaction with the environment. Common methods typically employ a globally shared reward mechanism, where multiple hunter drones receive the same team reward at the same time. These methods are relatively simple to implement and can guide the drone swarm towards a common goal to some extent. However, in tightly coupled cooperative capture tasks, it is difficult to accurately distinguish the true marginal contribution of individual drones, which can easily lead to problems such as credit allocation failure, redundant behavior of individual drones, and degradation of cooperative strategies.
[0004] Regarding existing counterfactual credit allocation methods, some methods estimate the marginal contribution of agents by replacing individual actions and performing multiple samplings, while others learn the counterfactual baseline by constructing an additional value network. Although these methods can improve the credit allocation problem under the shared reward mechanism, they still have the following drawbacks: Action sampling-based methods require a large amount of sampling computation in the joint action space, and the computational complexity increases significantly with the number of agents; methods based on learning-based value networks require additional training of the counterfactual value network, increasing training costs and implementation complexity, and may introduce new training instabilities. Summary of the Invention
[0005] This invention addresses the shortcomings of existing multi-UAV cooperative encirclement methods in terms of credit allocation, computational complexity, and training stability. Based on the spatial geometric relationships in multi-UAV encirclement tasks, it designs a credit allocation method for multi-UAV cooperative encirclement based on a geometric counterfactual baseline. This method relies only on the local positional information of the hunter UAV and the target UAV. By constructing the geometric centroids of the remaining hunter UAV positions as counterfactual baselines, it calculates the marginal contribution of a single hunter UAV to the dispersion of the encirclement angle, thereby achieving individual-level differentiated credit allocation. It aims to solve problems such as the difficulty in distinguishing the contributions of individual UAVs, credit allocation failure, and cooperative strategy degradation under traditional global shared reward mechanisms. This improves the stability, cooperation, and convergence of multi-UAV cooperative encirclement strategy training, effectively enhancing the autonomous cooperative encirclement capability of multi-UAV systems in complex environments. This method requires no action sampling or additional value network training, featuring low computational overhead, simple implementation, and strong training stability, effectively improving the level of autonomous decision-making and cooperative control in multi-UAV cooperative encirclement tasks.
[0006] This invention addresses the credit allocation problem in multi-UAV cooperative encirclement missions by proposing a credit allocation method based on a geometric counterfactual baseline. The implementation process of this method is as follows: Figure 1 As shown, the specific implementation steps are as follows: Step (1): Construct a simulation environment for multi-UAV collaborative encirclement and capture. Constructing within a two-dimensional continuous space A Hunter drone ( ), 1 target drone and A static obstacle ( In a collaborative encirclement environment, a kinematic model of the hunter drone and an escape model of the target drone are established. These models can utilize existing general models. Based on this general model structure, an observation space is constructed, consisting of the hunter drone's own state information, teammate state information, lidar perception information, and the target's relative position information. The hunter's maximum speed is... Target maximum speed Satisfy speed ratio ; Step (2): Calculate the geometric counterfactual marginal angle contribution reward Let the first The position vector of the Hunter drone in the two-dimensional plane is The position vector of the target UAV is Using the target UAV as the polar coordinate reference center, calculate the... The azimuth angle of the hunter drone relative to the target drone For those containing For coordinated capture missions using hunter drones, the ideal uniform angular interval is defined as... .
[0007] To measure the angular dispersion of the Hunter drones around the target, the calculations are performed for any two Hunter drones. and The shortest angular distance between the circles , and The first and The azimuth angle of the hunter drone relative to the target drone is obtained, and further, the first... Minimum angular distance between a Hunter drone and other Hunter drones Based on this, the minimum angular spacing is made dimensionless relative to the ideal uniform angular spacing, resulting in the first... The actual angle dispersion score of the hunter drone .
[0008] To evaluate the first The marginal contribution of the Hunter drone to the overall enclosure configuration is used to construct its geometric counterfactual baseline position. Specifically, the first... The geometric centroid of the position of the hunter drone is replaced by the position of the other hunter drones, that is: (5) Indicates the first The counterfactual baseline position of the Hunter drone, i.e., the geometric centroid of the positions of the other Hunter drones. Indicates except the first Two-dimensional planar position vectors of other Hunter drones besides the Hunter drone.
[0009] The counterfactual position indicates the first The hunter drone does not provide independent spatial collaborative contributions and degenerates into the baseline state when it becomes a group average behavior.
[0010] Substituting the geometric counterfactual baseline position into the aforementioned angle dispersion calculation process, we obtain the counterfactual azimuth angle. Then, the minimum counterfactual angular spacing is calculated using the counterfactual azimuth. The counterfactual angular dispersion score is further obtained from the minimum counterfactual angular spacing. .
[0011] When the counterfactual baseline position coincides with the target UAV position, or the distance between them is less than a preset threshold ε, the minimum counterfactual angular spacing is set to 0 to avoid singularities in the counterfactual azimuth calculation. In one implementation, ε is set to 10. - 6 m.
[0012] Indicates the use of the evaluation of the first The y-axis coordinate of the counterfactual baseline position of the Hunter drone. For the first The x-axis coordinate of the counterfactual baseline position of the Hunter drone. For the first The counterfactual baseline position of the hunter drone relative to the azimuth angle of the target drone.
[0013] Finally, the difference between the actual angle dispersion score and the counterfactual angle dispersion score is used as the first... The marginal angle contribution of the hunter drone is multiplied by the angle reward weight. They will receive a contribution reward from a marginal perspective.
[0014] (6) when When, explain the first The actual position of the hunter drone, compared to the geometric counterfactual baseline position, improves the uniformity of the encirclement angle, resulting in a positive reward; when This indicates that the actual position of the hunter drone has a redundant or weakening effect on the encirclement configuration, resulting in negative feedback.
[0015] Step (3): Construct a differentiated reward function In step (2), the first Marginal angle contribution reward for the Hunter drone Then, it is used as the core differentiated reward term in the comprehensive reward function, and combined with auxiliary terms such as distance reward, proximity reward, capture reward, collision avoidance penalty, boundary penalty, and time penalty to obtain the first... Total reward for using the Hunter drone: (6) in, The marginal angle contribution reward calculated based on the geometric counterfactual baseline in step (2) is the core reward item for this method to achieve individual-level differentiated credit allocation; As a distance reward, Used to guide the hunter drone toward the target drone; Used to award a mission completion reward upon successful capture; , and These are used to constrain teammate spacing, obstacle collision, and boundary behavior, respectively. This is a time-based penalty item.
[0016] Specifically, with Let be the current Euclidean distance between the i-th hunter drone and the target drone. This is the diagonal half-length of the court. Current side length of the court. , Distance rewards are calculated based on... Calculation, where ;when Less than When the time is positive, greater than When the time is negative, it equals The time is 0. Because The normalization factor belongs to Therefore .
[0017] To approach the reward, the change in distance between adjacent time steps is used. Calculation. When hour, ;when hour, The value is 0 when the distance remains constant, where the reward weight is closer. Assuming the hunter's maximum speed is 5.0 m / s, the target's maximum speed is 4.5 m / s, and the simulated step length is 0.1 s, then... ,therefore .
[0018] Contributing rewards from a marginal perspective is the core reward element for achieving individual-level differentiated credit allocation in this method. The position of the i-th hunter is replaced with the centroid of the positions of the other hunters. Calculate the enclosing angle dispersion score for the actual position and the counterfactual position respectively, and then... Calculation, where the marginal contribution reward weight is... Currently under implementation The value is 0 when the three hunters are approximately evenly distributed around the target, and 0 when any two hunters are collinear with the target. 1, therefore A positive value is used when the actual position expands the bounding angle more than the counterfactual baseline, and a negative value is used otherwise. The counterfactual position is only used for reward calculation; the drone will not actually fly to that position.
[0019] The task completion reward is triggered only upon successful target capture; otherwise, it is zero. In the current implementation, when at least... The hunter enters the capture radius A successful capture is determined upon capture; hunters within the capture range receive the full reward. Hunters outside the designated area will receive participation rewards. Its upper limit is Target achieved Punishment. Therefore, for the hunter, .
[0020] For teammate spacing constraints, when teammate distance Less than the safety distance At that time, according to Apply a penalty, where the teammate spacing constraint has a weight. Otherwise, it is 0. Currently, with three hunters by default, each hunter is affected by at most two teammates, with a single teammate's penalty range being... Therefore .
[0021] This is a collision penalty. It takes a fixed negative value when the collision detection flag is true (in the current code, this includes entering an obstacle or going out of bounds), and 0 when there is no collision. .
[0022] Penalty for boundaries: when And it is not yet close to the target ( When approaching the target (allowing to stick to the wall), press... Boundary penalty weight , Calculate; otherwise, return 0.
[0023] A fixed time penalty is applied for each simulation step. The current simulation code also includes milestone shaping rewards: when... At that time, each hunter triggers each milestone once per round, with the reward calculated as follows: The calculations correspond to approximately 23.57, 35.36, and 70.71. This item represents the distance-dependent shaping reward in engineering implementation and does not alter the core role of the aforementioned geometric counterfactual marginal angle reward.
[0024] To further maintain differentiated reward distribution during the capture phase, a set of hunter drones entering the capture radius can be defined. , For the capture radius, when the following conditions are met hour, A successful capture is determined by setting a capture threshold. For each element entering the set... The hunter drones are given a higher capture reward; for hunter drones that do not enter the group but participate in the encirclement, the reward is reduced based on the distance between them and the target drone. This design is used to complement the marginal angle contribution reward in step (2), so that the reward feedback during the training process reflects both the spatial encirclement contribution and the degree of participation in the capture phase.
[0025] Where di is the Euclidean distance between the i-th hunter drone and the target drone, dcapture is a positive number, and ncapture is an integer satisfying 2 ≤ ncapture ≤ N. In one implementation, dcapture is 3.0 m and ncapture is 2; when the capture is successful, the hunter drone that enters the capture range receives the full capture reward, and the hunter drone that does not enter the capture range receives a reward that decays exponentially with the distance to the target.
[0026] Step (4): Training a cooperative encirclement strategy Based on the differentiated reward function constructed in step (3), a reinforcement learning strategy optimization algorithm is used to train a multi-UAV cooperative capture strategy network. Let the strategy network of the hunter UAV be... ,in For the first Observational information from the Hunter drone, Output actions for it. These are the parameters of the policy network. During training, the environment calculates the first reward function based on the reward function in step (3). Rewards for flying the Hunter drone Based on this, the strategy network parameters are updated, and finally a multi-UAV collaborative encirclement strategy model is obtained.
[0027] To avoid insufficient exploration in the early stages of training and excessive randomness in the later stages, this method can implement a policy entropy coefficient annealing mechanism. During training, this mechanism maintains a high entropy coefficient in the early stages to encourage exploration, and decreases the entropy coefficient in the later stages to promote policy convergence. This mechanism is an auxiliary training setting and does not alter the differentiated credit allocation mechanism centered on contribution rewards from a geometric counterfactual marginal perspective.
[0028] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a multi-UAV cooperative encirclement credit allocation method based on a geometric counterfactual baseline. This method constructs a counterfactual baseline using the centroids of the remaining hunter UAVs, based on the spatial geometric relationships in a multi-UAV cooperative encirclement task. It calculates the marginal angular contribution of a single hunter UAV by the difference between the actual angular dispersion score and the counterfactual angular dispersion score. Its advantages are mainly reflected in two aspects: First, this method does not require counterfactual action sampling in the joint action space, nor does it require additional construction and training of a counterfactual value network, resulting in low computational overhead, simple implementation, and strong training stability. Second, this method can allocate rewards differently based on the actual contribution of a single hunter UAV to the encirclement configuration, alleviating problems such as difficulty in distinguishing individual contributions, credit allocation failure, and cooperative strategy degradation under the traditional global shared reward mechanism. This effectively improves the synergy, robustness, and task execution efficiency of the multi-UAV cooperative encirclement strategy. Attached Figure Description
[0029] Figure 1 This is a flowchart of the multi-UAV collaborative capture credit allocation method based on geometric counterfactual baselines of the present invention.
[0030] Figure 2 A schematic diagram illustrating the calculation principle of contribution reward for geometric counterfactual marginal angles.
[0031] Figure 3 This is the dynamic evolution process of multi-UAV collaborative encirclement and capture under the counterfactual reward mechanism of this invention. Detailed Implementation
[0032] See Figures 1 to 3 The method proposed in this invention will be further illustrated below through an example of multi-drone collaborative encirclement and capture of a target drone. The specific steps of this method are as follows: Step (1): Construct a simulation environment for multi-UAV collaborative encirclement and capture. This embodiment sets up three hunter drones, one target drone, and two static obstacles within a two-dimensional continuous space with a side length of 50.0 m. The simulation step size is 0.1 s. The maximum speeds of the hunter and target drones are 5.0 m / s and 4.5 m / s, respectively, with upper limits for linear acceleration of 2.0 m / s² and 1.8 m / s², respectively. The control actions of the hunter drones are linear acceleration and heading angle change rate, with an upper limit of π rad / s for the heading angle change rate. The target drone employs an artificial potential field escape strategy. The basic observations for each hunter drone are 28-dimensional, including 5-dimensional self-state, 4-dimensional relative positions of the two teammates, 16-dimensional information from 16-beam lidar sensing, and 3-dimensional relative distance and direction to the target. Historical trajectories and predicted trajectories can be used as observation extensions but are not necessary inputs for the geometric counterfactual credit allocation mechanism. The overall process of this method is as follows: Figure 1 As shown.
[0033] Step (2): Calculate the geometric counterfactual marginal angle contribution reward For any hunter drone, taking the current position of the target drone as the reference center, calculate the azimuth angle of the hunter drone relative to the target drone, and calculate the shortest circumferential angular distance between it and other hunter drones to obtain the actual angular dispersion score. For example... Figure 2 As shown, the target drone is denoted as The locations of the Hunter drones are denoted as follows: , , To evaluate the independent contribution of a particular hunter drone to the enclosure configuration, the position of that hunter drone is replaced with the geometric centroid of the positions of the other hunter drones. This constructs a geometric counterfactual baseline. This counterfactual baseline represents the state where the hunter drone degenerates into the group's average behavior when it does not provide an independent spatial contribution. The actual angular dispersion score and the counterfactual angular dispersion score are further calculated, and the difference between them is taken as the hunter drone's marginal angular contribution. When the actual angular dispersion score is higher than the counterfactual angular dispersion score, it indicates that the hunter drone's actual position improves the enclosure configuration and should be positively rewarded; conversely, it indicates that the position has a redundant or weakening effect on the enclosure configuration.
[0034] Step (3): Construct a differentiated reward function The marginal angle contribution reward obtained in step (2) is used as the core differentiated reward item in the comprehensive reward function, and combined with auxiliary reward items such as distance approach, capture completion, teammate collision avoidance, obstacle avoidance, boundary constraints, and time penalty to form a reward signal for strategy training. The reward weights used in this embodiment are consistent with the specific data example mentioned above. Among them, the angle reward is used to evaluate the azimuth dispersion of the hunter drone around the target, and the distance reward, approach reward, and capture reward are used to supplement the guidance of the radial approach process. The distance-related shaping reward also includes a one-time milestone reward triggered when the distance between the hunter drone and the target is first less than 15 m, 10 m, and 5 m. In this embodiment, the capture radius is 3.0 m, and the capture is considered successful when at least 2 hunter drones enter the capture range; the remaining hunter drones receive exponentially decaying rewards according to their distance from the target drone.
[0035] Step (4): Training a cooperative encirclement strategy Based on the differentiated reward function constructed in step (3), this embodiment uses the near-end policy optimization algorithm (PPO) with parameter sharing to train the multi-UAV cooperative encirclement policy network. Each hunter UAV shares the same policy network and updates its policy based on its own observation information and individual rewards, without setting a centralized value network. The initial value of the policy entropy coefficient is 0.008, which decreases linearly after the 50th training iteration, with a lower limit of 0.0005. PPO and entropy coefficient annealing are both training implementation parameters and do not change the geometric counterfactual credit allocation mechanism.
[0036] Post-training testing and validation: After training, with fixed policy network parameters, a multi-drone cooperative encirclement strategy is run in a test environment. For example... Figure 3 As shown, the hunter drones gradually form a spatial encirclement configuration around the target drone and complete the cooperative encirclement and capture of the target drone in a static obstacle environment. This result demonstrates that marginal angle contribution rewards based on a geometric counterfactual baseline can guide multiple hunter drones to form a dispersed and stable encirclement behavior, thereby improving the effectiveness and stability of the cooperative encirclement strategy.
[0037] The above-described embodiments are merely some implementation methods of this application. For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of this application, and these all fall within the protection scope of this application.
Claims
1. A multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baseline, characterized in that: Includes the following steps: Step (1): Construct a simulation environment for multi-UAV collaborative encirclement and capture. A collaborative hunting environment containing hunter drones, target drones, and static obstacles is constructed in a two-dimensional continuous space. A kinematic model of the hunter drone, an escape model of the target drone, and an observation space are established. Step (2): Calculate the geometric counterfactual marginal angle contribution reward Calculate the actual angular dispersion score for each hunter drone, construct the geometric counterfactual baseline position for each hunter drone, and obtain the counterfactual angular dispersion score; The difference between the actual angle dispersion score and the counterfactual angle dispersion score is taken as the marginal angle contribution of each hunter drone, and multiplied by the angle reward weight. They will receive a contribution reward from a marginal perspective; Step (3): Construct a differentiated reward function After obtaining the marginal angle contribution reward for each hunter drone in step (2), it is used as the core differentiated reward item in the comprehensive reward function and combined with other auxiliary items to obtain the total reward for each hunter drone. Step (4) Training a cooperative encirclement strategy Based on the differentiated reward function constructed in step (3), the multi-UAV cooperative capture strategy network is trained using the near-end strategy optimization algorithm. During the training process, the environment calculates the reward of each hunter UAV according to the reward function in step (3) and updates the strategy network parameters accordingly, finally obtaining the multi-UAV cooperative capture strategy model.
2. The multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baseline as described in claim 1, characterized in that, In step (1), the observation space includes the hunter drone's own status information, teammate status information, lidar perception information, and target relative position information, as well as the hunter drone's maximum speed. The maximum speed of the target drone Satisfy speed ratio .
3. The multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baseline as described in claim 2, characterized in that, In step (2), the actual angle dispersion score of each hunter drone is calculated as follows: calculate the azimuth angle of the hunter drone relative to the target drone, and the ideal uniform angle interval; calculate the shortest circumferential angular distance between any two hunter drones, obtain the minimum angular distance between each hunter drone and other hunter drones, normalize the minimum angular distance relative to the ideal uniform angle interval, and obtain the actual angle dispersion score of each hunter drone.
4. The multi-UAV cooperative capture credit allocation method based on geometric counterfactual baselines according to claim 3, characterized in that, In step (2), the counterfactual angle dispersion score of each hunter drone is as follows: the geometric counterfactual baseline position is the geometric centroid of the position of each hunter drone replaced by the position of the other hunter drones. The counterfactual azimuth is obtained based on the geometric counterfactual baseline position. The minimum counterfactual angle spacing is calculated through the counterfactual azimuth, and the counterfactual angle dispersion score is further obtained from the minimum counterfactual angle spacing.
5. A multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baselines according to claim 4, characterized in that, Marginal contribution reward When, explain the first The actual position of the hunter drone, compared to the geometric counterfactual baseline position, improves the uniformity of the encirclement angle, resulting in a positive reward; when At that time, you may receive either positive or negative feedback.
6. The multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baseline as described in claim 5, characterized in that, In step (3), the auxiliary items of the total reward include distance reward, proximity reward, capture reward, collision avoidance penalty, boundary penalty and time penalty.
7. A multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baselines according to claim 6, characterized in that, In step (3), the set of hunter drones entering the capture radius is defined. When satisfied When the encirclement is deemed successful, those who enter the assembly point... Hunter drones that capture targets receive higher rewards; hunter drones that participate in the hunt but do not join the group receive diminished rewards based on their distance from the target drone.
8. A multi-UAV cooperative encirclement credit allocation method based on geometric counterfactual baselines according to claim 7, characterized in that, In step (4), a strategy entropy coefficient annealing mechanism is adopted during the training of the multi-UAV collaborative encirclement strategy network. In the early stage of training, a high entropy coefficient is maintained to encourage exploration, and in the middle and late stages of training, the entropy coefficient is reduced to promote strategy convergence.
