An unmanned aerial vehicle cluster cooperative target tracking algorithm based on curriculum learning and MAPPO algorithm

Abstract

This invention discloses a collaborative target tracking algorithm for UAV swarms based on curriculum learning and the MAPPO algorithm. The core of this invention addresses the problems of uneven task allocation and low training efficiency in multi-UAV collaborative tracking by constructing a three-stage curriculum learning framework. This framework guides agents from basic navigation to complex collaboration through dynamic adjustment of training weights. A reward function based on global task allocation is designed, which uses the Hungarian algorithm to perform globally optimal task allocation on a score matrix that integrates distance, heading, and tracking stickiness, and decouples the allocation result into individual reward signals, thereby reducing resource waste. The algorithm employs a multi-agent proximal strategy optimization algorithm with centralized training and distributed execution, and introduces an adaptive performance protection mechanism. When performance degradation is detected, the curriculum stage is dynamically rolled back to ensure training robustness. This invention effectively improves multi-target coverage and collaborative efficiency, and is suitable for reconnaissance, search and rescue, and other scenarios.

Description

Technical Field

[0001] This invention relates to the field of collaborative target tracking in unmanned aerial vehicle (UAV) swarms, specifically to a multi-agent collaborative target tracking algorithm based on course learning and global allocation. This method is particularly suitable for efficient, stable, and adaptive collaborative tracking of multiple moving targets by multiple UAV systems in dynamic, uncertain, and complex environments. Background Technology

[0002] With the rapid development of UAV technology and the continuous improvement of its autonomy, multi-UAV swarm systems are increasingly being used in military reconnaissance, border patrol, disaster area search and rescue, and security for large-scale events. Among these applications, collaborative tracking of moving or stationary targets is one of the core mission scenarios. This requires the UAV swarm to autonomously form a collaborative observation network in complex and dynamic environments to achieve efficient, stable, and continuous tracking of targets.

[0003] To address this challenge, research focus in academia and industry has shifted from traditional centralized control to distributed collaborative decision-making. In this context, Multi-Agent Deep Reinforcement Learning (MADRL) has become a mainstream technological approach for solving multi-UAV cooperative tracking problems due to its ability to autonomously learn cooperative strategies through interactions between agents and the environment. In particular, the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm, through centralized training and distributed execution, has demonstrated great potential for handling complex cooperative tasks. These methods aim to guide each UAV to learn local control strategies by designing reasonable reward functions, ultimately leading to the emergence of global cooperative behavior.

[0004] Despite the promising future of the MADRL framework, existing MADRL-based collaborative tracing methods still face some technical bottlenecks in practical applications.

[0005] In multi-target tracking scenarios, the balance of dynamic task allocation is severely lacking. This is because most existing methods indirectly incentivize drones to choose different targets by relying on global rewards and sharing observation information, lacking a learnable mechanism to directly optimize the crucial decision of "which drone should be responsible for tracking which target at what time." If the MADAL algorithm is used alone, uneven task allocation often occurs, resulting in multiple drones vying to track the same high-value target while other targets are overlooked. This is because during training, the sum of rewards from multiple drones tracking a single target numerically offsets the penalty for conflict, causing drones to fail to learn and fall into local optima. Moreover, in practice, unreasonable allocation can lead to some drones being overloaded and experiencing a sharp decrease in endurance, severely restricting the overall tracking performance and robustness of the system.

[0006] When transferring models using the MADRL algorithm, it is often necessary to reconfigure training parameters and the weights of various reward components, which limits the algorithm's practicality. Traditional MADRL training methods typically expose the agent to the final, complex task environment from the very beginning of training. Since reinforcement learning mimics human learning processes, this one-step training approach is akin to asking a novice to solve a complex problem directly, easily leading to low policy exploration efficiency, volatile learning curves, slow convergence, or even failure to converge. More importantly, policies trained in such high-intensity, high-complexity environments often exhibit poor generalization ability. When faced with new target movement patterns, unknown environmental interference, or changes in task parameters (such as the ratio of drones to targets) that were not encountered during training, the learned policy is prone to failure, demonstrating poor adaptability and flexibility.

[0007] Therefore, current multi-UAV cooperative tracking systems have not yet fully realized their theoretical potential. The industry urgently needs an innovative training framework that can systematically guide agents to master cooperative tracking skills progressively and gradually; simultaneously, an endogenous task allocation mechanism is also needed to optimize cluster resource scheduling. This invention innovatively and systematically combines course learning and traditional task allocation algorithms with multi-UAV target cooperative tracking, maximizing the utilization of UAV resources while enhancing the generalization ability of the MADRL model, thus filling this technological gap. Summary of the Invention

[0008] The purpose of this invention is to provide a collaborative target tracking algorithm for UAV swarms based on course learning and the MAPPO algorithm, to solve the problems of target allocation optimization, trajectory planning, and conflict avoidance in collaborative tracking of multiple targets by multiple UAVs in dynamic environments, and to improve the adaptive collaboration and persistent tracking capabilities of the swarm system. The technical solution proposed in this invention includes the following:

[0009] Step 1: Dynamic scheduling mechanism for course learning weights: Design a three-stage course learning framework, and dynamically adjust the weights of each component in the reward function through a weight scheduling algorithm to achieve progressive learning from simple navigation to complex collaboration.

[0010] Step 2: Design of a reward function based on global task allocation: Introduce a sticky reward mechanism to encourage maintaining the existing tracking relationship. Construct a scoring matrix with three dimensions: distance, heading, and stickiness. Employ a global bipartite graph matching algorithm with one-to-one constraints. Build a multi-layered reward function, decomposing the tracking task into different dimensions, and calculate individual rewards based on the globally optimal allocation result.

[0011] Step 3: MADRL Collaborative Training Framework: Integrates course learning, global allocation, and the MAPPO algorithm to form a complete training process.

[0012] Step 4: Adaptive performance protection mechanism: Real-time monitoring of cluster performance indicators and dynamic adjustment of course progress based on training results. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of a drone's motion model;

[0014] Figure 2 This is a schematic diagram of the global task allocation and reward function design;

[0015] Figure 3 This is the overall architecture diagram of a multi-agent collaborative target tracking algorithm based on course learning and global allocation. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0017] The specific embodiments of the present invention include the following aspects.

[0018] Step 1: First, construct a three-stage course learning framework, with the following settings:

[0019] Phase 1 (Basic Navigation Phase): Training steps from 0 to 500,000, focusing on learning basic navigation skills.

[0020] Phase 2 (Collaborative Tracking Phase): Training steps of 500,000-1,500,000 steps, introducing collaborative constraints.

[0021] Phase 3 (Adaptation to Complex Environments): More than 1,500,000 training steps to enhance sustained tracking ability.

[0022] Then, the inverse Sigmoid weight transition function is set:

[0023]

[0024] Among them, W stage (t) represents the stage weights at training step t, k = 0.8 is the kurtosis parameter, controlling the speed of weight transition, and m is the midpoint parameter, set at 70% progress as the transition midpoint. The stage progress function... T stage_start T is the number of training steps at the start of the current phase. stage_end This is the number of training steps at the end of the current phase.

[0025] Furthermore, a weight cap protection mechanism is designed, primarily to prevent performance crashes caused by excessive growth of penalty weights in the later stages of training. This mechanism uses mathematical constraints to ensure that the weights of each component in the reward function remain within a reasonable range, maintaining a balance between exploration and exploitation.

[0026] A maximum penalty weight of 0.25 is set for repeated tracking. Exceeding this limit will cause drones to excessively avoid conflict or even abandon effective tracking, leading to a significant decrease in coverage. This threshold ensures that the penalty intensity does not completely suppress exploration behavior. Continuous tracking is the core objective of the tracking task and must maintain sufficient incentives. A minimum penalty weight of 0.75 is set for tracking step reward to prevent the increase of other weights during the learning transition from course to course, diluting the tracking incentives and ensuring that tracking behavior is always adequately rewarded. A minimum penalty weight of 0.7 for distance reward ensures that drones maintain basic target approach capabilities, laying the foundation for effective collaboration.

[0027] Step 2: Create a rating matrix S of size N×M, where each element consists of three parts: distance rating, heading rating, and stickiness rating.

[0028] S ij =S dist +S heading +S stickiness (2)

[0029] The distance scoring formula is as follows:

[0030]

[0031] Where, d ij d is the distance between drone i and target j. perception It is the sensing radius of the drone.

[0032] The heading score formula is as follows:

[0033]

[0034] Where, Δθ ij It is the heading deviation between UAV i and target j.

[0035] The viscosity scoring formula is as follows:

[0036] S stickiness =min(max(5.0+α·C) track ,3.0),40.0) (5)

[0037] Where α is the weighting parameter, C track This is the number of steps that the current target j has been continuously tracked by drone i. A reward cap of 40.0 is set to prevent over-incentivization that could prevent target switching.

[0038] After calculating the scoring matrix, the Hungarian algorithm is used to calculate the maximum matching in the bipartite graph, and the matching results are stored. If the current drone is not assigned to the nearest target, a fixed distance penalty is imposed on the drone, and it is not included in the hierarchical reward calculation, which encourages the drone to find a new target and reduces conflict.

[0039] The reward function design is mainly divided into four modules: basic navigation, continuous tracking, collaborative constraints, and security constraints.

[0040] The basic navigation module is further divided into distance reward and heading reward.

[0041] The distance reward uses a linear decay function, meaning the closer the drone is to the target, the higher the reward, thus encouraging the drone to approach the target. The formula is as follows:

[0042]

[0043] Where, d min It is the distance between the drone and the nearest target.

[0044] To prevent target loss due to drones approaching with completely opposite headings, drones are encouraged to keep their headings as close to the target's as possible. A heading reward formula is as follows:

[0045]

[0046] in, (x uav y uav (x) represents the coordinates of the drone. nearest_target y nearest_target θ is the coordinate of the nearest target. current This is the drone's current heading. It is a linear approximation of cosΔθ.

[0047] The continuous tracking module includes motion smoothing rewards and tracking step rewards.

[0048] When a drone makes a sharp turn, the maneuverability requirements are high, and consequently, stability decreases. Therefore, during target tracking, the drone should be encouraged to make smooth movements as much as possible to avoid attitude instability. Thus, a smoothing reward is designed as follows:

[0049]

[0050] Where, ω max ω is the maximum angular velocity of the drone, and |Δω| is the absolute value of the change in the drone's angular velocity.

[0051] Actual target tracking is a continuous process. Therefore, introducing a tracking step length metric encourages continuous target tracking rather than occasional coverage. This reduces the frequent switching of targets by the drone and ensures tracking stability. The formula is as follows:

[0052]

[0053] Among them, t track To track step length, more than 45 steps are considered a successful tracking.

[0054] To maximize resource utilization and minimize redundant tracking (i.e., multiple drones tracking a single target), the redundant tracking penalty in the cooperation constraint module is set as follows:

[0055] R conflict = -0.3×n repeat (10)

[0056] Among them, the nearest target to the current drone is being measured by n. repeat A drone is being tracked.

[0057] The safety constraint module is divided into boundary penalties and conflict penalties.

[0058] To avoid drone collisions, the drone collision penalty formula is designed as follows:

[0059]

[0060] Where D is the distance between the current drone and the nearest drone, d safe It refers to the safe distance between drones.

[0061] The drone should search for targets within the designated mission area; therefore, a linear penalty is applied when the drone's distance to the boundary is less than the sensing distance. The boundary penalty formula is as follows:

[0062]

[0063] Where, d margin It is the shortest distance between the drone and the boundary.

[0064] In summary, the total reward formula is as follows:

[0065]

[0066] in, Each part of the reward has its own weight.

[0067] To prevent drones from getting stuck in local optima, i.e., a "lazy" state of spinning in place, the system first checks whether the nearest target is within the drone's perception range before entering the tiered reward mechanism. If not, the drone is encouraged to explore the entire system and reduce local turns, as shown in the following formula:

[0068]

[0069] Among them, h max This is the drone's maximum heading angular velocity. Conversely, if the drone's nearest target is within its perception range, then a decentralized, hierarchical total reward is calculated.

[0070] Step 3: Construct a multi-agent reinforcement learning collaborative training framework and optimize the target definition at the system level:

[0071]

[0072] Where, π * This represents the optimal collaborative strategy to be learned. A discount factor γ is used to balance immediate rewards and long-term gains. T is the maximum number of steps in a single training epoch. t a t These are the state space and action space of all drones at any given time, R total This is the total reward function after weight adjustments based on course learning. The MAPPO algorithm is used for centralized training and distributed execution. A centralized critic network evaluates the global state value to guide the updates of individual UAV policy networks, while ensuring that each UAV relies only on local observations during execution.

[0073] Step 4: Establish a real-time performance monitoring system and calculate the coverage change using the following formula:

[0074]

[0075] Where, Δ coverage It is a change in coverage, used to determine whether training performance has degraded. It is the average coverage over the last 50 rounds, reflecting the current training status. It is the average coverage over the first 50 rounds, used as a performance benchmark.

[0076] After each training epoch, update the coverage history. Maintain a sliding window of length 100 to store the coverage data for the most recent 100 epochs. Calculate the difference between the average of the most recent 50 epochs and the average of the previous 50 epochs, and set a performance degradation threshold. When Δ... coverage An adjustment mechanism is triggered when the threshold is exceeded. When performance degradation is detected, a weight rollback strategy is automatically executed, returning a certain proportion of the reward weights for each part of the current course learning stage to the weights of the previous course learning stage.

[0077] The contents not described in detail in this application are existing technologies known to those skilled in the art.

Claims

1. A collaborative target tracking algorithm for UAV swarms based on curriculum learning and the MAPPO algorithm, characterized in that, include: Step 1: First, construct a three-stage course learning framework, with the following settings: Phase 1 (Basic Navigation Phase): Training steps from 0 to 500,000, focusing on learning basic navigation skills; Phase Two (Collaborative Tracking Phase): Training steps of 500,000-1,500,000 steps, introducing collaborative constraints; Phase 3 (Adaptation to Complex Environments): More than 1,500,000 training steps to enhance sustained tracking ability; Then, the inverse Sigmoid weight transition function is set: Among them, W stage (t) represents the stage weights at training step t, k = 0.8 is the kurtosis parameter, controlling the speed of weight transition, and m is the midpoint parameter, set at 70% progress as the transition midpoint. The stage progress function... T stage_starl T is the number of training steps at the start of the current phase. stage_end It is the number of training steps at the end of the current phase; Furthermore, a weight cap protection mechanism is designed, the main purpose of which is to prevent performance collapse caused by excessive growth of penalty weights in the later stages of training; this mechanism ensures through mathematical constraints that the weights of each component in the reward function are always kept within a reasonable range, maintaining a balance between exploration and exploitation. The penalty weight for repeated tracking is capped at 0.

25. Exceeding this cap will cause drones to excessively avoid conflict or even abandon effective tracking, resulting in a significant decrease in coverage. This threshold ensures that the penalty intensity will not completely suppress exploration behavior. Continuous tracking is the core objective of the tracking task and must maintain sufficient incentives. The reward weight for tracking steps is capped at 0.75 to prevent the increase of other weights during the learning transition from course to course and to ensure that tracking behavior is always adequately rewarded. The distance reward weight is capped at 0.7 to protect and ensure that drones maintain basic target approach capabilities, laying the foundation for effective collaboration. Step 2: Create a rating matrix S of size N×M, where each element consists of three parts: distance rating, heading rating, and stickiness rating; S ij =S dist +S heading +S stickiness (2) The distance scoring formula is as follows: Where, d ij d is the distance between drone i and target j. perception It is the sensing radius of the drone; The heading score formula is as follows: Where, Δθ ij It is the heading deviation between UAV i and target j; The viscosity scoring formula is as follows: S stickiness =min(max(5.0+α·C track ,3.0),40.0) (5) Where α is the weighting parameter, C track This is the number of steps that the current target j has been continuously tracked by drone i; set 40.0 as the reward cap to avoid over-incentivizing the target and making it impossible to switch targets. After calculating the scoring matrix, the maximum matching of the bipartite graph is calculated using the Hungarian algorithm, and the matching results are stored. If the current drone is not assigned to the nearest target, a fixed distance penalty is imposed on the drone, and it is not included in the hierarchical reward calculation, which encourages the drone to find a new target and reduces conflicts. The reward function design is mainly divided into four modules: basic navigation, continuous tracking, collaborative constraints, and security constraints. The basic navigation module is further divided into distance reward and heading reward; The distance reward uses a linear decay function, meaning the closer the drone is to the target, the higher the reward, thus encouraging the drone to approach the target. The formula is as follows: Where, d min It is the distance between the drone and the nearest target; To prevent target loss due to drones approaching with completely opposite headings, drones are encouraged to keep their headings as close to the target's as possible. A heading reward formula is as follows: in, (x uav y uav (x) represents the coordinates of the drone. nearest_target y nearest_target θ is the coordinate of the nearest target. current This is the drone's current heading. It is a linear approximation of cosΔθ; The continuous tracking module includes motion smoothing rewards and tracking step rewards; When a drone makes a sharp turn, the maneuverability requirements are high, and consequently, stability decreases. Therefore, during target tracking, the drone should be encouraged to make smooth movements as much as possible to avoid attitude instability. Thus, the smoothness reward is designed as follows: Where, ω max ω is the maximum angular velocity of the drone, and |Δω| is the absolute value of the change in the drone's angular velocity; Actual target tracking is a continuous process. Therefore, introducing a tracking step length metric encourages continuous target tracking rather than occasional coverage. This reduces the frequent switching of targets by the drone and ensures tracking stability to some extent. The formula is as follows: Among them, t track To track step length, more than 45 steps are considered successful tracking; To maximize resource utilization and minimize redundant tracking (i.e., multiple drones tracking a single target), the redundant tracking penalty in the cooperation constraint module is set as follows: R conflict =-0.3×n repeat (10) Among them, the nearest target to the current drone is being measured by n. repeat One drone tracking; The security constraint module is divided into boundary penalties and conflict penalties; To avoid drone collisions, the drone collision penalty formula is designed as follows: Where D is the distance between the current drone and the nearest drone, d safe It refers to the safe distance between drones; The drone should search for targets within the designated mission area; therefore, a linear penalty is applied when the drone's distance to the boundary is less than the sensing distance. The boundary penalty formula is as follows: Where, d margin It is the shortest distance between the drone and the boundary; In summary, the total reward formula is as follows: in, Each part of the reward has its own weight. To prevent drones from getting stuck in local optima, i.e., a "lazy" state of spinning in place, the system first checks whether the nearest target is within the drone's perception range before entering the tiered reward mechanism. If not, the drone is encouraged to explore the entire system and reduce local turns, as shown in the following formula: Among them, h max It is the drone's maximum heading angular velocity; conversely, if the drone's nearest target is within its perception range, then a decentralized, hierarchical total reward is calculated. Step 3: Construct a multi-agent reinforcement learning collaborative training framework and optimize the target definition at the system level: Where, π * The optimal collaborative strategy to be learned; the discount factor γ, used to balance immediate rewards and long-term benefits; T is the maximum number of steps in a single training round; s t a t These are the state space and action space of all drones at any given time, R total It is the total reward function after weight adjustment based on course learning; it uses the MAPPO algorithm for centralized training and distributed execution; it utilizes a centralized critic network to evaluate the global state value, guide the update of each UAV policy network, and ensure that each UAV relies only on local observations during execution; Step 4: Establish a real-time performance monitoring system and calculate the coverage change using the following formula: Where, Δ coverage It is a change in coverage, used to determine whether training performance has degraded. It is the average coverage over the last 50 rounds, reflecting the current training status. It is the average coverage over the first 50 rounds, used as a performance benchmark; After each training epoch, update the coverage history; maintain a sliding window of length 100 to store the coverage data for the most recent 100 epochs, calculate the difference between the average of the most recent 50 epochs and the average of the previous 50 epochs, and set a performance degradation threshold. When Δ... coverage When the value exceeds the threshold, an adjustment mechanism is triggered; when performance degradation is detected, a weight rollback strategy is automatically executed, returning a certain proportion of the reward weights for each part of the current course learning stage to the weights of the previous course learning stage.

Images