Multi-uav multi-target cooperative task allocation and trajectory tracking method

Abstract

The application discloses a multi-UAV multi-target cooperative task allocation and trajectory tracking method, comprising the following steps: acquiring interaction information of N unmanned aerial vehicles (UAVs) with an environment in a tracking process of M dynamic targets; constructing a HAPPO-AGRN model; training the HAPPO-AGRN model based on the interaction information by using a centralized training and decentralized execution paradigm; and loading the trained HAPPO-AGRN model into the N UAVs, so that the N UAVs perform cooperative task allocation and trajectory tracking on the M dynamic targets. The application realizes intelligent and self-adaptive task allocation and real-time trajectory tracking of a multi-UAV system, and effectively improves the overall cooperative efficiency, tracking performance and task execution reliability of the system.

Description

Technical Field

[0001] This invention relates to the field of collaborative control and intelligent decision-making technology for unmanned aerial vehicles (UAVs), and in particular to a method for multi-UAV multi-target collaborative task allocation and trajectory tracking. Background Technology

[0002] With the rapid development of UAV technology, multi-UAV cooperative systems have been widely used in target monitoring, emergency rescue, environmental perception, trajectory tracking, and communication support. In dynamic multi-target environments, multi-UAV systems typically need to simultaneously perform complex collaborative operations such as target allocation, task scheduling, and trajectory tracking. Traditional multi-UAV task allocation methods mostly rely on pre-built environmental models or rule-based fixed scheduling strategies. These methods are prone to low allocation efficiency, high computational overhead, and difficulty in adaptability when the task scale is large or the target position changes rapidly. On the other hand, multiple UAVs need to continuously adjust their flight trajectories according to the target's motion characteristics to maintain stable tracking. However, existing trajectory planning algorithms generally lack the ability to model the target's dynamics and the interaction between multiple UAVs. When the number of targets is much greater than the number of UAVs or the environment changes frequently, it can easily lead to trajectory deviation, unstable tracking, or uneven resource allocation. Furthermore, in multi-UAV systems, uneven task allocation can cause some UAVs to be overloaded while others are idle, thereby reducing overall collaborative efficiency. In recent years, multi-agent reinforcement learning methods have been increasingly applied to UAV collaborative decision-making scenarios. While such methods offer advantages such as not requiring precise modeling and being able to learn strategies through interaction, traditional multi-agent algorithms still face challenges in complex environments, including training instability, slow policy convergence, insufficient utilization of historical information, and difficulty in revealing the relationships between targets. This makes it difficult to obtain efficient cooperative control strategies. How to achieve efficient, stable, and fair multi-UAV task allocation and trajectory tracking in dynamic multi-target environments remains a crucial problem that urgently needs to be solved. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a multi-UAV multi-target cooperative task allocation and trajectory tracking method, which can more effectively address issues such as uneven task allocation, unstable trajectory tracking, and difficulty in strategy convergence among multiple UAVs in a multi-target dynamic environment.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows:

[0005] A method for multi-UAV multi-target cooperative task allocation and trajectory tracking includes the following steps:

[0006] Acquire information about the interaction between N UAVs and the environment during the tracking of M dynamic targets;

[0007] A HAPPO–AGRN model is constructed and trained using a centralized training and decentralized execution paradigm based on interactive information.

[0008] The trained HAPPO–AGRN model is loaded into N drones, enabling the N drones to perform collaborative task allocation and trajectory tracking on M dynamic targets.

[0009] Preferably, the actions of the UAV include: target selection vector, task allocation action, UAV flight control action, and trajectory update vector, and the state of the UAV includes: the position of the allocated target, local observation of multiple surrounding targets, and some task allocation information of other UAVs.

[0010] During the training of the HAPPO–AGRN model, the state sequences and action sequences of all UAVs are input into the Critic network with an RNN-GAN structure to capture temporal dynamic dependencies and predict Q values. The output Q values ​​are then fed back to the Actor network based on a self-attention mechanism to focus on key target features for policy updates.

[0011] Preferably, the loss function of the Critic network is as follows:

[0012]

[0013] in, , These are the weighting coefficients. For counter-loss, This represents the Q-value of the Critic network's prediction for the current state-action pair. This is the target value.

[0014] Preferably, the strategy update is performed using the following formula:

[0015]

[0016] Where πᵢ* represents the optimal strategy of the i-th UAV, and πᵢ represents the strategy to be optimized for the i-th UAV; arg max₍πᵢ₎ represents selecting the strategy that maximizes the expected cumulative reward from all candidate strategies; s0 and a0 represent the initial global state and initial joint action, respectively; E₍s0,a0,…₎[·] represents the expected calculation of environmental state transition, strategy action sampling, and initial state distribution; t represents the discrete decision time; sᵢᵗ and aᵢᵗ represent the local state and executed action of the i-th UAV at time t, respectively; Rᵢ(sᵢᵗ,aᵢᵗ) represents the immediate reward obtained by the i-th UAV at time t; γ∈(0,1] represents the discount factor used to balance the current reward and future reward.

[0017] Preferably, the reward function in the HAPPO–AGRN model The formula is as follows:

[0018]

[0019]

[0020] in, For drones Allocate the target quantity in the current time slot; The UAV-target assignment matrix at time t; α and β are weighting coefficients; The Jains index, Indicates drone For the target Effective perception update rate.

[0021] Preferably, the optimization objective for N drones is:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] in, Let be the position of the drone at time t. For drones With the goal The Euclidean distance between times t.

[0028] Based on the above technical solution, the beneficial effects of this invention are as follows: This invention addresses the problems of uneven allocation, unstable tracking, and difficulty in policy training convergence during multi-UAV multi-target cooperative task allocation and trajectory tracking. It utilizes a Markov game framework to uniformly model the task allocation decision-making and trajectory optimization process of multi-UAV systems. This invention proposes a multi-UAV multi-target cooperative task allocation and trajectory tracking method. By introducing an RNN-GAN fusion structure into the Critic network, it effectively enhances the ability to model the time-dependent dynamic evolution characteristics of targets, improving the accuracy and robustness of value assessment. Simultaneously, this invention introduces a self-attention mechanism into the Actor network, enabling UAVs to automatically extract key correlation information from the high-dimensional features of multiple targets, significantly improving policy expression capabilities and cooperative decision-making accuracy. This invention further employs a hierarchical policy update mechanism, enabling multiple UAVs to achieve higher policy stability and convergence efficiency under a centralized training and distributed execution (CTDE) framework. In scenarios with dynamically changing tasks or enhanced target coupling, decision conflicts among UAVs are significantly reduced, and task allocation becomes more balanced. By comprehensively utilizing target distance, dynamic characteristics, and task load information, this invention achieves intelligent and adaptive task allocation and real-time trajectory tracking for multi-UAV systems, effectively improving the overall collaborative efficiency, tracking performance, and task execution reliability of the system. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a multi-UAV multi-target cooperative task allocation and trajectory tracking method in one embodiment;

[0030] Figure 2 This is a visualization of experimental results of a multi-UAV multi-target cooperative task allocation and trajectory tracking method in one embodiment;

[0031] Figure 3 The following is a data graph showing the experimental results of three algorithms, HAPPO, HATRPO, and HAPPO-AGRN, in one embodiment: (a) is a schematic diagram of the average reward curve of UAV 1 using HAPPO, HATRPO, and HAPPO-AGRN respectively; (b) is a schematic diagram of the average reward curve of UAV 2 using HAPPO, HATRPO, and HAPPO-AGRN respectively; and (c) is a schematic diagram of the average reward curve of UAV 3 using HAPPO, HATRPO, and HAPPO-AGRN respectively.

[0032] Figure 4This is a data graph of ablation experiment results in one embodiment. (a) is a schematic diagram of the average reward curves of UAV 1 using HAPPO-A, HAPPO-C, and HAPPO-AGRN respectively; (b) is a schematic diagram of the average reward curves of UAV 2 using HAPPO-A, HAPPO-C, and HAPPO-AGRN respectively; and (c) is a schematic diagram of the average reward curves of UAV 3 using HAPPO-A, HAPPO-C, and HAPPO-AGRN respectively. HAPPO-C represents the introduction of RNN-GAN only in the discriminator, and HAPPO-A represents the introduction of a self-attention mechanism only in the generator. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0034] like Figure 1 As shown, this embodiment provides a method for multi-UAV multi-target cooperative task allocation and trajectory tracking. The method first models the cooperative task allocation problem of multiple UAVs in a multi-target dynamic environment, where N UAVs are used to continuously track M randomly moving targets, and each UAV can make independent scheduling decisions. All UAVs share the same airspace resources and take independent actions in each decision time slot. The UAV set is defined as... The target set is defined as .

[0035] In a dynamic interactive environment with multiple drones, each drone needs to make sequential decisions through continuous interaction with the unknown environment. This collaborative decision-making process can be described as a stochastic game (Markov Game), also known as a multi-agent Markov decision process.

[0036] Therefore, the multi-target cooperative tracking task of multiple UAVs is modeled as a multi-agent Markov game, and interacts with the environment according to the following process: at time t, based on the current local or global state... drones According to its strategy Select Action The drone was subsequently awarded a prize. and state transition probability Transition to the next environmental state .

[0037] In this collaborative tracking system, all targets move randomly in a two-dimensional plane, while each UAV flies in three-dimensional space and continuously tracks the targets under given speed and acceleration constraints.

[0038] Step 1: Initialize the joint strategy for multiple drones Specifically: Within each time slot, multiple drones sequentially select targets to track according to their assigned numbers. Specifically, the drones first select targets based on the currently observable target's position, speed, and other characteristics, and then broadcast their selections to other drones. Subsequently, each drone plans its own flight trajectory in real time based on the assigned target position. This invention assumes that each target is tracked by only one drone at a time, and stipulates that each drone tracks at least one target to avoid resource waste or drones remaining idle for extended periods. The positions of the drones and targets are defined as three-dimensional coordinates. with two-dimensional coordinates ,in , .

[0039] To ensure flight safety, the operating altitude of the drones is limited to between 5 and 100 meters. To simplify the model and reduce environmental uncertainties, it is assumed that the drones fly in unobstructed airspace, and all targets move on a two-dimensional plane without obstacles such as buildings. Therefore, each drone can directly observe any ground target within its perception range.

[0040] In a three-dimensional Cartesian coordinate system, the velocity vector of the i-th UAV at time t is:

[0041] (1)

[0042] Where i, j, and k represent the unit vectors in the x, y, and z directions of the three-dimensional Cartesian coordinate system, respectively; , and Let x, y, and z represent the velocity components of the i-th UAV along the x, y, and z axes at time t, respectively.

[0043] The velocity vector of the j-th target at time t is:

[0044] (2)

[0045] in, and Let x and y represent the velocity components of the j-th target along the x and y axes at time t, respectively.

[0046] The motion model of the targets is defined. All targets are assumed to move randomly in a plane, with their velocities constrained by an upper limit. The UAV flies in three-dimensional space, with a minimum altitude set at 5 m to ensure safe flight. The velocity vector of the UAV is defined as: The position update equation is:

[0047] (3)

[0048] in, and Let represent the three-dimensional position vectors of the i-th UAV at times t and t+1, respectively; This represents its velocity vector at time t; The discrete time step; This represents the displacement increment of the drone within that time step.

[0049] The target motion is modeled as a two-dimensional random walk. Its motion model is as follows:

[0050] (4)

[0051] in, and Let represent the three-dimensional position vectors of the j-th target at times t and t+1, respectively; This represents its velocity vector at time t; The discrete time step; This represents the displacement increment of the target within that time step.

[0052] After target allocation is completed, the UAV needs to plan its flight trajectory based on its current state and the allocation results to minimize the expected tracking distance between itself and the assigned target, thereby reducing energy consumption and improving tracking efficiency. Each UAV calculates its trajectory based on the allocation results, minimizing the sum of its Euclidean distances to all assigned targets at the current moment. The relationship between the UAV and the target is represented by a binary allocation matrix. Let... Assign a UAV-target matrix at time t, where if the target Assigned to drones but ,otherwise .

[0053] (5)

[0054] At this moment, the drone With the goal Euclidean distance between times t Defined as:

[0055] (6)

[0056] To further optimize the system's task load balancing, the Jains fairness metric is introduced as a component of the reward function. This metric measures the uniformity of target allocation among UAVs, as follows:

[0057] (7)

[0058] The Jain index reaches its maximum value of 1 when all drones are assigned the same number of targets. Conversely, when the allocation is severely unbalanced (e.g., all targets are assigned to a single drone), the value approaches 1 / N. Therefore, the Jain fairness index is incorporated into the reward function as a target fairness constraint to promote fair task allocation and prevent specific drones from being idle for extended periods or overloaded. The reward function formula is as follows:

[0059] (8)

[0060] in, For drones i The number of targets is allocated in the current time slot; α and β are weighting coefficients. This represents the effective perception update rate of the UAV regarding the target, used to quantify the UAV's ability to acquire and update target state information.

[0061] To quantify drones i With target t j The perceived intensity between them is defined as the perceived gain g. ij Its value is inversely proportional to the squared distance:

[0062] (9)

[0063] in, It is a scaling factor that reflects the perception capabilities of the unmanned aerial vehicle (UAV) system. Each UAV must continuously perceive and track its assigned target.

[0064] Each drone is equipped with a fixed total sampling capacity. When drones At time t, a is allocated i When there are multiple targets, the sampling capability will be evenly distributed, with each target having a sampling frequency of [missing information]. Based on the sensor noise power spectral density N0 and the assigned sampling frequency drones With the goal The effective sensing update rate (unit: effective information bits per second) can be modeled as:

[0065] (10)

[0066] in, N represents the transmit power of the drone's sensors or communication modules, in W; N0 represents the noise power spectral density, in W / Hz. The larger the value, the more timely the drone updates the target's status information and the higher the quality of the perception link, thus it can obtain higher returns in the reward function.

[0067] This model reveals the relationship between target allocation and tracking performance: as the number of allocated targets increases, the sampling frequency of each target decreases, thus affecting the tracking effect. This provides a quantitative basis for system optimization.

[0068] Step 2: Train the HAPPO–AGRN model.

[0069] 1) Employing a centralized training, distributed execution (CTDE) paradigm: The Critic network has access to the global state to estimate the shared value function, while the Actor network relies solely on local observations to select actions during online execution. These formulas together constitute the learning signal for the multi-UAV system, enabling it to achieve efficient, balanced, and continuous patrol coverage under partially observable conditions. To ensure the feasibility, robustness, and safety of task allocation and path planning, the multi-UAV system must meet the following constraints:

[0070] (11)

[0071] (12)

[0072] (13)

[0073] (14)

[0074] (15)

[0075] After modeling and optimizing the problem description, an RNN-GAN fusion structure is introduced into the Critic network, effectively enhancing the ability to model the time-dependent dynamic evolution characteristics of the target, thereby improving the accuracy and robustness of value assessment. Simultaneously, a self-attention mechanism is introduced into the Actor network, enabling UAVs to automatically extract key correlation information from the high-dimensional features of multiple targets, significantly enhancing the expressive power of the policy and the accuracy of collaborative decision-making. This invention also employs a hierarchical policy update mechanism, enabling multiple UAVs to achieve higher policy stability and convergence efficiency under a centralized training and distributed execution (CTDE) framework. In scenarios with dynamic task changes or enhanced target coupling, decision conflicts among UAVs are significantly alleviated, and task allocation is more balanced. By comprehensively considering target distance, dynamic features, and task load information, this invention achieves intelligent, adaptive task allocation and real-time trajectory tracking for multi-UAV systems, effectively improving the overall collaborative efficiency, tracking performance, and task execution reliability of the system.

[0076] 2) Calculate the output of the Critic network, with the global state sequence as input. With action sequence The algorithm encodes temporal features using an RNN-GAN structure, generates a generator to predict Q-values, a discriminator to distinguish between true and false signals, and calculates the GAN loss. The output is the predicted Q-value and the predicted state value. The advantage function is calculated, and the output Q-value is fed back to the Actor network of the drone base station. The Actor network is based on the HAPPO Actor and incorporates a Self-Attention layer to calculate the dependencies between targets, enabling the drone to focus on key target features and output weighted features, thereby assisting the drone base station in policy updates. The loss function of the Critic network is... ,in, , These are the weighting coefficients. For counter-loss, For the current Critic network, the current state-action pair The predicted Q value, This is the target value.

[0077] 3) Update strategy Where πᵢ* represents the optimal strategy of the i-th UAV, and πᵢ represents the strategy to be optimized for the i-th UAV; arg max₍πᵢ₎ represents selecting the strategy that maximizes the expected cumulative reward from all candidate strategies; s0 and a0 represent the initial global state and initial joint action, respectively; E₍s0,a0,…₎[·] represents the expected calculation of environmental state transition, strategy action sampling and initial state distribution; t represents the discrete decision time; sᵢᵗ and aᵢᵗ represent the local state and executed action of the i-th UAV at time t, respectively; Rᵢ(sᵢᵗ,aᵢᵗ) represents the local state and executed action of the i-th UAV at time t. The immediate reward obtained; γ∈(0,1] represents the discount factor used to balance the current reward with the future reward. The training process iterates continuously until the training rounds are satisfied, and a trained model of the heterogeneous agent proximal policy optimization algorithm (HAPPO–AGRN) based on self-attention generative adversarial recurrent network is obtained.

[0078] Step 3: Load the trained HAPPO–AGRN model into N UAVs, enabling the N UAVs to perform collaborative task allocation and trajectory tracking for M dynamic targets.

[0079] The state of the UAV in the previous time slot is input into the Critic network, which uses an RNN-GAN structure to generate a state value function. The dominance function is calculated. If a drastic change in target motion, uneven UAV load, or disturbance in cooperative relationships is detected, the Actor network based on a self-attention mechanism is triggered, outputting a new action distribution and calculating the dominance function. Otherwise, the dominance function is estimated based on the local target state. The actions in the previous time slot are determined based on the dominance function, including target selection, allocation strategy, and 3D trajectory adjustment. The UAV updates its flight trajectory based on its actions and interaction with the environment. Simultaneously, the Critic network outputs the corresponding Q-value and feeds it back to the Actor network for policy updates.

[0080] experiment

[0081] Experimental results show that in multi-UAV, multi-target missions, Figure 2 The three drones can dynamically adjust their positions based on the trajectories of 12 randomly moving targets, achieving continuous coverage and effective matching. Figure 3 During training, the cumulative rewards of the three algorithms generally increased with each training round, stabilizing after approximately 200 rounds, indicating algorithm convergence. Among them, HAPPO-AGRN not only had the highest cumulative reward after convergence but also exhibited the smallest fluctuation in its reward curve, demonstrating superior stability and collaborative efficiency (see Table 1). Ablation Experiment Figure 4 The results show that the Critic network incorporating RNN-GAN significantly improves the average reward and convergence speed, while the Actor network combined with the self-attention mechanism further enhances the policy representation and task allocation capabilities after being combined with RNN-GAN, making HAPPO-AGRN the best performing network in terms of reward, convergence speed, task balancing, and tracking accuracy. Further generalization experiments show that this policy can converge stably in scenarios with different numbers of drones and targets, indicating that the method has strong environmental adaptability and generalization ability.

[0082] Table 1 Final Rewards for the Three Drones

[0083]

[0084] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0085] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A method for multi-UAV multi-target cooperative task allocation and trajectory tracking, characterized in that, Includes the following steps: Acquire information about the interaction between N UAVs and the environment during the tracking of M dynamic targets; A HAPPO–AGRN model is constructed and trained using a centralized training and decentralized execution paradigm based on interactive information. The trained HAPPO–AGRN model is loaded into N drones, enabling the N drones to perform collaborative task allocation and trajectory tracking on M dynamic targets.

2. The method for multi-UAV multi-target cooperative task allocation and trajectory tracking according to claim 1, characterized in that, The actions of the UAV include: target selection vector, task allocation action, UAV flight control action, and trajectory update vector. The state of the UAV includes: the position of the assigned target, local observation of multiple surrounding targets, and some task allocation information of other UAVs. During the training of the HAPPO–AGRN model, the state sequences and action sequences of all UAVs are input into the Critic network with an RNN-GAN structure to capture temporal dynamic dependencies and predict Q values. The output Q values ​​are then fed back to the Actor network based on a self-attention mechanism to focus on key target features for policy updates.

3. The method for multi-UAV multi-target cooperative task allocation and trajectory tracking according to claim 2, characterized in that, The loss function of the Critic network is as follows: ,in, , These are the weighting coefficients. For counter-loss, This represents the Q-value of the Critic network's prediction for the current state-action pair. This is the target value.

4. The method for multi-UAV multi-target cooperative task allocation and trajectory tracking according to claim 2, characterized in that, The strategy update formula is as follows: , where πᵢ* represents the optimal strategy of the i-th UAV, and πᵢ represents the strategy to be optimized for the i-th UAV; arg max₍πᵢ₎ represents selecting the strategy that maximizes the expected cumulative reward from all candidate strategies; s0 and a0 represent the initial global state and initial joint action, respectively; E₍s0,a0,…₎[·] represents the expected calculation of environmental state transition, strategy action sampling and initial state distribution; t represents the discrete decision time; sᵢᵗ and aᵢᵗ represent the local state and executed action of the i-th UAV at time t, respectively; Rᵢ(sᵢᵗ,aᵢᵗ) represents the immediate reward obtained by the i-th UAV at time t; γ∈(0,1] represents the discount factor, used to balance the current reward and future reward.

5. The method for multi-UAV multi-target cooperative task allocation and trajectory tracking according to claim 1, characterized in that, The reward function in the HAPPO–AGRN model The formula is as follows: ,in, For drones Allocate the target quantity in the current time slot; The UAV-target assignment matrix at time t; α and β are weighting coefficients; The Jains index, Indicates drone For the target Effective perception update rate.

6. The method for multi-UAV multi-target cooperative task allocation and trajectory tracking according to claim 1, characterized in that, The optimization objective for N drones is: ,in, Let be the position of the drone at time t. For drones With the goal The Euclidean distance between times t.

Images