Cooperative task allocation method for time-sensitive target tracking

Abstract

The invention discloses a cooperative task allocation method for time-sensitive target tracking. The method comprises the following steps: step 1, modeling an unmanned aerial vehicle cluster cooperative task allocation problem as a De-POMDP model; 2, performing state space design, action space design and reward function design on the De-POMDP model to obtain a final optimization target function of the multi-target tracking task; 3, solving an optimization objective function of the multi-target tracking task by using an FMASAC algorithm; and step 4, outputting an optimal allocation scheme. Finally, a simulation experiment shows that the method provided by the invention has better robustness and expansibility.

Description

Technical Field

[0001] The present invention relates to the technical field of upper-layer collaborative task allocation, and in particular to a collaborative task allocation method for time-sensitive target tracking. Background Art

[0002] There are often a variety of vehicle-mounted mobile targets on modern battlefields, such as HIMARS rocket launchers and mobile missile launchers, which are highly mobile and concealed. How to quickly and efficiently discover and allocate weapons to strike such time-sensitive dynamic targets is an important task that needs to be solved urgently.

[0003] Striking dynamic time-sensitive targets is a difficult task on the battlefield. First, because of the "fog of war," the target is difficult to find; second, striking dynamic time-sensitive targets requires a very high kill chain closure speed, and the entire kill chain F2T2EA process needs to be completed quickly. Therefore, the dynamic time-sensitive target task allocation problem urgently needs to be empowered by the latest artificial intelligence technology.

[0004] Task allocation is a classic combinatorial optimization problem, but traditional optimization methods face the "curse of dimensionality" and real-time problems when solving combinatorial optimization problems, and the speed of solution decreases exponentially as the scale of the problem increases. By converting the combinatorial optimization problem into a sequential decision-making problem, using the MDP model to simulate the dynamic optimization process, and using deep reinforcement learning to approximate the complex non-convex objective function, the task allocation problem can be solved more efficiently and in a real-time manner through data-driven autonomous learning of optimal or suboptimal strategies. For multiple drones and multiple targets, it is natural to use multi-agent deep reinforcement learning technology that matches them. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention takes multi-target tracking tasks as the background, studies the problem of task allocation of UAV cluster collaborative tracking of multiple mobile targets on the ground, and proposes a collaborative task allocation method for time-sensitive target tracking.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A collaborative task allocation method for time-sensitive target tracking, characterized in that it includes the following steps:

[0008] Step 1: Model the UAV cluster collaborative task allocation problem as a Dec-POMDP model;

[0009] Step 2: Design the state space, action space and reward function of the Dec-POMDP model to obtain the final optimization objective function of the multi-target tracking task;

[0010] Step 3: Use the FMASAC algorithm to solve the optimization objective function of the multi-target tracking task;

[0011] Step 4: Output the optimal allocation plan.

[0012] Furthermore, the specific steps of step 2 include:

[0013] Step 21: Record the drone's observation vector as in, For drone i to receive the communication information of drone j, is the speed of UAV j in the (x, y) direction; is the perception vector of drone i to target k;

[0014] Step 22: Discretize the angular velocity of the drone into N a Discrete values:

[0015]

[0016] Step 23: Set up Tracking Rewards i tracking , collision penalty r i collision , and out-of-bounds penalty r i bound , and then according to r i tracking 、r i collision and r i bound Get the team reward function of the drone cluster, and then combine it with the spatial entropy reward to get the final reward function;

[0017] Step 24: Based on the final reward function and temperature parameters, the optimized objective function of the multi-target tracking task is obtained.

[0018] Furthermore, the specific steps of step 23 include:

[0019] Step 231: The closer the distance between the drone and the target is, the higher the reward is, so the tracking reward is defined as:

[0020]

[0021] Among them, d i,k represents the distance between UAV i and target k;

[0022] Step 232: If the relative distance between the drones is less than the safe distance, they will be penalized, so the collision penalty is defined as:

[0023]

[0024] Among them, d i,j represents the distance between UAV i and UAV j, ds Indicates the anti-collision safety distance between drones;

[0025] Step 233: If the drone flies out of the boundary, it will be penalized for invalid search and tracking, so the out-of-bounds penalty is defined as:

[0026]

[0027] Among them, d i,O represents the distance from drone i to the center of the scene, d bound is the scene width;

[0028] Step 234: Based on the complete cooperation relationship between the drone clusters, the team reward shared by the entire cluster is defined as:

[0029]

[0030] Step 235: To make the distance between drones as large as possible, define the spatial entropy reward r SE for:

[0031]

[0032]

[0033] Among them, d SE Indicates the size of the space entropy equivalent distance, represents the expected minimum spatial entropy, represents the spatial entropy, represents the average distance of the drone cluster,

[0034] Step 236: The final reward function calculation formula is:

[0035]

[0036] Where κ = max(1-n e / 1000,0),n e represents the current round number, and κ uses linear annealing and decreases from 1 to 0 after 1000 rounds.

[0037] Furthermore, the optimization objective function of the multi-target tracking task established in step 24 is:

[0038]

[0039] in, represents the entropy of the strategy, r t represents the reward at time t, α represents the entropy coefficient, π θ Represents an evaluation strategy.

[0040] Furthermore, the specific steps of step 3 include:

[0041] Step 31: For each agent in the Dec-POMDP model, its actor network is based on the local observation vector and the previous action To select an action

[0042] Step S31: Decompose the centralized critic network into a linear combination of individual critic networks, and estimate the observation-action pair (τ i ,a i ) to obtain the joint action value function

[0043] Among them, φ represents the centralized critic The network parameters, φ i Represents individual critic The network parameters of g ω is a linear monotonic function; s represents the global state of the environment;

[0044] Step S32: The TD loss function shown in formula (19) is used to The function is optimized to obtain the optimized critic network, and the network is used to evaluate the strategy π θ :

[0045]

[0046]

[0047] Among them, formula (20) represents the optimized target critic network; and Represent the parameters of the target critic and hybrid network respectively; Represents the joint target actor parameters;

[0048] Step S33: Based on strategy π θ , the decomposed critic is used to train actors to achieve decentralized execution. The training formula is:

[0049]

[0050] Among them, g ω represents a single-layer monotone mixture network;

[0051] Step S34: Temperature adjustment is achieved by optimizing the loss function to update the actor network:

[0052]

[0053] in, is the minimum entropy constraint, satisfying The beneficial effects of the present invention are:

[0054] First, the present invention proposes an end-to-end FMASAC-based multi-target collaborative tracking task allocation scheme, which can realize the collaborative tracking of multiple targets by UAV clusters in a distributed manner;

[0055] Second, by learning a centralized but decomposed critic and decomposing the policy iteration, the algorithm learning efficiency is improved and the policy update variance is reduced. SER increases the detection coverage of the drone cluster and improves the task coordination performance;

[0056] Third, the FMASAC algorithm proposed in the present invention can realize multi-target collaborative tracking task allocation while avoiding collisions. It outperforms the existing baseline algorithms in terms of average reward and tracking success rate and is significantly better than the existing popular MARL baselines. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of collaborative task allocation for drone swarms.

[0058] Figure 2 It is the observation range and communication range of the UAV.

[0059] Figure 3 This is the overall framework of the FMASAC algorithm proposed in this invention.

[0060] Figure 4 Assign visual results to tasks.

[0061] Figure 5 Average reward learning curves for different algorithms.

[0062] Figure 6 The ablation experiment results.

[0063] Figure 7 (a)- Figure 7 (c) is the UAV target tracking result; Figure 7 (a) is the result of the change of the UAV-target distance during training. Figure 7 (b) is the relative distance change between drones during training. Figure 7 (c) Results of tracking success rate changes during training.

[0064] Figure 8 (a)- Figure 8 (b) is the test result of the UAV target tracking performance; Figure 8(a) is the test result of the relative distance performance between the drone and the target during the test. Figure 8 (b) is the relative distance performance test result between drones in the test.

[0065] Fig. 9 Relative distance and tracking success rate in robustness test.

[0066] Fig.10 (a)- Fig.10 (b) is the scalability test result; Fig.10 (a) is the average number of tracked targets in different UAV scenarios. Fig.10 (b) is the trend chart of tracking success rate for scenes with different numbers of drones. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0068] 1. Collaborative task allocation model for time-sensitive goals

[0069] The UAV cluster needs to collaboratively search, discover and assign tracking targets to complete the task, thereby achieving efficient, robust and scalable dynamic task allocation. The schematic diagram of UAV cluster collaborative task allocation is shown in the figure. Figure 1 shown.

[0070] The present invention describes the task allocation problem as a sequential decision-making problem, and models the UAV cluster collaborative task allocation problem as a Dec-POMDP model, and designs the state space, action space and reward function of Dec-POMDP respectively. Finally, in order to increase the search and tracking coverage of the intelligent agent, combined with the characteristics of the multi-target tracking task allocation scenario, a sub-goal reward function combined with spatial entropy is proposed to guide the intelligent agent to achieve task allocation better and faster.

[0071] (1) Overview of Dec-POMDP Model

[0072] In the UAV swarm collaborative multi-target tracking task, each UAV only interacts with the environment based on its own local observations and communication information to achieve the overall goal. This task can be modeled as a typical Dec-POMDP model, denoted by G =<I,S,A,P,R,Ω,O,n,γ> . Where I is the set of agents, S represents the global state of the environment, n is the total number of agents, and γ∈[0,1] is the discount factor. Due to partial observability, at each time step t, each agent i perceives a local observation o i ∈Ω, then choose action a i ∈A, forming a joint action a∈A nWhen the environment receives the joint action, each agent will transfer to the next state s' according to the state transfer function P(s'|s,a) and receive a team-shared reward r = R(s,a). The goal of each agent is to calculate the reward based on its local action-observation history τ i Let's learn a deterministic policy Or random strategy According to the joint random strategy The joint action value function can be derived Similarly, according to the joint determination strategy The joint action value function Q can also be derived tot (τ,a).

[0073] (2) Dec-POMDP modeling

[0074] It mainly includes state space design, action space design and reward function design.

[0075] ① State space design

[0076] Considering the limited radar detection range and communication distance of the UAV, it is assumed that the maximum observation radius of the UAV is d o , the maximum communication distance is d c ,like Figure 2 As shown. If the distance between UAV i and UAV j is less than d c , then the two drones can promote cooperation by sharing information through communication. The communication information received by drone i can be recorded as in is the velocity of drone j in the x and y directions, is the action of the drone at the last moment. Similarly, if the distance between drone i and target k is less than d o , then the UAV can sense the target and obtain the target position and speed information, and the target information can be recorded as Therefore, the observation vector of the UAV consists of its position, heading, target information and communication information, which can be expressed as

[0077] ②Action space design

[0078] The action space of the drone mainly refers to the angular velocity (lateral acceleration). In order to improve learning efficiency, the angular velocity is discretized into N a discrete values, the maximum heading angular velocity is The formula is:

[0079]

[0080] Among them, a represents the final heading angular velocity of the UAV agent, and its value range is n a Represents the output of a policy.

[0081] ③Reward function design

[0082] In reinforcement learning, the reward function is the core of guiding the agent's learning, and the reward function greatly affects the agent's learning speed and final performance.

[0083] For multi-target tracking tasks, each drone needs to learn to search and track the target, that is, keep the target within the drone's observation radius. In addition, drones should avoid collision or flying out of the environment boundary. Therefore, the reward function design consists of the following three parts: tracking reward r i tracking , collision penalty r i collision , and out-of-bounds penalty r i bound .

[0084] The tracking reward mainly considers the distance between the drone and the target. The closer the distance, the higher the reward. Therefore, the tracking reward is defined as:

[0085]

[0086] Among them, d i,k represents the distance between drone i and target k. If drone i perceives multiple targets, it is encouraged to track the target closest to itself, and the reward is

[0087] The collision penalty mainly considers the relative distance between drones. If the distance is less than the safe distance, a penalty will be imposed. Therefore, the collision penalty is defined as:

[0088]

[0089] Among them, d i,j represents the distance between UAV i and UAV j, d s Indicates the collision avoidance safety distance between drones.

[0090] If d i,j ≤d s , the drone will receive a collision penalty to improve safety.

[0091] Similarly, the out-of-bounds penalty is defined as:

[0092]

[0093] Among them, d i,O represents the distance from drone i to the center of the scene, d bound is the scene width. If drone i flies out of the boundary, it will be penalized due to invalid search and tracking.

[0094] Because the drone clusters are in full cooperation, the entire cluster shares a team reward

[0095] (3) Space Entropy Reward

[0096] However, due to insufficient exploration, some targets will always be missed, that is, the multi-target tracking task is prone to fall into local optimality. To solve this problem, the simplest and most direct method is to encourage drones to maintain a certain distance, thereby increasing the spatial distribution and detection coverage of the entire cluster. Therefore, the present invention proposes a spatial entropy reward (SER) to encourage drones to maintain a certain spatial distance to increase detection coverage, thereby improving the collaborative multi-target tracking capability.

[0097] First, let's define spatial entropy. Consider a two-dimensional polar coordinate plane (ρ, θ), where the origin is the position coordinate of the drone. Each position on the plane can be represented by polar coordinates (ρ, θ), where ρ represents the polar diameter and θ represents the relative angle. Then, the spatial density function is:

[0098] p(ρ,θ)=Ke -λρ (5)

[0099] Where λ is a parameter, K is a normalization constant, and:

[0100]

[0101] Therefore, according to equations (5) and (6), we can conclude that:

[0102]

[0103] in, Represents the generalized radius related to spatial density, similar to the definition of information entropy, spatial entropy It can be defined as:

[0104]

[0105] Obviously, Follow Monotonically increasing.

[0106] Therefore, the entropy can be adjusted to control Size:

[0107]

[0108] in, represents the expected minimum spatial entropy, d SE Indicates the size of the spatial entropy equivalent distance.

[0109] based on and d SE Definition, propose spatial entropy reward r SE The concept is to consider the distance between drones as large as possible. The calculation formula is:

[0110]

[0111] in, represents the average distance of the drone cluster. If Then r SE >1.

[0112] Therefore, the spatial entropy reward r SE Drones can be encouraged to stay as far away as possible to increase detection coverage.

[0113] In summary, the final reward function is:

[0114]

[0115] Where κ = max(1-n e / 1000,0),n e represents the current round number, and κ uses linear annealing, which decreases from 1 to 0 after 1000 rounds, thereby appropriately increasing the spatial entropy reward r SE to improve the tracking task performance.

[0116] Therefore, the optimization objective function of the multi-target tracking task can be expressed as:

[0117]

[0118] in, represents the entropy of the strategy, r t represents the reward at time t, and α represents the entropy coefficient.

[0119] Next, the present invention proposes a new maximum entropy MARL algorithm combining the value decomposition concept to solve this problem.

[0120] 2. FMASAC algorithm

[0121] (1) MASAC algorithm

[0122] SAC is an off-policy maximum entropy framework RL algorithm that maximizes the cumulative return and policy entropy to encourage exploration. SAC optimizes the critic network Q by minimizing the temporal difference (TD) error loss function. φ :

[0123]

[0124] in, D represents the playback of past experience, φ, and Represent the parameters of the critic, target critic and target actor networks respectively.

[0125] SAC introduces entropy term and adopts flexible action value function Strategy objective function J(π θ ) is optimized by gradient ascent using the following formula:

[0126]

[0127] MASAC is a MARL algorithm that extends SAC to a multi-agent framework. It adopts the CTDE paradigm and learns random strategies. It can solve both discrete action space problems and continuous action space problems. The loss function shown in formula (13) is transformed into the loss function shown in formula (15) using the parameter sharing joint optimization method:

[0128]

[0129] in, here represents the central action value function of agent i, and the individual strategy is updated by gradient ascent using the following formula:

[0130]

[0131] Among them, all actions are sampled from the current strategy of the agent to prevent the problem of over-generalization of the strategy. However, since the size of the joint action space will increase exponentially with the increase in the number of agents, MASAC cannot be directly extended to large-scale joint action space problems.

[0132] (2) FMASAC algorithm

[0133] To solve the scalability problem of MASAC, this paper introduces the idea of ​​value decomposition and proposes a factored multi-agent soft actor-critic (FMASAC) algorithm. In order to solve the high variance and credit allocation problems of the cooperative multi-agent actor-critic framework, the centralized critic is decomposed into individual critics, and credit is allocated to each agent to improve the strategy and increase the reward. Then, a decomposed flexible policy iteration is proposed to realize off-policy learning of multi-agent stochastic policy gradient. The complete process of the FMASAC algorithm is shown in the pseudo code in Table 1.

[0134] ① Learning to decompose the critic

[0135] Considering the explosion of the joint state-action space caused by the increase in the number of agents, it is very difficult to learn a centralized critic. This paper combines value decomposition and the MASAC framework to achieve scalable learning of centralized critics under Dec-POMDP. The centralized critic is decomposed into a linear combination of individual critics:

[0136]

[0137] Among them, φ represents the centralized critic The network parameters, φ i Represents individual critic Network parameters. ω represents a linear monotonic function, parameterized as a hybrid network without activation function, and s represents the global state of the environment. Among them, the linear monotonic function increases the expressive power of the network and can fit the utility of individual agents into a centralized critic or joint action value function. In addition, monotonicity guarantees the individual global maximum (IGM), that is, to ensure the consistency between the maximum joint action value function and the maximum individual action value function, as shown in formula (18).

[0138]

[0139] To evaluate the strategy π θ , the critic network is optimized using the following TD loss:

[0140]

[0141]

[0142] Among them, formula (20) represents the target critic network. and Represent the parameters of the target critic and hybrid network respectively. Represents the joint target actor parameters.

[0143] ② Decomposition of flexible strategy iteration

[0144] In an end-to-end manner, the decomposed critic is used to train actors to achieve decentralized execution, and a decomposed soft strategy iteration is proposed. The training formula is:

[0145]

[0146] Among them, g ωRepresents a single-layer monotone hybrid network, which is consistent with formula (17). The weights and biases of the neural network can be generated according to the input s, and then the utility is used to allocate credits. The above formula is derived in detail:

[0147]

[0148] From equation (23), we can see that this is consistent with the original / standard SAC update method.

[0149] Similar to the automatic update of individual temperature parameters in previous literature, here we consider transforming the automatic update problem of the global temperature parameter α into a constrained optimization problem, and achieve temperature adjustment by optimizing the loss function:

[0150]

[0151] in, is the minimum entropy constraint, satisfying Ensure the randomness of the policy. In addition, if the policy satisfies the entropy constraint, α will decrease; otherwise, α will increase to encourage the policy to explore more. Therefore, the entropy constraint encourages exploration while ensuring that the policy converges to the optimal.

[0152] ③ Multi-target tracking task allocation of UAV cluster based on FMASAC

[0153] The overall architecture of the FMASAC algorithm proposed in this invention is as follows: Figure 3 As shown. Combined Figure 1 , for each agent, its actor network is based on the local observation and the previous action Choose the action of this moment At the same time, there is a critic network estimation used to estimate the observation-action pair (τ i ,a i ), and finally form a joint action value function through a linear monotone mixing function combination Optimize through TD learning method in formula (19). Then, each optimized Iteratively update the actor network by decomposing the flexible policy.

[0154] The detailed process of the FMASAC algorithm is shown in Table 1, which is roughly divided into three parts: initialization phase, experience collection phase, and network update phase. First, all parameters are initialized. Then, in the experience collection phase, all drone agents that interact with the environment collect and store experience according to their own strategies. The actor and critic networks are optimized in the network update phase so that each drone learns to cooperate in tracking the target. After training convergence, each drone model can realize autonomous search and tracking target tasks, and the entire drone cluster can obtain the ability to coordinate multi-target tracking in an unknown environment.

[0155] Table 1 FMASAC algorithm pseudo code

[0156]

[0157]

[0158] Example

[0159] The FMASAC algorithm proposed in this paper is evaluated through simulation experiments, and its performance is compared with three popular MARL algorithms, including rewards in training and tracking performance in testing, to prove the effectiveness, robustness and scalability of the FMASAC algorithm.

[0160] 1. Simulation settings

[0161] Based on Ubuntu 20.04, PyCharm and multi-agent particle environment, this paper constructs a drone cluster multi-target tracking simulation environment, and the user interface and scene parameters can be customized. The number of drones and targets and the size of the map are variable and can be further modified according to needs. The positions of drones and targets are randomly initialized in each round, and other environmental parameters are shown in Table 2.

[0162] Table 2 Environmental parameter settings

[0163]

[0164]

[0165] Next, taking the scenario of three drones tracking three targets as an example, the performance comparison of the FMASAC algorithm and the MADDPG, MASAC, and MAPPO algorithms was tested. The simulation hyperparameter settings are shown in Table 3, and the baseline hyperparameter settings refer to reference

[135] . For fairness, each algorithm is tested on five random seeds.

[0166] Table 3 Hyperparameter settings

[0167]

[0168] 2. Effectiveness test

[0169] First, each algorithm was trained for 1,005,000 steps until the learning curve converged to test the algorithm performance. In order to intuitively understand whether the drones have learned to collaboratively search and track targets, the drone tracking process after algorithm training was visualized, as shown in Figure 2. Figure 4 shown.

[0170] Figure 4 The three drones were shown to automatically assign the nearest target for collaborative tracking, and each drone learned to autonomously search and track the target. Next, to compare the performance of the algorithm, the average reward of the drones during training, the relative distance between the drone and the target, and the tracking success rate were analyzed. Tracking success is defined as tracking more than 70 steps in each round. The tracking success rate is defined as the ratio of the number of successful tracking to the total number of successful tracking in 80 consecutive rounds. Figure 5 Reward learning curves for the four algorithms are shown.

[0171] from Figure 5 It can be seen that the average reward and convergence speed of the FMASAC algorithm are better than those of the other three algorithms, which means that the proposed decomposed critic and flexible policy iteration are effective. Compared with the original MASAC algorithm, the performance improvement of the FMASAC algorithm is more stable and the policy variance is smaller. This is because MASAC only greedily maximizes the team reward in policy learning, which makes the exploration of other agents or the suboptimality of the strategy fed back through the centralized critic, exacerbating the credit allocation problem. Therefore, by learning the decomposed critic and allocating the overall utility information to each agent to improve the strategy, the reasonable allocation of multi-agent credit is implicitly achieved.

[0172] To further demonstrate the effectiveness and value contribution of the corresponding improvements in the FMASAC algorithm, an ablation experiment was performed to compare the performance of the complete FMASAC, FMASAC_Fixed_α with fixed temperature parameters, and FMASAC_NoSER without SER reward. The results are as follows: Figure 6 shown.

[0173] from Figure 6 It can be seen that the performance of the full FMASAC algorithm is better than FMASAC_Fixed_α and FMASAC_NoSER. This shows that the automatic adjustment learning of temperature α can better balance exploration and exploitation, thereby improving the performance of the policy. In particular, SER significantly improves the stability and the final performance of the policy by encouraging increased detection coverage, which shows the importance of SER and its effectiveness in multi-target collaborative tracking tasks.

[0174] Next, we analyze the changes in the distance between drones, the distance between drones and targets, and the tracking success rate during the training process. The results are as follows: Figure 7 shown.

[0175] Figure 7 (a) shows that after 300,000 training steps, the relative distance between the drone and the target is smaller than the drone’s observation range, indicating that each drone has learned to search and track the target. Figure 7 (b) shows that after training, the relative distance between drones is greater than the minimum safe collision avoidance distance of drones, which means that the drones have learned to track and avoid collisions at the same time. Figure 7 (c) shows the changing trend of the tracking success rate, the most important indicator in the multi-target tracking task. After training, the tracking success rate of each drone reached 1. Therefore, the three drones learned to avoid collisions while autonomously searching and tracking targets, verifying the effectiveness of the FMASAC algorithm.

[0176] 3. Robustness test

[0177] Load the trained policy model (actor network parameters) without further fine-tuning and test the robustness of the model in an unknown dynamic environment. The positions of all drone targets are re-initialized and the test results are as follows: Figure 8 shown.

[0178] from Figure 8 As can be seen in (a), each drone can track an unknown target in about 10 steps, because the drone strategy model has "seen" thousands of different scenarios during the training process and stored them in the form of neural network parameters. Figure 8 (b) shows that the relative distance between drones is greater than the safe distance and effective communication is maintained. Therefore, it is concluded that the learned policy model can be directly generalized to unknown environments without further fine-tuning or retraining, achieving efficient and real-time tracking.

[0179] Fig. 9 The relative distance and test tracking success rate in 100 tests are shown. From the violin plot, it can be seen that the drone-target distance is almost smaller than the drone's observation range. This result shows that the trained drone model can track the target even in unknown scenes. In addition, the tracking success rate in each drone test is greater than 98%, indicating that the trained policy model can generalize to unknown scenes.

[0180] 4. Scalability testing

[0181] In order to further test the scalability of the FMASAC algorithm in large-scale scenarios, the simulation of multi-target collaborative tracking task allocation in 5-UAV scenarios and 10-UAV scenarios was further completed. Fig.10The changing trends of the average number of tracked targets and tracking success rates of different algorithms are shown. The detailed results are shown in Table 4.

[0182] Table 4 Comparison of statistical results of different scale scenes

[0183]

[0184] from Fig.10 From Table 4, we can conclude that in the above three scenarios, the FMASAC algorithm achieved the highest average reward and the lowest standard deviation. In addition, the average tracking target and tracking success rate in training are almost better than other baselines. Therefore, the performance of the FMASAC algorithm is better in the three scenarios. This also verifies that the decomposition of the critic and the flexible policy iteration reduce the complexity of the joint action value function, and by combining the individual utility based on local information, scalable learning under the CTDE paradigm is achieved. Therefore, it is shown that the FMASAC algorithm can maintain good performance as the number of agents and the joint action space increase, and can be expanded to large-scale scenarios.

[0185] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A collaborative task allocation method for time-sensitive target tracking, characterized in that: The following steps are involved: Step 1: Model the UAV cluster collaborative task allocation problem as a Dec-POMDP model; Step 2: Design the state space, action space and reward function of the Dec-POMDP model to obtain the final optimization objective function of the multi-target tracking task; Step 3: Use the FMASAC algorithm to solve the optimization objective function of the multi-target tracking task; Step 4: Output the optimal allocation plan.

2. A collaborative task allocation method for time-sensitive target tracking as claimed in claim 1, characterized in that: The specific steps of step 2 include: Step 21: Record the drone's observation vector as in, For drone i to receive the communication information of drone j, is the speed of UAV j in the (x, y) direction; is the perception vector of drone i to target k; Step 22: Discretize the angular velocity of the drone into N a Discrete values: Step 23: Set up Tracking Rewards i tracking , collision penalty r i collision , and out-of-bounds penalty r i bound , and then according to r i tracking 、r i collision and r i bound Get the team reward function of the drone cluster, and then combine it with the spatial entropy reward to get the final reward function; Step 24: Based on the final reward function and temperature parameters, the optimized objective function of the multi-target tracking task is obtained.

3. A collaborative task allocation method for time-sensitive target tracking as claimed in claim 2, characterized in that: The specific steps of step 23 include: Step 231: The closer the distance between the drone and the target is, the higher the reward is, so the tracking reward is defined as: Among them, d i,k represents the distance between UAV i and target k; Step 232: If the relative distance between the drones is less than the safe distance, they will be penalized, so the collision penalty is defined as: Among them, d i,j represents the distance between UAV i and UAV j, d s Indicates the anti-collision safety distance between drones; Step 233: If the drone flies out of the boundary, it will be penalized for invalid search and tracking, so the out-of-bounds penalty is defined as: Among them, d i,O represents the distance from drone i to the center of the scene, d bound is the scene width; Step 234: Based on the complete cooperation relationship between the drone clusters, the team reward shared by the entire cluster is defined as: Step 235: To make the distance between drones as large as possible, define the spatial entropy reward r SE for: Among them, d SE Indicates the size of the space entropy equivalent distance, represents the expected minimum spatial entropy, represents the spatial entropy, represents the average distance of the drone cluster, Step 236: The final reward function calculation formula is: Where κ = max(1-n e / 1000,0),n e represents the current round number, and κ uses linear annealing and decreases from 1 to 0 after 1000 rounds.

4. A collaborative task allocation method for time-sensitive target tracking as claimed in claim 3, characterized in that: The optimization objective function of the multi-target tracking task established in step 24 is: in, represents the entropy of the strategy, r t represents the reward at time t, α represents the entropy coefficient, π θ Represents an evaluation strategy.

5. A collaborative task allocation method for time-sensitive target tracking as claimed in claim 4, characterized in that: The specific steps of step 3 include: Step 31: For each agent in the Dec-POMDP model, its actor network is based on the local observation vector and the previous action To select an action Step S31: Decompose the centralized critic network into a linear combination of individual critic networks, and estimate the observation-action pair (τ i ,a i ) to obtain the joint action value function Among them, φ represents the centralization The network parameters, φ i Represents an individual The network parameters of g ω is a linear monotonic function; s represents the global state of the environment; Step S32: The TD loss function shown in formula (19) is used to The function is optimized to obtain the optimized critic network, and the network is used to evaluate the strategy π θ : Among them, formula (20) represents the optimized target critic network; and Represent the parameters of the target critic and hybrid network respectively; Represents the joint target actor parameters; Step S33: Based on strategy π θ , the decomposed critic is used to train actors to achieve decentralized execution. The training formula is: Among them, g ω represents a single-layer monotone mixture network; Step S34: Temperature adjustment is achieved by optimizing the loss function to update the actor network: in, is the minimum entropy constraint, satisfying

Images