A cooperative task allocation method for time-sensitive target tracking

Abstract

The application discloses a kind of collaborative task allocation methods for time-sensitive target tracking, comprising the following steps: step 1: unmanned aerial vehicle cluster collaborative task allocation problem is modeled as Dec-POMDP model;Step 2: the state space design, action space design and reward function design of Dec-POMDP model are carried out, and the optimization objective function of final multi-target tracking task is obtained;Step 3: the optimization objective function of multi-target tracking task is solved using FMASAC algorithm;Step 4: output optimal allocation scheme.Finally, the simulation experiment shows that the method disclosed by the application has better robustness and expansibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of upper layer cooperative task allocation, and in particular to a cooperative task allocation method for time-sensitive target tracking. BACKGROUND

[0002] There are often various vehicle-mounted mobile targets in modern battlefield, such as Harimau rocket launchers, mobile missile launchers and the like, which are highly mobile and highly concealed. How to quickly and efficiently discover and allocate weapons to attack such time-sensitive dynamic targets is an important task that needs to be solved urgently.

[0003] Attacking dynamic time-sensitive targets is a tricky task in the battlefield. First, the target is difficult to discover because of the "war fog"; second, attacking dynamic time-sensitive targets requires a very high speed of closing the kill chain, and the entire process of F2T2EA of the kill chain needs to be completed quickly. Therefore, the dynamic time-sensitive target task allocation problem needs to be empowered by the latest artificial intelligence technology.

[0004] Task allocation is a classic combinatorial optimization problem, but traditional optimization methods face the problem of "dimension disaster" and real-time when solving combinatorial optimization problems, and the solving speed decreases exponentially with the increase of the problem size. By converting the combinatorial optimization problem into a sequential decision problem, using the MDP model to simulate the dynamic optimization process, using deep reinforcement learning to approximately solve the complex non-convex objective function, and through data-driven autonomous learning of the optimal or suboptimal strategy, the solving efficiency and real-time of the task allocation problem can be improved. For multiple unmanned aerial vehicles and multiple targets, multi-agent deep reinforcement learning technology is naturally used. SUMMARY

[0005] In view of the above problems, the present application takes the multi-target tracking task as the background, studies the unmanned aerial vehicle cluster cooperative tracking of multiple mobile ground targets task allocation problem, and proposes a cooperative task allocation method for time-sensitive target tracking.

[0006] The technical scheme adopted by the present application is as follows:

[0007] A cooperative task allocation method for time-sensitive target tracking, characterized in that it comprises the following steps:

[0008] Step 1: model the unmanned aerial vehicle cluster cooperative task allocation problem as a Dec-POMDP model;

[0009] Step 2: design the state space, action space and reward function of the Dec-POMDP model to obtain the final optimization objective function of the multi-target tracking task;

[0010] Step 3: use the FMASAC algorithm to solve the optimization objective function of the multi-target tracking task;

[0011] Step 4: output the optimal allocation scheme.

[0012] Further, the specific steps of step 2 include:

[0013] Step 21: record the observation vector of the UAV as wherein, is the communication information received by the UAV i from the UAV j, is the speed of the UAV j in the (x, y) direction; is the perception vector of the UAV i to the target k;

[0014] Step 22: discretize the heading angle speed of the UAV into N a discrete values:

[0015]

[0016] Step 23: set the tracking reward r i tracking , the collision penalty r i collision , and the out-of-bound penalty r i bound , and then obtain the team reward function of the UAV cluster according to r i tracking , r i collision , and r i bound , and obtain the final reward function in combination with the spatial entropy reward;

[0017] Step 24: obtain the optimization objective function of the multi-target tracking task according to the final reward function and the temperature parameter.

[0018] Further, the specific steps of step 23 include:

[0019] Step 231: the closer the distance between the UAV and the target, the higher the reward, so the tracking reward is defined as:

[0020]

[0021] wherein d i,k represents the distance between the UAV i and the target k;

[0022] Step 232: if the relative distance between the UAVs is less than the safe distance, a penalty is imposed, so the collision penalty is defined as:

[0023]

[0024] wherein d i,j represents the distance between the UAV i and the UAV j, and ds represents the safe distance between UAVs for collision avoidance;

[0025] Step 233: If the UAV flies out of the boundary, it will be punished for invalid search and tracking, so the out-of-boundary punishment is defined as:

[0026]

[0027] where d i,O represents the distance from UAV i to the center of the scene, d bound is the width of the scene;

[0028] Step 234: Based on the complete cooperation relationship between the UAV cluster, the team reward shared by the entire cluster is defined as:

[0029]

[0030] Step 235: In order to make the distance between UAVs as large as possible, the spatial entropy reward r SE is defined as:

[0031]

[0032]

[0033] where d SE represents the spatial entropy equivalent distance size, represents the expected minimum spatial entropy, represents the spatial entropy, represents the average distance of the UAV cluster,

[0034] Step 236: The final reward function calculation formula is:

[0035]

[0036] where κ = max(1-n e / 1000, 0), n e represents the current round number, and κ adopts linear annealing, decreasing from 1 to 0 after 1000 rounds.

[0037] Further, the optimization objective function of the multi-target tracking task established in step 24 is:

[0038]

[0039] where H represents the entropy of the strategy, r t represents the reward at time t, and α represents the entropy coefficient. θ represents the evaluation strategy.

[0040] Further, the specific steps of step 3 include:

[0041] Step 31: For each agent in the Dec-POMDP model, the actor network selects an action based on the local observation vector and the action at the last time step.

[0042] Step S31: Decompose the centralized critic network into a linear combination of individual critic networks, estimate the Q value of the observation-action pair (τ i ,a i ) through the critic network, and obtain the joint action value function

[0043] where φ represents the network parameters of the centralized critic , φ i represents the network parameters of the individual critic ; g ω is a linear monotonic function; s represents the global state of the environment.

[0044] Step S32: Optimize the function through the TD loss function shown in equation (19) to obtain the optimized critic network, and use the network to evaluate the policy π θ :

[0045]

[0046]

[0047] where equation (20) represents the optimized target critic network; and represent the parameters of the target critic and the hybrid network, respectively; represent the joint target actor parameters;

[0048] Step S33: Based on the policy π θ , use the decomposed critic to train the actors, thereby realizing decentralized execution, and the training formula is:

[0049]

[0050] where g ω represents a single-layer monotonic hybrid network.

[0051] Step S34: Update the actor network by optimizing the loss function to adjust the temperature, thereby updating the actor network:

[0052]

[0053] wherein, is the minimum entropy constraint, satisfying The beneficial effects of the present application are:

[0054] Firstly, the present application proposes an end-to-end FMASAC-based multi-target cooperative tracking task allocation scheme, which can realize the cooperative tracking of multiple targets by the UAV cluster in a distributed manner.

[0055] Secondly, the present application improves the learning efficiency of the algorithm by learning a centralized but decomposed critic and a decomposed policy iteration, and reduces the policy update variance. The SER increases the detection coverage of the UAV cluster and improves the task cooperation performance.

[0056] Thirdly, the FMASAC algorithm proposed in the present application can realize multi-target cooperative tracking task allocation while preventing collision, and is superior to the existing baseline algorithm in terms of average return and tracking success rate, and is significantly superior to the existing popular MARL baseline. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 It is a schematic diagram of cooperative task allocation for a UAV cluster.

[0058] Figure 2 It is the observation range and communication range of the UAV.

[0059] Figure 3 It is the overall framework of the FMASAC algorithm proposed in the present application.

[0060] Figure 4 It is the task allocation visualization result.

[0061] Figure 5 It is the average reward learning curve of different algorithms.

[0062] Figure 6 It is the ablation experiment result.

[0063] Figure 7 (a)- Figure 7 (c) is the UAV-target tracking result; wherein Figure 7 (a) is the UAV-target distance change result in training, Figure 7 (b) is the relative distance change result between UAVs in training, Figure 7 (c) is the tracking success rate change result in training.

[0064] Figure 8 (a)- Figure 8 (b) is the UAV target tracking performance test result; wherein Figure 8(a) is the relative distance performance test result of UAV-target in the test, Figure 8 (b) is the relative distance performance test result of UAV-UAV in the test.

[0065] Figure 9 Relative distance and tracking success rate in robustness test.

[0066] Figure 10 (a)- Figure 10 (b) is the extensibility test result; wherein Figure 10 (a) is the average tracking target number trend chart of different number of UAV scenes, Figure 10 (b) is the tracking success rate trend chart of different number of UAV scenes. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions of the present application will be further described below in combination with the drawings and examples.

[0068] 1. Cooperative task allocation model for time-sensitive targets

[0069] The UAV cluster needs to cooperatively search, discover and allocate tracking targets to complete the task, so as to realize efficient, robust and scalable dynamic task allocation. The schematic diagram of cooperative task allocation of the UAV cluster is shown in Figure 1 .

[0070] The present application describes the task allocation problem as a sequential decision problem, and models the cooperative task allocation problem of the UAV cluster as a Dec-POMDP model, and designs the state space, action space and reward function of Dec-POMDP respectively. Finally, in order to increase the search and tracking coverage of the agent, a sub-target reward function combined with spatial entropy is proposed according to the characteristics of the multi-target tracking task allocation scene, which is used to guide the agent to better and faster realize task allocation.

[0071] (1) Overview of Dec-POMDP model

[0072] In the cooperative multi-target tracking task of the UAV cluster, each UAV only interacts with the environment according to its own local observation and communication information to realize the overall target. This task can be modeled as a typical Dec-POMDP model, denoted as G = <I, S, A, P, R, Ω, O, n, γ>. Wherein, I is the set of agents, S represents the global state of the environment, n is the total number of agents, and γ ∈ [0, 1] is the discount factor. Due to partial observability, at each time step t, each agent i perceives the local observation o i ∈ Ω, and then selects an action a i ∈ A, to form a joint action a ∈ A nWhen the environment receives a joint action, each agent will transition to the next state s′ according to the state transition function P(s′|s,a) and receive a team-shared reward r = R(s,a). Each agent's goal is to determine its local action-observation history τ. i Let's learn a deterministic strategy Or random strategy According to the joint random strategy The joint action value function can be derived. Similarly, based on the joint determination strategy The joint action value function Q can also be derived. tot (τ,a).

[0073] (2) Dec-POMDP modeling

[0074] It mainly includes state space design, action space design, and reward function design.

[0075] ①State Space Design

[0076] Considering the limited radar detection range and communication distance of UAVs, let's assume the maximum observation radius of the UAV is d. o The maximum communication distance is d c ,like Figure 2 As shown. If the distance between drone i and drone j is less than d... c Then, the two drones can promote cooperation by sharing information through communication. The communication information received by drone i can be denoted as... in It is the speed of drone j in the x and y directions. This refers to the drone's action at the previous moment. Similarly, if the distance between drone i and target k is less than d... o If the drone can sense the target and obtain its position and velocity information, then the target information can be denoted as... Therefore, the observation vector of a UAV consists of its position, heading, target information, and communication information, denoted as .

[0077] ② Action Space Design

[0078] The action space of a UAV mainly refers to its directional angular velocity (lateral acceleration). To improve learning efficiency, the directional angular velocity is discretized into N. a There are discrete values, and the maximum heading angular velocity is... The formula is:

[0079]

[0080] Where 'a' represents the final heading angular velocity of the UAV agent, and its value ranges from 1 to 10. n a The output representing the policy.

[0081] ③ Reward function design

[0082] In reinforcement learning, the reward function is the core of guiding the agent to learn, and the reward function greatly influences the learning speed and final performance of the agent.

[0083] For the multi-target tracking task, each UAV needs to learn to search and track targets, i.e., to keep the targets within the observation radius of the UAV. In addition, the UAVs should prevent collision or fly out of the environment boundary. Therefore, the reward function design consists of three parts: tracking reward r i tracking , collision penalty r i collision , and boundary penalty r i bound .

[0084] The tracking reward mainly considers the distance between the UAV and the target, and the closer the distance, the higher the reward. Therefore, the tracking reward is defined as:

[0085]

[0086] where d i,k denotes the distance between UAV i and target k. If UAV i perceives multiple targets, it is encouraged to track the closest target and thus gets a reward of

[0087] The collision penalty mainly considers the relative distance between the UAVs, and if the distance is less than the safe distance, a penalty is received. Therefore, the collision penalty is defined as:

[0088]

[0089] where d i,j denotes the distance between UAV i and UAV j, and d s denotes the safe distance for collision avoidance between UAVs.

[0090] If d i,j ≤ d s , the UAV will receive a collision penalty to improve safety.

[0091] Similarly, the boundary penalty is defined as:

[0092]

[0093] where d i,O denotes the distance from UAV i to the center of the scene, and d bound is the width of the scene. If UAV i flies out of the boundary, it will be penalized due to invalid search and tracking.

[0094] Because the UAV cluster is a complete cooperative relationship, the whole cluster shares a team reward

[0095] (3) Spatial Entropy Reward

[0096] However, due to insufficient exploration, some targets are always missed, that is, the multi-target tracking task is easy to fall into local optimum. To solve this problem, the simplest and most direct method is to encourage UAVs to maintain a certain distance, thereby increasing the spatial distribution and detection coverage of the whole cluster. Therefore, the present application proposes a spatial entropy reward (SER) to encourage UAVs to maintain a certain spatial distance to increase the detection coverage and improve the cooperative multi-target tracking capability.

[0097] First, the definition of spatial entropy is given. Consider a two-dimensional polar coordinate plane (ρ, θ), and the origin is the position coordinate of the UAV. Each position on the plane can be represented by polar coordinates (ρ, θ), where ρ represents the polar radius and θ represents the relative angle. Then, the spatial density function is:

[0098] p(ρ, θ) = Ke -λρ (5)

[0099] where λ is a parameter, K is a normalization constant, and:

[0100]

[0101] Therefore, according to equations (5) and (6), we have:

[0102]

[0103] where represents the generalized radius related to the spatial density, and the spatial entropy can be defined as:

[0104]

[0105] Obviously, increases monotonically with .

[0106] Therefore, the size of can be controlled by adjusting the size of the entropy:

[0107]

[0108] where represents the expected minimum spatial entropy, and d SE represents the size of the spatial entropy equivalent distance.

[0109] Based on and d SE Definition, the concept of spatial entropy reward r SE is proposed, mainly considering that the distance between UAVs is as large as possible, and the calculation formula is:

[0110]

[0111] wherein, represents the average distance of the UAV cluster. If r SE >1.

[0112] Therefore, the spatial entropy reward r SE can encourage UAVs to maintain a distance as far as possible to increase the detection coverage.

[0113] Therefore, the final reward function is obtained:

[0114]

[0115] wherein, κ=max(1-n e / 1000,0), n e represents the current round number, and κ adopts linear annealing, decreasing from 1 to 0 after 1000 rounds, so as to appropriately increase the influence of the spatial entropy reward r SE to improve the tracking task performance.

[0116] Therefore, the optimization objective function of the multi-target tracking task can be represented as:

[0117]

[0118] wherein, represents the entropy of the strategy, r t represents the reward at time t, and α represents the entropy coefficient.

[0119] Next, the present application proposes a new maximum entropy MARL algorithm combining the value decomposition idea to solve this problem.

[0120] 2、FMASAC algorithm

[0121] (1) MASAC algorithm

[0122] SAC is an off-policy maximum entropy framework RL algorithm, which realizes the simultaneous maximization of cumulative return and the entropy of the strategy to encourage exploration. SAC optimizes the critic network Q φ by minimizing the temporal difference (TD) error loss function:

[0123]

[0124] where, D denotes past experience replay, φ, and denote the parameters of critic, target critic and target actor networks, respectively.

[0125] SAC introduces an entropy term with a soft action-value function Policy objective function J(π θ ) is optimized by gradient ascent as follows:

[0126]

[0127] MASAC is an MARL algorithm that extends SAC to the multi-agent framework, adopts the CTDE paradigm, and learns stochastic policies, which can solve both discrete and continuous action space problems. The loss function shown in equation (13) is optimized jointly using the parameter sharing method as the loss function shown in equation (15):

[0128]

[0129] where, Here, denotes the central action-value function of agent i, and the individual policy is updated by gradient ascent as follows:

[0130]

[0131] where all actions are sampled from the current policy of the agent to prevent policy overfitting. However, since the size of the joint action space grows exponentially with the number of agents, MASAC cannot be directly extended to large-scale joint action space problems.

[0132] (2) FMASAC algorithm

[0133] To solve the scalability problem of MASAC, the invention introduces the value decomposition idea and proposes a factored multi-agent soft actor-critic (FMASAC) algorithm. To solve the high variance and credit assignment problem of the cooperative multi-agent actor-critic framework, the centralized critic is decomposed into individual critics, and each agent is assigned credit to improve the policy and increase the reward. Then, a decomposed soft policy iteration is proposed to implement off-policy learning of multi-agent stochastic policy gradient. The complete process of the FMASAC algorithm is shown in the pseudo-code in Table 1.

[0134] ① Learning-centered decomposition of critic

[0135] Considering the explosion of the joint state-action space due to the increasing number of agents, learning a centralized critic is extremely difficult. This invention combines value decomposition and the MASAC framework to achieve scalable learning of a centralized critic under Dec-POMDP. The centralized critic is decomposed into a linear combination of individual critics:

[0136]

[0137] Where φ represents the centered critic Network parameters, φ i Indicates individual criticism Network parameters. ω Let represent a linear monotonic function, parameterized as a hybrid network without activation functions, and s represent the global state of the environment. The linear monotonic function increases the network's expressive power, enabling the utility of individual agents to be fitted to a centered critic or joint action value function. Furthermore, monotonicity guarantees the individual global maximum (IGM) principle, ensuring consistency between maximizing the joint action value function and maximizing the individual action value function, as shown in Equation (18).

[0138]

[0139] To evaluate strategy π θ The critic network is optimized using the following TD loss:

[0140]

[0141]

[0142] Wherein, formula (20) represents the target critic network. and These represent the parameters of the target critic and the hybrid network, respectively. This represents the joint target actor parameter.

[0143] ② Decompose the flexible strategy iteration

[0144] Following an end-to-end approach, this paper utilizes a decomposed critic to train actors for decentralized execution, proposing a decomposed soft iterative strategy. The training formula is as follows:

[0145]

[0146] Among them, g ωThe single-layer monotonic mixing network is represented, which is consistent with formula (17). The weights and bias of the neural network can be generated according to the input s, and then the utility is used to allocate credit, and the above formula is derived in detail:

[0147]

[0148] As can be seen from formula (23), this is consistent with the original / standard SAC update method.

[0149] Similar to the automatic update of individual temperature parameters in the previous literature, the automatic update of the global temperature parameter alpha is considered as a constrained optimization problem, and the temperature adjustment is realized by optimizing the loss function:

[0150]

[0151] Wherein, The minimum entropy constraint satisfies Ensure the randomness of the strategy. In addition, if the strategy satisfies the entropy constraint, alpha will decrease; otherwise, alpha will increase to encourage the strategy to explore more. Therefore, the entropy constraint encourages exploration while ensuring that the strategy converges to the optimal.

[0152] ③Unmanned aerial vehicle cluster multi-target tracking task allocation based on FMASAC

[0153] The overall architecture of the FMASAC algorithm proposed by the application is as shown in Figure 3 Combining Figure 1 , for each agent, its actor network selects the action a according to the local observation and the action at the last moment. At the same time, there is a critic network to estimate the Q value of the observation-action pair (tau i ,a i ), and finally the joint action value function is formed by combining the linear monotonic mixing function. Optimization is performed through the TD learning method of formula (19). Then, each optimized updates the actor network through the decomposition of the flexible policy iteration.

[0154] The detailed flow of the FMASAC algorithm is shown in Table 1 and is roughly divided into three parts: an initialization stage, an experience collection stage, and a network updating stage. First, all parameters are initialized. Then, in the experience collection stage, all unmanned aerial vehicle agents interacting with the environment collect and store experiences according to their own strategies. The actor and critic networks are optimized in the network updating stage, so that each unmanned aerial vehicle learns to track the target cooperatively. After training converges, each unmanned aerial vehicle model can achieve the task of autonomous search and tracking of the target, and the entire unmanned aerial vehicle cluster can obtain the ability of cooperative multi-target tracking in an unknown environment.

[0155] Table 1 Pseudo code of FMASAC algorithm

[0156]

[0157]

[0158] Embodiment

[0159] The FMASAC algorithm proposed in the present application is evaluated through simulation experiments, and its performance is compared with three popular MARL algorithms, including rewards in training, tracking performance in testing, etc., to prove the effectiveness, robustness and scalability of the FMASAC algorithm.

[0160] 1. Simulation settings

[0161] Based on Ubuntu 20.04, PyCharm and multi-agent particle environment, the present application constructs a multi-target tracking simulation environment for an unmanned aerial vehicle cluster, and the user interface and scene parameters can be customized and modified. The number of unmanned aerial vehicles and targets and the size of the map can be variable, and can also be further modified according to requirements. The positions of the unmanned aerial vehicles and the targets are randomly initialized in each round, and other environmental parameters are shown in Table 2.

[0162] Table 2 Environmental parameter settings

[0163]

[0164]

[0165] Next, taking the scenario of tracking 3 targets with 3 unmanned aerial vehicles as an example, the performance of the FMASAC algorithm and the MADDPG, MASAC and MAPPO algorithms is tested. The simulation hyperparameter settings are shown in Table 3, and the baseline hyperparameter settings are referred to in reference

[135] . For fairness, the performance of each algorithm is tested on 5 random seeds.

[0166] Table 3 Hyperparameter settings

[0167]

[0168] 2. Effectiveness test

[0169] First, each algorithm is trained for 1005000 steps until the learning curve converges to test the algorithm performance. To intuitively understand whether the UAVs have learned to cooperatively search and track targets, the UAV tracking process after algorithm training is visualized, as shown in Figure 4 .

[0170] Figure 4 It is shown that three UAVs automatically distribute the nearest target to cooperatively track, and each UAV learns to autonomously search and track targets. Next, to compare the performance of the algorithms, the average reward of the UAVs, the relative distance between the UAVs and the target, and the tracking success rate during training are analyzed. Tracking success is defined as tracking more than 70 steps in each round. The tracking success rate is defined as the ratio of the number of successful tracking times to the total number of times in 80 consecutive rounds. Figure 5 The reward learning curves of the four algorithms are shown.

[0171] As can be seen from Figure 5 , the average reward and convergence speed of the FMASAC algorithm are better than those of the other three algorithms, which means that the proposed decomposition critic and flexible policy iteration are effective. Compared with the original MASAC algorithm, the performance improvement of the FMASAC algorithm is more stable, and the policy variance is smaller. This is because MASAC only greedily maximizes the team reward in policy learning, which exacerbates the credit assignment problem by returning the centralized critic to the exploration or suboptimality of the strategy of other agents. Therefore, by learning the decomposed critic, the overall utility information is distributed to each agent to improve the policy, which implicitly realizes the reasonable distribution of multi-agent credit.

[0172] To further demonstrate the effectiveness and value contribution of the corresponding improvements in the FMASAC algorithm, ablation experiments are completed for comparison, and the performance comparison of the complete FMASAC, FMASAC_Fixed_α with a fixed temperature parameter, and FMASAC_NoSER without the SER reward is obtained, as shown in Figure 6 .

[0173] As can be seen from Figure 6 , the performance of the complete FMASAC algorithm is better than that of FMASAC_Fixed_α and FMASAC_NoSER. This shows that the automatic adjustment of the temperature α can better balance exploration and utilization, thereby improving the performance of the policy. In particular, the SER significantly improves the stability and the final performance of the policy by encouraging increased detection coverage, which demonstrates the importance and effectiveness of the SER in the multi-target cooperative tracking task.

[0174] Next, the distance between UAVs and the distance between UAVs-targets and the tracking success rate are analyzed during the training process, and the results are shown in Figure 7

[0175] Figure 7 (a) shows the relative distance between UAVs and targets after 300,000 steps of training, which is less than the observation range of the UAVs, indicating that each UAV has learned to search and track the target. Figure 7 (b) shows that through training, the relative distance between UAVs is greater than the minimum safe collision distance of the UAVs, meaning that the UAVs have learned to track while avoiding collisions. Figure 7 (c) shows the trend of the tracking success rate, the most important indicator in the multi-target tracking task. Through training, the tracking success rate of each UAV reaches 1. Therefore, the three UAVs have learned to search and track targets autonomously while avoiding collisions, verifying the effectiveness of the FMASAC algorithm.

[0176] 3. Robustness test

[0177] Load the trained strategy model (actor network parameters) without further fine-tuning, and test the robustness of the model in an unknown dynamic environment. The positions of the targets of all UAVs are re-initialized and generated, and the test results are shown in Figure 8

[0178] From Figure 8 (a), it can be seen that each UAV can track the target in the unknown environment after about 10 steps, because the UAV strategy model has "seen" thousands of different scenarios during the training process and stores the memory in the form of neural network parameters. At the same time, Figure 8 (b) shows that the relative distance between UAVs is greater than the safe distance, and effective communication is maintained. Therefore, it is concluded that the learned strategy model can be directly generalized to unknown environments without further fine-tuning or retraining, achieving efficient and real-time tracking.

[0179] Figure 9 The relative distance and test tracking success rate in 100 tests are shown. From the violin plot, it can be seen that the UAV-target distance is almost less than the observation range of the UAVs. This result shows that even in unknown scenarios, the trained UAV model can track the target. In addition, the tracking success rate of each UAV in the test is greater than 98%, indicating that the trained strategy model can be generalized to unknown scenarios.

[0180] 4. Scalability test

[0181] To further test the scalability of the FMASAC algorithm in large-scale scenarios, further simulations of the multi-target cooperative tracking task assignment are completed in a 5-UAV scenario and a 10-UAV scenario. Figure 10 ​​The average tracking target number and tracking success rate of different algorithms are shown, and the detailed results are shown in Table 4.

[0182] Table 4 Comparison of statistical results of different scale scenarios

[0183]

[0184] From Figure 10 It can be concluded from Table 4 that in the above three scenarios, the FMASAC algorithm achieves the highest average reward and the lowest standard deviation. In addition, the average tracking target and tracking success rate during training are almost better than other baselines. Therefore, the performance of the FMASAC algorithm is better in the three scenarios. This also verifies that the decomposition of the critic and the flexible policy iteration reduce the complexity of the joint action value function, and realize the scalable learning under the CTDE paradigm by combining the individual utility based on local information. Therefore, it shows that the FMASAC algorithm can maintain good performance with the increase of the number of agents and the joint action space, and can be extended to large-scale scenarios.

[0185] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A collaborative task allocation method for time-sensitive target tracking, characterized in that, Includes the following steps: Step 1: Model the drone swarm collaborative task allocation problem as a Dec-POMDP model; Step 2: Design the state space, action space, and reward function of the Dec-POMDP model to obtain the final optimized objective function for the multi-target tracking task; Step 3: Solve the objective function of the multi-target tracking task using the FMASAC algorithm; Step 4: Output the optimal allocation scheme; Step 3 includes the following specific steps: Step 31: For each agent in the Dec-POMDP model, its actor network is based on the local observation vectors. Actions at the previous moment Choose an action ; Step S31: Decompose the centralized critic network into a linear combination of individual critic networks, and estimate the observation-action pairs through the critic networks. of The value is used to obtain the joint action value function. : （17） in, Centralized critic Network parameters, Indicates individual criticism Network parameters; It is a linear monotonic function; Represents the global state of the environment; Step S32: Apply the TD loss function shown in equation (19) to... The function is optimized to obtain an optimized critic network, and the strategy is evaluated using this network. : （19） （20） （21） Wherein, equation (20) represents the optimized target critic network; and These represent the parameters of the target critic and the hybrid network, respectively. Indicates the joint target actor parameter; Step S33: Policy-based Decentralized execution is achieved by training actors using a decomposed critic. The training formula is as follows: （22） in, This represents a single-layer monotonic hybrid network; Step S34: Temperature adjustment is achieved by optimizing the loss function, thereby updating the actor network. （24） in, It is a minimum entropy constraint, satisfying .

2. The collaborative task allocation method for time-sensitive target tracking as described in claim 1, characterized in that, Step 2 includes the following specific steps: Step 21: Denote the observation vector of the UAV as ,in, For drones Received drone Communication information, It is a drone exist Velocity in the direction; For drones For the target The perceptual vector; Step 22: Discretize the UAV's heading angular velocity into... Discrete values: （1）； Step 23: Set up tracking rewards Collision penalty and out-of-bounds penalty And then according to , as well as The team reward function of the drone swarm is obtained, and then combined with the spatial entropy reward to obtain the final reward function; Step 24: Based on the final reward function and temperature parameters, obtain the optimized objective function for the multi-target tracking task.

3. The collaborative task allocation method for time-sensitive target tracking as described in claim 2, characterized in that, Step 23 includes the following specific steps: Step 231: The closer the drone is to the target, the higher the reward; therefore, the tracking reward is defined as: （2） in, Indicates drone and target The distance between them; Step 232: If the relative distance between drones is less than the safe distance, a penalty is imposed. Therefore, the collision penalty is defined as: （3） in, Indicates drone and drones The distance between them Indicates the safe distance between drones to avoid collision; Step 233: If the drone flies out of the boundary, it will be penalized for invalid search and tracking. Therefore, the boundary violation penalty is defined as: （4） in, Indicates drone Distance to the center of the scene It is the scene width; Step 234: Based on the full cooperation among drone swarms, the team reward shared across the entire swarm is defined as follows: ； Step 235: To maximize the distance between drones, define a spatial entropy reward. for: （10） （9） in, This represents the equivalent distance magnitude of spatial entropy. This represents the expected minimum spatial entropy. Represents spatial entropy, This indicates the average distance of the drone swarm. ; Step 236: The final reward function calculation formula is as follows: （11） in, , Indicates the current round number. Linear annealing is used, and the value decreases from 1 to 0 after 1000 rounds.

4. The collaborative task allocation method for time-sensitive target tracking as described in claim 3, characterized in that, The objective function for the multi-target tracking task established in step 24 is: （12） in, Entropy represents the strategy. express Rewards at all times Represents the entropy coefficient. This indicates the evaluation strategy.

Images