Maac multi-objective tracking method based on self-distillation algorithm optimization

Abstract

The application discloses a MAAC multi-target tracking method based on a self-distillation algorithm optimization, and comprises the following steps: an intelligent agent is interacted with an environment to generate interactive experience playback, a teacher network and a student network with the same structure are used to learn through a double-strategy circulation mode, and multi-target tracking of the intelligent agent with a limited field of view is completed; wherein the double-strategy circulation mode is that the teacher network generates high-error and high-return experience through interaction with the environment, trains a centralized critic network and guides an actor network to update; the student network learns the value function and the action distribution of the teacher network through a smooth L1 loss and an inverse KL divergence in a distillation stage, and finally covers the parameters of the teacher network to realize multi-target tracking optimization. An independent teacher strategy target critic network is designed, does not participate in circulation covering in the training process, a special loss function design is adopted, the convergence process of the student network is accelerated, and the optimization effect of the strategy is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-target tracking, and particularly relates to a MAAC multi-target tracking method based on a self-distillation algorithm optimization. BACKGROUND

[0002] Multi-target tracking technology in complex environments has always been the focus and difficulty in the tracking field. In the multi-target tracking task, multi-agent reinforcement learning algorithm has gradually become one of the mainstream methods due to its strong adaptive ability and collaboration potential. However, due to the complexity and variability of the environment and the real-time interaction of the algorithm, the optimal training of the multi-agent reinforcement learning algorithm still faces many challenges. Among them, the primacy bias (PB) problem is one of the key factors affecting the tracking accuracy. Primacy bias refers to the fact that due to the imperfection of the initial strategy or the lack of local observation information, the agent may deviate from the optimal path in the early decision-making, triggering a chain reaction, leading to low learning efficiency, and even the strategy falls into a local optimal solution. Especially in the multi-agent collaboration scenario, the primacy bias problem will be amplified, not only weakening the performance of a single agent, but also possibly leading to inconsistency of the global belief space, significantly reducing the overall performance of the system. Therefore, solving the primacy bias problem is of great significance to improve the accuracy and robustness of multi-target tracking.

[0003] In recent years, with the rapid development of reinforcement learning, research on the primacy bias problem in single-agent environments has gradually emerged, and various solutions have been proposed. Nikishin et al. proposed a method of periodically reinitializing the last few layers of neural network parameters to alleviate the bias that may accumulate in the early training. This method weakens the excessive influence of early decision-making on the model by reinitializing part of the network parameters, effectively avoiding the problem of the model converging to a suboptimal strategy too early. Qiao et al. further explored the reinforcement learning framework based on the world model and proposed a world model resetting strategy, which periodically resets the model parameters to clear the overfitting memory, enabling the agent to fully utilize high-quality data later, thereby improving the effectiveness of strategy learning. In addition, Li et al. introduced a self-distillation mechanism in single-agent control tasks, which effectively alleviates the primacy bias problem and further improves the training efficiency and strategy performance through the alternative update of teacher-student strategies.

[0004] In summary, although much progress has been made in the research on the first-cause bias problem in single-agent reinforcement learning and various effective coping strategies have been proposed, the research in the field of multi-agent reinforcement learning, especially for multi-target tracking tasks, is still relatively scarce. Given the complexity of multi-target tracking tasks and the variability of their environment, it is urgent to design more refined and highly operational algorithm optimization methods to better cope with the challenges posed by the first-cause bias problem in multi-agent collaboration and multi-target tracking processes, and to improve the performance of algorithms in practical applications. SUMMARY

[0005] To solve the technical problems existing in the prior art, the present application proposes a MAAC multi-target tracking method based on self-distillation algorithm optimization, which uses a special loss function design to improve the learning efficiency of the student network, thereby accelerating the convergence process of the student network and effectively improving the optimization effect of the strategy.

[0006] To achieve the above purpose, the present application provides a MAAC multi-target tracking method based on self-distillation algorithm optimization, comprising:

[0007] interacting with the environment to generate interactive experience replay, learning by using a teacher network and a student network with the same structure through a double-strategy cycle mode, completing multi-target tracking of the agent with limited field of view;

[0008] Among them, the double-strategy cycle mode is that the teacher network generates first experience and second experience by interacting with the environment, trains the centralized critic network and guides the actor network to update; the student network learns the value function and action distribution of the teacher network through smooth L1 loss and inverse KL divergence in the distillation stage, and finally covers the teacher network parameters to realize multi-target tracking optimization.

[0009] Preferably, learning by using a teacher network and a student network with the same structure through a double-strategy cycle mode comprises:

[0010] initializing the teacher network and the student network respectively;

[0011] interacting the initialized teacher network with the environment to generate interactive experience, evaluating the interactive performance according to a preset reward mechanism, and storing the interactive experience to a priority experience replay pool, wherein the preset reward mechanism includes distance reward, anti-touch reward and boundary reward;

[0012] extracting first experience based on timing difference error and second experience based on reward value from the priority experience replay pool in priority order respectively;

[0013] training the teacher critic network using the first experience and the second experience, and updating the target critic network of the teacher strategy through a soft update strategy;

[0014] Based on the output of the teacher critic network, the action network of each agent is guided to perform policy gradient update;

[0015] In the distillation stage, the student network learns the value function of the teacher network through a smooth L1 loss and learns the action distribution of the teacher network through an inverse KL divergence;

[0016] When the distillation stage ends, the parameters of the student network are overwritten with the parameters of the teacher network, completing the double-policy loop iteration.

[0017] Preferably, the priority experience replay pool is provided with a weight based on a temporal difference error and a weight based on a reward value;

[0018] The weight of the temporal difference error is:

[0019] δ i = r + γ (Q (s t+1 ,a t+1 ; θ t+1 ) - Q (s t ,a t ; θ t ) ) ;

[0020] w δ = | δ i | ;

[0021] In the formula, r is a reward value, γ is a discount factor, Q (s t ,a t ; θ t ) is the evaluation value function of the current network, Q (s t+1 ,a t+1 ; θ t+1 ) is the evaluation value function at the next moment, δ i is the temporal difference error of the i th sample, and w δ is the experience weight of the i th sample.

[0022] The weight based on the reward value is:

[0023]

[0024] In the formula, represents the average reward value of all agents, w r represents the reward value weight of the current experience, N is the total number of agents, and r j is the independent reward value of agent j.

[0025] Preferably, training the teacher critic network using the first experience and the second experience comprises:

[0026] The first experience based on a time difference error and the second experience based on a reward value are trained respectively to obtain a time difference target and an action value function;

[0027] The policy gradient loss and the high reward experience loss of different interaction experiences are calculated respectively by the time difference target and the action value function, the policy gradient loss and the high reward experience loss are summed, and the sum result is used as a loss function of the teacher critic network.

[0028] Preferably, the loss function of the teacher critic network is:

[0029]

[0030] In the formula, L V (θ tea ) is the loss function of the teacher critic network at the current moment, D is the priority experience replay pool, L td (θ tea ) is a standard value loss, L reward (θ tea ) is an additional loss of positive reward.

[0031] Preferably, the soft update policy is:

[0032] θ target ←τ·θ critic +(1-τ)·θ target ;

[0033] In the formula, θ target is the parameter of the target critic network, τ∈(0,1) is an update factor, and θ critic is the parameter of the current critic network.

[0034] Preferably, the student network learns the value function of the teacher network through a smooth L1 loss, and the value function is:

[0035]

[0036] In the formula, Q tea (s t ,a t ) is the Q value of the teacher network at the current moment; Q stu (s t ,a t ) is the Q value estimation of the student policy at the current moment, and δ is a threshold of switching sum of squares and linear loss. is the loss function of the student policy critic network.

[0037] Preferably, the action distribution of the teacher network is learned through inverse KL divergence, and the action distribution is:

[0038]

[0039] wherein, alpha is an amplification factor, pi tea (a t |s t ) is the action probability distribution of the teacher policy, pi stu (a t |s t ) is the action probability distribution of the student policy, is the loss function of the student policy action network.

[0040] Preferably, the method for covering the parameters of the student network with the parameters of the teacher network is:

[0041] theta tea ← theta stu ;

[0042] wherein, theta stu is the parameter of the student policy, and theta tea is the parameter of the teacher policy.

[0043] Preferably, the method is applied to a distributed partially observable Markov decision process framework.

[0044] Compared with the prior art, the present application has the following advantages and technical effects:

[0045] (1) The present application uses the self-distillation technology to reduce the deviation and overfitting of the model in the training process by letting the model learn by itself, that is, using the output of its previous version as the "teacher" of the current model to guide the learning, thereby reducing the influence of the first cause bias problem in the traditional reinforcement learning, helping the agent to converge to an effective policy more quickly, and effectively improving the accuracy and generalization ability of the policy;

[0046] (2) In the simulation application of multiple unmanned aerial vehicles chasing multiple targets, the present application effectively solves the problems of limited field of view and multi-target cooperation, making the training process more efficient and stable. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their description are used to explain the application and are not intended to limit the application. In the drawings:

[0048] Figure 1 is a flow chart of the MAAC multi-target tracking method based on the self-distillation algorithm optimization of the embodiment of the present application;

[0049] Figure 2 is a schematic diagram of the complete distillation process of the embodiment of the present application;

[0050] Figure 3 Figure 1 is a limited field of view multi-agent actual tracking graph for an embodiment of the present application;

[0051] Figure 4 Figure 5 is a comparison chart of the average reward of the MAAC algorithm before and after optimization of the self-distillation algorithm for an embodiment of the present application;

[0052] Figure 5 Figure 6 is a comparison chart of the average tracking number of the MAAC algorithm before and after optimization of the self-distillation algorithm for an embodiment of the present application;

[0053] Figure 6 Figure 7 is a comparison chart of the student actor network loss function at different training stages for an embodiment of the present application. DETAILED DESCRIPTION

[0054] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0055] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0056] The technical terms in the present application are explained as follows:

[0057] MAAC (Multi-Agent Actor-Critic) is a multi-agent reinforcement learning algorithm that combines Actor-Critic method and policy gradient method, aiming to solve multi-agent problems. In MAAC, each agent has its own Actor (policy network) and Critic (value network), through which MAAC can learn the policy of each agent to make the whole system achieve optimal performance.

[0058] Core concepts and algorithm principles of MAAC:

[0059] Agent: an entity that can interact with the environment and take actions.

[0060] State: a description of the environment, used to represent the current situation.

[0061] Action: an action that an agent can take.

[0062] Reward: feedback given by the environment after an agent takes an action.

[0063] Policy: The probability distribution of actions an agent takes in a given state.

[0064] Value Function: The expected value of cumulative rewards under a given policy in a given state.

[0065] Policy Gradient Method: Learning by optimizing the policy through gradient descent.

[0066] Actor-Critic Method: Combining policy gradient method and value function estimation to learn both policy and value function.

[0067] The embodiment proposes a MAAC multi-target tracking method based on self-distillation algorithm optimization, including:

[0068] The agent interacts with the environment to generate interactive experience replay, and learns through a double policy cycle mode using a teacher network and a student network with the same structure to complete multi-target tracking of the limited field of view agent.

[0069] The double policy cycle mode is that the teacher network generates high error experience (i.e., first experience) and high return experience (i.e., second experience) through interaction with the environment, trains the centralized critic network and guides the actor network to update; the student network learns the value function and action distribution of the teacher network through smooth L1 loss and inverse KL divergence in the distillation stage, and finally covers the parameters of the teacher network to realize multi-target tracking optimization.

[0070] Specifically, learning through a double policy cycle mode using a teacher network and a student network with the same structure includes:

[0071] (1) initializing the teacher network and the student network, respectively;

[0072] By initializing two neural networks with the same structure, a parameter coverage cycle mechanism can be achieved, thereby meeting the requirements of the self-distillation algorithm. Specifically, this method updates the parameters of one network to another network at regular intervals, thereby achieving knowledge transfer and enhancement, and promoting the network to gradually optimize its performance during training.

[0073] (2) distillation interval, i.e., stopping the student network learning and only using the teacher network learning:

[0074] During the entire training process, only the teacher network needs to interact with the outside world to obtain experience data, while the student network does not need to directly interact with the environment. The learning process of the student network mainly relies on periodically obtaining parameters from the teacher network, and then optimizing its own policy or prediction ability by imitating the performance of the teacher network.

[0075] (3) After the multi-agent interacts with the environment, the interaction performance is evaluated according to the reward value, and two batches of interaction experiences are generated and stored in the experience replay pool:

[0076] The reward value is set as the distance reward R track , the anti-touch reward R collosion , and the boundary reward R boundary . The three reward values are used to evaluate the reward and punishment of the agent in each interaction, and the priority weight w i and the high return weight w r are used as value standards and stored in the experience replay pool.

[0077] (4) The experiences based on the time difference error and the high return experiences based on the reward value are taken out from the priority experience replay pool in the order of priority:

[0078] (5) The interaction experiences taken out are used to train the teacher critic network based on the following formula:

[0079]

[0080] Where L V (θ tea ) represents the loss function of the critic network at the current time, D is the priority experience replay pool, L td (θ tea ) is the standard value loss, and L reward (θ tea ) is the additional loss of positive reward.

[0081] Through the calculation of the standard value function and the additional value return calculation of the high return experience, the rapid convergence of the teacher network based on the current strategy can be promoted. The rapid convergence of the teacher network can provide more accurate and effective guidance for the student network, thereby accelerating the learning process of the student network and enabling it to better imitate the teacher network and improve its performance.

[0082] (6) The target critic network (Target critic) is updated using a soft update policy:

[0083] By setting the target critic network, the instability of the policy caused by frequent updates of the teacher network can be avoided, thereby improving the reliability of the distillation signal. The target network is updated by a soft update policy, which smooths the value estimation and avoids overestimation. The soft update policy is given by the following formula:

[0084] θ target ←τ·θ critic +(1-τ)·θ target ;

[0085] where θ target is the parameter of the target critic network, τ ∈ (0, 1) is the update factor, and θ critic is the parameter of the current critic network.

[0086] (7) The critic network guides the respective actor network of different agents to update based on the following formula:

[0087]

[0088] where L π (θ tea ) is the loss function of the teacher critic network, and L policy (θ tea ) is the standard policy gradient loss.

[0089] (8) The teacher network continues to update and waits for the distillation stage to arrive.

[0090] (9) The distillation interval ends, and the teacher network has accumulated enough training knowledge and reached a relatively stable performance. The distillation stage officially begins, and the student network starts to improve its ability by learning the knowledge of the teacher network and approaches the convergence state without interacting with the outside world.

[0091] (9a) The student network learns after the teacher network completes an update, and the teacher policy still maintains continuous updating. The difference is that the student network learns based on the current optimal state of the teacher network, i.e., after the teacher network completes an update cycle, the student network obtains knowledge from it and optimizes its own policy.

[0092] (9b) The student network learns the value function of the teacher policy based on the smooth L1 loss (Huber Loss), and the formula is as follows:

[0093]

[0094] where Q tea (s t ,a t ) is the Q value of the teacher network at the current time; Q stu (s t ,a t ) is the Q value estimate of the student policy at the current time, and δ is the threshold of the switching square and linear loss.

[0095] (9c) The student policy learns the action distribution of the teacher policy based on the inverse KL divergence, and the formula is as follows:

[0096]

[0097] where a is a scaling factor that increases the influence of the inverse KL divergence on the loss function, and tea (a t |s t ) is the action probability distribution of the teacher policy, and stu (a t |s t ) is the action probability distribution of the student policy.

[0098] (10) Determine whether it is the end of the distillation stage, if it is the end, use the student policy parameters to cover the teacher policy parameters.

[0099] Further, the method is applied to a distributed partially observable Markov decision process framework.

[0100] Embodiment one,

[0101] In the multi-target tracking task, multiple agents need to cooperate to effectively complete the tracking of different targets. To meet this demand, the technical solution of the embodiment is widely used in the following scenarios:

[0102] 1. Limited field of view multi-UAV pursuit of multiple targets: In the application of limited field of view multi-UAV pursuit of multiple targets, each UAV can only perceive target information within its limited range, and cannot directly obtain global information. In this case, each agent not only needs to have strong local perception ability, but also needs to realize effective tracking and coverage of multiple targets through cooperation between agents and fusion of local information. In order to improve the tracking performance of agents in such an environment, the new algorithm proposed in this embodiment can provide significant advantages for each agent. Self-distillation can enable each agent to efficiently learn in a local field of view, avoid the problem of confirmation bias, and continuously optimize the strategy through historical experience. At the same time, through the multi-agent cooperation mechanism of the MAAC algorithm, agents can share information, coordinate actions, and flexibly adjust strategies when facing dynamic targets. These advantages together enable agents to achieve efficient and stable multi-target tracking in complex environments.

[0103] 2. Limited field of view multi-intelligent car pursuit of multiple mobile targets: Each intelligent car needs to perceive and predict target trajectories in a complex dynamic environment, while overcoming the limitations of limited field of view, and through cooperation between intelligent cars, effectively track and capture multiple high-speed mobile targets. This task not only requires intelligent cars to have efficient perception ability, but also requires them to quickly respond when facing dynamic targets and cooperatively adjust strategies to ensure the completion of the task. Combined with the method of the embodiment, it can provide stronger strategy optimization and cooperation ability for this task.

[0104] 3. Multi-agent search and rescue: After a disaster such as an earthquake, fire, or flood, search and rescue tasks are usually performed by a multi-agent system composed of multiple drones, robots, or intelligent vehicles. Due to the complex environment of the disaster area, many areas are inaccessible, and information is incomplete, so a single agent cannot complete the entire task. Therefore, multiple agents need to work together, use their own perception capabilities and decision-making mechanisms, quickly and efficiently search, locate trapped personnel, and perform rescue tasks. The algorithm provided in this embodiment can help multiple agents work together to complete the pre-search task.

[0105] 4. Robot border patrol: Border patrol tasks usually involve multiple agents monitoring, patrolling, and responding in real time in a wide and complex border area. The patrol area is large and variable, and may contain multiple potential threats, so the agent must be able to perceive and respond to these threats in real time. Through the present invention, each agent can independently optimize its strategy in the patrol task, while also being able to share critical threat information.

[0106] The following will be described in detail with reference to the accompanying drawings Figure 1 and the accompanying drawings Figure 2 The steps of this embodiment will be further described in detail.

[0107] This embodiment mainly uses the MAAC algorithm combined with self-distillation technology for multi-target tracking. Since the agent is set to track with limited field of view, i.e. needs to change the global state through joint action, a distributed partially observable Markov decision process (Dec-POMDP) is used to build the framework:

[0108] Suppose the tracking process involves N agents interacting with the environment, where each agent can be modeled as a distributed partially observable Markov decision process, and the behavior of each agent is defined as a tuple: <N, S, A, T, R, O, Z, γ>; where N represents the set of n agents, each agent can make decisions based on its own observation information; S represents the global state space, which contains all possible states in the environment; A = {A (1) ,A (2) ,…A (n)} represents the joint action space of all agents, where the action of a single agent i a (i) ∈A (i) ; T is the state transition model, which represents the probability P(s'|s,a) of transitioning from state s to new state s' after performing joint action A; R is the joint reward function, which represents the reward obtained by performing joint action a in state s; O: {O (1) ,O(2) ,…O (n)} is the joint observation of all agents, and o (i) ∈O (i) ,o:{o (1) ,o (2) ,…,o (n)} is the local observation of agent i. Z: o (i) → Z(s, i) represents the local observation model of agent i given the global state s; γ is the discount factor, which measures the weight of future rewards.

[0109] Based on the framework of Dec-POMDP model, the agent obtains its partial observation o (i) , and then constructs a unified belief space through information fusion, that is:

[0110] O = {o (1) ,o (2) ,…,o (n)}(1).

[0111] In the formula, o (1) ,o (2) ,…,o (n) represent the partial observation of each agent with limited field of view, and O represents the unified global observation of all agents. By unifying the belief space of all agents, the unified detection result of all agents for the target can be obtained, and then the multi-agent algorithm processing can be carried out.

[0112] 1) Reward value setting:

[0113] After the basic environment is constructed, the training stage is entered. Assuming that the position of the agent is [x n ,y n ], n is the number of the agent, the position of the target is [x m ,y m ], and m is the number of the target. To guide the policy to converge to the optimal solution, a reward mechanism based on environmental feedback is designed, including distance reward, anti-collision reward, and boundary reward. Among them, the distance reward encourages the agent to approach the target position, guiding it to achieve effective tracking; the anti-collision reward punishes possible agent collision behavior, ensuring the safety of agent cooperation; the boundary reward punishes behavior close to the boundary, prompting the agent to remain within the task area.

[0114] The distance reward is set by the following formula (2):

[0115]

[0116] Where R track is the tracking reward of the current agent, d pThe detection range of the limited field of view of the agent, ||p m -p n || is the Euclidean distance between the agent and the tracking target, by introducing a distance-based reward component, the agent is encouraged to be as close to the target as possible, improving the accuracy of tracking.

[0117] The anti-collision reward is set by formula (3):

[0118]

[0119] where R collosion is the anti-collision reward, d safe is the defined safety distance, is the Euclidean distance between agent i and agent j, by introducing a safety distance-based penalty component, the agent is constrained to maintain a sufficient distance in the shared space, thereby effectively avoiding the risk of collision.

[0120] The boundary reward is set by formula (4):

[0121]

[0122] where R boundary is the boundary reward, d s is the safety distance threshold, d b is the closest distance of the agent to the boundary.

[0123] By introducing a boundary safety distance-based penalty component, the agent's activity range within the task area is constrained, thereby effectively avoiding the risk of border crossing behavior and potential risks.

[0124] 2) Experience replay pool setting:

[0125] After all agents determine the reward value according to the reward value setting, the four-element array <s, a, r, s'> of all agents is obtained, and these four-element array data are combined and stored in the experience replay pool for subsequent training.

[0126] The experience replay pool in reinforcement learning uses sample priority and sampling probability design to make important experience samples more likely to be used for training, thereby improving learning efficiency and stability. Due to the needs of the distillation algorithm, the experience replay reward pool in this embodiment sets two priority weight standards, namely the weight based on temporal difference error (td_error) and the weight based on reward value.

[0127] The temporal difference error weight of all agents is determined by the neural network according to the following formula:

[0128] δi =r+γ(Q(s) t+1 ,a t+1 ;θ t+1 )-Q(s t ,a t ;θ t ))(5);

[0129] w δ =|δ i |(6);

[0130] Where r is the reward value, γ is the discount factor, and Q(s) t ,a t ;θ t Let Q(s) be the current network's evaluation value function. t+1 ,a t+1 ;θ t+1 ) represents the evaluation value function at the next time step, δ i For timing difference error, w δ This represents the weight for the time-series difference error.

[0131] The initial temporal difference error weights are all set to 1, i.e., the maximum value. They are only updated after the neural network update is complete. The weights are changed using the temporal difference error of each empirical step calculated by the neural network, according to the following formula:

[0132]

[0133] in, For the time-series difference error, ε is a small constant to avoid zero weights when the error is zero, α∈(0,1) is the priority parameter, and w i This is the updated timing difference error weight.

[0134] The weight of each agent's reward value is determined by the average reward value of all agents at the same time. The specific rules are as follows: An experience is assigned a weight only when the average reward value is greater than 0; the weight value is equal to the average reward value of all agents for that experience. If the average reward value is less than or equal to 0, the weight is set to 0, indicating that the experience is not included in the weighted processing. The specific formula is as follows:

[0135]

[0136] in, w represents the average reward value of all agents. r This represents the reward value weight of the current experience.

[0137] According to the two priority weights w δ and w r Each value is taken separately, with the same sampling probability method, determined by the following formula:

[0138]

[0139] where w j is the priority weight, and a e (0, 1) is the adjustment priority, which determines the influence of priority on sampling, and P(i) is the sampling probability.

[0140] Since the sampling probability is affected by the priority, the sample distribution is not uniform, thereby introducing bias. Therefore, when updating the gradient, the importance sampling weight is introduced to correct it:

[0141]

[0142] where P(i) is the calculated sampling probability, N is the total number of samples, and b e (0, 1) controls the correction strength, which corrects the bias introduced by the non-uniform sampling probability distribution, so that the learning process remains unbiased.

[0143] According to formulas (5)-(10), two batches of experience D1 and D2 can be obtained, where D1 is the priority interaction experience determined by the time difference error, and D2 is the high reward value interaction experience determined by the reward value, for subsequent training.

[0144] 3) Distillation gap:

[0145] This embodiment adopts a double-strategy cycle mode, which is a teacher strategy and a student strategy, and the two strategies use the same network structure. Among them, the teacher strategy interacts with the environment throughout the process to obtain learning experience; while the student strategy only updates in the distillation stage by learning the parameters of the teacher strategy. Therefore, during the distillation gap before the start of the distillation stage, the teacher strategy needs to accumulate enough experience to meet the demand of the student strategy for high-quality learning samples. Therefore, this embodiment additionally introduces a high-return trajectory, that is, the experience with positive reward value is stored and played back preferentially, thereby accelerating the fitting speed of the teacher strategy for key experience and improving the overall learning efficiency.

[0146] The teacher strategy processes and optimizes by passing in two batches of experience D1 and D2, and the MAAC algorithm used is based on a centralized processing and decentralized execution framework (Centralized Training with Decentralized Execution, CTDE), so the centralized critic network is processed first.

[0147] The target critic network is used to calculate the time difference target, and the action value function is calculated for all possible actions of the next state s', and the maximum value is selected, which represents the optimal value estimate of the next state. Then add the immediate reward r to the discounted optimal value estimate to get the time difference target:

[0148] Q target (s t ,a t )=r+γ·maxQ target (s t+1 ,a t+1 )(11);

[0149] Where Q target (s,a) is the time difference target, r is the joint reward at the current time, and Q target (s t+1 ,a t+1 ) is the estimated action value function after executing the current selected action at the next time.

[0150] The current critic network is used to estimate the action value function (Q value) of the state and action. The input batch of state-action information (s, a) is used to calculate the corresponding current Q value through the critic network These values will be used to calculate the loss function and update the parameters of the critic network, and the Q value calculation formula is as follows:

[0151]

[0152] All experiences are uniformly input to the centralized critic network according to equations (13)-(15) for value estimation. Two batches of experiences are processed in the same way, and the results are summed after processing to serve as the critic network loss function:

[0153]

[0154] Where, represents the total loss function of the critic network, represents the policy gradient loss, represents the high reward value experience loss.

[0155] Calculate the time difference error as the weight update standard of the experience replay pool, and the specific formula is as follows:

[0156]

[0157] Where, is the time difference error, is the current Q value, Q target (s t,a t ) is the time difference objective, both of which have been calculated in the previous formulas (11) and (12).

[0158] The parameters of the current critic network are updated to those of the target critic network using soft update rules:

[0159] θ target ←τ·θ critic +(1-τ)·θ target (17);

[0160] Where, θ target Here are the parameters of the target critic network, τ∈(0,1) is the update factor, and θ critic These are the parameters of the current critic network.

[0161] After processing the centralized commentator network, the advantage function at the current moment is calculated according to the following formula:

[0162]

[0163] in, V(s) is the current Q value. t ) is the state value function, (a t |s t A(s) is the probability of taking an action in the current state. t ,a t ) is the dominant function.

[0164] Then, the actor network for each agent is processed individually, and the action probability distribution in the current state is calculated using the following formula:

[0165] π tea (a t |s t ) = softmax(f(s) t ;θ))(19);

[0166] Where θ represents the network parameters, f(s) t ;θ) is the output of the actor network, π tea (a t |s t ) represents the probability distribution of actions in the current state.

[0167] The actor network is then updated using the following formula:

[0168]

[0169] in, The loss function of the current actor network is logπ tea (a t |s t ) is the logarithm of the action probability, used to calculate the policy gradient; H(π) is the regularization term, which measures the uncertainty of the policy distribution, and encourages the network to output a more diverse action probability distribution; A(s t ,a t ) is the advantage function, and β is a hyperparameter that controls the weight of the regularization term; π tea (a t |s t ) is the action probability.

[0170] 4) Distillation phase:

[0171] In the distillation gap phase, the teacher policy accumulates enough policy experience through continuous interaction with the environment, laying the foundation for the distillation phase. In the subsequent distillation phase, the teacher policy still interacts with the environment to ensure dynamic updating of the policy, while the student policy gradually optimizes itself by learning the parameter distribution of the teacher policy. At the end of the distillation phase, the parameters of the student policy will cover those of the teacher policy, completing a complete distillation process and achieving policy transfer and optimization.

[0172] In this embodiment, the critic network of the student policy is used to mimic the teacher network to make the same decision, and the smooth L1 loss (Huber Loss) is used as the loss function of the critic network of the student policy:

[0173]

[0174] Where Q tea (s t ,a t ) is the Q value of the teacher network at the current time, Q stu (s t ,a t ) is the Q value estimate of the student policy at the current time, and δ is the threshold for switching the square and linear loss.

[0175] The actor network of the student policy learns the action probability distribution output by the teacher policy. Since the action probability is normalized, the value is relatively small, and the relative multiple inverse KL divergence (Reverse Kullback-Leibler Divergence) is selected as the loss function of the critic network:

[0176]

[0177] Where α is the amplification coefficient, which increases the influence of the inverse KL divergence on the loss function, and π tea (a t |st ) is the action probability distribution of the teacher policy, π stu (a t |s t ) is the action probability distribution of the student policy.

[0178] In the distillation phase, the student policy learns the teacher policy after each interaction of the teacher policy with the outside world until the distillation is completed, the parameters of the student policy are used to cover the teacher policy, and a complete distillation is realized. The parameter coverage is shown in the following formula:

[0179] θ tea ←θ stu (24);

[0180] Where θ stu is the parameter of the student policy, and θ tea is the parameter of the teacher policy.

[0181] Simulation environment:

[0182] The simulation running system of the embodiment is Intel(R) Core(TM) i5-10400 CPU@2.90GHz 2.90GHz, 64-bit Windows 10 operating system, and Python 3.8 environment is used for simulation;

[0183] The specific parameter settings are as follows:

[0184] The number of simulated unmanned aerial vehicles is 5, the number of simulated targets is 4, the size of the simulated map is 2500*2500, the limited field of view radius of the unmanned aerial vehicle is 350, the training round is 100 rounds, the training step length of each round is 2400 steps, the distillation interval is 1200 steps, the distillation phase is 1200 steps, the critic network learning rate is 5e-5, the actor network learning rate is 1e-5, the action space is a 12-dimensional discrete action space, and the number of Monte Carlo runs is 100.

[0185] The effects of the embodiment will be further described below in combination with simulation experiments and the accompanying drawings.

[0186] The accompanying Figure 3 demonstrates the application demonstration of the MAAC multi-target tracking algorithm based on self-distillation optimization in actual tracking. Five limited field of view agents track four targets in the scene, and the agents with limited circular field of view are shown by thick lines, and the targets are shown by thick lines. Through the experiment, it can be observed that when the agent identifies the target within its detection range, it will immediately start tracking the target; and the agent that fails to detect the target will perform autonomous search in the environment until the target is found and the tracking task is completed. This process fully demonstrates the effectiveness of the algorithm in the multi-target tracking task and the cooperative ability of the agents.

[0187] the accompanyingFigure 4 and the attached Figure 5 respectively show the comparison of the algorithm effect after self-distillation optimization and the original algorithm effect, the attached Figure 4 is the average reward value comparison, the attached Figure 5 is the average tracking target quantity comparison. In the figure, the triangular dotted line represents the optimized algorithm, and the square dotted line represents the original algorithm. The comparison is the average result after 100 times of Monte Carlo. From the figure, it can be seen that whether it is the average reward value, the average tracking target quantity, or the convergence speed of the algorithm, the optimized algorithm is better than the original algorithm. These results fully prove the significant effect of self-distillation optimization in improving the performance of the algorithm and verify its effectiveness. Especially the significant improvement of the convergence speed, which shows that the application has achieved good results in alleviating the first cause bias problem and can effectively reduce the training difficulty caused by improper initial strategy selection, and speed up the convergence of the learning process.

[0188] The attached Figure 6 shows the loss function (i.e. inverse KL divergence) comparison of the student behavior network at different stages of training. Specifically, the average value of the inverse KL divergence after 100 times of Monte Carlo sampling is selected as an index, and the inverse KL divergence of the 5th round and the 100th round is compared and analyzed. Since the action probability distribution is normalized, the original value of the inverse KL divergence is relatively small, so it is enlarged by 100 times to enhance its optimization effect when calculating the loss function. From the figure, it can be seen that in the early stage of training, the 5th round, the inverse KL divergence is large and the distribution is relatively chaotic, indicating that the difference between the student network and the teacher network is significant. With the deepening of training, in the 100th round, the inverse KL divergence is significantly reduced, and the distribution tends to be stable, indicating that the similarity of the behavior strategy between the student network and the teacher network is significantly improved, verifying the effectiveness and optimization effect of the training process.

[0189] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A MAAC multi-target tracking method based on self-distillation algorithm optimization, characterized in that, include: The agent interacts with the environment to generate interactive experience playback. It learns through a dual-strategy cyclic mode using teacher and student networks with the same structure to complete multi-target tracking of the agent with limited field of vision. The agent is an entity that interacts with the environment and takes actions. The dual-strategy cyclic mode involves the teacher network generating first and second experiences through interaction with the environment, training a centralized critic network, and guiding the actor network to update. During the distillation phase, the student network learns the value function and action distribution of the teacher network through smoothed L1 loss and inverse KL divergence, ultimately covering the teacher network parameters to achieve multi-objective tracking optimization. Learning using a dual-strategy cyclical model with identical teacher and student networks includes: Initialize the teacher network and the student network respectively; The initialized teacher network interacts with the environment to generate interactive experiences. The interactive performance is evaluated according to a preset reward mechanism, and the interactive experiences are stored in a priority experience replay pool. The preset reward mechanism includes distance reward, anti-touch reward, and boundary reward. Extract the first experience based on the temporal differential error and the second experience based on the reward value from the priority experience replay pool in order of priority. The teacher-critic network is trained using the first and second experiences, and the target-critic network of the teacher strategy is updated using a soft update strategy. Based on the output of the teacher critic network, the agent network of each agent is guided to update the policy gradient. During the distillation phase, the student network learns the value function of the teacher network through smoothed L1 loss and learns the action distribution of the teacher network through inverse KL divergence. When the distillation phase ends, the parameters of the student network are overwritten with the parameters of the teacher network to complete the dual-strategy iterative cycle. The distance bonus is set using the following formula (2): （2）； in, The current tracking reward for the agent. For the limited field of view detection range of the intelligent agent, The Euclidean distance between the agent and the target being tracked; The anti-touch reward is set using formula (3): （3）； in, To prevent the reward from being triggered, For the defined safe distance, For intelligent agents and intelligent agents Euclidean distance; The boundary reward is set using equation (4): （4）； in, For boundary rewards, For safe distance threshold, This is the shortest distance for the agent to reach the boundary.

2. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 1, characterized in that, The priority experience replay pool is configured with weights based on time-series difference error and weights based on reward value; The weight of the timing difference error is as follows: ； ； In the formula, As a reward value, As a discount factor, The current network's evaluation value function is... The evaluation value function for the next time step. Let be the temporal difference error of the i-th sample. The empirical weight of the i-th sample; The weight based on the reward value is: ； In the formula, This represents the average reward value for all agents. This represents the reward value weight of the current experience, where N is the total number of agents. Let be the independent reward value of agent j.

3. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 1, characterized in that, Training the teacher-critic network using the first and second experiences includes: The first empirical method based on temporal difference error and the second empirical method based on reward value are trained separately to obtain the temporal difference objective and action value function; The policy gradient loss and high reward value experience loss for different interaction experiences are calculated using the temporal difference objective and action value function, respectively. The policy gradient loss and the high reward value experience loss are summed, and the summation result is used as the loss function of the teacher critic network.

4. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 3, characterized in that, The loss function of the teacher critic network is: ； In the formula, Let be the loss function of the teacher critic network at the current moment. Prioritize the experience replay pool. For the standard value loss, Additional losses for positive rewards.

5. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 1, characterized in that, The soft update strategy is as follows: ； In the formula, For the parameters of the target critic network, For the update factor, These are the parameters of the current critic network.

6. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 1, characterized in that, The student network learns the value function of the teacher network through smoothed L1 loss as follows: ； In the formula, Let Q be the Q value of the teacher network at the current moment; This is an estimate of the Q-value of the student's strategy at the current moment. The threshold for switching the sum of squares linear loss; Let be the loss function for the student policy commentator network.

7. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 6, characterized in that, The action distribution of the teacher network is learned through inverse KL divergence: ； in, This is the magnification factor. The action probability distribution of the teacher's strategy. For the action probability distribution of the student's strategy, Let be the loss function for the student strategy actor network.

8. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 1, characterized in that, The method for overriding the parameters of the teacher network in the student network is as follows: ； In the formula, For parameters of student strategies, These are parameters for teacher strategies.

9. The MAAC multi-target tracking method based on self-distillation algorithm optimization according to claim 1, characterized in that, The method is applied to a distributed partially observable Markov decision process framework.

Images