Multi-target tracking method based on multi-agent collaborative deep reinforcement learning

Through the method of collaborative deep reinforcement learning of multi-agents, a tracking model is constructed using three-dimensional coordinate method and twin neural network, and a task allocation is combined with artificial potential field method, which solves the problem of slow online tracking speed and low accuracy in multi-objective tracking. Achieve more efficient target tracking effects.

WO2024250325A1PCT designated stage expired Publication Date: 2024-12-12YUNNAN MINZU UNIV

Patent Information

Application Number
PCT/CN2023/100423
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-09
Filing Date
2023-06-15
Publication Date
2024-12-12

AI Technical Summary

Technical Problem

The existing multi-objective tracking technology faces challenges such as background messy, lighting changes, motion blur and occlusion, and has slow online tracking speed and low accuracy.

Method used

Using a method based on multi-agent collaborative deep reinforcement learning, a three-dimensional coordinate method modeling environment is used to build a tracking model through a twin neural network, dynamic task allocation and tracking agent collaboration are combined with artificial potential field method, and deep reinforcement learning is used to optimize the tracking trajectory predict.

Benefits of technology

Effectively improve the speed and accuracy of online tracking, and make full use of environmental feedback information through the synergy of multiple agents to achieve more efficient target tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023100423_12122024_PF_FP_ABST
    Figure CN2023100423_12122024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present invention is a multi-target tracking method based on multi-agent collaborative deep reinforcement learning, comprising: using a three-dimensional coordinate method to model an environment to obtain an environment model; acquiring input data on the basis of the environment model; and inputting the input data into a tracking model, and determining a tracking trajectory of a tracking target, wherein the tracking model is used for using a tracking agent to dynamically track position information of the tracking target in the environment model, taking environment feedback information and a current frame of position information of the tracking target at a ratio of 4:6 as input, taking the next frame of position information as output, and determining the tracking trajectory of the tracking target. According to the present invention, the tracking model is constructed on the basis of a twin neural network, the synergistic effect of multiple agents is fully exerted, and the environment feedback information is fully utilized, so that the tracking speed and accuracy of online tracking are effectively improved.
Need to check novelty before this filing date? Find Prior Art

Description

A multi-target tracking method based on multi-agent collaborative deep reinforcement learning Technical Field

[0001] The present invention belongs to the field of target tracking technology, and in particular relates to a multi-target tracking method based on multi-agent collaborative deep reinforcement learning. Background Art

[0002] A multi-agent system is a distributed intelligent system that utilizes multiple agents to collaborate to achieve a global or overall goal, with individual agents responsible for completing local objectives. Due to their distributed nature and ability to handle more complex tasks, multi-agent systems have long been a key area of ​​research in artificial intelligence. Collaboration technology is crucial to multi-agent systems and plays a central role in their operation. Therefore, how to enable multi-agent collaboration to achieve tasks quickly and effectively remains a key concern for researchers.

[0003] The individuals and agents in multi-target tracking bear significant similarities, making it easy to transfer AI methods and models to the problem. Multi-target tracking involves tracking multiple objects of interest in a video while maintaining fixed identities. Simply put, the fundamental task of multi-target tracking is to identify and locate multiple objects of interest in a given video sequence, simultaneously correlating identical objects across different frames and generating their motion trajectories. This approach has proven particularly effective in a wide range of complex computer vision systems, including security, industry, transportation, and military applications. Multi-target tracking can be categorized as either offline or online. The difference lies in the fact that offline tracking leverages the entire video to provide comprehensive spatiotemporal information for tracking, while online tracking, while more widely used in daily life, relies solely on known information, such as the current and previous frames, to construct target trajectories. Both offline and online tracking present numerous challenges, such as background clutter, changing lighting, motion blur, and occlusion, making online tracking difficult, slow, and inaccurate.

[0004] Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-target tracking method based on multi-agent collaborative deep reinforcement learning to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention provides a multi-target tracking method based on multi-agent collaborative deep reinforcement learning, comprising:

[0007] Modeling the environment using a three-dimensional coordinate method to obtain an environment model;

[0008] Acquiring input data based on the environment model; the input data includes environmental feedback information and current frame position information of the tracking target;

[0009] The input data is input into the tracking model to determine the tracking trajectory of the tracking target; the tracking model is constructed based on the twin neural network; the tracking model is used to use the tracking agent to dynamically track the position information of the tracking target in the environmental model, and the environmental feedback information and the current frame position information of the tracking target are used as input in a ratio of 4:6, and the next frame position information is used as output to determine the tracking trajectory of the tracking target.

[0010] Optionally, the environmental model is a three-dimensional coordinate system with a unit length of 1m established with the position of the first tracking target as the origin. The positions of the tracking target and the tracking agent in the environmental model are both represented by six three-dimensional coordinates, and the tracking target and the tracking agent can both perform seven actions: up, down, left, right, shrink, expand, and stay still. In the environmental model, the Euclidean distance is used to measure the distance between the tracking target and the tracking agent.

[0011] Optionally, the environmental feedback information includes: a set of exploration agents, a set of tracking agents, a set of tracking targets, and a set of static obstacles.

[0012] Optionally, the tracking model construction process is: using a twin neural network as the evaluation network and target network in the multi-agent, DQN establishes an experience replay mechanism, and trains the reinforcement learning process through experience replay. The actions are set to up, down, left, right, shrink, expand, and stay still. After the training is completed, the tracking model is obtained.

[0013] Optionally, the training process of the tracking model is as follows: the negative value of the distance between the target and the tracking agent is used as a reward for environmental feedback; the farther the distance between the two, the greater the penalty; the closer the distance, the smaller the penalty; in particular, when the tracking agent reaches the specified target position, that is, the four squares above, below, left and right of the escaper, the reward is 0, that is, no penalty; the capacity of the experience replay pool is defined as C, and when its capacity is exceeded, the experience frames that have not been used or have been used less times are deleted.

[0014] Optionally, the tracking model building process further includes: determining local model parameters;

[0015] The specific steps of determining the local model parameters are as follows:

[0016] Based on the twin neural network model, the state space and behavior space of the tracking agent in the environment and the behavior reward function of the environment on the tracking agent are set;

[0017] Based on the DQN algorithm, initialize the corresponding estimation network and evaluation network for each tracking agent, and assign an initial tracking target to each tracking agent;

[0018] The tracking agent decides the next action to take based on its current state, interacts with the environment, obtains its next state, and dynamically adjusts the tracking target based on the gravity method of the artificial potential field method;

[0019] Storing the decision experience in an experience pool and updating the local twin neural network model according to the stochastic gradient descent method;

[0020] After a certain number of cycles, the local twin neural network model parameters are transmitted to the collaborative platform for subsequent deep reinforcement learning;

[0021] The collaborative platform aggregates the parameters uploaded by all tracking agents and returns the results to each tracking agent for parameter update;

[0022] Each tracking agent will soft-update the received return results and the local twin neural network parameters to obtain the latest local model parameters.

[0023] Optionally, the dynamically tracking the position information of the tracking target in the environment model by using a tracking agent specifically includes:

[0024] Dynamic task allocation in the tracking process is carried out based on the artificial potential field method. By calculating the Euclidean distance between the target and the tracking agent, the target is assigned to the specific tracking agent nearby. Task allocation is divided into two stages: the initial stage and the dynamic adjustment during the tracking process. In the initial stage, each tracking agent is first assigned a tracking target. The tracking agents assigned to the same tracking target reach the front, back, left, and right positions of the corresponding tracking target through internal collaboration, and achieve joint tracking of the target.

[0025] Optionally, the criterion for determining whether the tracking agent has successfully tracked the tracking target is:

[0026] The tracking target is surrounded by tracking agents in front, behind, left and right, or is surrounded by tracking agents at the boundary of the environment model.

[0027] Optionally, the target data set is input into a tracking model to determine the tracking trajectory of the tracking target. The specific process is as follows:

[0028] Step 1 Tracking initialization: Select the first frame, use the detection result as the target tracking position, and then obtain the target appearance features through the twin neural network; the tracking position and appearance features are used to initialize the tracking state to obtain the initial tracking state set;

[0029] Step 2: Motion prediction: Use the historical position of the tracked target to calculate the historical position difference set. Then use the enhanced correlation coefficient method (ECC) to calculate the affine vector between the current frame and the previous frame and add it to the historical affine vector set. Finally, based on the historical position difference set and the historical affine vector set, predict the target's position in the current frame.

[0030] Step 3: Position refinement: The predicted position and the environmental feedback information are input into the tracking model in a ratio of 4:6 to obtain the refined position.

[0031] Step 4: Dynamic task allocation: Based on the artificial potential field method, the tracking agent is made to track the target nearby and update the position information; if there is an unmatched tracking, it is marked as lost tracking;

[0032] Step 5: Person re-identification: Calculate the appearance feature distances between unmatched detections and lost tracks using Euclidean distance, and use the inverse of these appearance feature distances as affinity scores to obtain an affinity matrix. Based on this affinity matrix, perform matching using the artificial potential field method, restore the identities of matched detections, and update their tracking status using the corresponding detections. Terminate tracking if the number of lost frames exceeds a threshold.

[0033] Step 6: Add new tracks: Same as step 1, initialize the detections in step 5 that do not match any tracks as new tracks;

[0034] Step 7: Select the next frame and repeat steps 2-6 until all frames are tracked and the final tracking trajectory is constructed.

[0035] The technical effects of the present invention are:

[0036] The present invention constructs a tracking model based on a twin neural network, uses environmental feedback information and the current frame position information of the tracking target as model input in a ratio of 4:6, gives full play to the synergy of multiple intelligent agents, makes full use of environmental feedback information, and effectively improves the tracking speed and accuracy of online tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0038] FIG1 is a schematic diagram of a basic framework of offline tracking in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] This paper proposes a multi-target tracking method based on multi-agent collaborative deep reinforcement learning. To address the difficulties and low accuracy of online tracking caused by the diverse variations in the targets and their positions, as well as video background clutter, lighting variations, motion blur, and occlusions, the method first uses deep reinforcement learning to learn offline tracking. This method establishes an experience replay pool to store the algorithm's historical states, actions, and rewards. After building a model, it uses a dataset for training and testing. Based on this, trajectory prediction is performed for online tracking using a 4:6 ratio of experience to new environmental information, enabling the agents to take their next action. In multi-agent multi-target tracking, the presence of multiple targets necessitates task allocation. Using an artificial potential field method, targets are dynamically assigned to specific trackers, enabling collaboration between trackers to optimize resource utilization and reduce overall tracking costs. The target's activity area is assumed to be an infinitely large, bounded irregular grid. Successful tracking is criterion when the target is surrounded by tracking agents or is enclosed by tracked agents at the boundary. In order to facilitate the task, there are multiple intelligent agents whose task is to explore the road and intersection conditions ahead. The intelligent agents find an optimal action path through prediction and feedback.

[0041] Example 1

[0042] This multi-target tracking method, based on multi-agent collaborative deep reinforcement learning, first uses a twin neural network for offline tracking learning and stores it in an experience pool. It then uses the twin neural network for online tracking learning, combining the experience pool's knowledge with new environmental information in a 4:6 ratio to predict the online tracking trajectory. Finally, a complete tracking trajectory is obtained. The basic framework of offline tracking is shown in Figure 1.

[0043] First, the environment is modeled. This study uses a three-dimensional coordinate representation method to model the environment. Assuming there is an infinite and bounded three-dimensional space, a three-dimensional coordinate system with a unit length of 1m is established with the position of the first target as the origin. The target in the video frame can be represented by a cube. Therefore, the position information of the target in the environment can be represented by six three-dimensional coordinates. The same is true for the tracking agent. Whether it is the target or the tracking agent, it can perform seven actions such as up, down, left, right, shrink, expand, and stay still. The Euclidean distance is used to measure the distance between the target and the agent, and to prepare for the next task allocation. The basis for judging whether the agent is successfully tracking the target is that the tracking agent is surrounded by tracking agents in front, behind, left, and right, or the tracked agent is blocked at the boundary.

[0044] Secondly, tasks are assigned to the target and tracking agents. Dynamic task allocation during the tracking process is performed based on the artificial potential field method. By calculating the Euclidean distance between the target and the tracking agent, the target is assigned to the nearest specific tracking agent, and collaboration between the agents is achieved to optimize resource utilization and reduce overall tracking costs. Task allocation can be divided into two stages: the initial stage and dynamic adjustment during the tracking process. In the initial stage, each agent can be assigned a tracking target. Agents assigned to the same target can then collaborate internally to reach the front, back, left, and right positions of the corresponding target, thereby achieving a joint tracking of the target.

[0045] Then, a tracking model is built based on deep reinforcement learning. Using twin neural networks as the evaluation network and target network in the multi-agent system, DQN establishes an experience replay mechanism to train the reinforcement learning process through experience replay. The actions are set to 7 actions: up, down, left, right, shrink, expand, and stay still. The negative value of the distance between the target and the tracking agent is used as a reward for environmental feedback. The farther the distance between the two, the greater the penalty; the closer the distance, the smaller the penalty. In particular, when the tracking agent reaches the specified target position, that is, the four squares above, below, left, and right of the escaper, the reward is 0, that is, no penalty. The capacity of the experience replay pool is defined as C. When its capacity is exceeded, experience frames that have not been used or have been used less are deleted.

[0046] Finally, through deep reinforcement learning of offline tracking using the above method, a prediction algorithm is used to predict the online tracking route based on offline tracking. The ratio of accumulated offline tracking experience to new environmental feedback information is 4:6. The model construction for online multi-target tracking, such as environmental modeling and task allocation, is similar to that for offline tracking.

[0047] This study uses a three-dimensional coordinate system to model the environment, performs dynamic task allocation based on the artificial potential field method, builds a tracking model through deep reinforcement learning, and predicts the trajectory of online tracking based on offline tracking. Its core work includes: (1) Based on the position information of each agent and the target in the environment, a multi-agent deep reinforcement learning tracking algorithm is used to track multiple targets and store experience; (2) Based on the prediction model, the target trajectory is predicted with a feedback ratio of 4:6 between experience and new environmental information, which facilitates the multi-agent to perform the next tracking action.

[0048] Therefore, this study mainly includes the following two algorithms:

[0049] 1. Multi-agent deep reinforcement learning tracking algorithm. The specific steps are as follows:

[0050] Step 1: Each tracking agent is based on the twin neural network model, and sets the state space, behavior space, and reward function of the tracking agent in the environment.

[0051] Step 2: Initialize the corresponding estimation network and evaluation network for each tracking agent according to the specific DQN algorithm, and assign the starting tracking target to each tracking agent.

[0052] In step 3, the tracking agent decides the next action to be taken according to its current state, interacts with the environment, obtains its next state, and dynamically adjusts the tracking target based on the gravity method of the artificial potential field method.

[0053] Step 4 stores the decision experience in the experience pool and updates the local twin neural network model in a timely manner according to the stochastic gradient descent method.

[0054] In step 5, after a certain number of cycles, the local twin neural network model parameters are transmitted to the collaborative platform for subsequent deep reinforcement learning.

[0055] Step 6: The collaborative platform aggregates the parameters uploaded by all tracking agents and returns the results to each tracking agent for parameter update.

[0056] In step 7, each tracking agent soft-updates the received return results and the local twin neural network parameters to obtain the latest local model parameters, and repeats the process until all targets are successfully tracked.

[0057] 2. Multi-target tracking trajectory prediction algorithm, the specific steps are as follows:

[0058] Step 1: Tracking Initialization: Select the first frame and use the detection results as the target's tracking position. Then, use the Siamese neural network to obtain the target's appearance features. The tracking position and appearance features are used to initialize the tracking state, which includes the historical position and appearance features of the target. This results in the initial tracking state set.

[0059] Step 2: Motion prediction: The historical position difference set is calculated using the tracked target's historical position. The enhanced correlation coefficient (ECC) method is then used to calculate the affine vector between the current and previous frames and append it to the historical affine vector set. Finally, the proposed algorithm is used to predict the target's position in the current frame based on the historical position difference set and the historical affine vector set.

[0060] Step 3: Position refinement: The predicted position and environmental feedback information are input into the dynamic decision module of deep reinforcement learning at a ratio of 4:6 to obtain the refined position.

[0061] Step 4: Dynamic task allocation: Based on the artificial potential field method, the tracking agent is made to track the target nearby and update the position information. If there is an unmatched track, it is marked as lost track.

[0062] Step 5: Person Re-ID: Calculate the appearance feature distances between unmatched detections and lost tracks using Euclidean distance. The inverse of these appearance feature distances is used as the affinity score to generate an affinity matrix. Based on this affinity matrix, match the detected person using an artificial potential field method, recover the identity of the matched detection, and update the tracking state using the corresponding detection. Terminate the track if the number of lost frames exceeds a threshold.

[0063] Step 6: Add new tracks: Same as step 1, initialize the detections in step 5 that do not match any tracks as new tracks.

[0064] Step 7: Select the next frame and repeat steps 2-6 until all frames are tracked and the final tracking trajectory is constructed.

[0065] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-target tracking method based on multi-agent collaborative deep reinforcement learning, characterized in that: The following steps are involved: Modeling the environment using a three-dimensional coordinate method to obtain an environment model; Acquiring input data based on the environment model; the input data includes environment feedback information and current frame position information of the tracking target; Inputting the input data into a tracking model to determine a tracking trajectory of the tracking target; The tracking model is constructed based on a twin neural network; the tracking model is used to utilize a tracking agent to dynamically track the position information of the tracking target in the environmental model, taking the environmental feedback information and the current frame position information of the tracking target as input in a ratio of 4:6, and taking the next frame position information as output to determine the tracking trajectory of the tracking target.

2. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 1 is characterized in that: The environmental model is a three-dimensional coordinate system with a unit length of 1m established with the position of the first tracking target as the origin. The positions of the tracking target and the tracking agent in the environmental model are both represented by six three-dimensional coordinates, and the tracking target and the tracking agent can perform seven actions: up, down, left, right, shrink, expand, and stay still. In the environmental model, the Euclidean distance is used to measure the distance between the tracking target and the tracking agent.

3. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 1 is characterized in that: The environmental feedback information includes: a detection agent set, a tracking agent set, a tracking target set, and a static obstacle set.

4. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 1 is characterized in that: The tracking model construction process is as follows: using twin neural networks as the evaluation network and target network in multi-agents, DQN establishes an experience replay mechanism, and trains the reinforcement learning process through experience replay. The actions are set to up, down, left, right, shrink, expand, and stay still. After the training is completed, the tracking model is obtained.

5. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 4 is characterized in that: The training process of the tracking model is as follows: the negative value of the distance between the target and the tracking agent is used as the reward of the environment feedback; the farther the distance between the two, the greater the penalty; the closer the distance, the smaller the penalty; in particular, when the tracking agent reaches the specified target position, that is, the four squares above, below, left and right of the escaper, the reward is 0, that is, no penalty; the capacity of the experience replay pool is defined as C, and when it exceeds its capacity, the experience frames that have not been used or have been used less times are deleted.

6. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 1 is characterized in that: The tracking model building process also includes: determining local model parameters; The determination of the local model parameters is specifically as follows: Based on the twin neural network model, the state space and behavior space of the tracking agent in the environment and the behavior reward function of the environment on the tracking agent are set; Based on the DQN algorithm, a corresponding estimation network and an evaluation network are initialized for each tracking agent, and an initial tracking target is assigned to each tracking agent; The tracking agent decides the next action to be taken according to the current state of itself, interacts with the environment, obtains the next state of itself, and dynamically adjusts the tracking target based on the gravity method of the artificial potential field method; The decision experience is stored in the experience pool, and the local twin neural network model is updated according to the stochastic gradient descent method; After a certain number of cycles, the local twin neural network model parameters are transmitted to the collaboration platform. Conduct subsequent deep reinforcement learning; The collaboration platform aggregates the parameters uploaded by all tracking agents and returns the results to each tracking agent for parameter update; Each tracking agent will soft-update the received return results and the local twin neural network parameters to obtain the latest local model parameters.

7. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 1 is characterized in that: The step of dynamically tracking the position information of the tracking target in the environment model using the tracking agent specifically includes: Dynamic task allocation in the tracking process is carried out based on the artificial potential field method. The target is allocated to a specific tracking agent nearby by calculating the Euclidean distance between the target and the tracking agent. Task allocation is divided into two stages: the initial stage and the dynamic adjustment in the tracking process. In the initial stage, a tracking target is first assigned to each tracking agent. The tracking agents assigned to the same tracking target reach the front, back, left and right positions of the corresponding tracking target through internal collaboration, and work together to track the target.

8. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 7 is characterized in that: The criterion for determining whether the tracking agent is successful in tracking the tracking target is: The tracking target is surrounded by tracking agents in front, behind, left and right, or is surrounded by the tracking agents at the boundary of the environment model.

9. The multi-target tracking method based on multi-agent collaborative deep reinforcement learning according to claim 1 is characterized in that: The target data set is input into the tracking model to determine the tracking trajectory of the tracking target. The specific process is as follows: Step 1 Tracking initialization: Select the first frame, use the detection result as the target tracking position, and then obtain the target appearance features through the twin neural network; The tracking position and appearance features are used to initialize the tracking state to obtain the initial tracking state set; Step 2 Motion prediction: Use the historical position of the tracked target to calculate the historical position difference set, then use the enhanced correlation coefficient method ECC to calculate the affine vector between the current frame and the previous frame, and add it to the historical affine vector set; finally, based on the historical position difference set and the historical affine vector set, predict the position of the target in the current frame; Step 3: Position refinement: input the predicted position and the environmental feedback information into the tracking model at a ratio of 4:6 to obtain a refined position; Step 4: Dynamic task allocation: Based on the artificial potential field method, the tracking agent is enabled to track the target nearby and update the position information; If there is an unmatched track, it is marked as a lost track; Step 5 Person Re-ID: Calculate the appearance feature distances between unmatched detections and lost tracks by Euclidean distance, take the inverse of these appearance feature distances as affinity scores, and obtain an affinity matrix; Based on the affinity matrix, use the artificial potential field method to perform matching, restore the identity of the matching detection, and use the corresponding detection to update its tracking status; Terminate the tracking when the number of lost frames exceeds the threshold; Step 6: Add new tracks: Same as step 1, initialize the detections in step 5 that do not match any tracks as new tracks; Step 7: Select the next frame and repeat steps 2-6 until all frames are tracked and the final tracking trajectory is constructed.

Citation Information

Patent Citations

  • Object tracking method and device based on twin matching network

    CN109446889A

  • Multi-target pedestrian trajectory tracking method based on reinforcement learning

    CN112581496A

  • Multi-target cooperative tracking method based on unmanned aerial vehicle group

    CN114756052A

  • Methods for tracing fluid migration

    US20160131626A1

Cited By

  • Pipeline plugging test system and method of plugging device

    CN120102128A

  • A pipeline plugging test system and method for a plugging device

    CN120102128B

  • Multi-objective optimized photoelectric energy-saving system and method

    CN120145881A

  • Task scheduling method and device, medium and product

    CN120353561A

  • MADDPG multi-agent motion control method mixed with three types of experience

    CN120354876A