A multi-target online tracking method with automatic variable tracking mode

By employing an online multi-target tracking method with automatic variable tracking modalities, this method utilizes target detection and neural network evaluation, combined with geometric features and pedestrian re-identification features, to solve the accuracy and efficiency problems of multi-target tracking in complex scenarios, achieving efficient and robust target matching.

CN114332165BActive Publication Date: 2024-11-08SOUTHEAST UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111661412.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-30
Publication Date
2024-11-08
Estimated Expiration
2041-12-30

AI Technical Summary

Technical Problem

Existing multi-target tracking technologies face varying tracking difficulties in complex scenarios due to target interaction and occlusion, and the complexity of the algorithms limits accuracy and efficiency.

Method used

An online multi-target tracking method with automatic variable tracking modality is adopted. This method achieves target matching under different interaction states by target detection, trajectory initialization and preprocessing, Kalman filter prediction, multilayer perceptron network evaluation of tracking modality, selection of simple or complex trackers and data association, and neural network fusion of geometric features and pedestrian re-identification features.

Benefits of technology

It improved tracking accuracy, reduced identity switching errors, enhanced algorithm execution efficiency, and improved robustness in complex scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114332165B_ABST
    Figure CN114332165B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-target online tracking method with automatic variable tracking modes, and the steps are as follows: first, use a target detection algorithm to obtain detection data of each target; perform trajectory initialization and trajectory preprocessing based on a geometric association feature map according to the current frame detection; use Kalman filtering to predict the predicted position of the target on the current frame, and determine the current tracking mode (simple or complex) according to the training mode selector based on the geometric association feature map; finally, the target selects different trackers according to the corresponding tracking mode for target association, and obtains the multi-target tracking trajectory of the current frame. The present invention realizes automatic determination of tracking modes under different interactive occlusion conditions by performing online interactive analysis on the detection data of multiple targets, decouples the complex multi-target tracking problem, and enables targets of different tracking difficulties to adaptively select corresponding tracking modes to efficiently perform data association, thereby improving the efficiency of the tracking algorithm while ensuring the tracking accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of machine learning and computer vision, and in particular to a multi-target online tracking method with automatic variable tracking modes, which is particularly suitable for complex tracking scenarios with severe target interaction occlusion. Background Art

[0002] With the continuous development of computer technology, target tracking technology has gradually become a key development technology direction in intelligent video analysis and multimedia applications (such as intelligent monitoring, motion analysis and autonomous driving). Detecting and tracking multiple targets in complex scenes, performing real-time analysis and prediction of target trajectories, and providing richer data information for practical applications.

[0003] Thanks to the high iteration of computer hardware and deep learning technology, image-based target detection technology has become more and more mature. Accurate target detection results provide more powerful data support for multi-target tracking technology. In actual real-world scenarios, the deployment of multi-target tracking algorithms is often limited by the complexity of the algorithm. Due to the complex interactions and occlusions between targets, the tracking difficulty of different targets varies. If the targets with simple and complex tracking difficulties can be decoupled according to the different interaction states of the targets, and corresponding tracking methods are adopted for different tracking modes, the tracking accuracy of the algorithm can be guaranteed while improving the execution efficiency of the algorithm.

[0004] To address the problem of multimodal multi-target tracking, researchers have proposed a detection-based tracking algorithm paradigm that performs different target matching association algorithms by calculating the geometric features between different targets or using neural networks to extract the similarities between appearance features. Summary of the invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the existing multi-target tracking technology, the present invention provides a multi-target online tracking method with automatic variable tracking mode. The method has high recognition and tracking accuracy, fast algorithm operation speed, and has certain robustness to complex interactions and occlusion changes of the targets.

[0006] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A multi-target online tracking method with automatic variable tracking mode comprises the following steps:

[0008] Step 1: Use the target detector to detect the same category of targets in the current video frame and obtain detection data;

[0009] Step 2: Initialize and preprocess the trajectory of the current frame;

[0010] Step 3: Predict the position of the tracking target in the current frame based on the Kalman filter, and infer the tracking mode (simple or complex mode) of the target based on the multi-layer perceptron network based on the multi-dimensional geometric association feature map;

[0011] Step 4: Select a simple or complex multi-target tracker to perform corresponding data association according to the tracking mode of the target to be tracked;

[0012] Step 5: Post-process the associated trajectories, perform linear interpolation between frames on the trajectories associated across frames, and restore the motion trajectory of the target at the entire timestamp.

[0013] Furthermore, the step 1 specifically includes the following steps:

[0014] Select the target detector YOLOX to perform target detection on the current input video frame. The training of the target detector is based on the public datasets COCO, CrowdedPerson, MOT17, Cityperson, ETHZ and MOT20. The detection category is pedestrian. Get the center point and width and height data of the current M detection data bounding boxes: (x i ,y i ,w i ,h i ), i = 1, ..., M. Where x and y represent the horizontal and vertical coordinates of the center point of the bounding box, and w and h represent the width and height of the bounding box.

[0015] Furthermore, the step 2 specifically includes the following steps:

[0016] Before tracking, the track to be tracked is initialized. If the current frame is the initial frame of tracking, each detection is initialized to the first frame of the track. The track is preprocessed and the tracking quality of the tracked track is evaluated. The track score output by the trajectory evaluator based on the neural network is greater than 0.5, which is a high-quality track. high And the trajectory score less than 0.5 is a low-quality trajectory T low ; Among them, the tracking quality method for evaluating the tracked trajectory is:

[0017] Step 2.1: Based on the detection data in the tracked trajectory set, establish the state equation of the current tracked target, which consists of 7 variables: Where u and v represent the horizontal and vertical coordinates of the center of the object, and the scale and aspect ratio of the detection bounding box are described by s and r respectively; the aspect ratio is considered constant; finally, the Kalman filter is applied to predict the center point of the predicted bounding box of the trajectory in the current frame as well as the width and height data: (x T ,y T ,w T ,h T ).

[0018] Step 2.2: Based on the predicted position (x T ,y T ,w T ,h T ) and the target detection data (x D ,y D ,w D ,h D ) between the geometric correlation feature vector V TD And the geometric correlation feature sequence V between the current trajectory correlation detection TT . V TD and V TT The input is sent to the neural network to evaluate the tracking quality of the current trajectory and output the trajectory score. The geometric association feature consists of 7-dimensional features: g6=IoU(T,D),g7=t D -t T The geometric association feature is expressed as follows:

[0019] G=(g1,g2,g3,g4,g5,g6,g7,g8) (1)

[0020] The trajectory score P is calculated by the following formula, where N1, N2, and N3 are multi-layer perceptrons (MLPs):

[0021] P=N3(N1V TD +N2V TT ) (2)

[0022] Step 2.3: Based on the high-quality trajectory T obtained in step 2.2 high With low quality trajectory T low , if 5 consecutive frames are evaluated as low-quality trajectories, the tracking of the current target is terminated. When executing the subsequent tracking process, the high-quality trajectory T high Execute first, low quality trajectory T low Then execute.

[0023] Furthermore, the step 3 specifically includes the following steps:

[0024] Step 3.1: According to step 2.1, the predicted trajectory position (x T ,y T ,w T ,h T ) and the geometric correlation feature vector V between the candidate target detection data TD And the geometric correlation feature vector V between the last frame of the current tracking trajectory and other detections on this frame TL .

[0025] Step 3.2: Substitute the V obtained in step 3.1TD Update to get vector With V TL After splicing, it is input into the classification neural network to judge the tracking mode that the trajectory should select according to the actual interaction state of the trajectory, and realize the automatic tracking mode selection function. When the category score is less than 0.5, the mode is simple, and when the category score is greater than 0.5, the mode is complex; the mode category score P M Calculated by the following formula, N4, N5, N6 are multi-layer perceptrons MLP:

[0026]

[0027] Furthermore, the step 4 specifically includes the following steps:

[0028] Step 4.1: If the mode obtained in step 3 is simple, select the simple tracker to complete the detection association of the track to be tracked on the current video frame, and obtain the predicted position of the track (x T ,y T ,w T ,h T ) and the geometric correlation feature vector between the updated candidate target detection data Input into the matching neural network to obtain the matching score matrix

[0029] Finally, the score matrix is ​​solved by the minimum flow algorithm to optimize the detection matching and thus realize trajectory tracking. Implemented by Multilayer Perceptron N7:

[0030]

[0031] According to the association matrix A obtained by matching the neural network, the minimum cost flow graph problem is solved; the constraints of the optimization problem include: if the detection is a true positive TP, the detection must be initialized as a new trajectory or matched with the previous trajectory; for a detection in the previous frame and all detections in the current frame; these two constraints can encode all valid trajectories in matrix form; then the data association problem is expressed as an integer linear program as follows:

[0032]

[0033] stAY=0,Y∈{0,1} |Y|

[0034] is a vector consisting of all the prediction scores of the association matrix, and Y represents the linear constraints between the binary variables.

[0035] Step 4.2: If the mode obtained in step 3 is complex, then select the complex tracker to complete the detection association of the track to be tracked on the current video frame, and use the pedestrian re-identification (ReID) neural network to extract the appearance features of each candidate detection to obtain the appearance feature vector V ReID , and then get the predicted trajectory position (x T ,y T ,w T ,h T ) and the geometric correlation feature vector between the updated candidate target detection data Splicing is performed and then input into the matching neural network to obtain the matching score matrix Finally, similar to the simple tracker, the score matrix is ​​solved by the minimum flow algorithm to optimize the detection matching and thus realize trajectory tracking; Implemented by Multilayer Perceptron N8:

[0036]

[0037] Furthermore, the step 5 specifically includes the following steps:

[0038] According to the associated trajectory obtained in step 4, trajectory post-processing is performed, inter-frame linear interpolation is performed on the trajectory associated across frames, the motion trajectory of the target over the entire timestamp is restored, and the interpolated trajectory detection sequence is smoothed.

[0039] Beneficial effects: The multi-target online tracking method with automatic variable tracking mode provided by the present invention has the following advantages.

[0040] 1) Based on the accurate detection results of YOLOX, compared with the intersection-over-union ratio between bounding boxes, multi-dimensional geometric features can better distinguish the geometric similarities between trajectories and different candidate detection boxes.

[0041] 2) Evaluating the tracking quality of the current tracking trajectory based on a neural network can further reduce the identity switching errors of the trajectory.

[0042] 3) By utilizing the actual interactive situation of the trajectory, the neural network-based mode selector can reasonably distinguish tracking targets of different tracking difficulties and reasonably select different trackers, thus greatly improving the execution efficiency of the algorithm while ensuring tracking accuracy.

[0043] 4) The neural network based on the attention mechanism fuses the similarity of geometric features with pedestrian re-identification features. The fused similarity can better match the candidate detections on the connection, enhancing the robustness of the algorithm in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1It is an overall flow chart of the multi-target online tracking method with automatic variable tracking mode provided by the present invention;

[0045] Figure 2 It is a specific flow chart of the multi-target online tracking method with automatic variable tracking mode provided by the present invention;

[0046] Figure 3 It is a schematic diagram of geometric feature extraction between trajectories and candidate detections;

[0047] Figure 4 It is an algorithm execution flow chart of the multi-target online tracking method with automatic variable tracking mode provided by the present invention;

[0048] Figure 5 This is a schematic diagram of geometric feature extraction for person re-identification;

[0049] Figure 6 It is a tracking effect diagram and a data set tracking result of the multi-target online tracking method with automatic variable tracking mode provided by the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] Implementation Cases

[0052] The multi-target online tracking method of the joint sparse automatic variable tracking mode of the present invention obtains detection data on the video frame based on the YOLOX target detector (the backbone network is selected as Resnet-50), and the target type of multi-target tracking is selected as pedestrians. The detector is trained based on a mixed data set generated from five data sets: MOT17, MOT20, CrowdHuman, Cityperson, and ETHZ. The mixed data set includes 15,674 pictures and 207,996 pedestrians. The algorithm's trajectory evaluator, tracking mode selector, simple tracker and complex tracker are a unified training framework. The backbone network selects the multi-layer perceptron MLP, the training loss is the cross entropy loss, and the tracking data set is based on MOT17 and MOT20, containing a total of 11 videos. The backbone network for pedestrian re-identification feature extraction is Resnet-50, the training loss is the ternary loss, and the training data set selects Market1501 and MSMT17.

[0053] Experiment: Based on the test of the algorithm on the MOT dataset, the effectiveness and real-time performance of the multi-target tracking method proposed in this paper are verified.

[0054] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-target online tracking method with automatic variable tracking mode, characterized by: Step 1: Use the target detector to detect the same category of targets in the current video frame and obtain detection data; Step 2: Initialize and preprocess the trajectory of the current frame; Step 3: Predict the position of the tracking target in the current frame based on Kalman filtering, and infer the tracking mode of the target based on the multi-layer perceptron network based on the multi-dimensional geometric association feature map; Step 4: Select a simple or complex multi-target tracker to perform corresponding data association according to the tracking mode of the target to be tracked; Step 5: Post-process the associated trajectories, perform linear interpolation between frames on the associated trajectories across frames, and restore the motion trajectory of the target at the entire timestamp; The specific method of step 2 is: Before tracking, the track to be tracked is initialized. If the current frame is the initial frame of tracking, each detection is initialized to the first frame of the track. The track is preprocessed and the tracking quality of the tracked track is evaluated. The track score output by the trajectory evaluator based on the neural network is greater than 0.5, which is a high-quality track. high And the trajectory score less than 0.5 is a low-quality trajectory T low ; Among them, the tracking quality method for evaluating the tracked trajectory is: Step 2.1: Based on the detection data in the tracked trajectory set, establish the state equation of the current tracked target, which consists of 7 variables: Where u and v represent the horizontal and vertical coordinates of the center of the object, and the scale and aspect ratio of the detection bounding box are described by s and r respectively; the aspect ratio is considered constant; finally, the Kalman filter is applied to predict the center point of the predicted bounding box of the trajectory in the current frame as well as the width and height data: (x pre ,y pre ,w pre ,h pre ); Step 2.2: Based on the predicted position (x T ,y T ,w T ,h T ) and the target detection data (x D ,y D ,w D ,h D ) between the geometric correlation feature vector V TD And the geometric correlation feature sequence V between the current trajectory correlation detection TT ; V TD and V TT The input is sent to the neural network to evaluate the tracking quality of the current trajectory and output the trajectory score. The geometric association feature consists of 7-dimensional features: g6=IoU(T,D),g7=t D -t T ; The geometric association feature G is expressed as follows: G=(g1,g2,g3,g4,g5,g6,g7,g8) (1) The trajectory score P is calculated by the following formula, where N1, N2, and N3 are multi-layer perceptrons (MLPs): P=N3(N1V TD +N2V TT ) (2) Step 2.3: Based on the high-quality trajectory T obtained in step 2.2 high With low quality trajectory T low , if 5 consecutive frames are evaluated as low-quality trajectories, the tracking of the current target is terminated; when executing the subsequent tracking process, the high-quality trajectory T high Execute first, low quality trajectory T low After execution; The specific method of step 3 is: Step 3.1: According to step 2.1, the predicted trajectory position (x T ,y T ,w T ,h T ) and the geometric correlation feature vector V between the candidate target detection data TD And the geometric correlation feature vector V between the last frame of the current tracking trajectory and other detections on this frame TL ; Step 3.2: Substitute the V obtained in step 3.1 TD Update to get vector With V TL After splicing, the data are input into the classification neural network to determine the tracking mode that the trajectory should select according to the actual interaction state of the trajectory, and realize the automatic tracking mode selection function. When the category score is less than 0.5, the mode is simple, and when the category score is greater than 0.5, the mode is complex. Pattern category score P M Calculated by the following formula, N4, N5, N6 are multi-layer perceptrons MLP:

2. The multi-target online tracking method with automatic variable tracking mode according to claim 1 is characterized in that: The specific method of step 1 is: Select the target detector YOLOX to perform target detection on the current input video frame. The training of the target detector is based on the public datasets COCO, CrowdedPerson, MOT17, Cityperson, ETHZ and MOT20. The detection category is pedestrian. Get the center point and width and height data of the current M detection data bounding boxes: (x i ,y i ,w i ,h i ), i = 1, ..., M, where x and y represent the horizontal and vertical coordinates of the center point of the bounding box, and w and h represent the width and height of the bounding box.

3. The multi-target online tracking method with automatic variable tracking mode according to claim 2 is characterized in that: The specific method of step 4 is: Step 4.1: If the mode obtained in step 3 is simple, select the simple tracker to complete the detection association of the track to be tracked on the current video frame, and obtain the predicted position of the track (x T ,y T ,w T ,h T ) and the geometric correlation feature vector between the updated candidate target detection data Input into the matching neural network to obtain the matching score matrix Finally, the score matrix is ​​solved by the minimum flow algorithm to optimize the detection matching and thus realize trajectory tracking; Implemented by Multilayer Perceptron N7: Solve the minimum cost flow graph problem based on the association matrix A obtained by matching the neural network; The constraints of its optimization problem include: if a detection is a true positive TP, the detection must be initialized as a new trajectory or matched with a previous trajectory; For one detection in the previous frame and all detections in the current frame; these two constraints can encode all valid trajectories in matrix form; then the data association problem is formulated as an integer linear program as follows: stAY=0,Y∈{0,1} |Y| Θ(X) T is a vector consisting of all the prediction scores of the association matrix, and Y represents the linear constraints between binary variables; Step 4.2: If the mode obtained in step 3 is complex, then select the complex tracker to complete the detection association of the track to be tracked on the current video frame, and use the pedestrian re-identification neural network to extract the appearance features of each candidate detection to obtain the appearance feature vector V ReID , and then get the predicted trajectory position (x T ,y T ,w T ,h T ) and the geometric correlation feature vector between the updated candidate target detection data Splicing is performed and then input into the matching neural network to obtain the matching score matrix Finally, similar to the simple tracker, the score matrix is ​​solved by the minimum flow algorithm to optimize the detection matching and thus realize trajectory tracking; Implemented by Multilayer Perceptron N8:

4. The multi-target online tracking method with automatic variable tracking mode according to claim 1, characterized in that: The specific method of step 5 is: According to the associated trajectory obtained in step 4, trajectory post-processing is performed, inter-frame linear interpolation is performed on the trajectory associated across frames, the motion trajectory of the target over the entire timestamp is restored, and the interpolated trajectory detection sequence is smoothed.