A hierarchical multi-target tracking method based on trajectory interval

Abstract

The application provides a hierarchical multi-target tracking method based on track interval. The method comprises the following steps: detecting all target positions for a video; initializing each detection frame as a track; calculating target size consistency overlap similarity; constructing a camera motion consistency algorithm to compensate for the offset caused by camera motion in the matching stage; constructing a hierarchical consistency algorithm to compensate for the motion information lost in the first level; constructing a single-layer track association algorithm to realize one-to-one matching between tracks; constructing a hierarchical tracking framework based on track interval to gradually match all tracks; and post-processing the matching results to obtain the final tracking results. The hierarchical multi-target tracking method based on track interval designed by the application uses a heuristic method to gradually associate all targets, and is free from the dependence of multi-target tracking algorithms on target appearance features and neural networks, and is suitable for scenes with insufficient computing power and scarce training data in real-world applications. Moreover, the application can be seamlessly combined with other tracking algorithms to further improve the tracking effect at a very low computing cost.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer software, and relates to a video understanding technology, in particular to a hierarchical multi-target tracking method based on track interval. BACKGROUND

[0002] The multi-target tracking task aims to detect and associate all targets in a video sequence. With the development of deep learning and target detection technology, the current mainstream method often follows a detection-based hybrid tracking framework, that is, all targets are first detected, then the association between most target boxes is completed in an online manner, and finally the association result is optimized in a post-processing manner using offline technology. Although these methods have certain flexibility and accuracy, the inherent unreliability of the online algorithm restricts the upper limit of the performance of the overall algorithm. Therefore, some schemes model the multi-target tracking task as a two-stage clustering problem, that is, reliable track segments are first generated using strict spatiotemporal and appearance constraints, and then clustering is performed based on graph partitioning algorithms or iterative recommendation algorithms. However, the clustering-based algorithm needs to design different strategies for the two stages, thereby restricting the availability and scalability of such schemes.

[0003] In addition, the existing method often constructs a set of exponentially growing non-overlapping time domain windows, and then completes the association within the corresponding time domain window for each level. However, the way of predefining the time domain window fails to consider the differences in reliability between different tracks, thereby restricting the accuracy of the association process.

[0004] Unlike the above method, the present application aims to propose a new hierarchical tracking framework, which uses a unified association strategy for different levels, that is, uses the track segment interval as the basis for level division, so that short-time association and long-time association can be completed simultaneously using a single model or algorithm, while improving the reliability of intra-level association. The present application finds three inconsistency problems in the hierarchical association process, that is, the inconsistency of association similarity caused by target size difference, the inconsistency of motion information caused by camera motion, and the inconsistency of level information caused by hierarchical framework design, and proposes corresponding solutions. The present application can obtain high-precision tracking results without the need for tracking annotation data and GPU computing resources, reducing the dependence of multi-target tracking algorithms on data and computing resources. SUMMARY

[0005] The present application provides a hierarchical multi-target tracking method based on track interval, characterized in that the method comprises the following steps:

[0006] Step 1, for a video, detecting all target positions;

[0007] Step 2, initializing each detection box as a track segment;

[0008] Step 3, calculate the target size consistency overlap similarity;

[0009] Step 4, perform camera motion consistency compensation;

[0010] Step 5, perform hierarchical information consistency compensation;

[0011] Step 6, construct a single-layer track association algorithm to realize one-to-one matching between tracks;

[0012] Step 7, construct a hierarchical tracking framework based on track interval to gradually match all tracks;

[0013] Step 8, post-process the matching results to obtain the final tracking results.

[0014] Specifically, in step 1, all targets are located based on the YOLOX detector for each frame of the video sequence to obtain detection boxes, and filtered based on a confidence threshold s h = 0.6, and non-maximum suppression is performed using an IoU threshold o nms = 0.8.

[0015] Specifically, in step 2, given a detection box set D1 containing N1 targets, each detection box is initialized as a track segment, where the detection box coordinates are used as the tracking box of the frame, the detection box time domain frame f is used as the start and end frames of the track segment, the motion estimation algorithm Kalman filter is initialized, the prediction box set is initialized as an empty set, and each track segment is assigned a unique ID.

[0016] Specifically, in step 3, given two detection boxes b i = (x i , y i , w i , h i ) and b j = (x j , y j , w j , h j ), if w i < W and w j < W, they are expanded as follows

[0017]

[0018]

[0019]

[0020] where W = 64 is a preset width parameter and τ = 0.2 is a scaling coefficient. The expanded detection boxes and are used to calculate the IoU.

[0021] Specifically, in step 4, for the k-th video sequence, a first-level association is first performed based on the trajectory segment interval threshold Δt = 1. Then, the IoU between all matched detection box pairs is calculated and averaged to obtain O(n). k , which serves as a measure of the motion scale of the k-th sequence camera.

[0022] If O k If the value is less than the preset threshold ΔO = 0.65, the sequence is considered to have significant camera motion. To determine this, all matching detection box pairs are calculated. The average offset between them, as a measure of camera motion (ΔX) t ,ΔY t Estimate:

[0023]

[0024]

[0025] (ΔX) t ,ΔY t Add it to the position estimation step for all level trajectory segments.

[0026] Specifically, in step 5, a first-level association is performed based on the trajectory segment interval threshold Δt = 1. Then, for each detection box, forward and backward linear motion estimation is performed based on the matched detection boxes to obtain the predicted boxes. Finally, the IoU between the predicted boxes and other detection boxes is calculated and associated, thereby completing the motion information compensation for the first level.

[0027] Specifically, in step 6, for level l, motion estimation is performed on all trajectories based on the Kalman filter to obtain the predicted bounding box.

[0028] For frame t, given the trajectory segment interval threshold Δt l Get all trajectory segments ending in the current frame. And the interval between the current frame t and the current frame t does not exceed the threshold Δt l trajectory fragments

[0029] Based on the locations of the detection boxes and predicted boxes, a set of trajectory segments is calculated. and The IoU between the segments is used as the association similarity to obtain the association matrix C, and bipartite graph matching between trajectory segments is completed based on the Hungarian algorithm.

[0030] For level l, the above association process is performed iteratively for each frame to obtain the matching result of the entire video sequence, and the trajectory segment set is updated.

[0031] Specifically, in step 7, given a T-frame video sequence, for the 1st level, 1 ≥ 1, N l tracklets are associated into N l+1 longer tracklets T l+1 , N l ≥ N l+1 .

[0032] Each tracklet is represented as: where is the bounding box of the t-th frame, and are the minimum frame index and maximum frame index of the tracklet .

[0033] A set of tracklet gap thresholds For the 1st level, only when the gap between a pair of tracklets is smaller than the threshold, their association is considered, i.e.:

[0034]

[0035] During the association process, first, a Kalman filter is used to do bidirectional motion prediction, and then the IoU of the real bounding boxes and the predicted bounding boxes between the two tracklets is calculated as the association similarity.

[0036] All levels use a unified association threshold Δo, and the Hungarian algorithm is used to complete the matching.

[0037] By iteratively performing the above association steps on the 1st to the Lth level, and gradually increasing the tracklet gap threshold Δt l , N L+1 tracklets are output.

[0038] Specifically, in step 8, for the position of the lost detection box inside each tracklet, the tracking box of the previous frame and the next frame is used for linear interpolation, and the whole tracklet is nonlinearly fitted based on the Gaussian process regression algorithm. Finally, the overlap between the tracks is calculated based on the IoU, and the tracks with an overlap higher than the threshold 0.65 are fused. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0040] Figure 1 It is a hierarchical multi-target tracking framework based on tracklet gap.

[0041] Figure 2 Flowchart for single-level multi-target tracking. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0043] At present, multi-target tracking algorithms based on detection are widely used in video analysis tasks. Although these algorithms have achieved good performance in various scenes, they still face problems such as low association reliability, dependence on a large amount of labeled data, and dependence on high computing resources. The present application proposes a hierarchical multi-target tracking method based on track interval to obtain high-precision multi-target tracking results without relying on tracking labeled data and GPU computing resources. As shown in Figure 1 The specific steps of the present application are as follows:

[0044] S101: Based on the YOLOX detector, locate all targets in the video sequence, output the detection frame and the confidence score.

[0045] Specifically, in the detector training stage, YOLOX-X is used as the backbone network, and COCO pre-training weights are used as initialization, and then 80 cycles are trained. The image input size is 1440 pixels wide and 800 pixels high, and multi-scale training is used, with a short side range of 576 to 1024 pixels. Mosaic and Mixup are used as data augmentation strategies, and the training batch size is 48. The optimizer selects SGD, with a weight decay of 5x10 -4 , and a momentum of 0.9. The initial learning rate is 10 -3 , and a cycle of warmup is performed, and the learning rate strategy uses the cosine annealing algorithm. In the detector inference stage, the detector confidence threshold is set to 0.6, and non-maximum suppression is performed to delete redundant frames, with an IoU threshold of 0.8.

[0046] S102: Given two detection boxes b i =(x i ,y i ,w i ,h i ) and b j =(x j ,y j ,w j ,h j), set threshold W = 64, if w i < W and w j < W, compute target size consistency IoU for the two detection boxes as the correlation similarity.

[0047] Specifically, first extend the two detection boxes as follows:

[0048]

[0049]

[0050]

[0051] where τ = 0.2 is the scaling factor. Then compute the IoU for the extended detection boxes and as the correlation similarity.

[0052] S103: If a video sequence k has significant camera motion, estimate its camera motion scale for compensating the target motion estimation.

[0053] Specifically, perform the first level correlation based on the trajectory segment interval threshold Δt = 1. Compute the IoU between all matched detection box pairs and take the average to get O k as the kth sequence camera motion scale measure.

[0054] If O k is less than the preset threshold ΔO = 0.65, the sequence is determined to have significant camera motion. Compute the average offset between all matched detection box pairs as the estimate of the camera motion (ΔX t , ΔY t ):

[0055]

[0056]

[0057] Add (ΔX t , ΔY t ) to the position estimation step of all level trajectory segments for compensating the camera motion in the target motion estimation process.

[0058] S104: Perform the first level correlation based on the trajectory segment interval threshold Δt = 1. Then for each detection box, perform forward and backward linear motion estimation based on the matched detection boxes to get the predicted box. Finally use the predicted box to compute the IoU with other detection boxes and do the correlation, thus completing the motion information compensation for the first level.

[0059] S105: Constructing a single-layer track association algorithm to realize track association under a single track segment interval threshold.

[0060] Specifically, for a level l, first, motion estimation is performed on all tracks based on a Kalman filter to obtain a prediction box. For the t-th frame, given a track segment interval threshold Δt l , all track segments ending at the current frame are obtained All track segments with an interval not exceeding the threshold Δt l from the current frame t are obtained Then, based on the positions of the detection box and the prediction box, the IoU between the track segment set and is calculated as the association similarity, an association matrix C is obtained, and bipartite graph matching between track segments is completed based on the Hungarian algorithm. The above association process is iterated for each frame to obtain the matching result of the entire video sequence, and the track segment set is updated.

[0061] S106: Constructing a hierarchical track association algorithm to realize iterative track association under a multi-track segment interval.

[0062] Specifically, for the l-th level, l≥1, N l track segments are associated into N l+1 longer track segments N l ≥N l+1 . Each track segment is represented as:

[0063]

[0064] wherein is a detection box of the t-th frame, and are the minimum frame index and the maximum frame index of the track segment .

[0065] A set of preset track interval thresholds For the l-th level, a pair of track segments will only be considered for association when the interval between them is less than the threshold, i.e.

[0066] In the association process, first, a Kalman filter is used for bidirectional motion prediction, and then the IoU of the real detection box and the prediction box between the two track segments is calculated as the association similarity.

[0067] All levels use a unified association threshold Δo, and the Hungarian algorithm is used to complete the matching.

[0068] The above association step is iteratively performed on the 1st to Lth levels, and the trajectory interval threshold At is gradually increased l , and outputting N L+1 segment trajectories.

[0069] S107: Post-processing the level tracking results.

[0070] Specifically, for the position of the bounding box loss in each segment trajectory, linear interpolation is performed using the front and rear frame tracking boxes, and the whole segment trajectory is nonlinearly fitted based on the Gaussian process regression algorithm. Finally, the overlap degree between trajectories is calculated based on the IoU, and the trajectories with an overlap degree higher than the threshold 0.65 are fused. Finally, the tracking results are output.

[0071] As shown in FIG. 1, it is a single-level multi-target tracking flowchart, and the specific steps are as follows: Figure 2

[0072] S201: For each segment trajectory, bidirectional motion estimation is performed based on the Kalman filter.

[0073] Specifically, it includes two steps of state prediction and state update. The state prediction step predicts the current state mean based on the last state mean x t-1 :

[0074]

[0075] wherein F t is the state transition matrix.

[0076] And the last state covariance P t-1 predicts the current state covariance:

[0077]

[0078] wherein Q t is the process noise covariance.

[0079] The state update step first calculates the Kalman gain K t :

[0080]

[0081] wherein R t is the observation noise covariance.

[0082] Update the current state mean:

[0083]

[0084] wherein z t is the observation value from the detector.

[0085] Update the current state covariance:​

[0086]

[0087] Based on the above state prediction and state update steps, forward and backward motion estimation can be made for each trajectory segment to obtain a prediction box.

[0088] S202: For the current frame t, trajectory screening is completed based on the trajectory segment interval threshold At.

[0089] Specifically, all trajectories are traversed, and all trajectory segments with the maximum frame index t are retained as the pre-order target in the association process:

[0090]

[0091] All trajectory segments with the minimum index within the threshold At from the current frame are retained as the post-order target in the association process:

[0092]

[0093] S203: Calculate the IoU of the detection box and the prediction box between the trajectory segments as the association similarity.

[0094] Specifically, given the pre-order trajectory segment And the post-order trajectory segment Set the time domain prediction length len = 5, calculate the IoU between the last len detection boxes of the pre-order trajectory and the first len prediction boxes of the post-order trajectory, and calculate the IoU between the first len detection boxes of the pre-order trajectory and the last len prediction boxes of the post-order trajectory. All IoU are averaged as the association similarity between the two trajectory segments.

[0095] S204: Screen all high-score trajectory segments and associate.

[0096] Specifically, for each trajectory segment, calculate the average of all detection box confidences as the trajectory confidence. Set the trajectory confidence threshold s h = 0.6, screen all trajectory segments with a confidence higher than s h Based on the association similarity and the Hungarian algorithm, a one-to-one matching is made to obtain a matched trajectory set and an unmatched trajectory set.

[0097] S205: Screen all low-score trajectory segments and associate with high-score unmatched trajectory segments.

[0098] Specifically, set the trajectory confidence threshold s l = 0.1, screen all trajectory segments with a confidence lower than the threshold s h But higher than the threshold s lThe high-confidence unmatched trajectory and the trajectory segment are matched one by one based on a Hungarian algorithm.

[0099] The trajectory information is updated based on the matching result, and the above steps are repeated for a time domain frame t=t+1.

[0100] The trajectory interval-based hierarchical multi-target tracking method can be directly applied to existing video understanding models and target detection algorithms, realizes efficient and accurate multi-target tracking, does not require additional tracking data labeling and model training, does not require GPU computing resources, and is simple and convenient. Meanwhile, the method can also be directly embedded into other multi-target tracking algorithms to realize performance improvement at a very low computing cost.

Claims

1. A hierarchical multi-target tracking method based on trajectory interval, characterized in that, The method comprises the following steps: Step 1, for a video, detecting all target positions; Step 2, initializing each detection frame as a track; Step 3, calculate the target size consistency overlap similarity; specifically: given two detection boxes b i =(x i ,y i ,w i ,h i ) and b j =(x j ,y j ,w j ,h j ), if w i <W and w j <W, then extend them as follows, where W = 64 is a preset width hyper-parameter, τ = 0.2 is a scaling factor, and the expanded detection box and for computing the IoU to complete the tracklet association; Step 4, camera motion consistency compensation is performed; specifically, for the kth video sequence, first, first-level association is performed based on a trajectory segment interval threshold At = 1, then the average of the IoU between all matched detection box pairs is obtained as O k , which is a measure of the camera motion scale of the kth sequence. If O k is less than a preset threshold AO = 0.65, the sequence is determined to have significant camera motion, and the average offset between all matched detection box pairs is calculated as an estimate of the camera motion (AX t , AY t ), Finally, (ΔX t ,ΔY t ) is added to the position estimation step of all hierarchical track fragments, compensating for the bias introduced by the camera motion; Step 5, performing hierarchical information consistency compensation; specifically, first, performing first-level association based on a track segment interval threshold value Δt=1, then for each detection frame, performing forward and reverse linear motion estimation based on the matched detection frame to obtain a predicted frame, and finally using the predicted frame and other detection frames to calculate IoU and perform association, thereby completing the motion information compensation of the first level; Step 6, build a single-layer track association algorithm to achieve one-to-one matching between tracks; specifically: for level l, first complete motion estimation on all tracks based on the Kalman filter to obtain a prediction box; for the t-th frame, given the track segment interval threshold Δt l , obtain all track segments ending at the current frame Obtain all track segments with a time interval of no more than the threshold Δt l from the current frame t Then, based on the positions of the detection box and the prediction box, calculate the IoU between the track segment set and as the association similarity to obtain the association matrix C, and complete bipartite graph matching between track segments based on the Hungarian algorithm; iterate the above association process for each frame to obtain the matching result of the entire video sequence and update the track segment set; Step 7, construct a hierarchical tracking framework based on trajectory interval, match all trajectories step by step; specifically: given a T-frame video sequence, its detection box set D1 contains N1 target boxes, and N1 trajectory segments are initialized based on this; then for the lth level, l≥1, N l trajectory segments are associated as N l+1 longer trajectory segments N l ≥N l+1 ; each trajectory segment is represented as: wherein is the detection box of the t-th frame, and is the minimum frame index and the maximum frame index of the trajectory segment ; a set of trajectory interval thresholds is preset For the l-th level, only when the interval between a pair of trajectory segments is less than the threshold, the association between them is considered, that is In the association process, first, a Kalman filter is used for bidirectional motion prediction, and then the IoU of the real detection box and the predicted box between the two trajectory segments is calculated as the association similarity; a unified association threshold Δo is used for all levels, and the Hungarian algorithm is used to complete the matching; through iteratively performing the above association steps on the 1st to the Lth levels, and gradually increasing the trajectory interval threshold Δt l , the final output is N L+1 segment trajectories; Step 8, post-processing the matching result to obtain the final tracking result.

2. The trajectory interval-based hierarchical multi-object tracking method of claim 1, wherein, Based on YOLOX detection algorithm to locate all target positions, using confidence threshold s h = 0.6 for filtering, using IoU threshold o nms = 0.8 for non-maximum suppression.

Images