Hierarchical multi-target tracking method based on trajectory interval

Abstract

The invention provides a hierarchical multi-target tracking method based on trajectory intervals. The method comprises the following steps: detecting all target positions for a video; initializing each detection frame into a section of track; target size consistency overlapping similarity is calculated; constructing a camera motion consistency algorithm, and compensating offset caused by camera motion in a matching stage; constructing a hierarchy consistency algorithm, and compensating motion information lost by the first hierarchy; constructing a single-layer trajectory association algorithm to realize one-to-one matching between trajectories; constructing a hierarchical tracking framework based on trajectory intervals, and gradually matching all trajectories; and performing post-processing on the matching result to obtain a final tracking result. According to the hierarchical multi-target tracking method based on the trajectory interval, all the targets are associated step by step by using a heuristic method, the dependence of a multi-target tracking algorithm on target appearance characteristics and a neural network is eliminated, and the hierarchical multi-target tracking method is suitable for scenes with insufficient computing power and scarce training data in practical application. The method can be seamlessly combined with other tracking algorithms, and the tracking effect is further improved with extremely low calculation cost.

Description

Technical Field

[0001] The present invention belongs to the field of computer software technology, and relates to video understanding technology, in particular to a hierarchical multi-target tracking method based on trajectory intervals. Background Art

[0002] The multi-target tracking task aims to detect and associate all targets in a video sequence. With the development of deep learning and target detection technology, the current mainstream methods often follow a detection-based hybrid tracking framework, that is, first detect all targets, then complete the association between most target boxes in an online manner, and finally use offline technology to optimize the association results in a post-processing manner. Although these methods have certain flexibility and accuracy, the inherent unreliability of online algorithms restricts the performance upper limit of the overall algorithm. To this end, some schemes model the multi-target tracking task as a two-stage clustering problem, that is, first generate reliable trajectory segments using strict spatiotemporal and appearance constraints, and then perform clustering based on graph partitioning algorithms or iterative recommendation algorithms. However, clustering-based algorithms require different strategies for the two stages, which restricts the availability and scalability of such schemes.

[0003] In addition, existing methods often construct a set of exponentially growing non-overlapping time windows, and then constrain each level to complete the association within the corresponding time window. However, the predefined time window method fails to take into account the differences in reliability between different trajectories, thus constraining the accuracy of the association process.

[0004] Different from the above methods, the present invention aims to propose a new hierarchical tracking framework, which uses a unified association strategy for different levels, that is, using the interval of trajectory segments as the basis for hierarchical division, so that a single model or algorithm can be used to simultaneously complete short-term association and long-term association, while improving the reliability of intra-level association. The present invention discovered three inconsistency problems in the hierarchical association process, namely, the inconsistent association similarity caused by target size differences, the inconsistent motion information caused by camera motion, and the inconsistent hierarchical information caused by the hierarchical framework design, and proposed corresponding solutions. The present invention can obtain high-precision tracking results without the need to track annotated data and GPU computing resources, reducing the dependence of multi-target tracking algorithms on data and computing resources. Summary of the invention

[0005] The present invention provides a hierarchical multi-target tracking method based on trajectory interval, characterized in that the method comprises the following steps:

[0006] Step 1: Detect all target locations for a video;

[0007] Step 2, initialize each detection box as a trajectory;

[0008] Step 3, calculate the target size consistency overlap similarity;

[0009] Step 4, perform camera motion consistency compensation;

[0010] Step 5: Perform level information consistency compensation;

[0011] Step 6: construct a single-layer trajectory association algorithm to achieve one-to-one matching between trajectories;

[0012] Step 7: Build a hierarchical tracking framework based on trajectory intervals and gradually match all trajectories;

[0013] Step 8: Post-process the matching results to obtain the final tracking results.

[0014] Specifically, in step 1, all targets are located based on the YOLOX detector for each frame of the video sequence to obtain the detection box, and based on the confidence threshold s h = 0.6 for filtering, using the IoU threshold o nms = 0.8 for non-maximum suppression.

[0015] Specifically, in step 2, given a set D1 of target detection frames containing N1 target frames, each detection frame is initialized as a trajectory segment, where the detection frame coordinates are used as the frame tracking frame, the detection frame time domain frame f is used as the start and end frames of the trajectory segment, the motion estimation algorithm Kalman filter is initialized, the prediction frame set is initialized to an empty set, and a unique ID is assigned to each trajectory segment.

[0016] Specifically, in step 3, given two detection boxes b i =(χ i ,y i , w i ,h i ) and b j =(χ j ,y j , w j ,h j ), if w i <W and w j <W, then expand it as follows

[0017]

[0018]

[0019]

[0020] Where W = 64 is the preset width hyperparameter, τ = 0.2 is the scaling factor. Then the expanded detection box and Used to calculate IoU.

[0021] Specifically, in step 4, for the kth video sequence, the first level association is first performed based on the track segment interval threshold Δt=1. Then the IoU between all matching detection box pairs is calculated and averaged to obtain O k , as a measure of the camera motion scale of the kth sequence.

[0022] If O k If the value is less than the preset threshold ΔO=0.65, the sequence is judged to have significant camera motion. The average offset between the two is taken as the average offset of the camera motion (ΔX t , ΔY t )’s estimate:

[0023]

[0024]

[0025] (ΔX t , ΔY t ) is added to the position estimation step of all level tracklets.

[0026] Specifically, in step 5, the first level association is performed based on the trajectory segment interval threshold Δt=1. Then, for each detection frame, forward and reverse linear motion estimation is performed based on the matched detection frame to obtain a predicted frame. Finally, the predicted frame is used to calculate the IoU with other detection frames and associate them, thereby completing the first level of motion information compensation.

[0027] Specifically, in step 6, for level l, motion estimation is performed on all trajectories based on the Kalman filter to obtain a predicted box.

[0028] For the tth frame, given the trajectory segment interval threshold Δt l , get all trajectory segments ending at the current frame and the interval with the current frame t does not exceed the threshold Δt 1 Track fragment

[0029] Calculate the set of trajectory segments based on the detection box and prediction box positions and The IoU between them is used as the correlation similarity to obtain the correlation matrix C, and the bipartite graph matching between trajectory segments is completed based on the Hungarian algorithm.

[0030] For level l, the above association process is iterated for each frame to obtain the matching result of the entire video sequence and update the trajectory segment set.

[0031] Specifically, in step 7, given a T-frame video sequence, for the l-th level (l≥1), N l A track segment Associated with N l+1 A longer trajectory segment T l+1 (N l ≥N l+1 ).

[0032] Each trajectory segment is represented as: in is the detection box of the tth frame, and Is a track segment The minimum and maximum frame indexes of .

[0033] Preset a set of track interval thresholds For level l, only when a pair of trajectory segments The association is considered only when the interval effect is less than the threshold, that is:

[0034]

[0035] During the association process, the Kalman filter is first used to perform bidirectional motion prediction, and then the IoU between the real detection box and the predicted box between the two trajectory segments is calculated as the association similarity.

[0036] All levels use a unified correlation threshold Δo and use the Hungarian algorithm to complete the matching.

[0037] By iteratively performing the above association steps on the 1st to Lth levels and gradually increasing the trajectory interval threshold Δt l , output N L+1 Segment track.

[0038] Specifically, in step 8, for the position where the detection frame is lost inside each track, linear interpolation is performed using the tracking frames of the previous and next frames, and nonlinear fitting is performed on the entire track based on the Gaussian process regression algorithm. Finally, the overlap between tracks is calculated based on IoU, and tracks with an overlap higher than the threshold of 0.65 are fused. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 This is a diagram of the hierarchical multi-target tracking framework based on trajectory intervals;

[0041] Figure 2 Flowchart of multi-target tracking within a single layer. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] At present, detection-based multi-target tracking algorithms are widely used in video analysis tasks. Although these algorithms have achieved good performance in various scenarios, they still face problems such as low correlation reliability, dependence on a large amount of labeled data, and dependence on high computing resources. In view of the above problems, the present invention proposes a hierarchical multi-target tracking method based on trajectory intervals, which obtains high-precision multi-target tracking results without relying on tracking labeled data and GPU computing resources. Figure 1 As shown, the specific steps of the present invention are as follows:

[0044] S101: Based on the YOLOX detector, locate all objects in the video sequence and output the detection box and confidence score.

[0045] Specifically, in the detector training phase, YOLOX-X is used as the backbone network and COCO pre-trained weights are used as initialization, and then trained for 80 cycles. The image input size is 1440 pixels wide and 800 pixels high, and multi-scale training is used, where the short side ranges from 576 to 1024 pixels. Mosaic and Mixup are used as data enhancement strategies, and the training batch size is 48. SGD is used as the optimizer, and the weight decay is 5×10 -4 , momentum is 0.9. The initial learning rate is 10 -3 , and do a cycle of warmup, and the learning rate strategy uses the cosine annealing algorithm. In the detector reasoning stage, the detector confidence threshold is set to 0.6, and non-maximum suppression is performed to remove redundant boxes, with an IoU threshold of 0.8.

[0046] S102: Given two detection boxes b i =(χ i ,y i ,w i ,h i ) and b j =(x j ,y j ,w j ,h j) Set the threshold W = 64. If w i < W and w j < W, then calculate the target size consistency IoU as the association similarity for the two detection boxes.

[0047] Specifically, first expand the two detection boxes in the following manner:

[0048]

[0049]

[0050]

[0051] where τ = 0.2 is the scaling factor. Then calculate the IoU for the expanded detection boxes and as the association similarity.

[0052] S103: If a certain video sequence k has significant camera motion, then estimate its camera motion scale for compensating the target motion estimation.

[0053] Specifically, perform the first-level association based on the trajectory segment interval threshold Δt = 1. Calculate the IoU between all pairs of matching detection boxes and take the average to obtain O k , as a measure of the camera motion scale of the k-th sequence.

[0054] If O k is less than the preset threshold ΔO = 0.65, then the sequence is determined to have significant camera motion. Calculate the average offset between all pairs of matching detection boxes as an estimate of the camera motion (ΔX t , ΔY t ):

[0055]

[0056]

[0057] Add (ΔX t , ΔY t ) to the position estimation step of all levels of trajectory segments for compensating the camera motion during the target motion estimation.

[0058] S104: Perform the first-level association based on the trajectory segment interval threshold Δt = 1. Then for each detection box, perform forward and backward linear motion estimation based on the matched detection boxes to obtain the predicted boxes. Finally, calculate the IoU between the predicted boxes and other detection boxes and perform the association to complete the compensation of the motion information at the first level.

[0059] S105: construct a single-layer trajectory association algorithm to achieve trajectory association under a single trajectory segment interval threshold.

[0060] Specifically, for level l, firstly, motion estimation is performed on all trajectories based on the Kalman filter to obtain the predicted frame. For the tth frame, a trajectory segment interval threshold Δt is given. 1 , get all trajectory segments ending at the current frame Get all frames whose interval with the current frame t does not exceed the threshold Δt 1 The trajectory fragment Then, based on the detection box and prediction box positions, the trajectory segment set is calculated and The IoU between them is used as the correlation similarity to obtain the correlation matrix C, and the bipartite graph matching between the trajectory segments is completed based on the Hungarian algorithm. The above association process is iterated for each frame to obtain the matching result of the entire video sequence and update the trajectory segment set.

[0061] S106: constructing a hierarchical trajectory association algorithm to achieve iterative trajectory association under multiple trajectory segment intervals.

[0062] Specifically: for the lth level (l≥1), N l Track segments Associated with N l+1 Longer trajectory segments Each trajectory segment is represented as:

[0063]

[0064] in is the detection box of the tth frame, and Is a track segment The minimum and maximum frame indexes of .

[0065] Preset a set of track interval thresholds For level l, only when a pair of trajectory segments Only when the interval between them is less than the threshold will they be considered for association, that is,

[0066] During the association process, the Kalman filter is first used to perform bidirectional motion prediction, and then the IoU between the real detection box and the predicted box between the two trajectory segments is calculated as the association similarity.

[0067] All levels use a unified correlation threshold Δo and use the Hungarian algorithm to complete the matching.

[0068] By iteratively performing the above association steps on the 1st to Lth levels and gradually increasing the trajectory interval threshold Δt l , output N L+1 Segment track.

[0069] S107: Post-processing the hierarchical tracking results.

[0070] Specifically, for the position where the detection frame is lost inside each track, linear interpolation is performed using the tracking frames of the previous and next frames, and nonlinear fitting is performed on the entire track based on the Gaussian process regression algorithm. Finally, the overlap between tracks is calculated based on IoU, and tracks with overlaps higher than the threshold of 0.65 are fused. The tracking result is finally output.

[0071] like Figure 2 As shown in the figure, it is a flowchart of multi-target tracking in a single layer. The specific steps are as follows:

[0072] S201: For each trajectory, perform bidirectional motion estimation based on the Kalman filter.

[0073] Specifically, it includes two steps: state prediction and state update. The state prediction step is based on the previous state mean χ t-7 Predict the current state mean:

[0074]

[0075] where F t is the state transfer matrix.

[0076] And based on the previous state covariance P t-1 Predict the current state covariance:

[0077]

[0078] Where Q t is the process noise covariance.

[0079] The state update step first calculates the Kalman gain K t :

[0080]

[0081] Where R t is the observation noise covariance.

[0082] Update the current state mean:

[0083]

[0084] where z t is the observation value from the detector.

[0085] Update the current state covariance:

[0086]

[0087] Based on the above state prediction and state update steps, forward and reverse motion estimation can be performed for each trajectory to obtain a predicted box.

[0088] S202: For the current frame t, complete trajectory screening based on the trajectory segment interval threshold Δt.

[0089] Specifically, we traverse all trajectories and retain all trajectory segments with the maximum frame index t as the pre-order targets in the association process:

[0090]

[0091] All trajectory segments whose minimum index is less than the threshold Δt from the current frame are retained as subsequent targets in the association process:

[0092]

[0093] S203: Calculate the IoU between the detection box and the prediction box between the trajectory segments as the association similarity.

[0094] Specifically, given a preceding trajectory segment and subsequent trajectory fragments Set the time domain prediction length len=5, calculate the IoU between the len predicted boxes in the previous trajectory and the len detection boxes at the front of the subsequent trajectory, and calculate the IoU between the len predicted boxes in the forward direction of the subsequent trajectory and the len detection boxes at the back of the previous trajectory. The average of all IoUs is taken as the correlation similarity between the two trajectory segments.

[0095] S204: Filter and associate all high-scoring trajectory segments.

[0096] Specifically, for each track segment, the average confidence of all detection boxes is calculated as the track confidence. Set the track confidence threshold s h = 0.6, filter all confidence levels higher than s h Based on the correlation similarity and Hungarian algorithm, a one-to-one matching is performed on the trajectory segments to obtain a set of matched trajectories and a set of unmatched trajectories.

[0097] S205: Filter all low-scoring trajectory segments and associate them with high-scoring unmatched trajectory segments.

[0098] Specifically, set the trajectory confidence threshold s l = 0.1, filter all confidence values ​​below the threshold s h But above the threshold S lThe trajectory fragment is matched one-to-one with the above high-confidence unmatched trajectory based on the Hungarian algorithm.

[0099] Update the trajectory information based on the matching result, update the time domain frame t=t+1 and repeat the above steps.

[0100] The hierarchical multi-target tracking method based on trajectory intervals of the present invention can be directly applied to existing video understanding models and target detection algorithms to achieve efficient and accurate multi-target tracking without the need for additional tracking data annotation and model training, and without the need for GPU computing resources, which is simple and convenient. At the same time, the method can also be directly embedded in other multi-target tracking algorithms to achieve performance improvement at a very low computing cost.

Claims

1. A hierarchical multi-target tracking method based on trajectory interval, characterized in that: The method comprises the following steps: Step 1: Detect all target locations for a video; Step 2, initialize each detection box as a trajectory; Step 3, calculate the target size consistency overlap similarity; Step 4, perform camera motion consistency compensation; Step 5: Perform level information consistency compensation; Step 6: construct a single-layer trajectory association algorithm to achieve one-to-one matching between trajectories; Step 7: Build a hierarchical tracking framework based on trajectory intervals and gradually match all trajectories; Step 8: Post-process the matching results to obtain the final tracking results.

2. The hierarchical multi-target tracking method based on trajectory interval according to claim 1, characterized in that: Based on the YOLOX detection algorithm, all target positions are located, using the confidence threshold s h = 0.6 for filtering, using the IoU threshold o nms = 0.8 for non-maximum suppression.

3. The hierarchical multi-target tracking method based on trajectory interval according to claim 1, characterized in that: Based on the time domain interval between trajectory segments, a multi-level progressive association algorithm is constructed. Specifically, given a T-frame video sequence, its detection box set D1 contains N1 target boxes, and N1 trajectory segments are initialized based on this; then for the l-th level (l≥1), N l Track segments Associated with N l+1 Longer trajectory segments Each trajectory segment is represented as: in is the detection box of the Nth frame, and Is a track segment The minimum frame index and maximum frame index of . A set of trajectory interval thresholds are preset For level l, only when a pair of trajectory segments Only when the interval between them is less than the threshold will they be considered for association, that is, During the association process, the Kalman filter is first used to make bidirectional motion predictions, and then the IoU between the real detection box and the predicted box between the two track segments is calculated as the association similarity. All levels use a unified association threshold Δo, and the Hungarian algorithm is used to complete the matching. The above association steps are iteratively performed on the 1st to Lth levels, and the track interval threshold Δt is gradually increased. l , the final output is N L+1 Segment track.

4. The hierarchical multi-target tracking method based on trajectory interval according to claim 3, characterized in that: Each level uses a unified association threshold Δo and uses the Hungarian algorithm to complete the matching. Specifically, for level l, first complete the motion estimation of all trajectories based on the Kalman filter to obtain the predicted box. For the tth frame, a given trajectory segment interval threshold Δt l , get all trajectory segments ending at the current frame Get all frames whose interval with the current frame t does not exceed the threshold Δt l Track fragment Then, based on the detection box and prediction box positions, the trajectory segment set is calculated and The IoU between them is used as the correlation similarity to obtain the correlation matrix C, and the bipartite graph matching between the trajectory segments is completed based on the Hungarian algorithm. The above association process is iterated for each frame to obtain the matching result of the entire video sequence and update the trajectory segment set.

5. The hierarchical multi-target tracking method based on trajectory interval according to claim 1, characterized in that: Calculate the size consistency overlap similarity between targets, specifically: Given two detection boxes b i =(x i ,y i ,w i ,h i ) and b j =(χ j ,y j ,w j ,h j ), if w i <W and w j <W, then expand it in the following way Where W = 64 is the preset width hyperparameter, τ = 0.2 is the scaling factor. Then the expanded detection box and Used to calculate IoU to complete track segment association.

6. The hierarchical multi-target tracking method based on trajectory interval according to claim 1, characterized in that: Camera motion consistency compensation is performed as follows: for the kth video sequence, firstly, the first level association is performed based on the track segment interval threshold Δt=1. Then, the IoU between all matching detection box pairs is calculated and averaged to obtain O k , as a measure of the camera motion scale of the kth sequence. If O k If the value is less than the preset threshold ΔO=0.65, the sequence is judged to have significant camera motion. The average offset between the two is taken as the average offset of the camera motion (ΔX t ,ΔY t ), Finally, (ΔX t ,ΔY t ) is added to the position estimation step of all level track segments to compensate for the deviation caused by camera motion.

7. The hierarchical multi-target tracking method based on trajectory interval according to claim 1, characterized in that: Compensate for the consistency of hierarchical information. Specifically, first perform the first-level association based on the trajectory segment interval threshold Δt=1. Then, for each detection frame, perform forward and reverse linear motion estimation based on the matched detection frame to obtain the predicted frame. Finally, use the predicted frame to calculate the IoU with other detection frames and associate them, thereby completing the first-level motion information compensation.

Images