An anti-occlusion secondary association multi-target tracking method

Abstract

The present application relates to the technical field of image processing, in particular to a kind of anti-shielding secondary correlation multi-target tracking method, and tracking process includes establishing a track ID for each target frame that appears in the first frame of video sequence, and the target frame is regarded as the first track point under this track ID;The track of the same track ID under the existence of two frames or more than two frames of track point is regarded as active state, otherwise it is regarded as inactivated state, and a threshold is set, and the target frame confidence exceeding the threshold is regarded as high frame, other is regarded as low frame;First tracking is carried out, and the track of active state is tracked in the current frame high frame, if a high frame can be matched with a track, then the high frame is added to the track, and the first tracking is completed;Second tracking is carried out to the track that has not completed matching in the first tracking, i.e. the track of active state is tracked in the current frame low frame, and the track that has not completed matching in this time is marked as deleted track;If there is still unmatched detection frame after the current frame passes through first tracking and second tracking, the high frame in it is regarded as new track, and track ID is assigned, and the track point of low frame is deleted;The present application improves tracking effect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an occlusion-resistant secondary correlation multi-target tracking method. Background Technology

[0002] Currently, in multi-target tracking, target tracking can be achieved through local association algorithms between adjacent frames. This involves first detecting targets in each frame of the video and then using corresponding algorithms to temporally associate the detected boxes. Alternatively, target tracking can be achieved by using global association algorithms between all frames to associate all target detection boxes.

[0003] However, current multi-target tracking algorithms are prone to frequent switching of identity information when the target is occluded or the target's movement pattern is varied. For a target that reappears after being occluded, it cannot correctly retrieve the original identity information, or the target's trajectory is easily lost when the target's movement speed changes significantly. Summary of the Invention

[0004] To address the problem in multi-target tracking where the original identity information of the target cannot be correctly retrieved after it is occluded, this invention proposes an occlusion-resistant secondary association multi-target tracking method. This method utilizes a target detector to detect bounding boxes in a video sequence, tracks the target based on the detected bounding boxes, and forms a tracking trajectory. The tracking process includes the following steps:

[0005] For each bounding box appearing in the first frame of the video sequence, a trajectory ID is created, and the bounding box is used as the first trajectory point under that trajectory ID;

[0006] Trajectories with two or more consecutive trajectory points under the same trajectory ID are considered active, otherwise they are considered inactive. A threshold is set, and boxes with a confidence score exceeding the threshold are considered high-scoring boxes, while others are considered low-scoring boxes.

[0007] The first tracking is performed by tracking the high-resolution bounding boxes of the active trajectory in the current frame. If a high-resolution bounding box matches a trajectory, the high-resolution bounding box is added to the trajectory to complete the first tracking.

[0008] For trajectories that did not complete the match in the first tracking, a second tracking is performed. That is, in the current frame, the low-resolution bounding box of the active trajectory is tracked, and the trajectory that did not complete the match in the first tracking is marked as a deleted trajectory.

[0009] If there are still unmatched detection boxes after the first and second tracking of the current frame, the high-scoring boxes are treated as new trajectories and assigned trajectory IDs, while the trajectory points of the low-scoring boxes are deleted.

[0010] Furthermore, the initial tracking process includes:

[0011] Trajectories that were successfully tracked in the previous frame are considered tracked trajectories, and trajectories that were lost to be tracked in the previous n frames are considered lost-track trajectories.

[0012] When tracking is performed in the current frame, the tracked trajectory and the lost trajectory are merged into a preliminary tracking trajectory;

[0013] Based on Kalman filtering, the position and size of the target box of the preliminary tracking trajectory in the next frame are predicted, and the intersection-union ratio of the predicted target box with all high-resolution boxes in the current frame is calculated;

[0014] Calculate the loss between the two intersection-union ratios and construct the loss matrix;

[0015] Based on the loss matrix, the Hungarian algorithm is used to match the initial tracking trajectory with the high-resolution bounding box of the current frame. If the match is successful, the target box is updated with the latest trajectory point.

[0016] Furthermore, the process of conducting a second tracking includes:

[0017] The trajectory successfully tracked in the previous frame is considered as a tracked trajectory. When tracking is performed in the current frame, the trajectory that was successfully matched for the first time is deleted from the tracked trajectory.

[0018] Based on Kalman filtering, the position and size of the target box of the track point in the next frame are predicted, and the intersection-union ratio of the predicted target box with all low-resolution boxes in the current frame is calculated as the first intersection-union ratio;

[0019] Calculate the intersection-union ratio (IUU) of the predicted target box with the successfully matched trajectory after the first tracking, and perform matching based on the Hungarian algorithm to find the occluder of each unmatched trajectory;

[0020] The cross-union ratio (CUB) of the predicted target box of the occluder with all low-resolution boxes in the current frame is calculated as the second CUB, and the weighted sum of the first CUB and the second CUB is used as the loss matrix.

[0021] The Hungarian algorithm is used to match the loss matrix with the low-resolution bounding box of the current frame. The trajectory of the matched trajectory updates the trajectory points of the low-resolution bounding box.

[0022] Furthermore, the weighted sum of the first and second intersection-union ratios, as the loss matrix, is expressed as:

[0023]

[0024] Among them, c ij This represents the element in the i-th row and j-th column of the loss matrix; μ is the weighting factor. The intersection-union ratio (IoU) between the i-th predicted bounding box and the j-th low-resolution bounding box in the current frame; Let be the intersection-union ratio (IoU) of the predicted target box of the i-th occluder and the j-th low-resolution box in the current frame.

[0025] Furthermore, the process of predicting the position and size of the target box for the trajectory point in the next frame based on Kalman filtering includes:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] in, Let x represent the predicted value at time t. t-1 Let F represent the optimal estimate at time t-1, F represent the state transition matrix, and B represent the control transition matrix of the system. P represents the prior estimate of the covariance. t-1 U represents the posterior estimated covariance, and Q is the estimated noise covariance; t-1 K represents the control input of the Kalman filter system at time t-1; t H represents the Kalman gain at time t; H represents the observation transition matrix; R is the covariance of the detection noise; and y represents the Kalman gain at time t. t This indicates the position of the target when it is mapped into the detection space at time t.

[0032] Furthermore, the matching process using the Hungarian algorithm includes:

[0033]

[0034] Constraints:

[0035]

[0036] x ij =0or x ij =1

[0037] Where, x ij For a binary function, x is a multi-target tracking task. ij =1 indicates that the i-th target and the j-th trajectory have been matched; c ij This represents the correlation metric between the i-th target and the j-th trajectory in a multi-target tracking task. The loss matrix is ​​composed of the correlation metrics of n targets and m trajectories.

[0038] Furthermore, if the confidence score of the target box exceeds 0.5, the target box is considered a high-scoring box; otherwise, it is considered a low-scoring box.

[0039] Compared with existing technologies, this invention incorporates the information weight of the occluder into the secondary matching process, enabling the occluded target to correctly retrieve the correct identity information before the occlusion occurred, thereby improving the tracking effect of the tracker. Attached Figure Description

[0040] Figure 1 This is a flowchart of a secondary correlation multi-target tracking method with anti-occlusion according to the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] This invention proposes an occlusion-resistant secondary correlation multi-target tracking method. It utilizes a target detector to detect bounding boxes in a video sequence, tracks the detected bounding boxes, and forms a tracking trajectory. Figure 1 The tracking process includes the following steps:

[0043] For each bounding box appearing in the first frame of the video sequence, a trajectory ID is created, and the bounding box is used as the first trajectory point under that trajectory ID;

[0044] Trajectories with two or more consecutive trajectory points under the same trajectory ID are considered active, otherwise they are considered inactive. A threshold is set, and boxes with a confidence score exceeding the threshold are considered high-scoring boxes, while others are considered low-scoring boxes.

[0045] The first tracking is performed by tracking the high-resolution bounding boxes of the active trajectory in the current frame. If a high-resolution bounding box matches a trajectory, the high-resolution bounding box is added to the trajectory to complete the first tracking.

[0046] For trajectories that did not complete the match in the first tracking, a second tracking is performed. That is, in the current frame, the low-resolution bounding box of the active trajectory is tracked, and the trajectory that did not complete the match in the first tracking is marked as a deleted trajectory.

[0047] If there are still unmatched detection boxes after the first and second tracking of the current frame, the high-scoring boxes are treated as new trajectories and assigned trajectory IDs, while the trajectory points of the low-scoring boxes are deleted.

[0048] In this embodiment, a tracking trajectory is created starting from the first frame of the video frame. A trajectory includes all trajectories for continuous tracking and interrupted tracking. When tracking the trajectory, the target detection box is obtained from the current frame by the target detector.

[0049] In this embodiment, trajectories that have been tracked for more than two frames are considered active trajectories. Trajectories that appear for the first time in the video and have not yet matched a second trajectory point are considered inactive trajectories. For the current frame, trajectories that were successfully tracked in the previous frame are considered tracked trajectories. If a trajectory has no matching trajectory point in the previous n frames, it is considered a lost trajectory. Trajectories that have lost tracking in the current frame are deleted, meaning that no matching trajectory point appears in the previous m frames. Generally, n is less than m, and in this embodiment, n ≤ 30 and m > 30. In summary, for a trajectory, if there is a matching point in the previous frame, the trajectory is considered a tracked trajectory for the current frame. If there is no matching trajectory point in the previous n frames (but there was in the previous m frames), the trajectory is considered a lost trajectory for the current frame. If there is no matching trajectory point in the previous m frames, the trajectory is considered a lost trajectory for the current frame.

[0050] This embodiment downloads the SportsMOT dataset and uses it to train the object detector. When scanning the first frame, since no trajectories have yet appeared, trajectory objects are created for all the bounding boxes in the first frame and stored, with each bounding box serving as a trajectory point. Starting from the second frame, trajectories are constructed step by step, specifically including the following steps:

[0051] 1. The detector performs athlete detection on the current frame of the video sequence and obtains bounding boxes of several athletes.

[0052] 2. Classify tracking trajectories and bounding boxes. All tracking trajectories are divided into two categories: active and inactive (active tracks are target boxes that have been tracked for more than two frames (including the trajectory newly created by the target box in the first frame)). All current frame bounding boxes are divided into two categories: high score and low score (classified according to the bounding box score threshold (0.5)).

[0053] 3. Perform the first tracking of the trajectory (high-resolution matching only for active trajectories). Merge all tracked and untracked trajectories to obtain the preliminary tracking trajectory. Predict the possible position and size of the bounding box in the next frame of the preliminary tracking trajectory (using Kalman filtering to predict the bounding box). Calculate the IoU (Intersection over Union) value between the predicted bounding box in the next frame and the high-resolution bounding box in the current frame, obtaining a pairwise IoU relationship loss matrix (the smaller the IoU, the stronger the correlation; the maximum IoU is 1, representing no intersection between bounding boxes). Based on the IoU loss matrix, use the Hungarian algorithm to match the preliminary tracking trajectory with the high-resolution bounding box in the current frame, obtaining three results: matched trajectories and bounding boxes, unmatched trajectories, and unmatched bounding boxes in the current frame. (The Hungarian algorithm can perform one-to-one matching based on the loss matrix, returning the results of successful and unsuccessful matching). Update the preliminary tracking trajectory using the successfully matched bounding boxes in the current frame (changing the boxes in the preliminary tracking trajectory to the current frame bounding boxes, keeping the original IDs).

[0054] 4. Perform a second tracking of the trajectories (low-resolution matching only for active trajectories). Identify the trajectories that were not matched in the first matching and filter out the tracked trajectories (low-resolution matching does not match those trajectories that have already been lost to tracking). Since the bounding boxes of these trajectories have already been predicted for the next frame, no further prediction is needed here. Calculate the IoU1 between the predicted trajectory of the above trajectories and the low-resolution bounding box of the current frame. Identify the occluders of the unmatched trajectories: Calculate the IoU value between the above trajectories and the trajectories that were successfully matched in step 3. Use this IoU matrix to perform Hungarian matching to find the occluder of each unmatched trajectory. Calculate the IoU2 using the predicted trajectory of the occluder and the low-resolution bounding box of the current frame. Here, μ represents the weight factor, and the IoU1 and IoU2 are used for weighted calculation.

[0055]

[0056] Among them, c ij This represents the element in the i-th row and j-th column of the loss matrix; μ is the weighting factor. The intersection-union ratio (IoU) between the i-th predicted bounding box and the j-th low-resolution bounding box in the current frame; Let be the intersection-union ratio (IoU) of the predicted target box of the i-th occluder and the j-th low-resolution box in the current frame.

[0057] In object detection, the Intersection over Union (IoU) is proposed to measure the relationship between predicted and ground truth bounding boxes. IoU first calculates the areas of the predicted and ground truth bounding boxes, then calculates the union area of ​​the two boxes, and finally calculates the intersection and union areas. The calculation formulas are shown below, where A represents the predicted bounding box and B represents the ground truth bounding box.

[0058] The Kalman filter is an optimization estimation algorithm. Most of the work of Kalman filtering can be simplified to predicting and updating the Gaussian function and covariance. First, the filter predicts the next state based on the provided state. Then, noise measurement information is applied to the update stage. This process is repeated cyclically to obtain the optimal estimated state of the target. The two stages of Kalman filtering specifically include the following processes:

[0059] Prediction phase:

[0060] The optimal estimate from the previous moment is used to derive the predicted value for this moment:

[0061]

[0062] The variance / covariance of the optimal estimate at the previous time step and the hyperparameter Q derive the variance / covariance of the predicted value at this time step:

[0063]

[0064] in Let x represent the predicted value at time t. t-1 Let F represent the optimal estimate at time t-1, F represent the state transition matrix, and B represent the control transition matrix of the system. P represents the prior estimate of the covariance. t-1 Let Q represent the posterior estimated covariance, and let Q be the estimated noise covariance.

[0065] Update phase:

[0066] At this moment, the predicted variance / covariance and the hyperparameter R yield the Kalman gain K. t :

[0067]

[0068] The predicted value at this moment, the observed value at this moment, and the Kalman gain yield the optimal estimate at this moment:

[0069]

[0070] The variance / covariance of the predicted value at this moment, and the Kalman gain, yield the variance / covariance of the optimal estimate at this moment:

[0071]

[0072] Where H represents the observation transition matrix, R is the covariance of the detection noise, and y t This represents the position of the target mapped into the detection space at time t, that is, the predicted coordinates (X, Y) of the target's detection box in the image in the next frame, as well as the height H and width W of the detection box.

[0073] The Hungarian algorithm primarily addresses the linear trajectory assignment problem in multi-target tracking, finding the optimal trajectory match for all targets in the current frame. The matching process includes:

[0074]

[0075] Constraints:

[0076]

[0077] x ij =0or x ij =1

[0078] Where, x ij For a binary function, x is a multi-target tracking task. ij =1 indicates that the i-th target and the j-th trajectory have been matched; c ij This represents the correlation metric between the i-th target and the j-th trajectory in a multi-target tracking task. The loss matrix is ​​composed of the correlation metrics of n targets and m trajectories.

[0079] 5. Track the inactive trajectories, find the current frame bounding boxes that were not successfully matched in step 4 (high-resolution bounding boxes that were not successfully matched), and find the inactive trajectories. Calculate the IoU between the above trajectories and the current frame bounding boxes. Use the Hungarian algorithm to match the above tracked trajectories and bounding boxes. Update the above tracked trajectories with the successfully matched current frame bounding boxes. At this time, the inactive trajectories that were not successfully tracked are directly marked as deleted trajectories.

[0080] 6. Create a new trajectory. If there is no high-scoring bounding box that has been successfully matched by now, it is considered a new object and a new trajectory and a new ID will be assigned to it (low-scoring bounding boxes are simply discarded as false positives and no new trajectory will be generated).

[0081] 7. Return results: Return all tracked tracks (excluding tracks that have been lost or deleted).

[0082] 8. Repeat steps 2 to 7 until the video frame ends.

[0083] In this invention, the Hungarian matching algorithm is used to perform three matching operations. The first matching operation is performed during the first tracking process. The Hungarian matching algorithm is used to match the initial tracking trajectory and the high-resolution bounding box of the current frame to obtain three results: the matched trajectory and bounding box, the unmatched trajectory, and the unmatched bounding box of the current frame. The purpose of this matching operation is to perform the first association between the trajectory and the target box.

[0084] The second matching is performed during the second tracking process. This tracking only matches low-scoring matches of trajectories in the active state. In this matching, the Hungarian matching algorithm is used to identify the occluder that is matched in the second tracking process. This matching uses the IOU value of the trajectory predicted by Kalman filter to match the trajectory that was successfully matched in the first tracking. The occluder is found from the successfully matched trajectory. That is, the successfully matched trajectory is regarded as the occluder.

[0085] The third matching is to track inactive trajectories, which involves performing Hungarian matching based on the IoU between trajectory points in inactive trajectories and trajectory points in the current frame. If the trajectory is still inactive after this matching, it needs to be deleted.

[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An occlusion-resistant secondary correlation multi-target tracking method, comprising using a target detector to detect target boxes in a video sequence, tracking the detected target boxes, and forming a tracking trajectory, characterized in that, The tracking process includes the following steps: For each bounding box appearing in the first frame of the video sequence, a trajectory ID is created, and the bounding box is used as the first trajectory point under that trajectory ID; Trajectories with two or more consecutive trajectory points under the same trajectory ID are considered active, otherwise they are considered inactive. A threshold is set, and boxes with a confidence score exceeding the threshold are considered high-scoring boxes, while others are considered low-scoring boxes. The first tracking is performed by tracking the high-resolution bounding boxes of the active trajectory in the current frame. If a high-resolution bounding box matches a trajectory, the high-resolution bounding box is added to the trajectory to complete the first tracking. For trajectories that did not complete a match in the first tracking, a second tracking is performed. Specifically, in the current frame, low-resolution bounding boxes are tracked for active trajectories, and those that still do not complete a match are marked as deleted trajectories. This includes: The trajectory successfully tracked in the previous frame is considered as a tracked trajectory. When tracking is performed in the current frame, the trajectory that was successfully matched for the first time is deleted from the tracked trajectory. Based on Kalman filtering, the position and size of the target box of the track point in the next frame are predicted, and the intersection-union ratio of the predicted target box with all low-resolution boxes in the current frame is calculated as the first intersection-union ratio; Calculate the intersection-union ratio (IUU) of the predicted target box with the successfully matched trajectory after the first tracking, and perform matching based on the Hungarian algorithm to find the occluder of each unmatched trajectory; The intersection-union ratio (IUR) of the predicted target box of the occluder with all low-resolution boxes in the current frame is calculated as the second IUR. The weighted sum of the first IUR and the second IUR is used as the loss matrix, i.e.: in, This represents the element in the i-th row and j-th column of the loss matrix; μ is the weighting factor. The intersection-union ratio (IoU) between the i-th predicted bounding box and the j-th low-resolution bounding box in the current frame; The intersection-union ratio (IoU) of the predicted bounding box of the i-th occluder and the j-th low-resolution bounding box in the current frame; The Hungarian algorithm is used to match the loss matrix with the low bounding box of the current frame. The trajectory of the completed matching is updated with the trajectory points of the low bounding box. If there are still unmatched detection boxes after the first and second tracking of the current frame, the high-scoring boxes are treated as new trajectories and assigned trajectory IDs, while the trajectory points of the low-scoring boxes are deleted.

2. The occlusion-resistant secondary correlation multi-target tracking method according to claim 1, characterized in that, The process of conducting the first tracking includes: Trajectories that were successfully tracked in the previous frame are considered tracked trajectories, and trajectories that were lost to be tracked in the previous n frames are considered lost-track trajectories. When tracking is performed in the current frame, the tracked trajectory and the lost trajectory are merged into a preliminary tracking trajectory; Based on Kalman filtering, the position and size of the target box of the initial tracking trajectory in the next frame are predicted, and the intersection-union ratio of the target box with all high-resolution boxes in the current frame is calculated; Calculate the loss between the two intersection-union ratios and construct the loss matrix; Based on the loss matrix, the Hungarian algorithm is used to match the initial tracking trajectory with the high-resolution bounding box of the current frame. If the match is successful, the target box is updated with the latest trajectory point.

3. The occlusion-resistant secondary correlation multi-target tracking method according to claim 1, characterized in that, The process of predicting the position and size of the target box for the trajectory point in the next frame based on Kalman filtering includes: + ( ) in, This represents the predicted value at time t. Let F represent the optimal estimate at time t-1, F represent the state transition matrix, and B represent the system's control transition matrix. This indicates a priori estimation of the covariance. This represents the posterior estimated covariance, and Q is the estimated noise covariance. This represents the control input of the Kalman filter system at time t-1; H represents the Kalman gain at time t; H represents the observation transition matrix; and R is the covariance of the detection noise. This indicates the position of the target when it is mapped into the detection space at time t.

4. The occlusion-resistant secondary correlation multi-target tracking method according to claim 1, characterized in that, The matching process using the Hungarian algorithm includes: Constraints: in, It is a binary function, for multi-target tracking tasks, This indicates that the i-th target and the j-th trajectory have been matched; This represents the correlation metric between the i-th target and the j-th trajectory in a multi-target tracking task. The loss matrix is ​​composed of the correlation metrics of n targets and m trajectories.

5. The occlusion-resistant secondary correlation multi-target tracking method according to claim 1, characterized in that, If the confidence score of the target box exceeds 0.5, the target box is considered a high-scoring box; otherwise, it is considered a low-scoring box.

Images