Multi-target tracking method based on forward and reverse trajectory fusion

Abstract

The invention provides a multi-target tracking method based on forward and reverse trajectory fusion. The method is especially suitable for a multi-target tracking task in a complex scene. According to the method, forward and reverse target tracking is carried out respectively through a sliding window technology, and the consistency of a target ID in the whole tracking process is ensured by utilizing matching and fusion strategies of forward and reverse tracks. For short-term target loss caused by shielding or other reasons, trajectory prediction and completion are carried out through a Kalman filter, and continuity and accuracy of the trajectory are ensured. And between the sliding windows, target IDs are kept consistent through track matching of overlapped intervals, so that ID hopping or misrecognition is effectively avoided. The method can maintain an efficient and stable multi-target tracking effect in complex scenes of shielding, motion change and background interference, and is especially suitable for tracking tasks in dense pedestrian or dynamic scenes. Through the method, the precision and robustness of multi-target tracking are remarkably improved.

Description

Technical Field

[0001] The present application belongs to the technical field of computer vision and video analysis, and specifically relates to a multi-target tracking method based on forward and reverse trajectory fusion, which is particularly suitable for complex scenes with occlusion or temporary loss. Background Art

[0002] As a core task in computer vision, multi-target tracking (MOT) has a wide range of application prospects, including security monitoring, unmanned driving, behavior analysis, and posture estimation. Its task is to identify and output the motion trajectory of the target from the video sequence. In the specific application of multi-target tracking, multi-pedestrian tracking has become a hot topic in this field due to the complex scenes and severe occlusion.

[0003] In recent years, with the improvement of the performance of target detection algorithms, detection-based multi-pedestrian tracking frameworks have gradually become mainstream. This type of method first detects targets in each frame of the video, and then performs data association based on motion information and appearance features to generate a complete trajectory for each target. Current tracking algorithms usually rely on learning short-term motion patterns to match the same target in adjacent frames, while using appearance features to identify targets that have been lost for a long time. However, in scenes with dense crowds or frequent occlusion of targets, relying solely on motion prediction and appearance matching can easily lead to unstable target tracking and make it difficult to accurately associate target trajectories over long time spans. Summary of the invention

[0004] Purpose of the invention: In view of the shortcomings of existing multi-target tracking methods in dealing with occlusion by dense crowds, motion changes and background interference, the present invention proposes a multi-target tracking method based on forward and reverse trajectory fusion. This method performs forward and reverse trajectory tracking on video sequences and combines Kalman filters for motion prediction to achieve long-term association and accurate tracking of target trajectories. It is particularly suitable for dealing with short-term loss and occlusion problems in complex scenes. The present invention aims to improve the accuracy and robustness of target tracking by effectively fusing forward and reverse tracking results, ensuring the consistency of target ID and continuity of trajectory.

[0005] The present application provides a multi-target tracking method based on forward and reverse trajectory fusion, the method comprising: Step S1, for each video sequence to be tracked, use a deep learning-based target detector to detect the target in each frame, and obtain the coordinates and confidence score of each target detection frame; Step S2, processing the video sequence using a sliding window of a fixed size according to the coordinates and confidence scores of each detection frame; Step S3, in each sliding window, first perform forward tracking, then perform reverse tracking, through forward tracking and reverse tracking, obtain the forward target trajectory and the reverse target trajectory, and provide a data basis for subsequent matching and fusion; Step S4, in the overlapping area of ​​the sliding window, based on the position information and motion information of the target, a preset matching strategy is adopted to determine whether the target trajectory of the forward direction and the target trajectory of the reverse direction belong to the same target; Step S5, if the target trajectory of the forward direction and the target trajectory of the reverse direction belong to the same target, the forward and reverse trajectories that are successfully matched are fused to obtain a fused trajectory; Step S6, splicing the fused trajectories to obtain the tracking trajectory of the target. Preferably, the method further comprises: Step S7: In case of short-term target loss, a Kalman filter is used to predict the trajectory based on the historical motion state of the target.

[0006] Preferably, the forward tracking includes: From the initial frame to the last frame of the sliding window, the SORT algorithm is used to track each target frame by frame based on the above detection results to generate a positive target trajectory.

[0007] Preferably, the reverse tracking includes: performing reverse tracking from the tail frame of the sliding window to the initial frame using a SORT algorithm to generate a reverse target trajectory.

[0008] Preferably, the preset matching strategy includes: Position consistency: The positions of the object detection boxes in overlapping frames should be close, which can be determined by calculating the center point distance or IoU of the boxes. Motion consistency: The motion direction and speed in the forward and reverse trajectories should be similar. By comparing the velocity vectors between adjacent frames, it is determined whether the motion trend of the target is consistent.

[0009] Preferably, step S5 comprises: For the trajectories in the overlapping intervals, the positive target trajectory is directly used as the fused trajectory; If the forward target trajectory is missing but the reverse target trajectory exists, the reverse target trajectory is used as the fused trajectory; If both the forward and reverse target trajectories are missing, the Kalman filter is used for trajectory prediction, and the predicted trajectory is used as the fused trajectory.

[0010] Preferably, step S6 comprises: During the trajectory stitching process, trajectory updating, re-identification, trajectory termination and trajectory initialization are adopted to handle abnormal situations such as occlusion, target loss and target reappearance.

[0011] Preferably, step S7 comprises: The forward and backward Kalman filters calculate the prediction values ​​separately, and select the more accurate prediction results in the fusion process to fill the trajectory of the lost frame.

[0012] This application has the following technical effects: 1) When dealing with complex scenes, traditional multi-target tracking methods often encounter problems such as tracking trajectory interruption or ID jump, especially when there are dense pedestrians, severe occlusion, or violent movement of the target. The present invention can effectively solve these problems through the bidirectional fusion of forward and reverse trajectories, combined with the motion prediction of the Kalman filter. Specifically, through the forward and reverse trajectory matching and completion, the lost frames can be completed through the reverse trajectory when the target is temporarily occluded or disappears, greatly improving the continuity and stability of the trajectory. At the same time, the prediction of the Kalman filter further enhances the trajectory accuracy in the case of short-term loss, ensuring that the target can be accurately predicted and tracked even in the case of occlusion or detection loss.

[0013] 2) In multi-target tracking, maintaining the consistency of target IDs is crucial to achieve long-term stable tracking, especially during window sliding. The present invention ensures that the ID of each target remains consistent between different sliding windows by matching and associating the forward and reverse trajectories in overlapping frames, avoiding the phenomenon of reallocation of IDs after ID jumps or target loss. This not only improves the consistency of tracking, but also effectively avoids possible mismatches during target re-identification. In addition, the overlapping area design in window sliding also helps to verify the tracking results of the front and rear windows, making the target ID more robust throughout the video sequence, which is particularly suitable for scenes with dense pedestrians or high-speed motion.

[0014] 3) Existing multi-target tracking technologies often have difficulty accurately re-identifying targets when faced with long-term occlusion or the reappearance of pedestrians, resulting in trajectory interruption or loss. To solve this problem, the present invention designs a trajectory management module. Through the re-identification strategy in the trajectory management, it can efficiently identify and restore the previous ID when the target reappears, avoiding the problem of reallocating a new ID. In addition, the complementary characteristics of the forward and reverse trajectories can better handle the loss during occlusion. In the case of short-term loss, the Kalman filter is used to accurately predict the target and achieve a smooth transition of the trajectory; when the target reappears, accurate association and re-identification of the same target are achieved by matching with the previous trajectory. This makes the present invention more adaptable and stable when faced with long-term occlusion, complex motion or background interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of a multi-target tracking method based on forward and reverse trajectory fusion provided in an embodiment of the present application. DETAILED DESCRIPTION The present invention provides a multi-target tracking method based on forward and reverse trajectory fusion, which is particularly suitable for multi-target tracking tasks in complex scenes. The method uses sliding window technology to perform forward and reverse target tracking respectively, and uses the matching and fusion strategy of forward and reverse trajectories to ensure the consistency of target ID in the entire tracking process. For short-term target loss due to occlusion or other reasons, trajectory prediction and completion are performed through a Kalman filter to ensure the continuity and accuracy of the trajectory. Between sliding windows, the target ID is kept consistent through trajectory matching in overlapping intervals, effectively avoiding ID jumps or misidentification. The present invention can maintain efficient and stable multi-target tracking effects in complex scenes with occlusion, motion changes and background interference, and is particularly suitable for tracking tasks in densely populated or dynamic scenes. Through this method, the accuracy and robustness of multi-target tracking are significantly improved.

[0016] See also Figure 1 ,The multi-target tracking method based on forward and reverse trajectory fusion includes the following steps: Step S1, target detection: For each video sequence to be tracked, use a deep learning-based target detector (such as YOLOX) to detect the target in each frame. The YOLOX detector has high detection accuracy and real-time performance, and can effectively detect multiple targets in the video sequence and obtain the coordinates and confidence score of each detection box (Bounding Box).

[0017] Step S2, sliding window: Use a sliding window of fixed size (e.g., 30 frames) to process the video sequence. The sliding window slides forward at a certain step size (e.g., 10 frames) to ensure that there is an overlapping area between each window. Through this sliding method, it is ensured that the trajectories between different windows can be verified and matched with each other.

[0018] Step S3, target tracking (forward and reverse); in each sliding window, first perform forward tracking (frame 1 to frame 30), then perform reverse tracking (frame 30 to frame 1). Through these two steps, the forward and reverse target trajectories are obtained, providing a data basis for subsequent matching and fusion. Based on target detection, the SORT (Simple Online and Realtime Tracking) algorithm is used for forward and reverse tracking. Specifically: Forward tracking: From frame 1 to frame 30, the SORT algorithm is used to track each target frame by frame based on the above detection results to generate a forward motion trajectory.

[0019] Reverse tracking: From frame 30 to frame 1, the SORT algorithm is used to perform reverse tracking to generate a reverse motion trajectory. Through this forward and reverse bidirectional tracking, the motion trajectory of the target in the current sliding window is captured as the basis for subsequent trajectory fusion.

[0020] Step S4, forward and reverse trajectory matching and association: In the overlapping area of ​​the window, based on the location information and motion information of the target, determine whether the forward and reverse trajectories belong to the same target. The specific preset matching strategies include: Position consistency: The positions of the target detection boxes in overlapping frames should be close, which can be determined by calculating the distance between the center points of the boxes or IoU (Intersection over Union).

[0021] Motion consistency: The motion direction and speed in the forward and reverse trajectories should be similar. By comparing the velocity vectors between adjacent frames, it is determined whether the target's motion trend is consistent.

[0022] When the forward and reverse trajectories are matched successfully, the consistency of the target ID in the forward and reverse trajectories is ensured.

[0023] Step S5, forward and reverse trajectory fusion: for the forward and reverse trajectories that have been successfully matched, the trajectory fusion process is performed: Overlapping interval fusion: For overlapping intervals (such as frames 1-17 and 20-28), the data of the forward trajectory is directly used. In frames where the forward trajectory is missing but the reverse trajectory exists (such as frames 29 and 30), the reverse trajectory is used to complete the trajectories.

[0024] Missing segment completion: When the forward or reverse trajectory is lost in some frames, if the trajectory in the other direction exists, its data is used to complete it. If both the forward and reverse trajectories are missing, the Kalman filter is used for trajectory prediction.

[0025] Step S6, Kalman filter prediction: For the case of short-term target loss (less than 10 frames lost), the Kalman filter is used to predict the trajectory based on the target's historical motion state (such as speed and acceleration). The forward and reverse Kalman filters calculate the prediction values ​​respectively, and the more accurate prediction result is selected in the fusion process to fill the trajectory of the lost frame.

[0026] Step S7, the window sliding is consistent with the target ID; when the sliding window is processed, ensure that the target ID is consistent in the new window. Through the trajectory matching results in the overlapping interval of the previous window, ensure that the target ID in the new window is consistent with the ID in the previous window to avoid ID jumps or confusion. Each time the window moves, use the tracking results of the previous window as the initialization condition to track the target in the new window to ensure the continuity of the target trajectory and ID.

[0027] Step S8, track management and re-identification: In the track splicing process, the present invention designs a track management module to handle occlusion, target loss, target reappearance, etc. Track management includes track update, re-identification, track termination and track initialization.

[0028] In order to improve the long-term relevance and accuracy of multi-pedestrian tracking, this paper proposes a target tracking method based on forward and reverse trajectory fusion. This method tracks the target in the video sequence forward and backward, and uses motion information to fuse the forward and reverse trajectories, which can effectively solve the trajectory discontinuity problem caused by occlusion or temporary loss. The fusion of forward and reverse trajectories can not only improve the robustness of short-term tracking, but also use the Kalman filter to accurately predict the target position when occlusion occurs, thereby maintaining the consistency of the trajectory.

[0029] Specifically, when processing videos, this method uses a sliding window to perform forward and reverse tracking of the target in each window. In each window, forward tracking is first performed to generate a forward trajectory from the first frame to the last frame; then reverse tracking is performed to generate a reverse trajectory from the last frame back to the first frame. The forward and reverse trajectories are matched in the overlapping area of ​​the window and fused using the motion consistency and position consistency criteria. For targets lost during forward or reverse tracking, the Kalman filter is used to predict the position of the target that was temporarily lost, thereby completing the trajectory and ensuring the continuity of tracking.

[0030] In addition, when the window slides, this method ensures that the target ID remains consistent during the window switching process by matching the targets in the overlapping area, avoiding ID confusion or jumps. This method can not only effectively deal with the problem of target occlusion in complex scenes, but also improve the robustness and accuracy of long-term target tracking, and is particularly suitable for tracking tasks in dense multi-pedestrian or complex dynamic backgrounds.

[0031] In summary, the present invention provides a robust solution for multi-target tracking in complex scenes through the fusion of forward and reverse trajectories and short-term prediction of Kalman filtering, which significantly improves the accuracy and stability of long-term target tracking.

[0032] The present invention provides a multi-target tracking method based on forward and reverse trajectory fusion. By tracking the targets in a video sequence forward and backward and combining a Kalman filter for motion prediction, a stable and continuous target motion trajectory is generated, which is suitable for the problems of target occlusion, motion change and background interference in complex scenes. In this method, the video sequence is firstly tracked forward and backward to obtain the forward trajectory and reverse trajectory of the target in each sliding window. Through the forward and reverse dual trajectory tracking mechanism, the influence of short-term occlusion and detection error on the trajectory can be effectively reduced. The trajectory of each sliding window is matched by position and motion information, and the trajectory is fused in the overlapping frame area to ensure the consistency of the target ID in the forward and reverse tracking. In the forward and reverse trajectory matching process, the present invention judges whether the trajectory belongs to the same target based on the principle of position consistency and motion information consistency. By calculating the center point position, velocity vector and other features of the trajectory, it is ensured that the spatial position and motion trend of the forward and reverse trajectories are consistent. When the forward and reverse trajectories are successfully matched, the target ID remains consistent and is fused according to the available frame segments of the trajectory. For overlapping areas, more stable trajectory segments are preferentially selected for combination, while for lost frame segments, Kalman filtering is used for prediction and completion to maintain trajectory continuity.

[0033] The present invention specially designs a Kalman filter prediction mechanism to solve the problem of tracking interruption caused by short-term occlusion or detection error of the target. In the case where the target is lost in some frames, the Kalman filter makes a short-term prediction of the target position based on the motion state (such as speed and acceleration) of the historical trajectory to complete the lost trajectory segment. The Kalman filter prediction results of forward and reverse tracking are calculated separately, and the result with higher accuracy is selected when the trajectory is fused. This mechanism is particularly suitable for scenes where the target disappears in a short period of time or the occlusion time is short (such as less than 10 frames), which greatly improves the stability and continuity of tracking. The present invention adopts a fixed-size sliding window technology to generate the target trajectory by processing the video sequence frame by frame. The sliding window slides forward with a fixed step size to ensure that the front and rear windows have an overlapping interval. By matching and fusion of trajectories in the overlapping interval, it is ensured that the target ID remains consistent between the sliding windows to avoid ID jumps or mismatches caused by window switching. In addition, each time the sliding window is processed, the tracking result of the previous window will be used as the initial condition of the next window to ensure the continuity and accuracy of the trajectory.

[0034] In dense scenes or complex environments, target occlusion is a common problem. The present invention handles the occlusion and reappearance of the target through a trajectory management module. When the target is temporarily occluded, the Kalman filter can make predictions based on the historical motion trajectory of the target to maintain the continuity of the trajectory; when the target reappears, it automatically identifies and restores the target ID by matching with the original trajectory to ensure the continuity of the trajectory. This function shows stronger robustness when dealing with multi-target occlusion in dense crowds. The present invention also designs a trajectory management and optimization module to handle complex situations such as long-term occlusion of the target, trajectory interruption and target reappearance. By combining forward and reverse trajectories, motion prediction and re-identification technology, the motion trajectory of each target can be effectively managed to avoid loss or incorrect association. At the same time, an incremental learning mechanism is adopted to automatically adjust the tracking strategy according to the dynamic changes of the scene, so that the system can adapt to complex and changing environments and improve tracking accuracy.

[0035] The multi-target tracking method based on forward and reverse trajectory fusion and Kalman filter prediction has strong adaptability and is particularly suitable for multi-target tracking tasks in complex scenes. It can not only effectively improve the accuracy and stability of target tracking, but also significantly reduce the incidence of false tracking, and is suitable for scenes with dense crowds, frequent occlusions, and complex backgrounds.

[0036] The specific implementation includes the following steps: Furthermore, in step 1, the video sequence of the dataset Target detection is required to obtain the position of the target in each frame. Here we select the YOLOX detector as the benchmark detector (you can choose it according to your actual situation) to obtain the coordinates, confidence score and category label of the detection box.

[0037] in, is the video frame at the nth moment.

[0038] Furthermore, in step 2, the sliding window is used to process the video sequence using a sliding window of a fixed size (e.g., 30 frames). The sliding window slides forward at a certain step size (e.g., 10 frames) to ensure that there is an overlapping area between each window. This sliding method ensures that the trajectories between different windows can be verified and matched with each other. The window settings are:

[0039] Among them, the number of overlapping frames is O, the window size is W, and the step size is S. Tracking is performed by sliding the window until the entire video sequence ends.

[0040] Furthermore, in step 3, target tracking (forward and reverse) is performed; in each sliding window, forward tracking (frame 1 to frame 30) is performed first, followed by reverse tracking (frame 30 to frame 1). Through these two steps, forward and reverse target trajectories are obtained, providing a data basis for subsequent matching and fusion. Based on target detection, the SORT (Simple Online and Realtime Tracking) algorithm is used for forward and reverse tracking.

[0041] Forward tracking: From frame 1 to frame 30, the SORT algorithm is used to track each target frame by frame based on the above detection results to generate a forward motion trajectory. From frame 30 to frame 1, the SORT algorithm is used to track in reverse order to generate a reverse motion trajectory. The tracking result is expressed as:

[0042] in, Indicates the forward tracking result. Indicates the forward tracking result. is the target ID, t is the timestamp, Through this forward and reverse bidirectional tracking, the motion trajectory of the target in the current sliding window is captured as the basis for subsequent trajectory fusion.

[0043] Furthermore, in step 4, the forward and reverse trajectories are matched and associated; in the overlapping area of ​​the window, based on the position information and motion information of the target, it is determined whether the forward and reverse trajectories belong to the same target. The specific matching strategies include: Position consistency: The positions of the target detection boxes in overlapping frames should be close, which can be determined by calculating the distance between the center points of the boxes or IoU (Intersection over Union). It can be expressed as:

[0044] in, and Respectively represent the center coordinates of the forward and reverse tracking boxes of the i-th frame, is the position threshold.

[0045] Motion consistency: The motion direction and speed in the forward and reverse trajectories should be similar. By comparing the velocity vectors between adjacent frames, we can determine whether the motion trend of the target is consistent. It can be expressed as:

[0046] in, and Represent the velocity vectors of the forward and reverse trajectories respectively, is the motion similarity threshold.

[0047] Furthermore, in step 5, the forward and reverse trajectories are fused; for the forward and reverse trajectories that are successfully matched, the trajectory fusion process is performed: Overlapping interval fusion: For overlapping intervals (such as frames 1-17 and 20-28), the data of the forward trajectory is directly used. In frames where the forward trajectory is missing but the reverse trajectory exists (such as frames 29 and 30), the reverse trajectory is used to complete the trajectories.

[0048]

[0049] by The more stable trajectory is selected as the retained one, and when the forward or reverse trajectory is lost in some frames, if the trajectory in the other direction exists, its data is used to complete it. If both the forward and reverse trajectories are missing, the Kalman filter is used for trajectory prediction.

[0050] Furthermore, in step 6, Kalman filter prediction is used to predict the trajectory of the target based on its historical motion state (such as speed and acceleration) in the case of short-term target loss (less than 10 frames lost). The forward and reverse Kalman filters calculate the prediction values ​​respectively, and the more accurate prediction result is selected in the fusion process to fill in the trajectory of the lost frame.

[0051] Furthermore, in step 7, the window sliding is consistent with the target ID; when the sliding window is processed, ensure that the target ID is consistent in the new window. Through the trajectory matching results in the overlapping interval of the previous window, ensure that the target ID in the new window is consistent with the ID in the previous window to avoid ID jumps or confusion. Each time the window moves, use the tracking results of the previous window as the initialization condition to track the target in the new window to ensure the continuity of the target trajectory and ID. ID consistency strategy:

[0052] in, Indicates that the ID of the same target remains unchanged during the window sliding process. It indicates that a new target has appeared during the window sliding process and a new ID is assigned to it.

[0053] Furthermore, in step S8 of step 8, track management and re-identification: in the track splicing process, the present invention designs a track management module to handle situations such as occlusion, target loss and target reappearance. Track management includes track update, re-identification, track termination and track initialization.

Claims

1. A multi-target tracking method based on forward and reverse trajectory fusion, characterized in that: The method comprises: Step S1, for each video sequence to be tracked, use a deep learning-based target detector to detect the target in each frame, and obtain the coordinates and confidence score of each target detection frame; Step S2, processing the video sequence using a sliding window of a fixed size according to the coordinates and confidence scores of each detection frame; Step S3, in each sliding window, first perform forward tracking, then perform reverse tracking, through forward tracking and reverse tracking, obtain the forward target trajectory and the reverse target trajectory, and provide a data basis for subsequent matching and fusion; Step S4, in the overlapping area of ​​the sliding window, based on the position information and motion information of the target, a preset matching strategy is adopted to determine whether the target trajectory of the forward direction and the target trajectory of the reverse direction belong to the same target; Step S5, if the target trajectory of the forward direction and the target trajectory of the reverse direction belong to the same target, the forward and reverse trajectories that are successfully matched are fused to obtain a fused trajectory; Step S6, splicing the fused trajectories to obtain the tracking trajectory of the target.

2. The method according to claim 1, characterized in that The method further comprises: Step S7: In case of short-term target loss, a Kalman filter is used to predict the trajectory based on the historical motion state of the target.

3. The method according to claim 1, characterized in that The forward tracking includes: From the initial frame to the last frame of the sliding window, the SORT algorithm is used to track each target frame by frame based on the above detection results to generate a positive target trajectory.

4. The method according to claim 1, characterized in that: The reverse tracking includes: using the SORT algorithm to perform reverse tracking from the tail frame of the sliding window to the initial frame to generate a reverse target trajectory.

5. The method according to claim 1, characterized in that: The preset matching strategy includes: Position consistency: The positions of the object detection boxes in overlapping frames should be close, which can be determined by calculating the center point distance or IoU of the boxes. Motion consistency: The motion direction and speed in the forward and reverse trajectories should be similar. By comparing the velocity vectors between adjacent frames, it is determined whether the motion trend of the target is consistent.

6. The method according to claim 1, characterized in that The step S5 comprises: For the trajectories in the overlapping intervals, the positive target trajectory is directly used as the fused trajectory; If the forward target trajectory is missing but the reverse target trajectory exists, the reverse target trajectory is used as the fused trajectory; If both the forward and reverse target trajectories are missing, the Kalman filter is used for trajectory prediction, and the predicted trajectory is used as the fused trajectory.

7. The method according to claim 1, characterized in that The step S6 comprises: During the trajectory stitching process, trajectory updating, re-identification, trajectory termination and trajectory initialization are adopted to handle abnormal situations such as occlusion, target loss and target reappearance.

8. The method according to claim 2, characterized in that: The step S7 comprises: The forward and backward Kalman filters calculate the prediction values ​​separately, and select the more accurate prediction results in the fusion process to fill the trajectory of the lost frame.

Images