A multi-target tracking method based on forward and backward trajectory fusion

Abstract

The application provides a multi-target tracking method based on forward and reverse trajectory fusion, which is particularly suitable for multi-target tracking tasks in complex scenes. The method uses sliding window technology to perform forward and reverse target tracking respectively, and uses forward and reverse trajectory matching and fusion strategies to ensure the consistency of target ID throughout the tracking process. For short-term target loss caused by occlusion or other reasons, Kalman filter is used for trajectory prediction and completion to ensure the continuity and accuracy of the trajectory. In the sliding window, the trajectory matching in the overlapping interval keeps the target ID consistent, effectively avoiding ID jump or misidentification. The application can maintain efficient and stable multi-target tracking effect in complex scenes with occlusion, motion change and background interference, and is particularly suitable for tracking tasks in dense pedestrian or dynamic scenes. Through this method, the accuracy and robustness of multi-target tracking are significantly improved.

Description

Technical Field

[0001] This application belongs to the field of computer vision and video analysis technology, specifically relating to a multi-target tracking method based on forward and reverse trajectory fusion, which is particularly suitable for complex scenarios with occlusion or temporary loss. Background Technology

[0002] Multi-object tracking (MOT), a core task in computer vision, has broad application prospects, including security monitoring, autonomous driving, behavior analysis, and pose estimation. Its task is to identify and output the motion trajectory of targets from video sequences. Among specific applications of MOT, multi-pedestrian tracking has become a research hotspot in this field due to complex scenes and severe occlusion.

[0003] In recent years, with the improvement of object detection algorithm performance, detection-based multi-pedestrian tracking frameworks have gradually become mainstream. These methods first detect objects in each frame of a video, then correlate motion information and appearance features to generate a complete trajectory for each object. Current tracking algorithms typically rely on learning short-term motion patterns to match identical objects in adjacent frames, while using appearance features to identify objects that have been missing for extended periods. However, in scenarios with dense crowds or frequent object occlusion, relying solely on motion prediction and appearance matching can easily lead to unstable object tracking and difficulty in accurately correlating target trajectories over long periods. Summary of the Invention

[0004] Objective: To address the shortcomings of existing multi-target tracking methods in handling dense crowd occlusion, motion changes, and background interference, this invention proposes a multi-target tracking method based on forward and reverse trajectory fusion. This method performs forward and reverse trajectory tracking on a video sequence and combines it with Kalman filter for motion prediction, achieving long-term correlation and accurate tracking of target trajectories. It is particularly suitable for handling short-term loss and occlusion problems in complex scenes. This invention aims to improve the accuracy and robustness of target tracking by effectively fusing forward and reverse tracking results, ensuring the consistency of target IDs and the continuity of trajectories.

[0005] This application provides a multi-target tracking method based on forward and reverse trajectory fusion, the method comprising:

[0006] Step S1: For each video sequence to be tracked, use a deep learning-based object detector to detect the target in each frame and obtain the coordinates and confidence score of each target detection box.

[0007] Step S2: Based on the coordinates and confidence scores of each detection box, a fixed-size sliding window is used to process the video sequence;

[0008] Step S3: Within each sliding window, first perform forward tracking, then perform reverse tracking. Through forward and reverse tracking, obtain the forward target trajectory and the reverse target trajectory to provide a data foundation for subsequent matching and fusion.

[0009] Step S4: Within the overlapping area of ​​the sliding window, based on the target's position and motion information, a preset matching strategy is used to determine whether the forward target trajectory and the reverse target trajectory belong to the same target.

[0010] Step S5: If the target trajectory of the forward direction and the target trajectory of the reverse direction belong to the same target, perform trajectory fusion processing on the successfully matched forward and reverse trajectories to obtain the fused trajectory.

[0011] Step S6: Piece together the fused trajectories to obtain the target's tracking trajectory.

[0012] Preferably, the method further includes:

[0013] Step S7: For the case of short-term target loss, use a Kalman filter to predict the trajectory based on the target's historical motion state.

[0014] Preferably, the forward tracking includes:

[0015] From the initial frame to the last frame of the sliding window, the SORT algorithm is used to track each target frame by frame based on the aforementioned detection results, generating a positive target trajectory.

[0016] Preferably, the reverse tracking includes: performing reverse tracking from the last frame of the sliding window to the first frame using the SORT algorithm to generate a reverse target trajectory.

[0017] Preferably, the preset matching strategy includes:

[0018] Positional consistency: The target detection boxes in overlapping frames should be close in position, which is determined by calculating the distance between the center points of the boxes or the IoU.

[0019] Motion consistency: The direction and speed of motion in forward and reverse trajectories should be similar. By comparing the velocity vectors between adjacent frames, it can be determined whether the motion trend of the target is consistent.

[0020] Preferably, step S5 includes:

[0021] For trajectories in overlapping regions, the positive target trajectory is directly used as the fused trajectory;

[0022] If the forward target trajectory is missing, but the reverse target trajectory exists, then the reverse target trajectory is used as the fused trajectory.

[0023] If both forward and reverse target trajectories are missing, a Kalman filter is used for trajectory prediction, and the predicted trajectory is used as the fused trajectory.

[0024] Preferably, step S6 includes:

[0025] During the trajectory stitching process, trajectory update, re-identification, trajectory termination, and trajectory initialization are employed to handle abnormal situations such as occlusion, target loss, and target reappearance.

[0026] Preferably, step S7 includes:

[0027] Forward and inverse Kalman filtering are used to calculate predictions, and the more accurate prediction is selected during the fusion process to fill in the trajectory of the missing frames.

[0028] This application has the following technical advantages:

[0029] 1) Traditional multi-target tracking methods often encounter problems such as track trajectory interruption or ID jumps when dealing with complex scenes, especially in densely populated areas, severe occlusion, or when the target undergoes violent movement. This invention effectively solves these problems by fusing forward and reverse trajectories bidirectionally and combining them with Kalman filter motion prediction. Specifically, through forward and reverse trajectory matching and completion, lost frames can be completed using reverse trajectories when the target is briefly occluded or disappears, greatly improving the continuity and stability of the trajectory. At the same time, the prediction of the Kalman filter further enhances the trajectory accuracy in short-term loss cases, ensuring accurate target prediction and tracking even in cases of occlusion or detection loss.

[0030] 2) In multi-target tracking, maintaining target ID consistency is crucial for achieving long-term stable tracking, especially during window sliding. This invention ensures that the ID of each target remains consistent across different sliding windows by matching and associating forward and reverse trajectories in overlapping frames, avoiding ID jumps or reallocation of IDs after target loss. This not only improves tracking coherence but also effectively avoids mismatches that may occur during target re-identification. Furthermore, the overlapping region design during window sliding also helps verify the tracking results of preceding and following windows, making the target ID more robust throughout the video sequence, particularly suitable for scenes with dense pedestrian traffic or high-speed movement.

[0031] 3) Existing multi-target tracking technologies often struggle to accurately re-identify targets when faced with prolonged occlusion or the reappearance of pedestrians, leading to trajectory interruptions or loss. To address this issue, this invention designs a trajectory management module. Through a re-identification strategy within trajectory management, it can efficiently identify and recover the previous ID when the target reappears, avoiding the problem of reassigning a new ID. Furthermore, utilizing the complementary characteristics of forward and reverse trajectories, it can better handle loss during occlusion. In the case of short-term loss, a Kalman filter is used to accurately predict the target, achieving a smooth trajectory transition; when the target reappears, matching with the previous trajectory enables accurate association and re-identification of the same target. This makes this invention exhibit stronger adaptability and stability when facing prolonged occlusion, complex motion, or background interference. Attached Figure Description

[0032] Figure 1 This is a flowchart of a multi-target tracking method based on forward and reverse trajectory fusion provided in an embodiment of this application. Detailed Implementation

[0033] This invention provides a multi-target tracking method based on forward and reverse trajectory fusion, particularly suitable for multi-target tracking tasks in complex scenes. This method employs a sliding window technique to track targets in both forward and reverse directions separately, utilizing a matching and fusion strategy for forward and reverse trajectories to ensure target ID consistency throughout the tracking process. For short-term target loss due to occlusion or other reasons, a Kalman filter is used for trajectory prediction and completion, ensuring trajectory continuity and accuracy. Between sliding windows, trajectory matching in overlapping intervals maintains target ID consistency, effectively avoiding ID jumps or misidentification. This invention maintains efficient and stable multi-target tracking performance in complex scenes with occlusion, motion changes, and background interference, and is especially suitable for tracking tasks in densely populated or dynamic scenes. This method significantly improves the accuracy and robustness of multi-target tracking.

[0034] Please see Figure 1 A multi-target tracking method based on forward and reverse trajectory fusion includes the following steps:

[0035] Step S1, Object Detection: For each video sequence to be tracked, a deep learning-based object detector (e.g., YOLOX) is used to detect objects in each frame. The YOLOX detector has high detection accuracy and real-time performance, and can effectively detect multiple objects in a video sequence, obtaining the coordinates and confidence score of each bounding box.

[0036] Step S2, Sliding Window: The video sequence is processed using a fixed-size sliding window (e.g., 30 frames). The sliding window slides forward in steps (e.g., 10 frames) to ensure overlapping areas between each window. This sliding method ensures that the trajectories of different windows can be mutually verified and matched.

[0037] Step S3, Target Tracking (Forward and Backward): Within each sliding window, forward tracking is performed first (frames 1 to 30), followed by backward tracking (frames 30 to 1). These two steps acquire the forward and backward target trajectories, providing a data foundation for subsequent matching and fusion. Based on target detection, the SORT (Simple Online and Realtime Tracking) algorithm is used for forward and backward tracking respectively. Specifically:

[0038] Forward tracking: From frame 1 to frame 30, the SORT algorithm is used to track each target frame by frame based on the aforementioned detection results, generating a forward motion trajectory.

[0039] Reverse tracking: From frame 30 to frame 1, the SORT algorithm is used for reverse tracking to generate a reverse motion trajectory. Through this bidirectional tracking, the target's motion trajectory within the current sliding window is captured, serving as the basis for subsequent trajectory fusion.

[0040] Step S4, forward and reverse trajectory matching and association; within the overlapping area of ​​the window, based on the target's position and motion information, determine whether the forward and reverse trajectories belong to the same target. Specific preset matching strategies include:

[0041] Positional consistency: The target detection boxes in overlapping frames should be close in position, which can be determined by calculating the distance between the center points of the boxes or by IoU (Intersection over Union).

[0042] Motion consistency: The direction and velocity of motion in both forward and reverse trajectories should be similar. The consistency of the target's motion trend is determined by comparing the velocity vectors between adjacent frames.

[0043] Once the forward and reverse trajectories are successfully matched, ensure that the target ID is consistent in both forward and reverse trajectories.

[0044] Step S5, forward and reverse trajectory fusion; for successfully matched forward and reverse trajectories, trajectory fusion processing is performed:

[0045] Overlapping region fusion: For trajectories in overlapping regions (such as frames 1-17 and 20-28), the forward trajectory data is used directly. In frames where the forward trajectory is missing but the reverse trajectory exists (such as frames 29 and 30), the reverse trajectory is used to complete the trajectories.

[0046] Lost segment completion: When a forward or reverse trajectory is lost in some frames, if the trajectory in the other direction exists, its data is used to complete the missing segment. If both forward and reverse trajectories are missing, a Kalman filter is used for trajectory prediction.

[0047] Step S6, Kalman filter prediction: For short-term target loss (less than 10 frames lost), a Kalman filter is used to predict the trajectory based on the target's historical motion state (such as velocity and acceleration). Forward and reverse Kalman filters calculate the predicted values ​​respectively, and the more accurate prediction result is selected during the fusion process to fill in the trajectory of the lost frames.

[0048] Step S7: Maintain consistency between window sliding and target ID; during window sliding, ensure the target ID remains consistent within the new window. Using the trajectory matching results within the overlapping area of ​​the previous window, ensure the target ID in the new window is consistent with the ID in the previous window, avoiding ID jumps or confusion. Each time the window moves, use the tracking results of the previous window as initialization conditions to track the target in the new window, ensuring the continuity of the target trajectory and ID.

[0049] Step S8, Trajectory Management and Re-identification; During the trajectory stitching process, this invention designs a trajectory management module to handle situations such as occlusion, target loss, and target reappearance. Trajectory management includes trajectory update, re-identification, trajectory termination, and trajectory initialization.

[0050] To improve the long-term consistency and accuracy of multi-pedestrian tracking, this paper proposes a target tracking method based on forward and reverse trajectory fusion. This method tracks targets in video sequences in both forward and reverse directions and fuses the forward and reverse trajectories using motion information, effectively addressing trajectory discontinuities caused by occlusion or brief loss of track. The fusion of forward and reverse trajectories not only improves the robustness of short-term tracking but also utilizes a Kalman filter to accurately predict the target position when occlusion occurs, thus maintaining trajectory consistency.

[0051] Specifically, this method employs a sliding window approach to track targets in both forward and reverse directions within each video processing window. Within each window, forward tracking is performed first, generating a forward trajectory from the first frame to the last; then, reverse tracking is performed, reversing the order from the last frame back to the first frame to generate a reverse trajectory. The forward and reverse trajectories are matched in the overlapping area of ​​the window and fused using motion consistency and position consistency criteria. For targets lost during forward or reverse tracking, a Kalman filter is used to predict the location of the temporarily lost target, thereby completing the trajectory and ensuring tracking continuity.

[0052] Furthermore, this method ensures that the target ID remains consistent during window switching by matching targets within overlapping areas when the window slides, thus avoiding ID confusion or jumps. This method not only effectively addresses target occlusion issues in complex scenes but also improves the robustness and accuracy of long-term target tracking, making it particularly suitable for tracking tasks in densely populated or complex dynamic backgrounds.

[0053] In summary, this invention provides a robust solution for multi-target tracking in complex scenarios by fusing forward and reverse trajectories and using Kalman filtering for short-term prediction, significantly improving the accuracy and stability of long-term target tracking.

[0054] This invention provides a multi-target tracking method based on forward and reverse trajectory fusion. By tracking targets in a video sequence in both forward and reverse directions and combining this with Kalman filter motion prediction, stable and continuous target motion trajectories are generated, making it suitable for complex scenes with target occlusion, motion changes, and background interference. In this method, forward and reverse tracking is first performed on the video sequence to obtain the forward and reverse trajectories of the target within each sliding window. This dual trajectory tracking mechanism effectively reduces the impact of short-term occlusion and detection errors on the trajectory. The trajectory of each sliding window is matched using position and motion information, and trajectories are fused within overlapping frame regions to ensure the consistency of the target ID in both forward and reverse tracking. During the forward and reverse trajectory matching process, this invention determines whether the trajectories belong to the same target based on the principles of position consistency and motion information consistency. By calculating features such as the center point position and velocity vector of the trajectory, the spatial position and motion trend of the forward and reverse trajectories are ensured to be consistent. When the forward and reverse trajectories are successfully matched, the target ID remains consistent, and the trajectories are fused based on available frame segments. For overlapping areas, relatively stable trajectory segments are selected for combination first, while for lost frames, Kalman filtering is used for prediction and completion to maintain the continuity of the trajectory.

[0055] This invention employs a Kalman filter prediction mechanism to address tracking interruptions caused by brief target occlusion or detection errors. For cases where the target is lost in certain frames, the Kalman filter performs short-term predictions of the target's position based on the motion state (e.g., velocity, acceleration) of historical trajectories, filling in the missing trajectory segments. The Kalman filter prediction results for both forward and reverse tracking are calculated separately, and the more accurate result is selected during trajectory fusion. This mechanism is particularly suitable for scenarios where the target disappears briefly or the occlusion time is short (e.g., less than 10 frames), significantly improving tracking stability and continuity. This invention utilizes a fixed-size sliding window technique, generating the target trajectory by processing the video sequence frame by frame. The sliding window moves forward with a fixed step size, ensuring overlapping intervals between windows. Trajectory matching and fusion within these overlapping intervals ensure that the target ID remains consistent across sliding windows, avoiding ID jumps or mismatches caused by window switching. Furthermore, during each sliding window processing, the tracking result of the previous window is used as the initial condition for the next window, ensuring trajectory continuity and accuracy.

[0056] Target occlusion is a common problem in dense scenes or complex environments. This invention addresses target occlusion and re-emergence through a trajectory management module. When a target is temporarily occluded, a Kalman filter predicts its trajectory based on its historical motion trajectory, maintaining trajectory continuity. When the target re-emerges, it automatically identifies and recovers its target ID by matching it with the original trajectory, ensuring trajectory consistency. This function demonstrates stronger robustness when handling multi-target occlusion in dense crowds. The invention also includes a trajectory management and optimization module to handle complex situations such as prolonged target occlusion, trajectory interruption, and target re-emergence. By combining forward and reverse trajectories, motion prediction, and re-identification techniques, the system effectively manages the motion trajectory of each target, avoiding loss or incorrect association. Simultaneously, an incremental learning mechanism is employed to automatically adjust the tracking strategy based on dynamic scene changes, enabling the system to adapt to complex and ever-changing environments and improving tracking accuracy.

[0057] This invention presents a multi-target tracking method based on forward and reverse trajectory fusion and Kalman filter prediction, which has strong adaptability and is particularly suitable for multi-target tracking tasks in complex scenes. It not only effectively improves the accuracy and stability of target tracking but also significantly reduces the incidence of erroneous tracking, making it applicable to scenes with dense crowds, frequent occlusion, and complex backgrounds.

[0058] The specific implementation includes the following steps:

[0059] Furthermore, in step 1, the video sequences of the dataset... Target detection is required to obtain the position of the target in each frame. Here we select the YOLOX detector as the baseline detector (you can choose according to the actual situation) to obtain the coordinates of the detection box, confidence score and category label.

[0060] in, Let be the video frame at time n.

[0061] Furthermore, in step 2, a sliding window is used; a fixed-size sliding window (e.g., 30 frames) is used to process the video sequence. The sliding window slides forward in a certain step (e.g., 10 frames) to ensure that there is an overlapping area between each window. This sliding method ensures that the trajectories between different windows can be mutually verified and matched. The window is set as follows:

[0062]

[0063] The number of overlapping frames is O, the window size is W, and the step size is S. Tracking is performed using a sliding window until the entire video sequence ends.

[0064] Further, step 3 involves target tracking (forward and reverse). Within each sliding window, forward tracking is performed first (frames 1 to 30), followed by reverse tracking (frames 30 to 1). These two steps acquire the forward and reverse target trajectories, providing a data foundation for subsequent matching and fusion. Based on target detection, the SORT (Simple Online and Realtime Tracking) algorithm is used for both forward and reverse tracking.

[0065] Positive tracking:

[0066] From frame 1 to frame 30, the SORT algorithm is used to track each target frame by frame based on the aforementioned detection results, generating a forward motion trajectory. From frame 30 to frame 1, the SORT algorithm is used for reverse tracking, generating a reverse motion trajectory. The tracking results are represented as follows:

[0067]

[0068] in, This indicates a positive tracking result. This indicates a positive tracking result. Let t be the target ID and t be the timestamp. This represents the target detection bounding box. Through this bidirectional tracking, the motion trajectory of the target within the current sliding window is captured, serving as the basis for subsequent trajectory fusion.

[0069] Further, step 4 involves matching and associating forward and reverse trajectories. Within the overlapping area of ​​the window, based on the target's position and motion information, it is determined whether the forward and reverse trajectories belong to the same target. Specific matching strategies include:

[0070] Positional consistency: Target detection bounding boxes in overlapping frames should be close in position, which can be determined by calculating the distance between the center points of the boxes or by using the Intersection over Union (IoU). This can be represented as:

[0071]

[0072] in, and These represent the center coordinates of the forward and reverse tracking boxes in the i-th frame, respectively. This is the location threshold.

[0073] Motion consistency: The direction and velocity of motion in both forward and reverse trajectories should be similar. This is determined by comparing the velocity vectors between adjacent frames to determine if the target's motion trend is consistent. This can be represented as:

[0074]

[0075] in, and These represent the velocity vectors of the forward and reverse trajectories, respectively. This is the motion similarity threshold.

[0076] Furthermore, step 5 merges the forward and reverse trajectories; for successfully matched forward and reverse trajectories, trajectory fusion processing is performed:

[0077] Overlapping region fusion: For trajectories in overlapping regions (such as frames 1-17 and 20-28), the forward trajectory data is used directly. In frames where the forward trajectory is missing but the reverse trajectory exists (such as frames 29 and 30), the reverse trajectory is used to complete the trajectories.

[0078]

[0079] by The more stable trajectory is selected for preservation, and if a forward or reverse trajectory is lost in some frames, its data is used to complete the trajectory if it exists in the other direction. If both forward and reverse trajectories are missing, a Kalman filter is used for trajectory prediction.

[0080] Furthermore, step 6 involves Kalman filtering prediction. For cases of short-term target loss (less than 10 frames lost), a Kalman filter is used to predict the trajectory based on the target's historical motion state (e.g., velocity, acceleration). Forward and inverse Kalman filtering are used to calculate predictions, and the more accurate prediction is selected during the fusion process to fill in the trajectory of the lost frames.

[0081] Furthermore, in step 7, the window sliding maintains consistency with the target ID; during the sliding window processing, ensure that the target ID remains consistent within the new window. By using the trajectory matching results within the overlapping area of ​​the previous window, ensure that the target ID in the new window is consistent with the ID in the previous window, avoiding ID jumps or confusion. Each time the window moves, use the tracking results of the previous window as initialization conditions to perform target tracking in the new window, ensuring the continuity of the target trajectory and ID. ID consistency strategy:

[0082]

[0083] in, This indicates that the ID of the same target remains unchanged during window sliding. This indicates that a new target has appeared during the window sliding process, and a new ID has been assigned to it.

[0084] Furthermore, in step S8 of step 8, trajectory management and re-identification; during the trajectory stitching process, this invention designs a trajectory management module to handle situations such as occlusion, target loss, and target reappearance. Trajectory management includes trajectory update, re-identification, trajectory termination, and trajectory initialization.

Claims

1. A multi-target tracking method based on forward and reverse trajectory fusion, characterized in that, The method includes: Step S1: For each video sequence to be tracked, use a deep learning-based object detector to detect the target in each frame and obtain the coordinates and confidence score of each target detection box. Step S2: Based on the coordinates and confidence scores of each target detection box, a fixed-size sliding window is used to process the video sequence; Step S3: Within each sliding window, forward tracking is performed first, followed by reverse tracking. Through forward and reverse tracking, the forward and reverse target trajectories are obtained, providing a data foundation for subsequent matching and fusion. Step S4: Within the overlapping area of ​​the sliding window, based on the target's position and motion information, a preset matching strategy is used to determine whether the forward target trajectory and the reverse target trajectory belong to the same target. Step S5: If the target trajectory of the forward direction and the target trajectory of the reverse direction belong to the same target, perform trajectory fusion processing on the successfully matched forward and reverse trajectories to obtain the fused trajectory. Step S6: Piece together the fused trajectories to obtain the target's tracking trajectory.

2. The method according to claim 1, characterized in that, The method further includes: Step S7: For the case of short-term target loss, use a Kalman filter to predict the trajectory based on the target's historical motion state.

3. The method according to claim 1, characterized in that, The forward tracking includes: From the initial frame to the last frame of the sliding window, the SORT algorithm is used to track each target frame by frame based on the coordinates and confidence scores of each target detection box obtained in step S1, generating a positive target trajectory.

4. The method according to claim 1, characterized in that, The reverse tracking includes: performing reverse tracking from the last frame of the sliding window to the first frame, using the SORT algorithm to generate a reverse target trajectory.

5. The method according to claim 1, characterized in that, The preset matching strategy includes: Positional consistency: The target detection boxes in overlapping frames should be close in position, which is determined by calculating the distance between the center points of the boxes or the IoU. Motion consistency: The direction and speed of motion in forward and reverse trajectories should be similar. By comparing the velocity vectors between adjacent frames, it can be determined whether the motion trend of the target is consistent.

6. The method according to claim 1, characterized in that, Step S5 includes: For trajectories in overlapping regions, the positive target trajectory is directly used as the fused trajectory; If the forward target trajectory is missing, but the reverse target trajectory exists, then the reverse target trajectory is used as the fused trajectory. If both forward and reverse target trajectories are missing, a Kalman filter is used for trajectory prediction, and the predicted trajectory is used as the fused trajectory.

7. The method according to claim 1, characterized in that, Step S6 includes: During the trajectory stitching process, trajectory update, re-identification, trajectory termination, and trajectory initialization are employed to handle abnormal situations such as occlusion, target loss, and target reappearance.

8. The method according to claim 2, characterized in that, Step S7 includes: Forward and inverse Kalman filtering are used to calculate predictions, and the more accurate prediction is selected during the fusion process to fill in the trajectory of the missing frames.

Images