Dynamic scene multi-target tracking method fusing trajectory association technology

By employing the D-IoU matching algorithm, trajectory link fusion model, and multiple additive regression algorithm, the problems of trajectory interruption and fragmentation in multi-target tracking in dynamic environments are solved, achieving higher tracking accuracy and continuity.

CN119048557APending Publication Date: 2024-11-29NANTONG UNIV
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202411156580.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In dynamic environments, multi-target tracking algorithms face challenges such as target diversity, increased complexity, rapid movement, target occlusion, and environmental noise interference, leading to trajectory interruption and fragmentation. Existing methods struggle to accurately track and recover the true motion path.

Method used

The D-IoU matching algorithm is used to expand the matching space and construct spatiotemporal similarity to compensate for motion estimation bias; the trajectory linking and fusion model transforms short trajectories into long trajectories through spatiotemporal correlation; and the multiple additive regression algorithm adaptively approximates the real trajectory and reconnects fragmented trajectories.

Benefits of technology

It improves tracking accuracy and continuity in complex and dynamic scenarios, better adapts to various environmental changes, and enhances the ability to track pedestrian targets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048557A_ABST
    Figure CN119048557A_ABST
Patent Text Reader

Abstract

The invention provides a dynamic scene multi-target tracking method fusing a trajectory association technology, and belongs to the technical field of computer vision and artificial intelligence. The technical problem that in the prior art, the pedestrian tracking capability is insufficient when complex, diversified and crowded scenes are processed under the dynamic background is solved. According to the technical scheme, the method comprises the following steps: S1, video frame preprocessing and target detection; s2, tracking optimization of the fast moving target; s3, track linking and fusion; and S4, track interruption and fragmentization are repaired. The method has the beneficial effects that in order to improve the tracking capability of multi-target pedestrians under a complex dynamic background, accurate extraction and tracking of pedestrian features are realized by adopting an advanced deep learning algorithm and a computer vision technology, and more reliable technical support is provided for the fields of intelligent monitoring, automatic driving and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a dynamic scene multi-target tracking method integrating trajectory association technology. Background Art

[0002] Multi-object tracking (MOT) is a key task in the field of computer vision and plays an important role in applications such as intelligent transportation systems and video surveillance [1]. In particular, in the field of video surveillance, MOT technology is widely used in crime analysis and prevention, providing strong support for maintaining social security. In addition, MOT is the basis of advanced computer vision tasks such as posture estimation, behavior recognition, and behavior analysis [2,3]. Performing the MOT task requires detecting the input video sequence frame by frame to obtain the target position information, extracting the target's re-identification appearance features, combining motion and re-identification models to calculate similarity, and assigning a fixed-length ID to each target, and finally obtaining the trajectory information of the tracked target. With the development of deep learning technology, deep learning-based MOT methods have received great attention, especially due to the advancement of object detection technology. High-performance object detection models [4] have triggered interest in detection-based tracking methods. At present, deep learning-based methods decompose the MOT task into subtasks such as object detection, feature extraction, and data association, making the tracking algorithm more efficient and accurate.

[0003] However, in dynamic environments, MOT faces a series of challenges. First, the diversity and complexity of targets increase the difficulty of tracking. The changes in target shape and the interference of similar targets often blur and distort pedestrian motion patterns, posing challenges to tracking algorithms as they find it difficult to accurately distinguish and track these targets. This may lead to misidentification of one target as multiple tracks or misclassification of multiple targets under the same track. This problem is particularly common in online trackers that lack global correlation information. In order to better utilize the information between tracks, some methods [5,6] propose to use link models to connect short tracks with tracks. However, these methods often result in incomplete track fragments. Although these methods can improve tracking performance, they rely on computationally expensive models, especially appearance embedding models. Second, the rapid movement of targets can also lead to tracking deviations. When tracking target categories such as pedestrians, these factors often cause tracking interference, making multi-target tracking more difficult. Generally, IOU is widely used to measure geometric consistency [7-9]. Some methods [11,12] introduce transformers

[13] into multi-target tracking research, achieving accurate measurement of cross-frame appearance similarity, thereby achieving good tracking performance. However, in practice, motion models may not always be accurate, especially when pedestrians move irregularly. For example, in the MOT20 dataset, where pedestrian density is higher and motion patterns are more complex and diverse, motion estimation models often fail. Finally, target occlusion and environmental noise interference lead to trajectory interruption and fragmented trajectory problems. Linear interpolation

[14] is widely used to compensate for these missed detections, but it ignores motion information during the interpolation process, limiting the tracking accuracy. Recently, Strongsort

[15] proposed to use Gaussian process regression to model nonlinear motion to alleviate this problem. However, these interpolation methods do not take into account the actual trajectory of the real path, because objects in video frames do not move along smooth curves. In order to solve the above problems, more effective algorithms are needed that can better simulate the real motion path of the target and adapt to various complex environments and dynamic changes, thereby improving the robustness and accuracy of multi-target tracking. In the past decade, multi-target tracking has received widespread attention and produced many important research results. Two main tracking paradigms have emerged in multi-target tracking: Tracking-by-Detection (TBD) and Joint-Detection-Tracking (JDT). TBD uses a detector to detect targets in each frame, and then uses a data association method to associate the detection results with the tracked targets in the previous frame to obtain the target's motion trajectory. JDT performs target detection and tracking simultaneously in each frame. Among these works, Alex et al.

[16] proposed the SORT algorithm, which combines the Kalman filter and the Hungarian algorithm to construct a TBD framework.Nicolai et al.

[17] proposed the DeepSORT method, which uses a deep appearance feature extraction model and a cascade matching strategy using the weighted sum of Mahalanobis distances to successfully reduce the number of ID switches by 45%. Sun et al.

[18] proposed the TransTrack method, which uses the attention mechanism to simultaneously perform three tasks of target detection, feature extraction, and data association in a single network, establishing a new JDT paradigm. Zeng et al.

[19] solved the drift noise problem by improving the Kalman filter and proposed the NCT method. In addition, Zhang et al.

[14] proposed the ByteTrack algorithm, which uses Yolox [4] as a detector, tracks by associating almost all detection frames, and uses the similarity between low-scoring detection frames and trajectories to recover the true target and filter background detections. Aharon et al.

[20] proposed the BoT-SORT method, which improves the robustness of the tracker by improving the Kalman filter state vector and camera motion compensation. Cao et al.

[21] proposed OC-SORT, which uses an observation-based update strategy to reduce the accumulated error. Inspired by the above, a multi-target tracking algorithm DSTrack suitable for dynamic scenes is proposed to better adapt to the needs of various complex scenes and dynamic changes.

[0004] When dealing with missing tracks, some methods [5-6, 22] adopt global association models to improve the performance of multi-object tracking (MOT). These methods usually generate incomplete track segments based on appearance information and then use global information for association to alleviate track interruption caused by occlusion and interference of similar targets. For example, ReMOT

[22] segments and merges incomplete track segments by learning improved appearance features through self-supervision. TPM [6] introduces a track plane matching process to assign visually confusing tracks to different track planes, thereby reducing confusion between similar tracks. GIAOTrack [5] uses an improved ResNet50-TP model to obtain the appearance features of the track and extract global and partial spatial features of the track for a more robust representation. Although these methods have achieved certain results, they heavily rely on the appearance features of the target and bring significant computational costs. At the same time, track interruption may lead to track fragmentation, drift and other track noise problems. When severe occlusion or blur occurs, the detector will inevitably lead to a large number of missed detections, which not only affects the continuous tracking of the target but also may degrade the performance of the entire tracking system. Existing linear interpolation methods

[14] ignore motion information and do not consider the true trajectory, which limits the accuracy of bounding box recovery. The GSI algorithm

[15] proposed by Strongsort converts fragmented trajectories into smooth curves through Gaussian nonlinear interpolation, achieving a good balance between accuracy and efficiency, but it ignores the trend of the true trajectory. Therefore, it is necessary to design a new trajectory link fusion model and interpolation method that can process target trajectories more accurately and better deal with problems such as occlusion and blur. Summary of the invention

[0005] The purpose of the present invention is to provide a dynamic scene multi-target tracking method that integrates trajectory association technology. The invention is to improve the tracking ability of multi-target pedestrians in complex dynamic backgrounds. By adopting advanced deep learning algorithms and computer vision technology, accurate extraction and tracking of pedestrian features can be achieved, providing more reliable technical support for intelligent monitoring, autonomous driving and other fields.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a dynamic scene multi-target tracking method integrating trajectory association technology, comprising the following steps:

[0007] S1. Video frame preprocessing and target detection: Given a multi-target tracking MOT video sequence {V t ∈R w×H×3} T t=1 Each frame of V t The images are sequentially transmitted to Yolox for target detection, and the bounding box of each frame is generated. tImprove the robustness of tracking to camera motion through camera motion compensation model;

[0008] S2. Tracking optimization of fast-moving targets: The detection box detected by the target is matched with the tracking box by D-IoU matching algorithm to achieve the best match. The algorithm expands the matching space, builds spatiotemporal similarity between the original non-overlapping detection results and the trajectory, compensates for the matching deviation caused by motion estimation, and achieves better cross-frame geometric consistency, thereby accurately tracking fast-moving targets.

[0009] S3, track linking and fusion: The tracking results are input into the track link fusion model TLFModel, and the tracks that meet a certain time and space distance are associated, so as to transform the pedestrian's short track into a complete long track. This effectively solves the interference problem caused by similar targets and improves the accuracy and reliability of the tracking system.

[0010] S4. Restoration of track interruption and fragmentation: All tracks are processed through the multiple additive regression algorithm MARI. The algorithm accumulates prediction results and adaptively approximates the basic truth track to achieve the reconnection of track fragments. This can achieve tracking of track interruption and fragmentation problems caused by target occlusion and environmental noise interference.

[0011] Furthermore, the step S1 includes the following steps:

[0012] S11, given a MOT video sequence {V t ∈R w×H×3} T t=1 First, each frame of the video sequence V t Transmitted to Yolox in sequence for target detection, generate bounding boxes for each frame, and divide the bounding boxes into high-score boxes and low-score boxes.

[0013] S12, the frame image V t The robustness of tracking to camera motion is improved through a camera motion compensation model.

[0014] S13. Next, tracklets of frame t-1 are t-1 Together with the result after camera motion compensation, it is input into the noise adaptive Kalman filter (NSA) to predict the current frame.

[0015] Furthermore, the step S2 includes the following steps:

[0016] S21. The high-score frame of the detection frame is matched with the tracking frame predicted by the Kalman filter using the D-IoU matching algorithm. The D-IoU matching algorithm expands the matching space to build spatiotemporal similarities between the original non-overlapping detection results and trajectories without changing the original center point of the bounding box, so as to compensate for the deviation caused by motion estimation in the matching space and achieve better cross-frame geometric consistency.

[0017] S22. The low-resolution frame of the detection frame is matched with the tracking frame predicted by the Kalman filter using the D-IoU matching algorithm, and the low-resolution frame is incorporated into the tracking process to improve the integrity of the overall tracking.

[0018] Furthermore, the step S21 includes the following steps:

[0019] S211, in Figure 3 In this case, the value of bounding box A is calculated as w2*h2, and the value of bounding box B is calculated as w1*h1.

[0020] S212, while keeping the center points of the two bounding boxes unchanged, expand them according to a certain ratio, which is the solid line part in the figure, and calculate the value of the expanded bounding box C as Dw2*Dh2, and the value of the bounding box D as Dw1*Dh1.

[0021] Furthermore, the step S211 includes the following steps:

[0022] S2111. For the traditional IoU matching method, only the spatiotemporal similarity is formed between the overlapping detection results and trajectories, which is calculated as follows:

[0023]

[0024] Among them, A and B represent two original boxes respectively, the denominator is composed of the union of the two boxes A and B, the numerator is composed of the intersection of the two boxes, and the IoU value is obtained by the ratio between the two boxes. After calculating the IoU value between the two boxes, the best match is completed according to the IoU value through the Hungarian algorithm.

[0025] Furthermore, the step S212 includes the following steps:

[0026] S2121, D-IoU algorithm introduces a proportional expansion area based on the original box to build spatiotemporal similarity for the original non-overlapping detections and trajectories. Since the expansion area is proportional to the original detections and trajectories, D-IoU does not change their position center. The calculation is as follows:

[0027]

[0028] Among them, C and D represent the two extended boxes after the expansion of the two original boxes A and B respectively. The denominator is composed of the union of the two boxes C and D, and the numerator is composed of the intersection between the two extensions. The D-IoU value is obtained by the ratio between the two.

[0029] S2122, the width of the expanded bounding box is Dw1, Dw2, and the height is Dh1, Dh2. At this time, the expansion center is shared with the original coordinates. Then the expansion ratio coefficients α, β are calculated as:

[0030]

[0031] Among them, α, β represent the expansion ratio coefficients, Dw1, Dw2 represent the width after expansion, w1, w2 represent the width of the original box, Dh1, Dh2 represent the height after expansion, and h1, h2 represent the height of the original box. Assume that the original match is represented as:

[0032] A=(x,y,w,h) (5)

[0033] Among them, x, y represent the coordinates of the upper left corner of the bounding box, w, h represent the height and width of the bounding box, and according to the extended area ratio parameters α, β, the extended area is calculated as:

[0034]

[0035] Among them, σ∈{α,β} is the selection of parameters α and β in the present invention, and a grid search strategy is adopted to search for combinations of parameters α and β between -0.1 and 0.2. The types of combinations are limited, and the optimal values ​​of parameters α and β are obtained through grid search.

[0036] Furthermore, the S3 step includes the following steps:

[0037] S31, TLFModel performs data association by making full use of the global information between tracklets. The algorithm adopts a link model that is not associated with appearance and only relies on spatiotemporal information to predict whether multiple tracks belong to the same ID.

[0038] S32. In this process, the algorithm traverses the entire trajectory set, selecting multiple trajectories for processing each time. Each trajectory must meet the following spatiotemporal criteria to be considered for association prediction:

[0039] T x <Dis t (i,j)<T y (7)

[0040] T S >Dis s (i,j) (8)

[0041] Among them, Dis t (i,j) represents the time interval between trajectories, Dis s (i,j) represents the spatial distance between trajectories, T x Indicates the minimum value of the time threshold, T y Indicates the maximum value of the time threshold; T s Indicates the maximum value of the spatial threshold.

[0042] S33. Through the optimal allocation of the Hungarian algorithm, the trajectories that meet the above-mentioned spatiotemporal criteria are associated together, and the short trajectories of pedestrians are associated into complete long trajectories.

[0043] Furthermore, the step S32 includes the following steps:

[0044] S321, N trajectories T1, T2, ..., T n-1 ,T n Input into the model. The trajectory information can be expressed as f i * represents a trajectory of the i-th frame, (x i * ,y i * ) represents the position of a target box in the i-th frame. By default, the value of N is set to 30.

[0045] S322, for short trajectories T with less than 30 frames m , fill with 0.

[0046] S323, the time block containing the CNN structure is used to extract features of the trajectory, and the time module uses a 7×7 convolution kernel to perform preliminary feature extraction and time-related feature processing. After activation by the activation function, the processed time block features are obtained.

[0047] S324, the results are mapped and fused into a fusion block with a CNN structure to integrate information of different dimensions. The fusion module consists of a 3×3 convolution kernel, an average pooling layer, and a fully connected layer.

[0048] S325, integrating different feature information, i.e., f, x and y, pooling, compressing and connecting the features of these tracklets. Then, the obtained multiple feature layers are spliced ​​using Cat(·) to integrate into a feature vector containing rich spatiotemporal information.

[0049] S326, predict the associated confidence score through a multi-layer perceptron and obtain the final output result. In the whole process, the weights of each module branch are independent. Through the above process, the global connection problem is transformed into a linear allocation task and the estimated result is obtained.

[0050] S327. During the training process, the association task is expressed as a binary classification task and is calculated using the binary cross entropy loss:

[0051]

[0052] Among them, c i ∈[0,1] is the predicted association probability of sample pair n, o i ∈{0,1} represents the ground truth. In this association process, unreasonable trajectory pairs with spatiotemporal constraints are filtered out. Then, the global link is solved as a linear assignment task with predicted connectivity scores.

[0053] Furthermore, the S4 step includes the following steps:

[0054] S41. The output trajectory set is sent to MARI for processing.

[0055] S42, MARI constructs a set of weak learning numbers and accumulates the results of multiple decision trees as the final prediction output. By adaptively approximating the trajectory of the basic truth value, it can achieve a good trade-off between accuracy and efficiency and ultimately obtain the information of the tracking target.

[0056] Furthermore, the step S42 includes the following steps:

[0057] S421, MARI constructs a set of weak learning numbers and accumulates the results of multiple decision trees as the final prediction output, adaptively approximating the ground truth trajectory in the direction of residual reduction. Based on interpolation theory, the model of MARI's i-th frame trajectory is as follows:

[0058]

[0059] x represents each frame, y x Represents the position coordinate variable corresponding to the x frame. ε (i) represents Gaussian noise, which obeys N(0,σ 2 )distributed.

[0060] S422, the trajectory (t i ,y i )1 n Input into formula (12), and finally obtain the reconnected trajectory through residual recursion.

[0061]

[0062] Among them, γ i To fit the regression value, R i is the terminal area, and L is the differentiable loss function. The additive model and forward step algorithm are used to iteratively optimize the ground truth. MARI can solve the problem of tracking noise jitter, and has higher accuracy and robustness in non-linear motion scenes.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. The present invention proposes a tracking algorithm for dynamic scenes and complex environments, which has good tracking effect. First, the present invention designs a dynamic scene tracker DSTrack, which enhances the ability to track pedestrians in complex, diverse and crowded dynamic scenes.

[0065] 2. In order to solve the tracking problem caused by similar target interference, the TLFModel model is proposed. By associating trajectories that meet a certain spatiotemporal distance, short pedestrian trajectories are transformed into complete long trajectories, thereby improving the integrity and continuity of the trajectory.

[0066] 3. In order to solve the problem of tracking fast-moving targets, a D-IoU matching algorithm is designed. By expanding the matching space, the spatiotemporal similarity between the original non-overlapping detection results and the trajectory is constructed to compensate for the matching deviation caused by motion estimation, thereby achieving better results in cross-frame geometric consistency.

[0067] 4. In order to solve the problem of trajectory interruption and fragmentation caused by target occlusion and environmental noise interference, the MARI algorithm is designed to accumulate prediction results and adaptively approximate the basic truth trajectory to reconnect the trajectory fragments, thereby improving the continuity and integrity of the trajectory. Through the above innovations, the present invention successfully identifies and accurately tracks pedestrian targets in dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0069] Figure 1 It is an overall framework diagram of the multi-target tracking system integrating the trajectory association strategy in the dynamic scene of the present invention.

[0070] Figure 2 This is a structural diagram of the trajectory link fusion model of the present invention.

[0071] Figure 3 This is a working principle diagram of the D-IoU matching algorithm of the present invention.

[0072] Figure 4 The figure is a schematic diagram of the working principle of the multiple additive regression algorithm of the present invention and its comparison with other algorithms.

[0073] Figure 5 Schematic diagram of the verification results of different α and β on the MOT17 dataset of the present invention.

[0074] Figure 6 This is a MAE comparison chart of the GSI module of the present invention and the MARI module.

[0075] Figure 7 This is a comparison chart of the detection effects of different methods in the present invention on the MOT17 dataset.

[0076] Figure 8 The present invention shows the pedestrian tracking effect diagram of the multi-target tracking system in a dynamic scene. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0078] Example 1

[0079] See also Figure 1 and Figure 7 The present invention provides a technical solution, which is a dynamic scene multi-target tracking method integrating trajectory association technology, comprising the following steps:

[0080] The invention is a multi-target tracking method that combines the D-IoU matching algorithm, the trajectory link fusion model and the multiple additive regression algorithm. The overall tracking process is as follows: Figure 1 As shown in the figure, the algorithm aims to improve the tracking capability of multi-target pedestrians in complex dynamic backgrounds. In the D-IoU matching algorithm, the algorithm expands the matching space to build spatiotemporal similarities between the original non-overlapping detection results and trajectories to compensate for the matching deviation caused by motion estimation, thereby achieving better cross-frame geometric consistency. In the trajectory link fusion model, short pedestrian trajectories are converted into complete long trajectories by linking trajectories that meet a certain spatiotemporal distance. In the multiple additive regression algorithm, the trajectory fragments are reconnected by accumulating the prediction results and adaptively approximating the trajectory with the true value. The overall workflow of the system is as follows:

[0081] Given a multi-target tracking MOT video sequence {V t ∈R w×H×3} T t=1 First, each frame of the video sequence V tThe images are sequentially transmitted to Yolox for target detection and the bounding box of each frame is generated. t The camera motion compensation model is used to improve the robustness of tracking to camera motion. Next, the tracklets of frame t-1 are t-1 The result after camera motion compensation is input into the noise adaptive Kalman filter to predict the current frame. In the data association stage, different from the approach of only retaining high-resolution frames, the study t The detection output results are classified into high confidence and low confidence. In the initial data association, detection boxes with high confidence are given priority. In order to better build spatiotemporal similarity, D-IoU is used to calculate the confidence score of the predicted box and the detection box, and the Hungarian algorithm is used to complete the best association match. For those detection boxes that match the existing trajectory and meet the set threshold, the matched detection boxes are then stored in the corresponding trajectory segments. For the detection boxes that fail to match the existing trajectory segments but meet the set threshold, new trajectory segments are created. Next, a second data association is performed between the unmatched trajectory and the low-confidence detection box, using the same strategy as the first association. If there are trajectory segments that fail to match any detection box, they are marked and saved. For trajectory segments that fail to match any detection box for more than 30 frames, they are deleted so that the detection results can be updated through continuous tracking. In addition, after obtaining all trajectory segments, TLFModel is used to associate the trajectories that meet specific spatiotemporal conditions, so as to transform the pedestrian's short trajectory into a complete long trajectory. Finally, MARI is used for processing to generate the output trajectory set.

[0082] D-IoU matching algorithm: In multi-target tracking tasks, the detection and tracking results cannot overlap due to the rapid movement of the target between consecutive frames, resulting in tracking failure. In particular, it is difficult to initialize and estimate the motion correctly when the target has irregular or nonlinear motion, which may cause the same object in adjacent frames to not overlap in geometric features. Since traditional IoU produces zero values ​​for all non-overlapping geometric features, using IoU for geometric consistency judgment will lead to matching failures during initial tracking and tracking failures during continuous tracking. This problem can be alleviated by appearance features when the appearance of objects can be distinguished, such as matching detections across frames based on the appearance similarity of the objects. However, when the irregular motion of the object is accompanied by an indistinguishable appearance, most existing MOT solutions may not be able to perform reliable tracking, so new solutions are needed.

[0083] In this study, a matching algorithm based on D-IoU is proposed. D-IoU is robust to tracking multiple targets with irregular motion and difficult-to-distinguish appearance. This method expands the matching space without changing the original center point of the bounding box, builds spatiotemporal similarity between the original non-overlapping detection results and trajectories, and compensates for the deviation caused by motion estimation in the matching space, thereby achieving better geometric consistency across frames.

[0084] exist Figure 3 In the case of , considering that the object moves horizontally in the spatial depth direction, this study focuses on the current frame object predicted by the noise adaptive Kalman filter based on the previous frame and the object in the current frame that intersects with the detection anchor frame. These two objects are not necessarily the same tracking target. For the target to be detected, this paper assumes that the left part of the pedestrian in the figure is the original bounding box A of the next frame target predicted by the noise adaptive Kalman filter, and the right part of the pedestrian is the original bounding box B generated by the current target detection. By analyzing the figure, it can be calculated that the value of bounding box A is w2*h2, and the value of bounding box B is w1*h1. Considering that pedestrians will produce irregular motion changes during movement, it is difficult for traditional IoU matching methods to correctly estimate motion. This study keeps the center points of the two bounding boxes unchanged and expands them according to a certain proportion, which is the solid line part in the figure. At this time, it can be calculated that the value of the expanded bounding box C is Dw2*Dh2, and the value of the bounding box D is Dw1*Dh1.

[0085] For the traditional IoU matching method, only the spatiotemporal similarity is formed between the overlapping detection results and trajectories, which can be written as follows:

[0086]

[0087] Among them, A and B represent two original boxes respectively, the denominator is composed of the union of the two boxes A and B, the numerator is composed of the intersection of the two boxes, and the IoU value is obtained by the ratio between the two boxes. After calculating the IoU value between the two boxes, the best match is completed according to the IoU value through the Hungarian algorithm.

[0088] The D-IoU proposed in this study introduces a proportional extended area based on the original box to build spatiotemporal similarities for the original non-overlapping detections and trajectories. Since the extended area is proportional to the original detections and trajectories, D-IoU will not change their position center. Based on the above properties, the D-IoU of this study alleviates the impact of irregular motion in two ways: one is that it can directly match the same but non-overlapping detections and trajectories in adjacent frames, and the other is to compensate for the motion estimation deviation in the matching space. By introducing the extended area, the model can better understand the target and scene, thereby achieving a more accurate tracking effect. Figure 3As shown, it can be written as follows:

[0089]

[0090] Among them, C and D represent the two extended boxes after the expansion of the two original boxes A and B respectively. The denominator is composed of the union of the two boxes C and D, and the numerator is composed of the intersection between the two extensions. The D-IoU value is obtained by the ratio between the two.

[0091] according to Figure 3 As shown, the width of the expanded bounding box is Dw1, Dw2, and the height is Dh1, Dh2. At this time, the expansion center is shared with the original coordinates. The expansion ratio coefficients α, β can be expressed as:

[0092]

[0093] Among them, α, β represent the expansion ratio coefficients, Dw1, Dw2 represent the width after expansion, w1, w2 represent the width of the original box, Dh1, Dh2 represent the height after expansion, and h1, h2 represent the height of the original box. Assume that the original match is represented as:

[0094] A=(x,y,w,h) (5)

[0095] Among them, x, y represent the coordinates of the upper left corner of the bounding box, w, h represent the height and width of the bounding box, and according to the extended area ratio parameters α, β, the extended area is expressed as:

[0096]

[0097] Among them, σ∈{α,β}. For the selection of parameters α and β in this study, a grid search strategy was adopted to search for the combination of parameters α and β between -0.1 and 0.2. The combination types are limited, and the optimal values ​​of parameters α and β are determined through grid search.

[0098] TLFModel: The tracklet link fusion algorithm performs data association by making full use of the global information between tracklets. The algorithm uses a link model that is not associated with appearance and relies only on spatiotemporal information to predict whether multiple tracks belong to the same ID. In this process, the algorithm traverses the entire tracklet set and selects multiple tracks for processing each time. Each tracklet must meet the following spatiotemporal criteria to be considered for association prediction:

[0099] T x <Dis t (i,j)<T y (7)

[0100] T S >Dis s(i,j) (8)

[0101] Among them, Dis t (i,j) represents the time interval between trajectories, Dis s (i,j) represents the spatial distance between trajectories, T x Indicates the minimum value of the time threshold, T y Indicates the maximum value of the time threshold. T s Represents the maximum value of the spatial threshold. Through the optimal allocation of the Hungarian algorithm, the trajectories that meet the above spatiotemporal criteria are associated together, and the short trajectories of pedestrians are associated into complete long trajectories.

[0102] Figure 2 The structure of the trajectory linking model is shown. In this model, N trajectories T1, T2, …, T n-1 ,T n As the input of the model. The trajectory information can be expressed as f i * represents a trajectory of the i-th frame, (x i * ,y i * ) represents the position of a target box in the i-th frame. By default, the value of N is set to 30. For short trajectories with less than 30 frames, m , filled with 0. Next, the time block containing the CNN structure is used to extract the features of the trajectory. The time module uses a 7×7 convolution kernel for preliminary feature extraction and processing of time-related features. After activation by the activation function, the processed time block features can be obtained. Then, the results are mapped and fused into the fusion block with the CNN structure to integrate information of different dimensions. The fusion module consists of a 3×3 convolution kernel, an average pooling layer, and a fully connected layer. By integrating different feature information (i.e., f, x, and y), the features of these tracklets are pooled, compressed, and connected. Then, the obtained multiple feature layers are spliced ​​using Cat(·) to integrate into a feature vector containing rich spatiotemporal information. Finally, a multi-layer perceptron is used to predict the associated confidence score and obtain the final output result. In the whole process, the weights of each module branch are independent. Through the above process, the global link problem is transformed into a linear assignment task, and the estimated result is obtained. During the training process, the association task is formulated as a binary classification task, and then it is calculated using the binary cross entropy loss:

[0103]

[0104] Among them, c i ∈[0,1] is the predicted association probability of sample pair n, oi ∈{0,1} represents the ground truth. In this association process, unreasonable trajectory pairs with spatiotemporal constraints are filtered out. Then, the global link is solved as a linear assignment task with predicted connectivity scores.

[0105] MARI: In the tracking dataset, some pedestrians are completely occluded or affected by environmental noise, which limits the visual information of the detector to detect these pedestrians. Interpolation methods are widely used to fill the gaps in trajectories caused by missing detections. Linear interpolation is limited in accuracy because it does not use motion information. Due to the interference of target occlusion and environmental noise, interpolation methods are widely used to fill the gaps in trajectories caused by missed detections. Linear interpolation ignores the motion of the target during tracking, which greatly limits the accuracy of bounding box generation. To solve this problem, this paper adopts a method using multiple additive regression interpolation. The multiple additive regression interpolation method constructs a set of weak learning trees and accumulates the results of multiple decision trees as the final prediction output. By adaptively approximating the trajectory of the ground truth, a good trade-off between accuracy and efficiency can be achieved.

[0106] Figure 4 The differences between linear interpolation (LI), MARI and GSI are illustrated. The original tracking results usually contain noise jitter, and LI ignores motion information. GSI is modeled through nonlinear motion. It can smooth the curve to approach the true trajectory, but ignores the trajectory trend of the ground truth, and the result still does not match the true trajectory. As shown in the ground truth (GT) in the figure, it is not a smooth curve. The ground truth trajectory of the target is a progressive curve affected by the frame rate and the direction of target motion. On this basis, our MARI constructs a set of weak learning numbers, and accumulates the results of multiple decision trees as the final prediction output, adaptively approximating the ground truth trajectory in the direction of residual reduction, and the result is highly consistent with GT. Based on the interpolation theory, the model of the i-th frame trajectory of the proposed MARI is as follows:

[0107]

[0108] x represents each frame, y x Represents the position coordinate variable corresponding to the x frame. ε (i) represents Gaussian noise, which obeys N(0,σ 2 )distributed.

[0109] The trajectory (t i ,y i )1 n Input into formula (12), and finally obtain the reconnected trajectory through residual recursion.

[0110]

[0111] Among them, γ i To fit the regression value, R i is the terminal area, and L is the differentiable loss function. The additive model and forward step algorithm are used to iteratively optimize the ground truth. The use of multiple additive regression interpolation can solve the problem of tracking noise jitter, while having higher accuracy and robustness in nonlinear motion scenes.

[0112] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0113] Furthermore, simulation experiments are performed to verify the effectiveness of the method proposed in the present invention.

[0114] Table 1 Comparison of different methods on the MOT17 dataset

[0115] method Time of submission HOTA(↑) MOTA(↑) IDF1(↑) FP(↓) FN(↓) IDs(↓) MOTR 2022 - 65.1 66.4 45486 149307 2049 TransCenter 2022 54.5 73.2 62.2 23112 123738 4614 FairMOT 2021 59.3 73.7 72.3 27507 117477 3303 RelationTrack 2022 61.0 73.8 74.7 27999 118623 1374 TransTrack 2020 54.1 75.2 63.5 50157 86442 3603 TrackFormer 2022 - 74.1 68.0 34602 108777 2829 ReMOT 2021 59.7 77.0 72.0 33204 93612 2853 CorrTracker 2021 60.7 76.5 73.6 29808 99510 3369 MAATrack 2022 62.0 79.4 75.9 - - 1452 ByteTrack 2022 63.1 80.3 77.3 25491 83721 2196 StrongSORT 2023 63.5 78.3 78.5 - - 1446 OC-SORT 2023 63.2 78.0 77.5 15100 108000 1950 Our Approach 2024 64.2 79.9 79.2 27866 84027 1278

[0116] Table 2 Comparison of different methods on the MOT20 dataset

[0117]

[0118]

[0119] To validate the effectiveness of our method DSTrack, we conducted a series of experiments on the publicly available MOT17 and MOT20 datasets. We used the publicly available Yolox detector weights. For the association component, the detection score threshold was set to the default 0.6. During the linear assignment process, matches with an IoU less than 0.2 between the detection and tracked box were rejected. In the training phase of TLFModel, the labeled tracks were used and classified into positive and negative samples in a ratio of 1:3, and the number of iterations was set to 50. At the same time, the cross entropy loss function and Adam optimizer were used, the initial learning rate of the model was set to 0.001, and the learning rate was gradually reduced during the training process in combination with the cosine annealing algorithm. During the inference process, the temporal and spatial distance thresholds of 30 frames and 75 pixels were set to eliminate unreasonable track pairs. In addition, a prediction score threshold of 0.6 was set to consider association. The GPUs used in the experiment were two NVIDIA GeForce RTX-1080ti, each equipped with 11GB RAM. The system memory was 32GB. The software environment of the experiment includes CUDA version 11.6, PyTorch version 1.12.0, Python version 3.7, and the operating system is Windows.

[0120] In order to more objectively illustrate the performance of the object tracking method in this invention, in this study, we compare our method with the state-of-the-art tracking algorithms on the MOT17 and MOT20 test sets, and the results are from the official MOTChallenge evaluation server. Table 1 shows that our method achieves a MOTA of 79.9, an IDF1 of 79.2, and a HOTA of 64.2. Compared with the popular ByteTrack, our method has a HOTA that is 1.1 higher and an IDF1 that is 1.9 higher. In addition, Table 2 highlights the performance of our method on the challenging MOT20 test set with large-scale pedestrian variations and occlusions. Our method achieves a HOTA of 64.1, a MOTA of 78.1, and an IDF1 of 78.6, which is generally better than other trackers.

[0121] In order to better demonstrate the performance of the algorithm, Figure 7 We show the actual tracking results of the popular tracker ByteTrack and our method DSTrack in dynamic backgrounds. Figure 7It can be clearly seen that there are some defects when using ByteTrack for pedestrian tracking on the MOT17 dataset, as shown by the arrows in the figure, that is, there is a deviation between the detected anchor box and the pedestrian. This is due to the matching deviation caused by motion estimation, which causes the anchor box to not completely overlap with the pedestrian image. In contrast, when using the DSTrack tracker for pedestrian tracking, the anchor box can achieve accurate matching with the pedestrian. The above results show that the proposed tracking algorithm shows better performance in handling pedestrian target tracking in complex dynamic backgrounds, and provides a reliable solution for solving target tracking tasks in complex dynamic backgrounds.

[0122] Table 3 Ablation experiment results on the MOT17 dataset

[0123] Method D-IoU MARI TLFModel CMC IDF1(↑) MOTA(↑) HOTA(↑) Baseline Model - - - - 79.77 77.66 67.88 Tracker V1 √ - - - 79.95 77.80 68.18 Tracker V2 √ √ - - 80.15 79.84 69.55 Tracker V3 √ √ √ - 80.60 79.78 69.48 Tracker V4 √ √ √ √ 82.60 80.65 70.56

[0124] Table 4 Comparison of GPRI module and MARI module results on the MOT17 dataset

[0125]

[0126]

[0127] In Tables 3 and 4, we conducted ablation experiments on the MOT17 validation set to evaluate the contribution of each module to the algorithm performance, as well as the overall performance of the tracking algorithm DSTrack. As shown in Table 3, in the ablation experiment, ByteTrack is used as the baseline model and Yolox is used as the target detection network. The values ​​with the highest indicator values ​​are shown in bold. As shown in Table 3, there are four core modules in DSTrack, and three key multi-target tracking indicators are reported on the MOT17 validation set. The baseline model obtains 77.66 on MOTA, 79.77 on IDF1, and 67.88 on HOTA. When D-IoU is added to the tracker, DSTrack improves on all three indicators. When D-IoU and MARI modules are added at the same time, MOTA improves to 79.84, IDF1 improves to 80.15, and HOTA improves to 69.55. Continuing to add TLFModel, IDF1 improves, which shows that TLFModel improves the relevance of the tracker. At the same time, as standard practice, DSTrack attaches a camera motion compensation module to improve tracking performance, and tracker V4 shows the effect of adding this module on the MOT17 dataset. At this point, the tracking algorithm achieves the best performance in all three important metrics.

[0128] In order to reduce missed detections, an effective trajectory reconnection method is essential. Table 4 compares the performance of the linear interpolation (LI) module, Gaussian regression (GSI) module, and multiple additive regression (MARI) module on the MOT17 dataset. In Table 4, these three modules are added to the baseline model tracker v1 respectively. Adding the MARI module improves the continuity and completeness of the trajectory, with HOTA and MOTA reaching the highest values ​​of 69.55 and 79.84 respectively. Compared with the baseline model, they increased by 1.37 and 2.04 respectively. In addition, the effectiveness of the model is verified by comparing the mean absolute error (MAE) of the GPRI module and the MARI module. The calculation method of MAE is shown in the formula:

[0129]

[0130] like Figure 6 As shown in Figure 1, the MAE of the GSI module and the MARI module are compared in 6 scenes of the MOT17 dataset. When the GSI module is used, due to the interruption and fragmentation of the trajectory caused by target occlusion and environmental noise interference, the trajectory drift represented by the spike appears in the figure, resulting in a severe jitter of the MAE. However, when the MARI module is used, this phenomenon does not occur, indicating that MARI is closer to the true ground trajectory and effectively prevents trajectory drift.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic scene multi-target tracking method integrating trajectory association technology, characterized in that: The following steps are involved: S1. Video frame preprocessing and target detection: Given a multi-target tracking MOT video sequence {V t ∈R w×H×3 } T t=1 Each frame of V t The images are sequentially transmitted to Yolox for target detection, and the bounding box of each frame is generated. t Improve the robustness of tracking to camera motion through camera motion compensation model; S2. Tracking optimization of fast-moving targets: The detection frame detected by the target and the tracking frame are optimally matched through the D-IoU matching algorithm. The algorithm expands the matching space, builds spatiotemporal similarity between the original non-overlapping detection results and the trajectory, compensates for the matching deviation caused by motion estimation, and achieves cross-frame geometric consistency, thereby achieving accurate tracking of fast-moving targets; S3, trajectory linking and fusion: The tracking results are input into the trajectory linking and fusion model TLFModel, and the trajectories that meet a certain time and space distance will be associated, so as to transform the pedestrian's short trajectory into a complete long trajectory; S4. Repair of trajectory interruption and fragmentation: All trajectories are processed by the multiple additive regression algorithm MARI. The algorithm accumulates prediction results and adaptively approximates the basic truth trajectory to reconnect the trajectory fragments, thereby achieving tracking of trajectory interruption and fragmentation problems caused by target occlusion and environmental noise interference.

2. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 1 is characterized in that: The S1 step includes the following steps: S11, given a MOT video sequence {V t ∈R w×H×3 } T t=1 First, each frame of the video sequence V t Sequentially transmit to Yolox for target detection, generate bounding boxes for each frame, and divide the bounding boxes into high-score boxes and low-score boxes; S12, the frame image V t Improve the robustness of tracking to camera motion through camera motion compensation model; S13. Next, tracklets of frame t-1 are t-1 Together with the result after camera motion compensation, it is input into the noise adaptive Kalman filter (NSA) to predict the current frame.

3. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 1 is characterized in that: The S2 step includes the following steps: S21, the high-resolution frame of the detection frame is matched with the tracking frame predicted by the Kalman filter by the D-IoU matching algorithm. The D-IoU matching algorithm builds spatiotemporal similarity between the original non-overlapping detection results and trajectories by expanding the matching space without changing the original center point of the bounding box, so as to compensate for the deviation caused by motion estimation in the matching space and achieve cross-frame geometric consistency; S22. The low-resolution frame of the detection frame is matched with the tracking frame predicted by the Kalman filter using the D-IoU matching algorithm, and the low-resolution frame is incorporated into the tracking process to improve the integrity of the overall tracking.

4. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 3 is characterized in that: The step S21 includes the following steps: S211, calculate the value of bounding box A as w2*h2, and the value of bounding box B as w1*h1; S212. While keeping the center points of the two bounding boxes unchanged, two expansion ratio coefficients α and β are introduced to construct the spatiotemporal similarity between the two bounding boxes. The optimal values ​​of the expansion ratio coefficients α and β are determined through a grid search strategy, and the value of the expanded bounding box C is calculated to be Dw2*Dh2, and the value of the bounding box D is Dw1*Dh1.

5. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 4 is characterized in that: The step S211 includes the following steps: S2111. For the traditional IoU matching method, only the spatiotemporal similarity is formed between the overlapping detection results and trajectories, which is calculated as follows: Among them, A and B represent two original boxes respectively, the denominator is composed of the union of the two boxes A and B, the numerator is composed of the intersection between the two boxes, and the IoU value is obtained by the ratio between the two boxes. After calculating the IoU value between the two boxes, the best match is completed according to the IoU value through the Hungarian algorithm.

6. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 4 is characterized in that: The step S212 includes the following steps: S2121, D-IoU algorithm introduces a proportional expansion area based on the original box to build spatiotemporal similarity for the original non-overlapping detections and trajectories. Since the expansion area is proportional to the original detections and trajectories, D-IoU does not change their position center and is calculated as follows: Among them, C and D represent the two extended boxes after the expansion of the two original boxes A and B respectively. The denominator is composed of the union of the two boxes C and D, and the numerator is composed of the intersection of the two extensions. The value of D-IoU is obtained by the ratio between the two. S2122, the width of the expanded bounding box is Dw1, Dw2, and the height is Dh1, Dh2. At this time, the expansion center is shared with the original coordinates, and the expansion ratio coefficients α, β are calculated as: Among them, α, β represent the expansion ratio coefficients, Dw1, Dw2 represent the width after expansion, w1, w2 represent the width of the original box, Dh1, Dh2 represent the height after expansion, h1, h2 represent the height of the original box, assuming that the original match is expressed as: A=(x,y,w,h) (5) Among them, x, y represent the coordinates of the upper left corner of the bounding box, w, h represent the height and width of the bounding box, and according to the extended area ratio parameters α, β, the extended area is calculated as: Among them, σ∈{α,β}, for the selection of parameters α and β, a grid search strategy is adopted to search for the combination of parameters α and β between -0.1 and 0.

2. The types of combinations are limited, and the optimal values ​​of parameters α and β are obtained through grid search.

7. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 1 is characterized in that: The S3 step includes the following steps: S31,TLFModel makes full use of the global information between tracklets for data association. The algorithm adopts a link model that is not related to appearance and only relies on spatiotemporal information to predict whether multiple tracks belong to the same ID; S32. In this process, the algorithm traverses the entire trajectory set, selecting multiple trajectories for processing each time. Each trajectory must meet the following spatiotemporal criteria to be considered for association prediction: T x <Dis t (i,j)<T y (7) T S >Dis s (i,j) (8) Among them, Dis t (i,j) represents the time interval between trajectories, Dis s (i,j) represents the spatial distance between trajectories, T x Indicates the minimum value of the time threshold, T y Indicates the maximum value of the time threshold, T s Indicates the maximum value of the spatial threshold; S33. Through the optimal allocation of the Hungarian algorithm, the trajectories that meet the above-mentioned spatiotemporal criteria are associated together, and the short trajectories of pedestrians are associated into complete long trajectories.

8. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 7 is characterized in that: The step S32 includes the following steps: S321, N trajectories T1, T2, ..., T n-1 ,T n Input into the model, where the trajectory information is represented as represents a trajectory of the i-th frame, Indicates the position of a target box in the i-th frame; S322, for short trajectories T with less than 30 frames m , fill with 0; S323, the time block containing the CNN structure is used to extract features of the trajectory. The time module uses a 7×7 convolution kernel to perform preliminary feature extraction and time-related feature processing. After activation by an activation function, the processed time block features are obtained. S324, the results are mapped and fused into a fusion block with a CNN structure to integrate information of different dimensions. The fusion module consists of a 3×3 convolution kernel, an average pooling layer, and a fully connected layer; S325, integrating different feature information, i.e., f, x and y, pooling, compressing and connecting the features of these tracklets, and then concatenating the obtained multiple feature layers using Cat(·) to integrate them into a feature vector containing rich spatiotemporal information; S326, predicting the associated confidence score through a multi-layer perceptron and obtaining the final output result. In the whole process, the weights of each module branch are independent. Through the above process, the global connection problem is transformed into a linear allocation task and the estimated result is obtained; S327. During the training process, the association task is expressed as a binary classification task and is calculated using the binary cross entropy loss: Among them, c i ∈[0,1] is the predicted association probability of sample pair n, o i ∈{0,1} represents the ground truth. In this association process, unreasonable trajectory pairs with spatiotemporal constraints are filtered out. Then, the global link is solved as a linear assignment task with predicted connectivity scores.

9. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 1 is characterized in that: The S4 step includes the following steps: S41, the output trajectory set is sent to MARI for processing; S42, MARI constructs a set of weak learning numbers and accumulates the results of multiple decision trees as the final prediction output. It achieves a good trade-off between accuracy and efficiency by adaptively approximating the trajectory of the basic truth value, and finally obtains the information of the tracking target.

10. The dynamic scene multi-target tracking method integrating trajectory association technology according to claim 9 is characterized in that: The step S42 includes the following steps: S421, MARI constructs a set of weak learning numbers and accumulates the results of multiple decision trees as the final prediction output, adaptively approximating the ground truth trajectory in the direction of residual reduction. Based on interpolation theory, the model of the i-th frame trajectory of MARI is as follows: x represents each frame, y x represents the position coordinate variable corresponding to the x frame, ε (i) represents Gaussian noise, which obeys N(0,σ 2 )distributed; S422, the trajectory (t i ,y i )1 n Input into formula (12), and finally obtain the reconnected trajectory through residual recursion; Among them, γ i To fit the regression value, R i is the terminal area, L is the differentiable loss function, The additive model and forward step-by-step algorithm are used to iteratively optimize the ground truth. MARI can solve the problem of tracking noise and jitter, while having higher accuracy and robustness in non-linear motion scenes.

Citation Information

Cited By

  • Online multi-target tracking method and system based on feature fusion and iterative association

    CN119693416A

  • End-to-end automatic driving prediction and planning method, system and device and medium

    CN120071286A

  • Unmanned vehicle automatic driving scene multi-target tracking method based on online spatial-temporal feature association

    CN120279069A

  • Multi-target tracking method based on long and short term trajectory association

    CN120747166A

  • Multi-target tracking method based on trajectory pipeline regression prediction and association

    CN120953322A