Detection and tracking integrated method based on data secondary correlation

Abstract

The invention discloses a detection and tracking integrated method based on data secondary association, which comprises the following steps of: S1, collecting target information, fusing multi-state information of a data set, and dynamically enhancing various data sets on a training set; s2, associating the detected pedestrians with existing tracking trajectories by adopting a cascade matching method, performing priority matching on the pedestrians with relatively high occurrence frequency, predicting a motion trajectory of the pedestrians of the current frame, performing minimum cosine distance calculation on the tracking trajectories which are not successfully matched after cascade matching and a pedestrian detection frame with a low confidence threshold value, and calculating a minimum cosine distance between the tracking trajectories which are not successfully matched after cascade matching and the pedestrian detection frame with a low confidence threshold value; searching shielded pedestrians in a pedestrian detection frame with a low confidence threshold, and updating a successfully matched track; s3, performing GIOU method matching on an unmatched trajectory, an unmatched high-confidence threshold pedestrian detection frame after cascade matching and an unconfirmed tracking trajectory, and judging how the detection frame and the prediction frame are overlapped by using a loss function GIOU; and S4, comparing the detection performance of the improved method with the detection performance of the original method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image target detection and tracking, and in particular, relates to an integrated detection and tracking method based on secondary association of data. Background Art

[0002] The integrated approach of visual detection and tracking in complex environments requires stable detection and tracking of targets in chaotic and dynamic situations. In recent years, multi-object tracking (MOT) has been widely studied, with the goal of associating detected targets across video frames to obtain the entire motion trajectory. Various methods ranging from graph clustering methods to graph neural networks can aggregate information across frames and objects; from tracking by detection mode to joint detection and tracking to improve detection performance across multiple frames; from Kalman filtering to recurrent neural networks (RNN) and long short-term memory (LSTM) to improve the association performance with motion cues. With the development of tracking methods, multi-object tracking (MOT) can be applied to many tasks, such as traffic flow analysis, human behavior prediction and pose estimation, autonomous driving assistance, etc.

[0003] Traditional target tracking methods are mainly divided into one-stage tracking (DFT) and two-stage tracking (DBT). Two-stage tracking requires detecting each frame of the video and then associating it with the trajectory. For example, the sort method proposed by Bewley et al. mainly uses Kalman filtering for prediction and matches the predicted results using the Hungarian method. However, the sort method only uses IOU matching for identification and matching without using features, which will cause ID switching when the pedestrian overlap is high. Wojke et al. proposed the Deepsort method, which adds cascade matching and state estimation to the sort method. It determines whether it is the same person by comparing the distance between two vectors, and uses appearance information to achieve tracking of pedestrians that are occluded for a long time. However, the DBT method cannot balance the speed and accuracy of tracking, and the tracking effect is affected by the detection effect. Therefore, multi-target tracking gradually turns to a model that combines detection and matching. In 2020, Wang et al. integrated the target detection link and feature extraction into one network for the first time, creating a precedent for one-stage tracking. This method can achieve real-time tracking speed, but because the detector uses the YOLOV3 method with anchor boxes, it is sensitive to hyperparameters and easily leads to imbalanced positive and negative samples during training. Based on this, in 2020, Zhang et al. proposed the DFT method FairMOT based on the anchor-free detector centernet, which effectively made up for the shortcomings of JDE. However, due to the scale changes, illumination changes, occlusion and other problems in the pedestrian tracking process, the detection network in the traditional FairMOT method has low feature resolution for shallow pedestrian networks and insufficient tracking accuracy. In 2021, Wu et al. used tracking clues to assist detection and designed a new online MOT framework based on the JDE paradigm, named TraDeS. This method effectively improved some problems of the JDE paradigm and achieved quite good performance. Although the above method has achieved significant SOTA results in the field of unified recognition, in real scenes, pedestrians being occluded will result in low detection frame scores for pedestrians. Often, when the detection frame score is less than 0.1, the detection frame will be eliminated, but this is not the best solution and can easily lead to trajectory interruption and ID.

[0004] The IOU method has high requirements for the position of the predicted bounding box and the real box. Its calculation formula works only when the two boxes have an intersection. For two boxes that do not intersect at all, it is impossible to measure the distance between the two target bounding boxes. The IOU loss calculation is 0, and the loss cannot be fed back to the neural network. Without gradient feedback, learning and training cannot be performed, which affects the update of network weights, making the network always near the local optimal value and unable to converge to the global optimal value. Therefore, using the IOU loss calculation method will fail to optimize the algorithm model in some special cases. Summary of the invention

[0005] The present invention proposes an integrated detection and tracking method based on secondary data association to solve the problems of traditional methods such as frequent ID switching, target loss, trajectory interruption, etc., and realizes feature extraction and high-precision detection and tracking of targets of different sizes.

[0006] Technical solution of the present invention

[0007] A detection and tracking integrated method based on secondary data association includes the following steps:

[0008] S1, collect target information, integrate multi-state information of data sets, and dynamically perform multiple data set enhancements on the training set;

[0009] S2. Use the cascade matching method to associate the detected pedestrians with the existing tracking trajectories, and give priority to matching pedestrians with higher frequency, predict the movement trajectory of the pedestrian in the current frame, calculate the minimum cosine distance between the tracking trajectory that fails to match after cascade matching and the pedestrian detection frame with a low confidence threshold, find the obscured pedestrians in the pedestrian detection frame with a low confidence threshold, update the successfully matched trajectory, and directly delete the low confidence threshold detection frame that has not been successfully matched;

[0010] S3: Match the unmatched trajectories, the high confidence threshold pedestrian detection frames that are not matched after cascade matching, and the unconfirmed tracking trajectories using the GIOU method. By using the loss function GIOU to determine how the detection frame and the prediction frame overlap, the problem of being unable to optimize the algorithm model in special cases is solved:

[0011]

[0012] Where A is the area of ​​the detection box, B is the area of ​​the prediction box, and C is the minimum area surrounding the detection box and the prediction box;

[0013] S4. Compare the detection performance of the improved method with that of the original method.

[0014] Preferably, the method of enhancing the data set in step S1 is random cropping and scaling, affine transformation, and adding noise.

[0015] Preferably, step S2 includes the following steps: S21, detecting the detection frames and apparent feature vectors of all pedestrians appearing in the picture through a detector, setting a confidence threshold, and retaining both high-score detection frames and low-score detection frames; S22, using a cascade matching method to associate the detected pedestrians with existing tracking trajectories, and giving priority to matching pedestrians with higher frequency of appearance, and predicting the motion trajectory of the pedestrians in the current frame.

[0016] Preferably, in step S2, the cascade matching module uses appearance features and motion features to calculate similarity and obtain a cost matrix, and trajectories that have not been lost are matched first, and trajectories that have been lost for a long time are matched later.

[0017] Preferably, the motion feature in step S2 is calculated by using the Mahalanobis distance to calculate the correlation between the high-resolution detection frame and the tracker predicted position.

[0018] Preferably, the multi-state information of the data set includes position information, shape information and posture information of the detected and tracked target.

[0019] Preferably, step S2 predicts the motion trajectory of the pedestrian in the current frame by setting a tracker for each existing pedestrian and the newly entered pedestrian. When the unscented Kalman filter can find a matching detection frame in three consecutive frames, it means that the pedestrian is a new pedestrian found, and the state of the tracker of the pedestrian will be set to the "confirmed" state, otherwise the tracker state is "unconfirmed". If the tracker does not find a matching detection frame within the time of "max_age>30", the corresponding tracker will be deleted, and it is assumed that the pedestrian tracked by this tracker has left the video area.

[0020] Preferably, in step S3, GIOU takes a maximum value of 1 when the two frames overlap, and takes a minimum value of -1 when the two frames have no intersection and are infinitely far apart, that is, the value range is [-1, 1], to avoid failure of back propagation when the gradient is 0.

[0021] Technical Effects of the Invention

[0022] The present invention uses GIOU to optimize the multi-scale network. The present invention can solve the problems of traditional methods such as frequent ID switching, target loss, and track interruption, and realizes feature extraction and high-precision detection and tracking of targets of different scales. It also has better tracking effects for some complex scenes and can significantly reduce the number of pedestrian track interruptions and ID switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an overall framework diagram of an integrated detection and tracking method based on secondary data association.

[0024] Figure 2 It is the 2GIOU space representation graph.

[0025] Figure 3 A comparison chart with existing methods on the MOT16 dataset.

[0026] Figure 4 This is a pre-occlusion effect diagram of an integrated detection and tracking method based on secondary data association.

[0027] Figure 5This is a diagram showing the occlusion effect of an integrated detection and tracking method based on secondary data association. Specific implementation methods

[0029] Step 1: Collect target information, fuse multi-state information of the data set, and dynamically perform multiple data set enhancements on the training set. In order to increase the diversity of the data, various data enhancement techniques are used to simulate target detection and tracking under different circumstances. The present invention adopts four image enhancement methods, namely, random cropping and scaling, randomly cropping part of the image and randomly scaling the size of the target. Mirror flipping, randomly flipping the image horizontally or vertically. Affine transformation, randomly rotating, translating and affine transforming the image to simulate different perspectives. Adding noise, adding random noise, such as Gaussian noise, to the image. Simulation diversity helps the model to more accurately capture the different appearances and postures of the target, and adapt to detection tasks in complex environments, such as lighting changes, target occlusion, etc.

[0030] Step 2: Improve the data association mechanism of the tracking algorithm to achieve secondary association detection and tracking. Figure 1 The figure shows the overall framework of an integrated detection and tracking method based on data secondary association. The enhanced image is used as the input X = [x1, x2, ...] of step 2. First, the detector detects the detection boxes and appearance feature vectors of all pedestrians in the image, and sets the confidence threshold to 0.01, retaining both high-scoring and low-scoring detection boxes. The purpose of the multi-pedestrian tracking method is to continuously locate multiple pedestrians in a video sequence, maintain the consistent identity of each pedestrian in the video frame, and generate pedestrian motion trajectories. The Kalman filter is used to process the video data frame by frame, and the Hungarian matching algorithm matches the pedestrian detection box and appearance feature vector output by the detection network in the current frame with the pedestrian position predicted by the Kalman filter in the current frame across frames. The cascade matching method is used to associate the detected pedestrians with the existing tracking trajectories, and the pedestrians with higher frequency are matched first, solving the problem of probability discreteness of continuous prediction. The Kalman filter is used to predict the movement trajectory of the pedestrian in the current frame, and a tracker is set for each existing pedestrian and the newly entered pedestrian. When the unscented Kalman filter can find a matching detection frame in three consecutive frames, it means that the pedestrian is a new pedestrian. The state of the tracker of this pedestrian will be set to "confirmed", otherwise the tracker state is "unconfirmed". If the tracker does not find a matching detection frame within the time of "max_age>30", the corresponding tracker will be deleted, and the pedestrian tracked by this tracker will be assumed to have left the video area. The high confidence threshold detection frame is matched with the trajectory through the cascade matching module.

[0031] Among them, the cascade matching module uses appearance features and motion features to calculate the similarity and obtain the cost matrix. Trajectories that have not been lost will be matched first, and trajectories that have been lost for a long time will be matched later. The appearance features are used to enhance the robustness of pedestrians when they reappear after disappearing. The appearance features will create a list for each pedestrian detected by the detection module, store the appearance feature vectors of the last 100 frames that have been successfully matched, and then calculate the minimum cosine distance between the last 100 successfully matched appearance features of the i-th tracker in the current frame and the appearance features of the j-th pedestrian detection box, as shown in Equation 1.

[0032]

[0033] The motion feature mainly uses the Mahalanobis distance to calculate the correlation between the high-resolution pedestrian detection frame and the tracker predicted position. As shown in Formula 2, the Mahalanobis distance uses the standard deviation between the mean pi of the position tracked by the unscented Kalman filter and the high-resolution pedestrian detection frame to calculate the uncertainty between state estimates. Xi is the covariance, and l_2(i,j) is the Mahalanobis distance between the i-th tracking distribution and the j-th high-resolution detection frame.

[0034]

[0035] Then, the appearance and motion features of the pedestrian are linearly weighted to calculate the matching degree between the detection frame and the tracking trajectory, as shown in Equation 3. When the calculated value of c_(i,j) is less than the specified threshold, it means that the high-score pedestrian detection frame matches the existing trajectory correctly, and the remaining detection frames and trajectories will be classified as mismatched.

[0036] c i,j =(1-λ)l1(i,j)+λl2(i,j) (3)

[0037] For the tracking trajectories that fail to match after cascade matching, the minimum cosine distance is calculated with the pedestrian detection frame with a low confidence threshold, and the obscured pedestrians are searched in the pedestrian detection frame with a low confidence threshold. For the successfully matched trajectories, the unscented Kalman filter is updated, and the low confidence threshold detection frame that has not been successfully matched will be directly deleted.

[0038] Step 3: Perform GIOU matching on the unmatched trajectories, the unmatched high confidence threshold pedestrian detection frames after cascade matching, and the unconfirmed tracking trajectories, and then calculate the cost matrix based on the results of GIOU matching. The obtained cost matrix is ​​used as the input of the Hungarian algorithm to obtain a linear matching result. For the matched trajectories, the unmatched high confidence threshold detection frames will be treated as newly appeared pedestrians. For the unmatched trajectories, if they are in the "confirmed" state and max_age<30, they will be retained and continue to match with the next frame of the video. If max_age>30, they will be directly deleted. The GIOU method is used to improve the loss function, that is, the position information is added when measuring the IOU distance to determine how the detection frame and the predicted frame overlap. In the IoU loss calculation process, some relationships between the predicted bounding box and the true bounding box cannot be determined, such as the direction relationship, that is, when the target object and the detection frame are in different horizontal directions and thus present a certain angle, detection cannot be performed. As shown in Formula 4, the positional relationship between the two frames is calculated by introducing a minimum bounding box, where C is the minimum area surrounding the detection box and the prediction box, A is the detection box area, and B is the prediction bounding box area. From the formula, it can be seen that the farther the distance between the detection box and the prediction box, the larger C is, the smaller the GIOU is, and the larger the corresponding GIOU distance is, indicating that the match between the two is not high, and vice versa. The value range of GIOU is [-1,1]. When the detection box and the prediction box overlap, it takes the maximum value of 1, and when the detection box and the prediction box have no intersection and are infinitely far away, it takes the minimum value of -1.

[0039]

[0040] like Figure 2 As shown in the figure, the gray area is CA∪B. When the positions of the detection box and the prediction box change, the gray area will also change accordingly. The position information between the detection box and the prediction box is indirectly added to the distance matrix by using the ratio of the gray area to the minimum area of ​​the surrounding detection box and the prediction box. GIoU has good operability in practical applications. First, it is insensitive to feature maps of different scales. Secondly, in special cases, when the two boxes overlap, GIoU can take the maximum value of 1, and when the two boxes have no intersection and are infinitely far away, it takes the minimum value of -1, that is, the value range is [-1,1]. This operation can effectively avoid the problem of not being able to back propagate when the gradient is 0, and the model can still be optimized under special circumstances. According to the above analysis, it can be clearly understood that GIoU's attention to overlapping areas is not as single as IoU. It pays attention to both overlapping and non-overlapping areas at the same time, which can better reflect the degree of overlap between the predicted bounding box and the true bounding box. Therefore, GIoU is a very good distance metric.

[0041] Step 4: Compare the detection performance of the improved method with that of the original method. Figure 3It can be seen that the tracking accuracy of the improved method on the MOT16 dataset is 2.4% higher than that of the FairMOT method, IDF1 is improved by 1.5%, MT is improved by 2.6%, ML is improved by 1.5%, and the number of ID switches is reduced by 102 times. Compared with the original method, the improved method on the MOT17 dataset has an improvement of 0.6% in MOTA, 0.4% in IDF1, 1.6% in MT, 0.5% in ML, and a significant reduction in the number of ID switches. This shows that the improved method has better tracking effects for some complex scenes and can significantly reduce pedestrian trajectory interruptions and ID switching times. Figure 4 and Figure 5 The result diagram of the improved FairMOT method is shown in the figure below. Figure 4 The image of pedestrian ID 2 before being blocked is shown in the figure. Figure 5 The image of the pedestrian with ID 2 after being blocked shows that the pedestrian’s ID has not changed.

[0042] Although the above is a description of the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. Where equivalent substitution or equivalent substitution is adopted, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A detection and tracking integrated method based on secondary data association, characterized in that: The steps include: S1, collect target information, integrate multi-state information of data sets, and dynamically perform multiple data set enhancements on the training set; S2. Use the cascade matching method to associate the detected pedestrians with the existing tracking trajectories, and give priority to matching pedestrians with higher frequency, predict the movement trajectory of the pedestrian in the current frame, calculate the minimum cosine distance between the tracking trajectory that fails to match after cascade matching and the pedestrian detection frame with a low confidence threshold, find the obscured pedestrians in the pedestrian detection frame with a low confidence threshold, update the successfully matched trajectory, and directly delete the low confidence threshold detection frame that has not been successfully matched; S3: Match the unmatched trajectories, the high confidence threshold pedestrian detection frames that are not matched after cascade matching, and the unconfirmed tracking trajectories using the GIOU method. By using the loss function GIOU to determine how the detection frame and the prediction frame overlap, the problem of being unable to optimize the algorithm model in special cases is solved: Where A is the area of ​​the detection box, B is the area of ​​the prediction box, and C is the minimum area surrounding the detection box and the prediction box; S4. Compare the detection performance of the improved method with that of the original method.

2. The integrated detection and tracking method based on secondary data association according to claim 1, characterized in that: The method of enhancing the data set in step S1 is random cropping and scaling, affine transformation, and adding noise.

3. The detection and tracking integrated method based on secondary data association according to claim 1, characterized in that: The step S2 comprises the following steps: S21, detects the detection frames and apparent feature vectors of all pedestrians appearing in the image through the detector, sets the confidence threshold, and retains both high-score detection frames and low-score detection frames; S22, uses the cascade matching method to associate the detected pedestrians with the existing tracking trajectories, and prioritizes matching for pedestrians with higher frequency, and predicts the motion trajectory of the pedestrians in the current frame.

4. The detection and tracking integrated method based on secondary data association according to claim 1, characterized in that: In step S2, the cascade matching module uses the appearance features and the motion features to calculate the similarity and obtain the cost matrix. Trajectories that have not been lost will be matched first, and trajectories that have been lost for a long time will be matched later.

5. The detection and tracking integrated method based on secondary data association according to claim 1, characterized in that: The motion feature in step S2 is calculated by using the Mahalanobis distance to calculate the correlation between the high-resolution detection frame and the tracker predicted position.

6. The integrated detection and tracking method based on secondary data association according to claim 1, characterized in that: The multi-state information of the data set includes position information, shape information and posture information of the detected and tracked target.

7. The detection and tracking integrated method based on secondary data association according to claim 1, characterized in that: The step S2 predicts the motion trajectory of the pedestrian in the current frame by setting a tracker for each existing pedestrian and the newly entered pedestrian. When the unscented Kalman filter can find a matching detection frame in three consecutive frames, it means that the pedestrian is a new pedestrian found, and the state of the tracker of the pedestrian will be set to the "confirmed" state, otherwise the tracker state is "unconfirmed". If the tracker does not find a matching detection frame within the time of "max_age>30", the corresponding tracker will be deleted, and it is assumed that the pedestrian tracked by this tracker has left the video area.

8. The detection and tracking integrated method based on secondary data association according to claim 1, characterized in that: In step S3, GIOU takes the maximum value of 1 when the two frames overlap, and takes the minimum value of -1 when the two frames have no intersection and are infinitely far apart, that is, the value range is [-1, 1], to avoid the inability to backpropagate when the gradient is 0.

Images