A DeepSORT target tracking method based on feature sharing

Abstract

The present invention discloses a DeepSORT target tracking method based on feature sharing. Aiming at the problems that the feature extraction network of the DeepSORT tracking algorithm is relatively simple and time-consuming, the Kalman filter has poor noise robustness, and the cascade matching has limited tracking accuracy in a powerful tracker, the algorithm is improved from three aspects respectively. The detection network YOLOv5 network is used in the feature extraction module, and the output of appearance information is added to the detection head part. In the motion estimation module, the noise matrix in the Kalman filter algorithm is adaptively calculated and a Gaussian process regression module is added to interpolate the tracking trajectory when the target detector fails. In the trajectory association module, a basic linear matching strategy is used to replace the original cascade matching. On this basis, the YNGB-DeepSORT network is proposed to achieve efficient target tracking.

Description

Technical Field

[0001] The present invention relates to the field of target tracking in deep learning. Aiming at the serious time-consuming problem of DeepSORT, a multi-target tracking algorithm, a DeepSORT target tracking method based on feature sharing is proposed. Background Art

[0002] As an important research direction in the field of computer vision, object tracking has a wide range of applications in various fields. The task of object tracking is to predict the size and position of the target in subsequent frames given the size and position of the target manually or by the previous detection algorithm in a given video sequence.

[0003] DeepSORT has made three improvements on SORT. First, DeepSORT uses a simple convolutional neural network to extract the deep appearance features of the target, and uses these features as the input of the tracking and stores them for subsequent similarity calculation, which greatly improves the ability of DeepSORT to reproduce and re-identify occluded targets. Secondly, DeepSORT replaces the Euclidean distance used for motion feature measurement in SORT with the Mahalanobis distance, and normalizes the variance by adding the operation of the covariance matrix, so that the network can better measure the motion features of the target. The cosine distance is different from the distance measurement of the Mahalanobis distance for the difference degree. It is a similarity measurement for the direction, which is used to calculate the similarity of appearance features. Finally, DeepSORT uses cascade matching, which will divide the order of trackers according to a preset parameter, matching the smaller parameters first and matching the larger parameters later. Although DeepSORT finally achieved a good tracking effect and greatly reduced the IDswitch problem in the SORT algorithm, the following problems still exist:

[0004] 1. The feature extraction network used in DeepSORT is relatively simple, consisting of only a CNN network composed of convolutional layers and residual components. Such a network can generally meet the requirements of the task in simpler scenarios, but if it is used in complex scenarios, the features it can extract are extremely limited, especially when the similarity between two targets is relatively high. Relying solely on the features extracted by such a feature extraction network as constraints often cannot meet the requirements. Secondly, although DeepSORT can meet the real-time requirements on some powerful GPUs, it is often difficult to meet the real-time requirements if it is ported to embedded boards.

[0005] 2. DeepSORT uses Kalman filter for motion estimation, while in traditional Kalman filter algorithm, the noise matrix is ​​a constant matrix. However, in some complex scenes, due to some instability of target detection algorithm, the noise scale should be constantly changing, that is, the noise scale should change with the change of detection confidence. Moreover, in the case of detector failure, if only Kalman filter is used to perform linear interpolation to simulate the motion trajectory of the target, the result is often inaccurate.

[0006] 3. Cascade matching is used to deal with the priority problem of tracking targets. As the tracker becomes more powerful, cascade matching becomes more powerful for confusing associations. Such additional prior constraints limit the performance and accuracy of the matching module. Summary of the invention

[0007] In order to solve the problems existing in the DeepSORT multi-target tracking algorithm, the present invention proposes a multi-target tracking algorithm YNGB-DeepSORT based on feature sharing. The algorithm is based on the DeepSORT algorithm, and is improved in three aspects: feature extraction, motion estimation and trajectory association. The algorithm consists of three modules: feature extraction module, motion estimation module and trajectory association module. In view of the problem that the feature extraction network has limited feature extraction ability and serious time consumption, the tracking algorithm can directly use the features extracted from the detection network without re-extracting features by improving the output information of the detection head in the target detection algorithm. In view of the poor robustness of the noise matrix in the Kalman filter and the trajectory interpolation problem in the case of detection network failure, an adaptive noise calculation method is introduced, and the trajectory in the case of detector failure is interpolated by Gaussian process regression. In view of the problem that cascade matching limits the performance of the tracker, an ordinary linear matching is used to replace the original cascade matching algorithm in DeepSORT.

[0008] The specific steps of the present invention are as follows:

[0009] Step 1: By improving the output information of the detection head in the target detection network, it can output the appearance information of the target, thereby obtaining the motion feature description and the appearance feature description;

[0010] Step 2: After obtaining the feature description of the target, the motion estimation module estimates the tracking frame of the next frame based on the motion characteristics of the target;

[0011] Step 3: According to the motion characteristics and appearance characteristics of the target, the detection frame and the tracking frame are associated with each other to obtain the tracking trajectories of different targets.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] (1) By sharing the features extracted by the detection network, the problem of time-consuming feature extraction in the DeepSORT algorithm is solved;

[0014] (2) The robustness of the tracker to detection noise is improved through noise adaptive updating of Kalman filter; BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Figure 2: Overall block diagram of YNGB-DeepSORT.

[0016] Figure 2 : Schematic diagram of the output information of the improved detection head. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings.

[0018] This paper proposes a wandering slice detection algorithm based on the attention mechanism, aiming to optimize the adaptability of the target detection algorithm to the UAV scene. The overall framework of the algorithm is as follows Figure 1 shown.

[0019] The algorithm consists of three modules: feature extraction module, motion estimation module and trajectory association module. The feature extraction module is responsible for providing the appearance description of the target; the motion estimation module aims to estimate the motion of the target and obtain the tracking estimation frame of the next frame; the trajectory association module is responsible for associating the tracking frame with the detection frame.

[0020] (1) Feature extraction module

[0021] The trajectory association part in DeepSORT uses Mahalanobis distance and cosine similarity distance to constrain motion features and appearance features respectively. The YOLO series of algorithms outputs both the bounding box regression information and classification information of the target bounding box in the detection head. For DeepSORT, the regression information can be used for motion estimation modeling, but the appearance information for cosine similarity calculation is not output in the detection head of the YOLO series of algorithms. The output information of the detection head of the YOLO series of algorithms cannot be directly provided to DeepSORT for similarity calculation. Therefore, in this section, the detection head of the YOLO series of target detection algorithms will be modified. On the basis of the original regression and classification learning tasks, an appearance feature learning task will be added for appearance feature modeling. The detection head of the target detection algorithm YOLOv5 is modified to provide appearance information for the tracking algorithm. The specific diagram of the detection head output information is shown in the figure. Figure 2 shown.

[0022] First, based on the original regression and classification tasks of the prediction head, a branch is added to learn the appearance features of the target. In the appearance feature learning task, the same object needs to be locked by the same tracking label in different frames. This requires that the network can transform the appearance features into sufficiently strong semantic information so that the network can distinguish which track ID the appearance feature belongs to. Here, we borrow the common ideas in target classification, treat each tracking ID as a category, and introduce a Dense fully connected layer to refit the appearance information to reduce the loss of feature information, thereby obtaining a feature vector describing the appearance.

[0023] (2) Motion Estimation Module

[0024] In DeepSORT, the motion of the object is modeled using a Kalman filter based on the linear motion assumption. It consists of a state estimation step and a state update step. In state estimation, the Kalman filter produces an estimate of the current state variables and their uncertainties, and then estimates these values ​​using a weighted average of the estimated state and the measured values. Specifically, it uses the measurement noise covariance R * Represents the noise level of the measured value. A larger noise level means a smaller measurement weight in the state update step due to its larger uncertainty. In the Kalman filter algorithm, the noise scale is a constant matrix. Different measured values ​​intuitively contain different noise levels, but in fact the measurement noise level should change with the change of detection confidence. The original Kalman filter is improved by an adaptive method of calculating the noise covariance. The mathematical representation of this method is shown in formula (1):

[0025]

[0026] Among them, R n is the constant detection noise covariance preset in the original Kalman filter algorithm, c n is the confidence score output by the target detector, represents the new noise covariance, and u is the moment.

[0027] The Kalman filter algorithm after adaptive correction of the noise matrix first calculates the measurement pre-fitting residual. The specific calculation formula is shown in formula (2):

[0028]

[0029] In the formula, z n Represents the detection result at time n, represents the estimated value of the state prediction from time n-1 to time n, H n represents the observation model, The prefit residuals are measured at time n.

[0030] Then the adaptive detection noise covariance is updated, as shown in formula (1). After the noise update is completed, the pre-fitting residual covariance is calculated, which is obtained by formula (3):

[0031]

[0032] Among them, P n|n-1 is the prediction residual covariance matrix from time n-1 to time n.

[0033] Next, the optimal Kalman gain is calculated by formula (4): For S n The inverse of:

[0034]

[0035] Finally, the state estimation and estimated variance are updated and outputted through equations (5) and (6) respectively.

[0036] At this point, the entire operation process of the Kalman filter algorithm improved based on the adaptive noise matrix is ​​completed.

[0037]

[0038] P n|n =(IK n H n ) n|n-1 (6)

[0039] in, is the updated estimate of the state at time n, P n|n is the updated residual covariance matrix at time n, and I is the identity matrix.

[0040] In many complex scenarios, the detection performance of the detector is not very stable. Often the same target can be detected in the current frame but not in the next frame. Therefore, how to fill the track gaps caused by the failure of the detector is a major challenge for long-term target tracking. Interpolation is widely used to fill the track gaps caused by missing detection. Linear interpolation is very popular due to its simplicity. However, its accuracy is limited because it does not use motion information. Although several strategies have been proposed to solve this problem, they usually introduce additional time-consuming modules, such as single target trackers, Kalman filters, and ECC. This paper adopts a lightweight interpolation algorithm that uses Gaussian process regression to simulate nonlinear motion.

[0041] The mathematical expression of the Gaussian process regression interpolation algorithm simulating the trajectory algorithm of the i-th tracking box is shown in formula (7):

[0042]

[0043] Where f represents the image frame, p f represents the position variable at the fth frame, represents Gaussian noise, and

[0044] Given a tracking trajectory of length L and a linear interpolation trajectory of the i-th tracking box By fitting the function g (i) To solve the modeling task of nonlinear motion, t (i) is the moment of the i-th tracking frame, is the position variable of the ith tracking frame. Assume that the fitting function g of the ith tracking frame is (i) Obeying Gaussian process g (i) ∈GP(0,k(t,t′)), where is a nonlinear radial basis kernel function, t and t′ represent two positions in space, λ is a smoothing parameter, and A represents the amplitude parameter of the kernel function. According to the properties of the Gaussian process, a new frame set F can be given: * and smoothed position observations P * The prediction relationship is shown in formula (8):

[0045] P * =K(F * , F)(K(F, F)+s 2 I) -1 P (8)

[0046] In the formula, K(F * ,F), K(F,F) is the covariance function based on k(t,t′), F is the existing frame set, P is the adjacent existing position observation value, I is the unit matrix, and s is the standard deviation of the observation noise.

[0047] (3) Trajectory association module

[0048] DeepSORT uses Mahalanobis distance and cosine distance to calculate the similarity of motion features and appearance features respectively. This algorithm still uses these two distance indicators as indicators for similarity calculation. Mahalanobis distance is used to calculate the correlation between the Kalman filter prediction result of the existing target and the detection result of the current frame. The specific formula is shown in formula (9).

[0049]

[0050] In the formula, a j represents the position of the jth detection box, x i represents the predicted position of the ith tracking trajectory, S iRepresents the covariance matrix between the detected position and the tracked position. If the Mahalanobis distance calculated in a certain association is less than the specified threshold, the association is considered successful.

[0051] The cosine similarity is calculated by j The minimum cosine distance is calculated between the appearance feature vector of and the feature set of the 100 most recently successfully associated features of the ith tracker. The specific calculation formula is shown in formula (10):

[0052]

[0053] where r j is the j-th detection box appearance feature vector, is the kth feature vector in the i-th detection box feature set.

[0054] If the cosine distance is less than the specified threshold, the association is considered successful. Finally, the linear weighted result of the Mahalanobis distance and the cosine distance is used as the final metric. The specific mathematical expression is shown in formula (11):

[0055] C=δA α +(1-δ)A m (11)

[0056] Among them, C represents the cost matrix, A α represents the appearance cost, A m represents the movement cost, and δ represents the weight factor.

Claims

1. A DeepSORT target tracking method based on feature sharing, characterized in that: The following steps are involved: Step 1: By improving the output information of the detection head in the target detection network, it can output the appearance information of the target, thereby obtaining the motion feature description and the appearance feature description; Step 2: After obtaining the feature description of the target, a motion estimation module of a Kalman filter algorithm with adaptive noise covariance based on the detection confidence score is used to predict the tracking box of the next frame, and a Gaussian process regression interpolation algorithm is used to simulate the tracking box trajectory when the detector fails. The adaptive noise covariance calculation formula based on the detection confidence score is: R n is the constant detection noise covariance preset in the original Kalman filter algorithm, c n is the confidence score output by the target detector, represents the new noise covariance, n is the time, The Kalman filter algorithm after adaptive correction of the noise matrix first calculates the measurement pre-fitting residual. The specific calculation formula is shown in formula (2): In the formula, z n Represents the detection result at time n, represents the estimated value of the state prediction from time n-1 to time n, H n represents the observation model, is the pre-fitting residual measured at time n, The update of the adaptive noise covariance based on the detection confidence score is shown in formula (1). After the adaptive noise covariance based on the detection confidence score is updated, the pre-fitting residual covariance is calculated, which is obtained by formula (3): Among them, P n|n-1 is the prediction residual covariance matrix from time n-1 to time n, Next, the optimal Kalman gain K at time n is calculated by formula (4): n , For S n The inverse of: Finally, the state estimation and estimated variance are updated and outputted through equations (5) and (6) respectively. At this point, the operation process of the Kalman filter algorithm improved by adaptive noise covariance based on detection confidence score is completed. P n|n =(I-K n H n )P n|n-1 (6) in, is the updated estimate of the state at time n, P n|n is the residual covariance matrix updated at time n, I is the unit moment, The mathematical expression of the Gaussian process regression interpolation algorithm simulating the trajectory algorithm of the i-th tracking box is shown in formula (7): Where f represents the image frame, p f represents the position variable at the fth frame, represents Gaussian noise, and σ 2 is the variance of Gaussian noise, The fitting function g of the i-th tracking box (i) Obeying Gaussian process g (i) ∈GP(0,k(t,t′)), here is a nonlinear radial basis kernel function, t and t′ represent two positions in space, λ is a smoothing parameter, A represents the amplitude parameter of the kernel function, and the new frame set F * and smoothed position observations P * The prediction relationship is shown in formula (8): P * =K(F * ,F)(K(F,F)+s 2 I) -1 P (8) In the formula, K(F * ,F), K(F,F) is the covariance function based on k(t,t′), F is the existing frame set, P is the adjacent existing position observation value, I is the unit matrix, and s is the standard deviation of the observation noise; Step 3: According to the motion characteristics and appearance characteristics of the target, the detection frame and the tracking frame are associated with each other to obtain the tracking trajectories of different targets.

2. The method according to claim 1, characterized in that: The feature extraction module in step 1 improves the output information of the detection head in the target detection algorithm so that the tracking algorithm can directly use the features extracted from the detection network without re-extracting the features, thus solving the problem of inefficient and time-consuming feature extraction.

3. The method according to claim 1, characterized in that: The motion estimation module in step 2 enhances the robustness of the tracker to noise by introducing an adaptive noise covariance calculation method based on the detection confidence score, and uses a Gaussian process regression interpolation algorithm to simulate the tracking box trajectory value when the detector fails.

4. The method according to claim 1, characterized in that: The trajectory association module in step 3 uses a common linear matching to replace the original cascade matching algorithm. When the tracker is more powerful, the limitation of the prior constraints of cascade matching on the tracker performance is removed.

Images