A multi-target tracking algorithm for target nonlinear motion scenes

Abstract

The present application belongs to the field of computer vision and multi-target tracking, and relates to a multi-target tracking algorithm for a target nonlinear motion scene. The present application improves the detection feature and target re-identification feature extraction mode of the existing multi-target tracker, and at the same time, in order to be more suitable for the characteristics of target-track association matching under the nonlinear motion condition, a new association strategy is proposed in the specific matching association stage. The present application improves the calculation cost, and improves the problem that the existing multi-target tracker is difficult to associate targets under the pedestrian nonlinear motion scene, and improves the tracking effect of the multi-target tracker under the pedestrian nonlinear motion scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and multi-target tracking, and relates to a multi-target tracking algorithm for a target nonlinear motion scene. BACKGROUND

[0002] Multi-target tracking has always been a research hotspot in the field of computer vision, and has many applications in security, autonomous driving, military and other fields. With the development of deep learning technology, multi-target tracking technology has made great progress, and gradually formed the steps of first target detection, then target association (target re-identification) of the detected target and the previous frame track to achieve multi-target tracking. With the development of detection technology and re-identification technology, multi-target tracking technology relying on these two technologies has also made great progress.

[0003] However, in some complex multi-target tracking scenes such as target overlap, occlusion, track intersection, and nonlinear motion, it is difficult to achieve ideal tracking effect. Especially when the target is in nonlinear motion, the existing multi-target tracking algorithm cannot accurately predict the position of the target in the next frame, and cannot effectively associate the target with its previous track. In addition, when the target is in nonlinear motion, it will also exacerbate the phenomenon of target overlap, occlusion, and track intersection, which brings great challenges to multi-target tracking.

[0004] To solve the above problems, the current research on multi-target tracking in nonlinear motion scenes mainly includes three ways: (1) introducing a motion model in the traditional multi-target tracking algorithm to predict the position. The addition of the motion model can help the tracker to more accurately predict the target position. (2) modifying the network structure or introducing attention mechanism, Transformer and other structures to extract more accurate features. (3) adjusting the tracking strategy.

[0005] If you want the multi-target tracking algorithm to achieve good tracking effect in the nonlinear motion scene, the related method should have: (1) precision retention: to fully guarantee the precision of the detection and target re-identification (RE-ID) tasks, so as to accurately achieve the accuracy of tracking in the nonlinear motion scene. (2) more accurate target-track matching algorithm: in order to cope with the rising difficulty of target-track matching in the nonlinear motion scene. SUMMARY

[0006] The present application aims to overcome the drawbacks of the multi-target tracking algorithm in the nonlinear motion scene as introduced above. In view of the motion characteristics of the target in the nonlinear motion scene, the present application improves the existing multi-target tracking algorithm and provides a multi-target tracking algorithm for a target nonlinear motion scene, in order to improve the tracking accuracy of the multi-target tracking algorithm in the target nonlinear motion scene.

[0007] The technical scheme of the present application mainly includes the following two parts

[0008] (1) In order to obtain more accurate detection features and RE-ID features in the feature extraction stage, the extraction method of the two features is improved. In the process of extracting detection features, a spatial attention module is introduced to better fuse features at different levels; In the process of extracting RE-ID features, a channel attention is added, and the required information is better selected from different channels when extracting RE-ID features.

[0009] (2) In order to improve the problem that the target-track is often not easy to match in the scene of target nonlinear motion, a KNN feature matching module is introduced in the target-track matching process. For each newly detected target, match the RE-ID features of the target with all RE-ID in the nearest N (N can be adjusted according to different tracking scenes, and in the present application, it is set to 30) frames in the track. If the K features with the closest cosine distance to the RE-ID features of the target belong to the same track, and the average value of the cosine distance between the RE-ID of the target and the K features is less than a predetermined threshold, then the target is associated with the matching track.

[0010] The technical means adopted by the present application can be summarized as follows: In the detection and RE-ID feature extraction stage, after extracting hierarchical feature information through the backbone network, the extracted multi-level feature information is respectively sent to the feature fusion module with spatial attention and the channel attention module to obtain the detection feature and the RE-ID feature. In the target-track association stage, a KNN feature matching module is added to the original target-track matching stage to associate the target and the track, thereby improving the matching accuracy while reducing the false matching rate.

[0011] A multi-target tracking algorithm for target nonlinear motion scene, the specific steps are as follows:

[0012] Step 1: input the picture into the backbone network, and extract multi-level feature information through the backbone network. Then the feature information is sent to the feature fusion module with spatial attention and the channel attention module to obtain the detection feature and the target recognition (RE-ID) feature.

[0013] Step 2: extract the target and its corresponding RE-ID feature in this frame of picture through the feature obtained in step 1, and divide the target into high confidence target and low confidence target according to its own confidence score. Then match the target in this frame with the previous track (including the track in tracking and the track lost within the set number of frames). If the matching is successful, the target and the track are associated, and if the matching is unsuccessful, go to step 3.

[0014] Third step: collect the target that fails to match in the second step and the target with low confidence and the track that does not match in the second step to perform KNN matching, calculate the cosine distance of the RE-ID feature of the target and all RE-ID features in the nearest set number of frames in the track, if the K features closest to the RE-ID feature of the target belong to the same track and the average value of the cosine distance between the RE-ID of the target and the K features is less than a preset threshold, then associate the target with the matched track.

[0015] Fourth step: for the track marked as lost, if the target is matched again in the second step or the third step, perform track repair according to the observation position and the last appearance position.

[0016] Fifth step: for the target that does not match in the previous four steps, initialize a new track.

[0017] Advantages of the present application: the present application improves the detection feature extraction method of the existing multi-target tracker and the RE-ID feature extraction method of the existing multi-target tracker, and proposes a new association strategy in the specific matching and association stage in order to better adapt to the characteristics of target-track association matching under the condition of nonlinear motion. The present application improves the calculation cost, improves the problem that the existing multi-target tracker is difficult to perform target association under the condition of pedestrian nonlinear motion, and improves the tracking effect of the multi-target tracker under the condition of pedestrian nonlinear motion. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is the network structure framework diagram of the whole application.

[0019] Figure 2 is a detection branch feature fusion module structure diagram.

[0020] Figure 3 is a spatial attention module schematic diagram.

[0021] Figure 4 is a Re-ID branch structure diagram.

[0022] Figure 5 is a channel attention module schematic diagram.

[0023] Figure 6 is a tracking algorithm flowchart of the present application.

[0024] Figure 7 is the tracking effect diagram of the embodiment of the present application on the dancetrack dataset. DETAILED DESCRIPTION

[0025] The specific embodiments of the present application are further illustrated in conjunction with the accompanying drawings and technical solutions.

[0026] A multi-target tracking algorithm for a target nonlinear motion scene, comprising the following steps:

[0027] First, input the picture into the backbone network, extract multi-level feature information through the backbone network, and then send the multi-level feature information into the detection branch and the RE-ID branch to obtain the required feature information for detection and RE-ID. The specific network framework diagram is as shown in Figure 1 The specific calculation can be described as:

[0028] For the detection branch, the input multi-level feature information is obtained through a feature fusion mechanism to obtain a heat map fused with multi-level features. The specific feature fusion process is as shown in Figure 2 Among them, F1, F2 and F3 are three layers of features extracted by the backbone network. Before the low-level feature is fused into the high-level feature, it is first up-sampled, and then the up-sampled feature is sent into the spatial attention (SAM) module to better extract useful information in this dimension and reduce the influence of background noise on detection.

[0029] The structure of the spatial attention is as shown in Figure 3 The specific operation process of the spatial attention is as follows: first, the input feature map is globally pooled and average-pooled, then the globally pooled and average-pooled results are concatenated according to the channel, and finally the concatenated results are convolved to obtain a feature map, which is then processed by an activation function. The summary formula is as shown in formula (1).

[0030] M s (F d )=σ(f 7×7 ([AvgPool(F d );MaxPool(F d )])) (1)

[0031] Among them, M s represents the spatial attention function, i.e. the specific operation of the entire spatial attention, F d represents the detection feature of each layer, σ represents the sigmoid activation function, AvgPool represents the average pooling function, MaxPool represents the maximum pooling function, and f 7×7 represents a 7x7 convolution function.

[0032] The RE-ID branch structure is as shown in Figure 4As shown. For the RE-ID branch, first, the multi-level feature information obtained from the backbone network is converted in dimension to the same dimension feature, and then these dimensional features are aggregated together. Then select through channel attention to get the final RE-ID required feature information. Among them, the main role of the channel attention module is to give different weights to features of different dimensions, so that more accurate RE-ID features can be obtained, and the specific structure is as shown in Figure 5

[0033] The specific operation steps of channel attention are shown in formula (2): first, the input feature map is globally pooled and averaged, and then the global and average pooling results are sent to the multi-layer perceptron MLP learning; finally, the MLP output result is added, and then mapped and processed through the Sigmoid function to obtain the final channel attention value.

[0034] M c (F c )=σ(MLP(AvgPool(F c ))+MLP(MaxPool(F c ))) (2)

[0035] Where, M c represents a specific channel attention function, F c represents the aggregated RE-ID feature, σ represents the sigmoid activation function, MLP represents the multi-layer perceptron function, AvgPool represents the average pooling function, and MaxPool represents the maximum pooling function.

[0036] Second step: after the first step, the detection feature map and the RE-ID feature map corresponding to the input picture are obtained, and the target in the picture needs to be obtained according to the detection feature map. In order to prevent the same target from being repeatedly identified, first, the maximum value suppression operation is performed on the detection feature map, and the K (K is set to 500 in this embodiment) points with the highest detection confidence score are selected as the target candidate for this time. At the same time, find the RE-ID features corresponding to these points from the corresponding RE-ID feature map. In this application, two thresholds are preset, which are high confidence threshold and low confidence threshold, respectively, representing the relatively clear detection target and the uncertain detection target in the picture, and the points with detection confidence score greater than the high confidence threshold are selected from the K points. The high confidence target is set to perform the trajectory target matching this time, and the points with detection confidence score between the two thresholds are set as low confidence targets to perform the matching step in the third step.

[0037] ​The process of matching high-confidence targets with existing trajectories is as follows: the two parties of the match are high-confidence targets detected in the current frame and trajectories currently in the tracking state and trajectories in the lost state within M (set to 30 in this embodiment) frames; the matching criterion is to determine whether the EG distance between the target and the trajectory is less than a preset threshold (in this embodiment, the threshold is set to 0.4 this time).

[0038] The EG distance contains two parts, which are as follows:

[0039] First, the cosine distance E of the RE-ID features of the target and the trajectory is calculated, as shown in formula (3), where and represent the RE-ID features of the target and the trajectory that need to be matched, respectively:

[0040]

[0041] In the formula, the numerator represents the dot product of the two vectors, and the denominator represents the product of the L1 norm of the two vectors.

[0042] Then, the GIOU distance LG between the target and the trajectory is calculated, which is mainly used to determine the similarity of the target and the trajectory, as shown in formula (4):

[0043]

[0044] Where A and B represent the target and the trajectory, respectively, and C represents the minimum enclosing matrix of A and B.

[0045] After obtaining the two distances, formula (5) is calculated to obtain the final matching distance, called EG distance:

[0046] EG = λ1E + λ2LG (5)

[0047] Where λ1 represents the weight coefficient of the RE-ID feature cosine distance, and λ2 represents the weight coefficient of the LG distance. E and LG represent the RE-ID feature cosine distance and the LG distance calculated above, respectively.

[0048] If the EG distance between the target and the trajectory is less than the preset threshold, it is considered to be a successful match, and the two are associated, and then the trajectory update operation is performed; the target and the trajectory that do not match are processed in the third step.

[0049] Step 3: Match the trajectories that do not match in Step 2 and the low-confidence targets obtained in Step 2 with the targets that do not match in Step 2, using the KNN nearest neighbor matching method, and the specific steps are as follows:

[0050] For each target RE-ID feature in this step, match it with all RE-ID features of all existing trajectories in this step (i.e. trajectories that have not been matched successfully), calculate the cosine distance between them, and if the following two conditions are met, consider that the target trajectory is matched successfully.

[0051] The RE-ID feature of the N (set to 3 in this embodiment) closest trajectories to the RE-ID feature of the target belongs to the same trajectory.

[0052] The RE-ID feature of the N (set to 3 in this embodiment) closest trajectories to the RE-ID feature of the target belongs to the same trajectory.

[0053] If the above two conditions are met, consider that the target and the trajectory are matched successfully; the target and the trajectory that have not been matched successfully are processed in the fifth step.

[0054] In the fourth step, if a trajectory in the lost state is successfully associated with the target in the current frame in the second or third step, in order to prevent error accumulation in the lost state, a virtual trajectory is established for trajectory repair. The starting position of the virtual trajectory is the position of the last frame of the current trajectory, and the ending position is the position of the current target through the virtual trajectory; the specific operation is as follows:

[0055] Suppose t1 is the time of the last frame before the trajectory is lost, and t2 is the current time, then the virtual trajectory in the period from t1 to t2 can be represented as follows:

[0056]

[0057] wherein, represents the predicted position of the trajectory at time t (t1 < t < t2), z t1 , z t2 represents the positions of the trajectory at t1 and t2, Traj virtual represents the predicted position function.

[0058] In the estimation process, it is assumed that the virtual trajectory is established at a constant speed, according to the uniform motion model:

[0059] x(t) = x(t0) + v(t-t0) (7)

[0060] wherein, x(t) represents the position of the trajectory at time t, x(t0) represents the position of the trajectory at time t0, and v represents the speed of the trajectory.

[0061] It can be obtained that:

[0062]

[0063] wherein, represents the predicted position of the trajectory at time t, z t1 , z t2 respectively represent the position of the trajectory at the corresponding time.

[0064] Substituted into the Kalman filter, we get:

[0065]

[0066] wherein, F t represents the state transition matrix at time t, H t is the observation matrix at time t, K t represents the Kalman coefficient at time t, is the actual observation value at time t. is the predicted value of the Kalman filter at time t, represents the predicted value derived by the Kalman filter from the state of the trajectory at time t-1, is the predicted value of the Kalman filter at time t-1.

[0067] Fifthly, if there are still unmatched targets in the second and third steps (generally the first appearing target), initialize the operation for this target, that is, regard this target as a new trajectory, and assign the state in the trajectory tracking.

[0068] The above is the operation of the tracking algorithm of the present application when a picture comes. Repeat the above operation to match every time a new frame comes.

[0069] Unlike previous multi-target tracking algorithms, the present application makes the following improvements in order to make the multi-target tracking algorithm more suitable for the characteristics of non-linear motion of the target. On the one hand, more accurate detection features and RE-ID features are extracted, and on the other hand, the matching mechanism between the target and the trajectory is further improved. In order to achieve the first goal, firstly, a spatial attention module is introduced on the basis of the original feature fusion upsampling mechanism. In order to better extract the information needed for target detection from different scales while suppressing the influence of background noise on us, better fuse features of different scales, and thus improve the accuracy of detection features. Secondly, by introducing a channel attention mechanism for feature channel selection instead of selecting features through feature fusion, more accurate RE-ID features are extracted. In order to achieve the second goal, a KNN matching mechanism is added in the original target-trajectory matching process. Compared with the previous matching mechanism, it is more suitable for the motion characteristics of the target in the non-linear motion scene, such as occlusion, deformation, and large-scale displacement, so as to improve the multi-target tracking accuracy.

[0070] Table 1: Comparison of algorithm effects

[0071] Algorithm HOTA MOTA IDF1 Fairmot 39.7 82.2 40.8 Centertrack 41.8 86.8 35.7 ours 44.9 86.8 51.0

[0072] The above table is a comparison of the performance of the present application on the dancetrack dataset and its two benchmark algorithms. The dancetrack dataset emphasizes some target tracking clues that have been ignored by existing datasets, promoting the subsequent algorithm to focus more on the appearance of the target in addition to target detection. The dancetrack collects 100 videos, including group dance, kungfu, gymnastics, etc., which are characterized by: 1) the target task wears similar or even consistent; 2) there are a lot of occlusions and position interchanges between targets; 3) the motion pattern of the target is very complex and diverse, showing obvious nonlinearity and accompanied by diverse body movements. In terms of evaluation indicators, HOTA is an index for evaluating the performance of multi-target tracking, aiming to overcome the limitations of previous indicators. HOTA evaluates the tracking task into three sub-tasks and calculates the score of each sub-task using the Iou formula. Then it combines the three IOU score of each sub-task into the final HOTA score. MOTA represents the accuracy of multi-target tracking, IDF1 represents the proportion of detected targets that obtain correct IDs among detected and tracked targets. It can be seen that the present application has a large improvement in the IDF1 indicator, indicating that the method proposed by the present application improves the accuracy of associating detected targets and trajectories when tracking in complex scenes, especially when the target is in a non-linear motion.

[0073] Figure 7 is a tracking effect display diagram of the present application. Experiments can prove that the present application can maintain good tracking effect in the case of non-linear motion of the target, and can effectively deal with complex tracking scenes such as target congestion, deformation, occlusion, and trajectory overlap. In addition, the present application can maintain a certain tracking speed while maintaining tracking accuracy, and can achieve real-time tracking effect on lower hardware devices.

Claims

1. A multi-target tracking algorithm for a target nonlinear motion scene, characterized in that, The specific steps are as follows: In the first step, the input picture enters the backbone network, and multi-level feature information is extracted through the backbone network. Then, the multi-level feature information is sent to the detection branch and the RE-ID branch to obtain the required feature information for detection and RE-ID. The specific description is as follows: For the detection branch, the input multi-level feature information is fused through a feature fusion mechanism to obtain a heat map that integrates multi-level features. The specific feature fusion process is as follows: F1, F2, and F3 are three layers of features extracted by the backbone network. Before the low-level features are fused into high-level features, they are first upsampled, and then the upsampled features are sent to a spatial attention module. The specific operation process of the spatial attention is as follows: First, the input feature map is globally pooled and averaged. Then, the globally pooled and averaged results are concatenated by channel. Finally, the concatenated results are convolved to obtain a feature map, which is then processed by an activation function. The formula is shown in equation (1). （1）； wherein, represents a specific operation of the spatial attention function, i.e., the entire spatial attention, represents the detection feature of each layer corresponding to, represents a sigmoid activation function, represents an average pooling function, represents a max pooling function, represents a 7x7 convolution function; For the RE-ID branch, the multi-level feature information obtained from the backbone network is first converted in dimension to features of the same dimension. Then, these dimensional features are aggregated together. Finally, the channel attention is selected to obtain the final RE-ID feature information required. The specific operation steps of the channel attention are shown in equation (2): First, the input feature map is globally pooled and averaged. Then, the globally pooled and averaged results are sent to a multi-layer perceptron (MLP) for learning. Finally, the MLP output results are added together, and then mapped and processed by a Sigmoid function to obtain the final channel attention value. （2）； wherein, denotes a specific channel attention function, denotes an aggregated RE-ID feature, denotes a multi-layer perceptron function; In the second step, after the first step, the detection feature map and the RE-ID feature map corresponding to the input picture are obtained. The target in the picture needs to be obtained based on the detection feature map. To prevent the same target from being repeatedly identified, first, the detection feature map is subjected to a maximum suppression operation, and the K points with the highest detection confidence scores are selected as the target candidates. At the same time, the RE-ID features corresponding to these points are found from the corresponding RE-ID feature map. Two thresholds, a high confidence threshold and a low confidence threshold, are preset to represent the relatively clear detection target and the uncertain detection target, respectively. From the K points, the points with detection confidence scores greater than the high confidence threshold are selected as high confidence targets for trajectory target matching, and the points with detection confidence scores between the two thresholds are selected as low confidence targets for the matching step in the third step. The process of matching the high confidence target with the existing trajectory is as follows: The matching parties are the high confidence target detected in the current frame and the trajectory currently in the tracking state and the trajectory in the M frames in the lost state. The matching standard is to judge whether the EG distance between the target and the trajectory is less than a preset threshold. The EG distance contains two parts, which are as follows: First, the cosine distance of the RE-ID features of the target and the trajectory is calculated As shown in equation (3), where and represent the target RE-ID feature and the trajectory RE-ID feature that need to be matched, respectively （3）； In the formula, the numerator represents the dot product of two vectors, and the denominator represents the product of the L1 norms of the two vectors. Then calculate the GIOU distance between the target and the trajectory The GIOU distance is used to judge the similarity of the target and the trajectory, as shown in equation (4): （4）； Wherein, A and B represent target and trajectory respectively, and C represents the minimum enclosing matrix of A and B; After obtaining the two distances, the calculation of formula (5) is performed to obtain the distance used for final matching, referred to as EG distance: （5）； wherein, a weight coefficient representing the RE-ID feature cosine distance, a weight coefficient representing the LG distance; and respectively represent the RE-ID feature cosine distance and the LG distance calculated above. If the EG distance is less than a preset threshold value in the target and the trajectory, it is determined that the matching is successful, the two are associated, and then the trajectory updating operation is performed; the target and the trajectory that are not matched are processed in the third step; Third step: matching the target with low confidence obtained in the second step and the target that is not matched successfully in the second step with the trajectory that is not matched successfully in the second step, and the matching method adopts KNN nearest neighbor matching method, and the specific steps are as follows: For the RE-ID feature of each target in this step, all RE-ID features of all existing trajectories are matched, and the cosine distance between them is calculated, and if the following two conditions are met, the target and the trajectory are considered to be matched successfully; The RE-ID features of the N nearest trajectories of the RE-ID feature of the target belong to the same trajectory; The RE-ID feature of the nearest trajectory of the RE-ID feature of the target is less than a preset threshold value; If the above two conditions are met, the target and the trajectory are considered to be matched successfully; the target and the trajectory that are not matched successfully are processed in the fifth step; Fourth step: if the trajectory in the lost state is associated successfully with the target in the current frame in the second step or the third step, in order to prevent error accumulation in the lost state, a virtual trajectory is established for trajectory repair, and the starting position of the virtual trajectory is the position of the last frame of the current trajectory, and the ending position is the position of the current target through the virtual trajectory; the specific operation is as follows: Assume is the time of the last frame before the trajectory is lost, is the current time, then the virtual trajectory is represented as follows during the time period to . （6）； wherein, represents the predicted position of the trajectory at time t, , , represents the position at which the trajectory is located at time t, and , represents the predicted position function; In the estimation process, it is assumed that the virtual trajectory is established by constant speed, and according to the uniform motion model: （7）； wherein, represents the position of the trajectory at the time instant, represents the position of the trajectory at the time instant, represents the velocity of the trajectory; It is obtained that: （8）； wherein, represents a predicted position of the trajectory at time t, , respectively represent a position of the trajectory at the respective time; Substituted into the Kalman filter, it is obtained that: （9）； in, represent The state transition matrix at time t, for Time-of-flight observation matrix The Kalman coefficients at time t; For Kalman filtering in Predicted value at time, The representative is based on the trajectory by Kalman filtering. The predicted value derived from the state at any given time. For Kalman filtering in The predicted value at any given time; Fifth step: if there are still unmatched targets in the second step and the third step, initialization operation is performed for the target, that is, the target is regarded as a new trajectory, and the state in tracking is assigned to the trajectory. The above is the operation when a picture comes, and the above operation is repeated for matching when a new frame comes.

Images