A chained multi-target tracking method with quadratically associated low-score detection boxes

Abstract

The present invention provides a chained multi-target tracking method of a secondary associated low-score detection frame, and the specific steps include: multi-scale feature extraction: using two adjacent frames as input to generate a multi-scale feature representation; feature segmentation: combining features to generate two feature maps through a cross-correlation attention module, which are fed into the classification and regression branches and the identity verification branch respectively; joint attention; data association: matching the high-score detection frame with the trajectory according to motion similarity, and then performing a second match between the unmatched trajectory and the low-score detection frame, and correctly matching the occluded person with a low detection score to the previous trajectory and removing the background. The present invention is a multi-target tracking method that improves missed detection and trajectory interruption, reduces the competitive conflict caused by multi-task joint learning during training, and in addition, while retaining the high-score detection results, removes the background from the low-score detection results, and mines the real object, thereby reducing missed detection and improving the coherence of the trajectory.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a chained multi-target tracking method of a secondary associated low-score detection frame. Background Art

[0002] Multi-target tracking has received increasing attention in recent years due to its wide range of application scenarios, such as autonomous driving, motion posture analysis, and traffic monitoring. With the improvement of detection accuracy, detection-tracking methods have become the mainstream in the field of multi-target detection. In the frame-by-frame detection and tracking framework, tracking is considered to be a data association problem in adjacent frames. With the rapid development of target detection, more and more methods have begun to use more powerful detectors to achieve higher tracking performance. CenterNet is the most popular detector used by most methods because it is simple and efficient. The YOLO series detector is also used by a large number of methods because it achieves an excellent balance between accuracy and speed. Most of these methods directly use detection boxes on a single image for tracking. However, when occlusion or motion blur occurs in actual videos, the number of missing detections and very low-scoring detections begins to increase. Therefore, information from previous frames is usually used to improve video detection performance.

[0003] On the other hand, tracking can also be used to help obtain more accurate detection boxes. Use single object tracking (SOT) or Kalman filter to predict the position of the tracked frame in the next frame, and fuse the predicted box with the detection box to enhance the detection result, or use the tracked box in the previous frame to enhance the feature representation of the next frame. Recently, Transformer-based detectors have been used by many methods because of its powerful ability to propagate boxes between frames.

[0004] After obtaining the detection frames through various detectors, most MOT methods only keep a threshold of 0.5 for the high-score detection frames and use these frames as data input. Detection results below this threshold are directly discarded. This is unreasonable. Low-score detection frames often indicate the existence of objects (such as objects with severe occlusion). Simply discarding these objects will cause irreversible errors to MOT, including a large number of missed detections and track interruptions, reducing the overall tracking performance. Summary of the invention

[0005] In view of the above problems, the present invention provides a chained multi-target tracking method for secondary associated low-score detection frames.

[0006] The technical solution of the present invention is:

[0007] A chained multi-target tracking method for secondary associated low-score detection boxes, the specific operation steps are as follows:

[0008] Step 1. Multi-scale feature extraction: Given an image sequence of the tracking target, use two adjacent frames of the image sequence as input, use ResNet-50 as the backbone network to extract high-level semantic features, and then generate a multi-scale feature map through a feature pyramid network;

[0009] Step 2. Feature segmentation: Connect the multi-scale feature maps from each frame together, and generate two attention maps after the combined features pass through the cross-correlation attention network. One attention map is fed into the classification and regression branch, and the other attention map is fed into the authentication branch.

[0010] Step 3. Joint attention: Multiply the predictions of the classification branch and the authentication branch to obtain a joint attention map, which is finally input into the regression branch to generate a bounding box for the object of interest and obtain a video sequence;

[0011] Step 4. Data association: For each frame in the video sequence, use the detector to predict the detection box and score to obtain high-score detection boxes and low-score detection boxes. The high-score detection boxes are first matched with the trajectory based on motion similarity. A Kalman filter is used to predict the position of the trajectory in the new frame. Then, a second match is performed between the unmatched trajectory and the low-score detection box, so that the occluded tracking target with a low detection score is correctly matched to the previous trajectory and the background is removed.

[0012] Furthermore, the specific operation steps of step 1 are as follows:

[0013] Step 1.1 Given an image sequence containing N frames The multi-object tracking task aims to output all bounding boxes for all objects in all frames where they appear. and identity tags F t represents the tth frame, g t The true bounding box representing the number of objects in the t-th frame, to indicate their identity;

[0014] The network requires two adjacent frames as input, called chain nodes. The first chain node is (F1, F2) and the last chain node is (F N ,F N+1 ), (F t-1 ,F t ) as input, the network can generate bounding box pairs for the same object in two frames. Where n t-1 The number of pairs with the same target, and Represents the node F t-1 With F t Similarly, we can also find two bounding boxes of the same object in the next node (Ft ,F t+1 ) to obtain paired bounding boxes and Represents the detected bounding boxes of the same object that are in the common frame of adjacent nodes;

[0015] Step 1.2 The network adopts ResNet-50 as the backbone to extract high-level semantic features, then, it integrates the feature pyramid network to generate multi-scale feature maps for subsequent prediction.

[0016] Furthermore, the specific operation steps of step 2 are as follows:

[0017] Step 2.1 connects the multi-scale feature maps from each frame together;

[0018] Step 2.2: After the combined features are passed through the cross-correlation attention network, two attention maps are generated, one of which is fed into the classification and regression branch, and the other is fed into the authentication branch;

[0019] The feature finally output by the detector is F∈R C×H×W First, a smaller feature F'∈R is obtained through a flat pooling operation C×H ' ×W ', and then input it into the convolutional layer to obtain feature maps T1 and T2 for different tasks,

[0020] Then reshape them into the form {M1,M2}∈R C×N' , where N'=H'×W', and finally multiply each task individually with its own transpose, and apply row-wise softmax to calculate the channel attention map for each task independently The calculation is as follows:

[0021]

[0022] in, It means the relationship between the i-th channel and the j-th channel in the task channel attention map;

[0023] Similarly, matrix multiplication is performed between the transpose of M1 and M2 to learn the commonalities between different tasks, followed by a Softmax layer to generate a cross-correlation weight map

[0024]

[0025] in represents the influence of the ith channel of task 1 / 2 on the jth channel of task 2 / 1. Through the trainable parameter λ, the autocorrelation and cross-correlation weights are finally fused to obtain {W1,W2}∈RC×C :

[0026]

[0027] Rearrange the original feature map F into the shape R C×N , where N = H × W. Then, matrix multiplication is performed between the reshaped features and the learned weight map to obtain the enhanced representation for each task, and the enhanced representation is fused with the original F through residual calculation to prevent information loss.

[0028] Furthermore, the specific operation steps of step 3 are as follows:

[0029] Step 3.1 The attention map of the classification branch makes the regression process pay more attention to the foreground area containing valid information; multiply the predictions of the classification branch and the authentication branch to obtain the joint attention map;

[0030] Step 3.2 broadcasts on the joint attention map so that the joint attention map and the two attention maps obtained after feature segmentation have compatible shapes. With the help of the attention auxiliary function, a bounding box regression branch with four convolutional layers is used to generate paired boxes for the objects of interest to obtain the video sequence.

[0031] Furthermore, the specific operation steps of step 4 are as follows:

[0032] Step 4.1 For each frame in the video, use the detector Det to predict the detection box and score, according to the detection score threshold τ high and τ low All detection boxes are divided into two parts, and for those with scores higher than τ high The detection box is placed into the high-score detection frame D high For those scores in τ high to τ low The detection boxes in the middle, put them into the low score detection box D low ;

[0033] Step 4.2 After separating the low-score detection frames from the high-score detection frames, use the Kalman filter KF to predict the new position of each trajectory in T, where T is the predicted frame;

[0034] Step 4.3 The first association is performed between the high-scoring detection boxes and all trajectories T, and the similarity is determined by the detection box D high The IoU between the detection box and the prediction box T of the trajectory is calculated, and then the Hungarian algorithm is used to complete the similarity-based matching. In particular, if the IoU between the detection box and the trajectory box is less than 0.2, the matching fails and the unmatched detection results are saved to D remain , save the unmatched trajectories to T remain ;

[0035] Step 4.4 The second association is in the low score detection box D low The remaining trajectory after the first association T remain Execute between the two, and save the unmatched trajectories into T re-remain And delete all unmatched low-confidence detection boxes because they are considered as background;

[0036] After step 4.5 association, unmatched trajectories will be removed from the trajectory, for T re-remain The unmatched tracks in T lost , for each T lost , only a certain number of frames are retained, and they are deleted from the trajectory T when they time out. During this period, T lost Remain in trajectory T;

[0037] Step 4.6 Finally, the high confidence detection D that was not matched in the first association remain Initialize it as a new trajectory. The specific operation is, for each remain If the detection box with confidence exceeding ε exists in two consecutive frames, it is initialized as a new trajectory.

[0038] The beneficial effect of the present invention is that: the present invention proposes a chained multi-target tracking method of a secondary associated low-resolution detection frame, which has the following advantages:

[0039] 1. Feature segmentation decouples the detection task and the identity recognition task to balance and reduce the competition between the two tasks. For specific learning, the feature representation of each task is enhanced by reflecting the autocorrelation between different feature channels; for common learning, the shared information between the two tasks is learned through the cross-relation mechanism.

[0040] 2. The irrationality of discarding low-score detection frames in previous methods is solved by using a simple, efficient and universal data association method to process high-score frames and low-score frames separately. The similarity between low-score detection frames and tracking trajectories is used to extract real objects from low-score frames, filter out backgrounds, and maintain trajectory continuity. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is an overall algorithm flow chart of an embodiment of the present invention;

[0042] Figure 2 A network structure diagram in an embodiment of the present invention;

[0043] Figure 3 This is the detailed architecture of the backbone network in the embodiment of the present invention.

[0044] Figure 4 Schematic diagram of a method for associating each detection frame in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to more clearly illustrate the technical solution of the present invention, the present invention will be further described below; obviously, the following description is only a part of the embodiments, and for ordinary technicians in the field, the technical solution of the present invention can also be applied to other similar scenarios based on these without creative work; in order to more clearly illustrate the technical solution of the present invention, the technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0046] like Figure 1 As shown, the present invention proposes a chain multi-target tracking method of secondary associated low-score detection frames, comprising the following steps:

[0047] Step 1. Multi-scale feature extraction: Given an image sequence of the tracked target, two adjacent frames are used as input, ResNet-50 is used as the backbone network to extract high-level semantic features, and then a multi-scale feature map is generated through the Feature Pyramid Network (FPN) for subsequent prediction.

[0048] Specifically, the following steps are included: Step 1.1 Given an image sequence containing N frames The multi-object tracking task aims to output all bounding boxes for all objects in all frames where they appear. and identity tags F t represents the tth frame, g t represents the target K in the tth frame t The true bounding box of the number, Indicate their identity.

[0049] like Figure 2 The network structure diagram of this embodiment is shown as follows: Figure 3 The detailed architecture of the backbone network is shown in Figure 1. The network requires two adjacent frames as input, called chain nodes. The first chain node is (F1, F2) and the last chain node is (F N ,F N+1 ). t-1 ,F t ) as input, the network can generate bounding box pairs for the same object in two frames. Where n t-1 The number of pairs with the same target, and Represents the node F t-1 With F t Similarly, we can also use the next node (F t ,F t+1 ) to obtain paired bounding boxes and Represents the detected bounding boxes of the same object that are in the common frame of neighboring nodes.

[0050] Step 1.2 The network adopts ResNet-50 as the backbone to extract high-level semantic features. Then, it integrates the Feature Pyramid Network (FPN) to generate multi-scale feature representations for subsequent predictions.

[0051] Step 2. Feature segmentation: The scale-level feature maps from each frame are concatenated together, and the combined features are passed through a cross-correlation attention network to generate two attention maps, one of which is fed into the classification and regression branches, and the other is fed into the authentication branch.

[0052] The specific operation steps of step 2 are as follows:

[0053] Step 2.1 concatenates the scale-level feature maps from each frame.

[0054] Step 2.2 generates two feature maps after the combined features pass through the cross-correlation attention network. One feature map is fed into the classification and regression branch, and the other feature map is fed into the authentication branch.

[0055] The feature finally output by the detector is F∈R C×H×W First, a smaller feature F'∈R is obtained through a flat pooling operation C×H'×W' . It is then input into the convolutional layer to obtain feature maps T1 and T2 for different tasks.

[0056] Then reshape them into the form {M1,M2}∈R C×N' , where N'=H'×W'. Finally, each task is multiplied by its own transpose and row-wise softmax is applied to calculate the independent channel attention map for each task. The calculation is as follows:

[0057]

[0058] in, It means the relationship between the i-th channel and the j-th channel in the task channel attention map.

[0059] Similarly, matrix multiplication is performed between the transpose of M1 and M2 to learn the commonalities between different tasks, followed by a Softmax layer to generate a cross-correlation weight map

[0060]

[0061] in represents the influence of the ith channel of task 1 / 2 on the jth channel of task 2 / 1. Through the trainable parameter λ, the autocorrelation and cross-correlation weights are finally fused to obtain {W1,W2}∈R C×C :

[0062]

[0063] Rearrange the original feature map F into the shape R C×N , where N = H × W. Then, matrix multiplication is performed between the reshaped features and the learned weight map to obtain the enhanced representation for each task, and the enhanced representation is fused with the original F through residual calculation to prevent information loss.

[0064] Step 3. Joint attention: Multiply the predictions of the classification branch and the authentication branch to obtain a joint attention map, which is finally input into the regression branch to generate a bounding box for the target of interest and obtain a video sequence; the attention map of the classification branch makes the association process pay more attention to the foreground area containing valid information, and the attention map of the authentication branch makes the network focus on the association of the same target detection box pairs, highlighting the valid information area in the detection box pairs.

[0065] Step 4. Data association: Figure 4 The diagram shows a method for associating each detection frame in an embodiment. For each frame in a video sequence, a detector is used to predict a detection frame and a score to obtain a high-score detection frame and a low-score detection frame. First, the high-score detection frame is matched with the trajectory for the first time according to motion similarity, and a Kalman filter is used to predict the position of the trajectory in the new frame. The motion similarity can be calculated by the IoU of the predicted frame and the detection frame, and the Hungarian algorithm is used to complete the similarity-based matching. Then, a second match is performed between the unmatched trajectory and the low-score detection frame, and the occluded person with a low detection score is correctly matched to the previous trajectory and the background is removed. The specific operation steps of step 4 are as follows:

[0066] Step 4.1 For each frame in the video, use the detector Det to predict the detection box and score. According to the detection score threshold τ high and τ low All detection boxes are divided into two parts. For scores higher than τ high The detection box is put into the high-score detection set D high For those scores in τ high to τ low The detection boxes in the middle are placed into the low-score detection box D low .

[0067] Step 4.2 After separating the low-score detection frames from the high-score detection frames, use the Kalman filter KF to predict the new position of each trajectory in T.

[0068] Step 4.3 The first association is performed between the high-scoring detection boxes and all tracks T. The similarity is determined by the detection box D high The IoU between the detection box and the prediction box T of the trajectory is calculated. Then, the similarity-based matching is performed using the Hungarian algorithm. In particular, if the IoU between the detection box and the trajectory box is less than 0.2, then the matching fails. The unmatched detection results are saved to D remain , save the unmatched trajectories to T remain .

[0069] Step 4.4 The second association is in the low score detection box D low The remaining trajectory after the first association T remain Save the unmatched trajectories into T re-remain And remove all unmatched low-confidence detection boxes, as they are considered as background.

[0070] After the association in step 4.5, the unmatched tracks are removed from the trajectory. For simplicity, the process of track regeneration is not listed in Algorithm 1. In practice, long-distance association requires the preservation of the track identity. For T re-remain The unmatched tracks in T lost For each T lost , only a certain number of frames, such as 30 frames, are retained, and when the timeout occurs, they are deleted from the trajectory T. During this period, T lost Remain in trajectory T.

[0071] Step 4.6 Finally, the high confidence detection D that was not matched in the first association remain Initialize to a new trajectory. The specific operation is, for each remain If the detection box with confidence exceeding ε exists in two consecutive frames, it is initialized as a new trajectory.

[0072] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A chained multi-target tracking method with secondary associated low-score detection frames, characterized by: The specific steps are as follows: Step 1. Multi-scale feature extraction: Given an image sequence of the tracking target, use two adjacent frames of the image sequence as input, adopt ResNet-50 as the backbone network to extract high-level semantic features, and then generate a multi-scale feature map through the feature pyramid network; Step 2. Feature Segmentation: Connect the multi-scale feature maps from each frame together, pass the combined features through the cross-correlation attention network to generate two attention maps, one of which is fed into the classification and regression branches, and the other attention feature map is fed into the authentication branch. The specific steps are as follows: Step 2.1 connects the scale-level feature maps from each frame together; Step 2.2: After the combined features are passed through the cross-correlation attention network, two attention maps are generated. One attention map is fed into the classification and regression branches, and the other attention map is fed into the authentication branch. The feature finally output by the detector is F∈R C×H×W First, a smaller scale feature F'∈R is obtained through the flat pooling operation C×H’×W’ , then input it into the convolutional layer to obtain feature maps T1 and T2 for different tasks, and then reshape them into the form {M1,M2}∈R C×N’ , where N'=H'×W', and finally multiply each task individually with its own transpose, and apply row-wise softmax to calculate the channel attention map for each task independently The calculation is as follows: in, It means the relationship between the i-th channel and the j-th channel in the task channel attention map; Similarly, matrix multiplication is performed between the transpose of M1 and M2 to learn the commonalities between different tasks, followed by a Softmax layer to generate a cross-correlation weight map in represents the influence of the i-th channel of task 1 / 2 on the j-th channel of task 2 / 1; through the trainable parameter λ, the autocorrelation and cross-correlation weights are finally fused to obtain {W1,W2}∈R C×C : Rearrange the original feature map F into the shape R C×N , where N = H × W, then, matrix multiplication is performed between the reshaped features and the learned weight map to obtain the enhanced representation of each task, and the enhanced representation is fused with the original F through residual calculation to prevent information loss; Step 3. Joint attention: Multiply the predictions of the classification branch and the authentication branch to obtain a joint attention map, which is finally input into the regression branch to generate a bounding box for the object of interest and obtain a video sequence; Step 4. Data Association: For each frame in the video sequence, use the detector to predict the detection box and score to obtain high-scoring detection boxes and low-scoring detection boxes. The high-scoring detection boxes are first matched with the tracks based on motion similarity. A Kalman filter is used to predict the position of the tracks in the new frame. Then, a second matching is performed between the unmatched tracks and the low-scoring detection boxes so that the occluded tracked targets with low detection scores are correctly matched to the previous tracks and the background is removed.

2. The chained multi-target tracking method of the secondary associated low-score detection frame according to claim 1 is characterized in that: The specific steps of step 1 are as follows: Step 1.1 Given an image sequence containing N frames The multi-object tracking task aims to output all bounding boxes for all objects in all frames in which they appear. and identity tags F t represents the tth frame, g t The ground-truth bounding box representing the number of targets in the t-th frame, to indicate their identity; The network requires two adjacent frames as input, called chain nodes. The first chain node is (F1, F2) and the last chain node is (F N ,F N+1 ), (F t-1 ,F t ) as input, the network can generate bounding box pairs for the same object in two frames. where n t-1 The number of identical target pairs, and Represents the node F t-1 With F t Similarly, two bounding boxes of the same object can also be found in the next node (F t ,F t+1 ) to obtain paired bounding boxes and Represents the detected bounding boxes of the same object that are located in the common frame of adjacent nodes; In step 1.2, the network adopts ResNet-50 as the backbone to extract high-level semantic features. Then, it integrates the feature pyramid network to generate multi-scale feature maps for subsequent prediction.

3. A chained multi-target tracking method for secondary associated low-score detection frames according to claim 1, characterized in that: The specific steps of step 3 are as follows: Step 3.1: The attention map of the classification branch makes the regression process pay more attention to the foreground area containing valid information; multiply the predictions of the classification branch and the authentication branch to obtain a joint attention map; Step 3.2 broadcasts on the joint attention map so that the joint attention map and the two attention maps obtained after feature segmentation have compatible shapes. With the help of the attention auxiliary function, a bounding box regression branch with four convolutional layers is used to generate paired boxes for the objects of interest to obtain the video sequence.

4. A chained multi-target tracking method for secondary associated low-score detection frames according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step 4.1 For each frame in the video, use the detector Det to predict the detection box and score, according to the detection score threshold τ high and τ low All detection boxes are divided into two parts, and for the detection boxes with scores higher than τ high The detection box is placed into the high-score detection frame D high For those scores in τ high to τ low The detection boxes in the middle, put them into the low score detection box D low ; Step 4.2 After separating the low-score detection frames from the high-score detection frames, use the Kalman filter KF to predict the new position of each trajectory in T, where T is the predicted frame; Step 4.3 The first association is performed between the high-scoring detection boxes and all trajectories T, and the similarity is determined by the detection box D high The IoU between the detection box and the prediction box T of the trajectory is calculated, and then the Hungarian algorithm is used to complete the similarity-based matching. In particular, if the IoU between the detection box and the trajectory box is less than 0.2, then the matching fails and the unmatched detection results are saved to D remain , save the unmatched trajectory to T remain ; Step 4.4 The second association is in the low score detection box D low The remaining trajectory T after the first association remain Execute between the two, and save the unmatched trajectories into T re-remain And delete all unmatched low-confidence detection boxes because they are considered as background; After step 4.5 association, unmatched trajectories will be deleted from the trajectory. re-remain The unmatched trajectory in T lost , for each T lost , only a certain number of frames are retained, and when they are timed out, they are deleted from the trajectory T. During this period, T lost Remain in trajectory T; Step 4.6 Finally, the high confidence detection D that was not matched in the first association is remain Initialize it as a new trajectory. The specific operation is, for each remain If the detection box with confidence exceeding ε exists in two consecutive frames, it is initialized as a new trajectory.

Images