A multi-target tracking method based on channel attention and tripartition target matching

Abstract

The application discloses a multi-target tracking method based on channel attention and three-part target matching, relates to the field of multi-target tracking in the field of image processing, and is different from the track association and feature matching of only high-score targets in most multi-target tracking models. The application adopts a three-part target score method to accurately and effectively associate features of high-score targets, slightly-shielded medium-score targets and medium- and heavily-shielded low-score targets through dual association of motion features and apparent features under different thresholds, solves the identity switching problem caused by not adding apparent feature matching in the ByteTrack method, and adds a PSA channel attention module facilitating multi-scale feature extraction in the backbone network, thereby reducing the training duration while maintaining target continuity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a multi-target tracking method based on channel attention and three-part target matching. BACKGROUND

[0002] Multi-target tracking algorithm is an important field in computer vision, which is used to solve the continuous tracking problem of targets under single camera or multi-camera. At present, this algorithm is mainly applied to intelligent security, automatic driving and traffic monitoring fields. Although great progress has been made in multi-target tracking algorithm in recent years, some recognized problems are still difficult to solve, such as the problem of missing detection and target identity switching after target occlusion, and the problem of feature matching under crowded data set.

[0003] One of the existing methods to solve the occlusion problem is the method of ByteTrack, which first uses a threshold to distinguish high-score targets and low-score targets that are weakly occluded. Unlike previous methods, it does not directly remove low-score targets, but performs feature matching with high-score targets. Although this method effectively reduces the missing detection problem caused by the occlusion problem, it does not add the correlation of appearance features when performing feature matching, so it is easy to cause false association and identity switching problem for some targets with close distance and similar motion features. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a multi-target tracking method based on channel attention and three-part target matching, which accurately and effectively associates the trajectories of detected non-background information targets, and solves the problems of missing detection, false detection and id switching in multi-target tracking algorithm.

[0005] To solve the above technical problems, the technical solution adopted by the present application is: a multi-target tracking method based on channel attention and three-part target matching, comprising the following steps:

[0006] Step S1, using one-shot multi-target tracking method, the target detection module and the identity embedding module are simultaneously performed in one network, so that the target detection module and the identity embedding module learn fairly and reduce the calculation time of the network by sharing the feature extraction;

[0007] Step S2, using DLA-34 network as the backbone network for feature extraction: by extracting the features of the target set in the t-th frame input picture, the motion features and appearance features are provided for the subsequent target detection module and identity embedding module;

[0008] Step S3, enhancing the effectiveness of the main network feature extraction: adopting the PSA module composed of the spc module and the se channel attention module to learn the correlation between channels in the network;

[0009] Step S4, adopting CenterNet as the target detection branch in the network to detect and label the detection frame of the non-background information to-be-detected target in the image;

[0010] Step S5, adopting a tripartite method to set the target score threshold to classify the detected target into a high-score target, a medium-score target and a low-score target;

[0011] Step S6, by the state of the detected target in the previous t-1 frames, adopting the Kalman filtering method, the IOU distance calculation method or the method combining the Kalman filtering method and the IOU distance calculation method to predict the detected target in different threshold values by different methods, predict the state of the detected target in the t-th frame, and adopt the Hungarian algorithm to match all the detected targets in the t-th frame with the previous trajectories to complete the multi-target tracking.

[0012] The further improvement of the technical scheme of the application is that: in step S3, a PSA module is added after the last basic block of the backbone network, so that the network can extract multi-scale features and learn the correlation between channel features, and the specific operation method is as follows:

[0013] Step S31, the feature map is divided into n parts, and n different size convolution kernels are used to extract features of different scales for the n parts, and then the extracted multi-scale feature maps are concatenated;

[0014] Step S32, the multi-scale feature map is taken as the input feature map of the se channel attention module, and the channel attention vector is obtained through the se module to obtain the multi-scale feature information;

[0015] Step S33, the obtained channel attention vector is spliced and normalized by using Softmax to process the attention vector;

[0016] Step S34, the normalized channel attention vector is distributed to the output feature map of the spc module to complete the correction of the multi-scale channel features.

[0017] The further improvement of the technical scheme of the application is that: in step S32, the channel attention vector obtained through the se module is specifically the attention weight vector of each channel, and the formula is as follows:

[0018] Z i =SE(F i ), i = 0, 1 …… s-1;

[0019] Wherein, Zi is the attention vector of the i-th channel, F i is the multi-scale feature map extracted by the spc module in step S31.

[0020] Further improvement of the technical scheme of the present application is that the formula for normalizing the channel attention vector output by the se module in step S33 is as follows:

[0021] S = Softmax (Z),

[0022] wherein, is the channel attention splicing operation, and S is the normalized attention vector.

[0023] Further improvement of the technical scheme of the present application is that the correction formula of the multi-scale channel feature in step S34 is as follows:

[0024] F n = F o S,

[0025] wherein, F and F n respectively represent the feature map after spc and the feature map after attention vector correction.

[0026] Further improvement of the technical scheme of the present application is that in step S4, parallel heads for estimating heat map, target center point offset and bbox size are added behind the target detection branch, and the detection of the target to be detected is realized by predicting the center point of the target to be detected, the position of the center point of the target is predicted by the heat map with the Gaussian kernel, and the size of the detection frame is set by the radius represented by the Gaussian kernel.

[0027] Further improvement of the technical scheme of the present application is that in step S5, the specific steps for setting the target score threshold by the tripartition method are as follows:

[0028] Step S51, set two thresholds s1 and s2 by the score of the target in the detection stage to classify high-score targets, medium-score targets and low-score targets, wherein the high-score targets, the medium-score targets and the low-score targets represent unobstructed targets, slightly obstructed targets and heavily obstructed targets respectively.

[0029] Step S52, 1-s1 represents unobstructed targets, s1-s2 represents slightly obstructed targets, s2-0.1 represents heavily obstructed targets, and 0.1-0 is regarded as background information.

[0030] Further improvement of the technical scheme of the present application is that in step S6, the step for predicting the targets in different thresholds by different methods is as follows:

[0031] Step S61, twice trajectory matching is performed on high-score targets with obvious appearance features and motion features: the first matching is based on the high-score target set in the t-1th frame and the tth frame , the detection state of each target and the historical trajectory τ t-1 of the target in the t-1th frame, the state of the high-score target set at the tth time is predicted by Kalman filtering, the calculated cost matrix is taken as the input matrix of the Hungarian algorithm, and the matching between the trajectory and the detection frame is completed, and the second matching calculates the similarity between the unmatched trajectory and the unmatched high-score target detection frame by the IOU distance and completes the trajectory matching by the Hungarian algorithm;

[0032] Step S62, once matching is performed on medium-score targets, based on the target set slightly occluded in the t-1th frame and the tth frame , the state of each target and the historical trajectory τ t-1 of the target in the t-1th frame, the state of the medium-score target at the tth time is predicted by Kalman filtering, the cost matrix is obtained based on the detected results of the target at the tth time, and the matching is completed by the Hungarian algorithm;

[0033] Step S63, once matching is performed on a low-score target set , since the low-score target is not obvious in appearance performance due to occlusion, the IOU distance between the trajectory τ t-1 and the low-score target is calculated, and the trajectory matching is performed by the Hungarian algorithm with a lower threshold.

[0034] Thanks to the above technical solutions, the technical progress achieved by the present application is:

[0035] 1. The present application solves the problem of target occlusion under a single camera or multiple cameras, solves the problem of missed detection and identity switching caused by occlusion, optimizes the feature matching method, and optimizes the efficiency of multi-scale feature extraction;

[0036] 2. The present application adds a PSA module in an algorithm combining a joint detection model and a re-identification model, which can effectively extract spatial information of different scales on each channel feature map by compressing the channel dimension of the input tensor, and reduce the training time;

[0037] 3. The present application uses a three-differentiated target score method to accurately and effectively associate features by different threshold-based motion features and appearance features for high-score targets, slightly occluded medium-score targets, and moderately and heavily occluded low-score targets, solving the identity switching problem caused by not adding appearance feature matching in the ByteTrack method. BRIEF DESCRIPTION OF DRAWINGS

[0038] Fig. 1 is a diagram of the method of the invention;

[0039] Fig. 2 is a backbone network diagram;

[0040] Fig. 3 is a feature matching and identity association flowchart; DETAILED DESCRIPTION

[0041] The present invention will be further described in conjunction with the following examples:

[0042] As shown in Figs. 1 to 3 , a multi-target tracking method based on channel attention and three-way target matching includes the following steps:

[0043] Step S1, using a one-shot multi-target tracking method, the target detection module and the identity embedding module are simultaneously performed in one network, so that the target detection module and the identity embedding module learn fairly and reduce the calculation time of the network by sharing feature extraction;

[0044] Step S2, using a DLA-34 network as a backbone network for feature extraction: by extracting the features of the target set in the t-th frame input picture, the motion features and appearance features are provided for the subsequent target detection module and identity embedding module; the DLA-34 network adds more skip connections in the Resnet-34 network, better fuses deep information and shallow information, and provides multi-scale features for the network.

[0045] Step S3, to strengthen the effectiveness of the backbone network feature extraction: a PSA module composed of spc module and se channel attention module is used to learn the correlation between channels in the network;

[0046] A PSA module is added after the last basic block of the backbone network, so that the network can extract multi-scale features and learn the correlation between channel features, and the specific operation method is as follows:

[0047] Step S31, divide the feature map into n parts, and n different size convolution kernels are used to extract features of different scales for the n parts, and then concatenate the extracted multi-scale feature maps;

[0048] Step S32, the multi-scale feature map is used as the input feature map of the se channel attention module, and the channel attention vector is obtained through the se module to obtain multi-scale feature information;

[0049] The specific performance of obtaining the channel attention vector through the se module is the attention weight vector of each channel, and the formula is as follows:

[0050] Zi = SE(F i ), i = 0, 1,..., s - 1;

[0051] wherein Z i is the attention vector of the i-th channel, and F i is the multi-scale feature map extracted by the spc module in step S31.

[0052] In step S33, the obtained channel attention vectors are spliced and normalized by using Softmax, and the formula is as follows:

[0053] S = Softmax(Z),

[0054] wherein, is the channel attention splicing operation, and S is the normalized attention vector.

[0055] In step S34, the normalized channel attention vector is assigned to the output feature map of the spc module, and the multi-scale channel feature is corrected, and the formula is as follows:

[0056] F n = F © S,

[0057] wherein F and F n represent the feature map after spc and the feature map after attention vector correction, respectively.

[0058] In step S4, CenterNet is used as a target detection branch in the network to detect and label the detection frame of the non-background information to be detected in the image; parallel heads for estimating heat map, target center point offset and bbox size are added behind the target detection branch, and the detection of the to-be-detected target is realized by predicting the center point of the to-be-detected target. The position of the target center point is predicted by the heat map with a Gaussian kernel, and the size of the detection frame is set by the radius represented by the Gaussian kernel.

[0059] In step S5, a three-part method is used to set the target score threshold to classify the detected target into high-score target, medium-score target and low-score target, and different correlation methods are used to realize different degrees of correlation of the high-score target, medium-score target and low-score target. The specific steps are as follows:

[0060] In step S51, two thresholds s1 and s2 are set by the score of the target in the detection stage to classify the high-score target, medium-score target and low-score target. The high-score target, medium-score target and low-score target represent the unobstructed target, slightly obstructed target and heavily obstructed target, respectively, so that all non-background targets can be associated with the trajectory;

[0061] Step S52: Let 1 to s1 represent unobstructed targets, s1 to s2 represent slightly obstructed targets, and s2 to 0.1 represent heavily obstructed targets, and regard 0.1 to 0 as background information.

[0062] Step S6: Based on the state of the detected target in the previous t-1 frames, use Kalman filtering and IOU distance calculation, or a combination of Kalman filtering and IOU distance calculation, to predict the state of the detected target at different thresholds, predicting the state of the detected target in frame t. Then, use the Hungarian algorithm to match all detected targets in frame t with the previous trajectories to complete multi-target tracking. The specific steps are as follows:

[0063] Step S61: Perform two trajectory matching operations on high-scoring targets with obvious appearance and motion features: The first matching is based on the set of high-scoring targets in frame t-1 and frame t. The detection status of each target and the historical trajectory τ of the target in frame t-1. t-1 The trajectory is first predicted by Kalman filter to determine the state of the high-scoring target set at time t-1, and the calculated cost matrix is ​​used as the input matrix of the Hungarian algorithm to complete the matching between the trajectory and the detection box. The second matching calculates the similarity between the unmatched trajectory and the unmatched high-scoring target detection box through the IOU distance and completes the trajectory matching through the Hungarian algorithm. Since the high-scoring target is an unoccluded target, its appearance and motion features are more complete than those of the occluded target. Therefore, there will be no large deviation in the matching between the target and the trajectory due to inadequate feature extraction. Performing two matchings can better enable the algorithm to achieve the optimal result in the matching stage between the unoccluded target and the trajectory.

[0064] Step S62: Perform a matching operation on the center-scoring target based on the set of slightly occluded targets in frame t-1 and frame t. The state of each target and its historical trajectory τ in frame t-1. t-1 The trajectory is first predicted by Kalman filter to determine the state of the target at time t-1. The cost matrix is ​​obtained based on the detection result of the target at time t, and the matching is completed by Hungarian algorithm. Since the target with medium score is a slightly occluded target, its appearance features are stronger than those of the low-scoring target that is heavily occluded. However, its motion features have a lower fault tolerance in dense datasets compared to high-scoring targets that can completely extract appearance features and perform two matchings of appearance features, motion features and trajectory. Therefore, instead of calculating the IOU distance between the target with medium score and the trajectory, Kalman filter prediction is performed for the target with medium score, and a matching is performed once by its appearance features and the trajectory.

[0065] Step S63: For the low-scoring target set Once matching is performed, since the low-score target is not obvious in appearance due to the influence of occlusion, the trajectory tau t-1 The IOU distance of the low-score target is calculated, and the trajectory matching is performed by setting a lower threshold value by the Hungarian algorithm.

[0066] Unlike most multi-target tracking models that only track high-score targets and perform feature matching, the present application classifies all score targets into three categories and uses different feature association methods and threshold values to match targets under different occlusions. In addition, the present application adds a PSA channel attention module to the backbone network to facilitate multi-scale feature extraction, which reduces the training time while maintaining target continuity.

[0067] The present application solves the problem of target occlusion under single camera or multiple cameras, and solves the problem of missed detection and identity switching caused by occlusion. In addition, the present application optimizes the feature matching method and the efficiency of multi-scale feature extraction. The present application adds a PSA module to a joint detection model and a re-identification model algorithm. This module can effectively extract spatial information of different scales on each channel feature map by compressing the channel dimension of the input tensor, reducing the training time. The present application uses a three-way target score method to accurately and effectively associate features by using different threshold values for high-score targets, slightly occluded medium-score targets, and medium and heavy occluded low-score targets through dual association of motion features and appearance features, solving the identity switching problem caused by the ByteTrack method without adding appearance feature matching.

Claims

1. A multi-target tracking method based on channel attention and triadic target matching, characterized in that: Includes the following steps: Step S1: The one-shot multi-object tracking method is adopted, which integrates the object detection module and the identity embedding module in one network, so that the object detection module and the identity embedding module can learn fairly and reduce the network's computation time by sharing feature extraction. Step S2: Use the DLA-34 network as the backbone network for feature extraction: By extracting the features of the target set in the input image of frame t, motion features and appearance features are provided for the subsequent target detection module and identity embedding module; Step S3: Enhance the effectiveness of backbone network feature extraction: Use a PSA module composed of an SPC module and a SE channel attention module to learn the correlation between channels in the network; Step S4: Using CenterNet as the target detection branch in the network, detect and label the target objects in the image that are not background information. Step S5: Use a three-part differentiation method to set the target score threshold, classify the detected targets into high-scoring targets, medium-scoring targets, and low-scoring targets, and use different association methods to achieve different degrees of association between high-scoring targets, medium-scoring targets, and low-scoring targets; Step S6: Based on the state of the detected target in the previous t-1 frames, use Kalman filtering method to calculate the IOU distance or a combination of Kalman filtering method and IOU distance to predict the state of the detected target at different thresholds, predict the state of the detected target in the t-th frame, and use the Hungarian algorithm to match all detected targets in the t-th frame with the previous trajectories to complete multi-target tracking. The steps for predicting targets at different thresholds using different methods are as follows: Step S61: Perform two trajectory matching operations on high-scoring targets with obvious appearance and motion features: The first matching is based on the set of high-scoring targets in frame t-1 and frame t. The detection status of each target and the historical trajectory τ of the target in frame t-1. t-1 The trajectory is first predicted by Kalman filter to predict the state of the high-scoring target set at time t-1 and the calculated cost matrix is ​​used as the input matrix of the Hungarian algorithm to complete the matching between the trajectory and the detection box. The second matching calculates the similarity between the unmatched trajectory and the unmatched high-scoring target detection box through the IOU distance and completes the trajectory matching through the Hungarian algorithm. Step S62: Perform a matching operation on the center-scoring target based on the set of slightly occluded targets in frame t-1 and frame t. The state of each target and its historical trajectory τ in frame t-1. t-1 The trajectory in the image is first predicted by Kalman filter to determine the state of the target at time t in time t-1, and the cost matrix is ​​obtained based on the detection result of the target at time t. The matching is then completed by the Hungarian algorithm. Step S63: For the low-scoring target set A matching operation is performed. Since the low-scoring target is not significantly affected by occlusion in its appearance, the trajectory τ is calculated. t-1 With low-scoring targets The IOU distance is used for trajectory matching by setting a low threshold using the Hungarian algorithm.

2. The multi-target tracking method based on channel attention and triadic target matching according to claim 1, characterized in that: In step S3, a PSA module is added after the last basic block of the backbone network, enabling the network to extract multi-scale features and learn the correlation between channel features. The specific operation method is as follows: Step S31: Divide the feature map into n parts, and use n convolutional kernels of different sizes to extract features from these n parts at different scales. Then concat the extracted multi-scale feature maps together. Step S32: Use the multi-scale feature map as the input feature map of the se channel attention module, and obtain the channel attention vector through the se module to obtain multi-scale feature information; Step S33: Concatenate the obtained channel attention vectors and normalize the attention vectors using Softmax; Step S34: Assign the normalized channel attention vectors to the output feature map of the SPC module to complete the correction of multi-scale channel features.

3. The multi-target tracking method based on channel attention and triadic target matching according to claim 2, characterized in that: In step S32, the channel attention vector obtained through the se module is specifically represented as the attention weight vector of each channel, as shown in the following formula: FROM i =SE(F i ),i=0.1......s-1; Among them, Z i Let F be the attention vector for the i-th channel. i This refers to the multi-scale feature map extracted by the spc module in step S31.

4. The multi-target tracking method based on channel attention and triadic target matching according to claim 2, characterized in that: The formula for normalizing the channel attention vector output by the se module using Softmax in step S33 is as follows: S = Softmax(Z), in, This is the attention concatenation operation for each channel, where S is the normalized attention vector.

5. A multi-target tracking method based on channel attention and triadic target matching according to claim 2, characterized in that: The correction formula for the multi-scale channel features in step S34 is as follows: F n =F⊙S, Among them, F and F n These represent the feature map after SPC and the feature map after attention vector correction, respectively.

6. The multi-target tracking method based on channel attention and triadic target matching according to claim 1, characterized in that: In step S4, parallel headers are added after the target detection branch to estimate the heatmap, target center point offset, and bounding box size, respectively. The detection of the target is achieved by predicting the target center point. The target center point position is predicted by the heatmap with Gaussian kernel, and the size of the detection box is set by the radius represented by the Gaussian kernel.

7. The multi-target tracking method based on channel attention and triadic target matching according to claim 1, characterized in that: The specific steps for setting the target score threshold using the triadic method in step S5 are as follows: Step S51: Set two thresholds, s1 and s2, based on the target's score in the detection phase to classify high-scoring targets, medium-scoring targets, and low-scoring targets. High-scoring targets, medium-scoring targets, and low-scoring targets represent unoccluded targets, slightly occluded targets, and heavily occluded targets, respectively, so that all non-background targets can be associated with the trajectory. Step S52: Let 1 to s1 represent unobstructed targets, s1 to s2 represent slightly obstructed targets, and s2 to 0.1 represent heavily obstructed targets, and regard 0.1 to 0 as background information.

Images