A joint detection and re-identification multi-target tracking method based on feature decoupling

Abstract

A joint detection and re-identification multi-target tracking method based on feature decoupling relates to the technical field of multi-target tracking. The present invention aims to solve the problem of low tracking accuracy caused by conflicts in optimization directions during the joint training of detection and re-identification tasks in existing joint detection and re-identification multi-target tracking methods. The present invention includes: preprocessing the multi-target tracking dataset; feeding the processed images and labels into the backbone network to extract features F; decoupling the features F in the channel and spatial dimensions to obtain features F suitable for the detection task. det and features F suitable for re-identification tasks id According to F det The subdivision detection branch obtains the position of the target detection bounding box in the image; according to F id The similarity matrix D is calculated by combining the IOU overlap and the similarity matrix D. The target detection bounding box in the image is matched with the existing trajectory using D and the Hungarian algorithm to finally obtain the tracking result. Experiments have shown that the present invention can improve the tracking performance when used for multi-target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a joint detection and re-identification multi-target tracking method based on feature decoupling. Background Art

[0002] Multi-object tracking is a key research area in computer vision. Its primary task is to accurately and in real time detect the positions of multiple objects in a continuous video sequence. Each object is then assigned a unique number, and objects with the same number in consecutive video frames are connected to form a track. Multi-object tracking serves as the foundation for a variety of downstream tasks, including but not limited to video surveillance analysis, autonomous driving, and human-computer interaction. It provides information such as object positions and tracks for higher-level visual understanding and decision-making.

[0003] Multi-target tracking technology is widely used, but achieving accurate multi-target tracking faces many challenges. First, the number of targets is uncertain and may change over time. Second, targets may be poorly detected due to factors such as mutual occlusion, background interference, and lighting variations, which in turn affects tracking performance. Finally, the complexity of target motion patterns and the similarity between targets also increase the difficulty of multi-target tracking.

[0004] To address these challenges, researchers have proposed applying deep learning techniques to multi-target tracking. The advantage of deep learning is that it can automatically learn useful features from large amounts of data without the need for manual design and selection. This allows deep learning to overcome the aforementioned challenges to a certain extent when tackling multi-target tracking. In deep learning-based multi-target tracking, algorithms are categorized into two main categories: detection-based tracking and joint detection-based tracking, depending on whether the detection and tracking stages are independent. Detection-based tracking first uses a detector to annotate the target of interest in each image with a rectangular box. The tracker then uses the detector's output to extract features such as the target's appearance and motion, and calculates the similarity with existing tracks in previous frames to associate the detected target with the track. However, this approach requires two computationally intensive components: the detector and the tracker. To reduce computational complexity, some researchers have attempted to fuse and optimize the neural network used in the detection phase with the neural network used to extract target appearance features in the tracking phase. The JDE-based target tracking method innovatively integrates the target detection and appearance feature extraction stages into a single network, and introduces a feature pyramid network in the detection stage. This greatly improves the inference speed of the multi-target tracking method. However, target detection and re-identification are essentially two different tasks. The detection task focuses more on the positioning of the target in the image and the target category, and pays more attention to global features and deeper semantic information. The re-identification task does not require object positioning, and mainly distinguishes different objects of the same category based on the target's appearance features. This process pays more attention to the fine-grained features of the object. Therefore, there is a conflict in the optimization direction of the target detection task and the re-identification task during the training process. Directly using the features extracted by the same backbone network for target detection and re-identification will reduce the overall tracking accuracy. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem of low tracking accuracy caused by conflict in the optimization directions of detection tasks and re-identification tasks during training in existing joint detection and re-identification multi-target tracking methods, and propose a joint detection and re-identification multi-target tracking method based on feature decoupling.

[0006] A multi-target tracking method for joint detection and re-identification based on feature decoupling includes the following steps:

[0007] Step 1: Obtain the i-th frame image of the video sequence to be tracked, and crop the image to a preset size to obtain the i-th frame target image to be detected;

[0008] Step 2: Input the target image of the i-th frame to be detected into the multi-target tracking model, and obtain the center point coordinates of the k'th target detection bounding box in the target image of the i-th frame to be detected Width and height of the object detection bounding box The offset of the center point coordinates of the target detection bounding box and the target identity vector L i (k');

[0009] Where, k'∈[1,K'], K' is the total number of targets in the target image of the i-th frame;

[0010] Step 3: Use the data obtained in step 2 Get the position of the k'th target detection bounding box in the i-th target image to be detected:

[0011]

[0012] Step 4: Determine whether i is equal to 1. If i=1, initialize all targets in the target image of the i-th frame to be detected as new tracks, and use the target identity vector of the target as the feature of the corresponding track. Then set i=i+1 and return to step 1. If i≠1, execute step 5.

[0013] Step 5: Use the Kalman filter algorithm and the target detection bounding box position in the i-1th frame target image to obtain the target prediction bounding box position in the i-th frame target image to be detected, and use the target prediction bounding box position in the i-th frame target image to be detected, the target detection bounding box position in the i-th frame target image to be detected, the target identity vector of each target in the i-th frame target image to be detected, and the target identity vectors representing different trajectories in the i-1th frame to obtain the similarity matrix D between the target in the i-th frame target image to be detected and the existing trajectories in the i-1th frame target image;

[0014] Step 6. Use D as the matching cost matrix, and use the Hungarian algorithm and the matching cost matrix to perform a first match between the target detection bounding box in the target image of the i-th frame to be detected and the existing track in the target image of the i-1-th frame. Then, use the IOU overlap threshold to perform a second match between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame. Finally, set the unmatched track to an unmatched mark. If an existing track is not matched for z consecutive frames, it is considered that the target corresponding to the track has disappeared, and the current existing track is deleted.

[0015] Wherein, z is a positive integer;

[0016] The IOU overlap threshold is used to perform a second match on the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame, specifically:

[0017] Get the IOU value C between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first matching and the target prediction bounding box corresponding to each unmatched track in the target image of the i-1-th frame. If C is greater than the IOU overlap threshold, it means the match is successful. If C is less than the IOU overlap threshold, it means the match is unsuccessful.

[0018] Step 7: Determine whether i is equal to I. If i=I, output the trajectory matched successfully in step 6. If i<I, set i=i+1 and return to step 1.

[0019] Where I is the total number of frames in the video sequence to be tracked.

[0020] Furthermore, in step 5, the similarity matrix D of the target in the target image of the i-th frame to be detected and the existing trajectories in the target image of the i-th frame to be detected is obtained by using the target prediction bounding box position in the target image of the i-th frame to be detected, the target detection bounding box position in the target image of the i-th frame to be detected, the target identity vector of each target in the target image of the i-th frame to be detected, and the target identity vectors representing different trajectories in the i-1 frame, which is specifically:

[0021] D=λD r +(1-λ)D m

[0022] Among them, D r It is a matrix composed of the cosine distances between the identity vectors of different targets in the target image of the i-th frame to be detected and the target identity vectors representing different trajectories in the target image of the i-1th frame, λ is the weight parameter, D m It is a matrix composed of the Mahalanobis distances between the different target prediction bounding box positions in the target image of the i-th frame to be detected and the different target detection bounding box positions in the target image of the i-th frame.

[0023] Furthermore, the multi-target tracking model in step 2 is obtained by:

[0024] S1. Preprocess the MOT17 dataset to obtain the preprocessed MOT17 dataset, specifically:

[0025] S101. Preprocess the images in the MOT17 dataset:

[0026] Each frame of the target image in the MOT17 dataset is cropped to a uniform size, and then the target image is subjected to data enhancement processing to obtain the preprocessed target image

[0027] The data enhancement processing includes: random rotation, scaling and flipping operations;

[0028] The MOT17 dataset includes a training set and a test set;

[0029] The MOT17 dataset includes: each frame of target image and image label in the target tracking video;

[0030] The image labels in MOT17 include: the identity ID of each target in each frame of the target image and the coordinates of the upper left corner and lower right corner of each target detection bounding box;

[0031] S102: Preprocess the image labels in the MOT17 dataset to obtain preprocessed image labels:

[0032] First, use the coordinates of the upper left corner and lower right corner of each target detection bounding box to obtain the coordinates of the center point of the target detection bounding box

[0033] in, is the coordinate of the upper left corner of the k-th target detection bounding box in the t-th frame target image, is the coordinate of the lower right corner of the k-th target detection bounding box in the t-th frame target image;

[0034] Then, get the width and height of the target detection bounding box

[0035] in, is the width of the k-th target detection bounding box in the t-th frame target image, is the height of the k-th target detection bounding box in the target image of the t-th frame;

[0036] Then, the identity id of the kth target in the tth frame target image is encoded using a one-hot vector to obtain the target identity vector L t (k);

[0037] Then, get the offset of the center point of the k-th target detection bounding box in the t-th frame target image on the feature map

[0038] Finally, the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, the target identity vector, and the offset of the center point of the target detection bounding box on the feature map are used as the preprocessed image label;

[0039] S103, using the preprocessed labels and preprocessed images to form a preprocessed MOT17 dataset;

[0040] The preprocessed MOT17 dataset includes: a preprocessed training set and a preprocessed test set;

[0041] S2. Use the preprocessed MOT17 dataset to train and test the multi-target tracking network to obtain the multi-target tracking model.

[0042] Furthermore, the multi-target tracking network in S2 includes: a backbone network module, a feature decoupling module, a target detection branch module, and a re-identification branch module;

[0043] The backbone network module is a DLA-34 network, which is used to obtain the feature F of the target image in the tth frame;

[0044] The feature decoupling module includes: a channel decoupling unit and a spatial decoupling unit;

[0045] The channel decoupling unit is used to decouple the feature F in the channel dimension, and obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id ;

[0046] The spatial decoupling unit is used to c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branches det and the spatially decoupled feature F for the re-identification branch id ;

[0047] The target detection branch module includes: a heat map sub-branch unit, a bounding box size sub-branch unit, and a center point offset sub-branch unit;

[0048] The heat map sub-branch unit includes a seventh convolution sub-unit Conv5;

[0049] The Conv5 is used to adjust the feature F det Dimension, generate the center point coordinates of the target detection bounding box;

[0050] The Conv5 includes: a twenty-first convolution layer with a 3×3 convolution kernel, a first ReLU activation function layer, and a twenty-second convolution layer with a 1×1 convolution kernel;

[0051] The center point offset sub-branch unit includes an eighth convolution sub-unit Conv6;

[0052] The eighth convolution subunit Conv6 is used to adjust the feature F det Dimensions to generate the offset of the center point of the target detection bounding box;

[0053] The eighth convolution subunit Conv6 includes: a twenty-third convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-fourth convolution layer with a 1×1 convolution kernel;

[0054] The bounding box size sub-branch unit includes a ninth convolution sub-unit Conv7;

[0055] The ninth convolution subunit Conv7 is used to adjust the feature F det Dimensions to generate the width and height of the target detection bounding box;

[0056] The ninth convolution subunit Conv7 includes: a twenty-fifth convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-sixth convolution layer with a 1×1 convolution kernel; the re-identification branch module utilizes the feature F id Get each target identity vector in the target image;

[0057] The re-identification branch module includes an eighth convolution subunit Conv8;

[0058] The Conv8 is used to adjust F id dimensionality to generate the target identity vector;

[0059] The tenth convolution subunit Conv8 includes: a twenty-seventh convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-eighth convolution layer with a 1×1 convolution kernel.

[0060] Furthermore, the channel decoupling unit includes: a maximum pooling layer subunit, an average pooling layer subunit, a first convolution subunit Conv1, a second convolution subunit Conv2, a first element summation subunit, a second element summation subunit, a first residual link subunit, and a second residual link subunit;

[0061] The maximum pooling layer subunit is used to perform a maximum pooling operation on the feature F;

[0062] The average pooling layer subunit is used to perform an average pooling operation on the feature F;

[0063] The first convolution subunit Conv1 includes: a first convolution layer, a second convolution layer, and a ReLU activation function;

[0064] The first convolution subunit is used to aggregate the channel information of the feature F detection task after maximum pooling and average pooling;

[0065] The second convolution subunit Conv2 includes: a third convolution layer, a fourth convolution layer, and a ReLU activation function;

[0066] The second convolutional layer subunit Conv2 is used to aggregate the channel information of the feature F after maximum pooling and average pooling for the re-identification task;

[0067] The first element summation subunit is used to sum the features output by the first convolutional layer subunit element by element and obtain the detection branch channel attention weight C through the Sigmoid activation function det ;

[0068] The second element summation subunit is used to sum the features output by the second convolutional layer subunit element by element and obtain the re-identification branch channel attention weight C through the Sigmoid activation function. id ;

[0069] The first residual link subunit is used to convert C det Multiply it with feature F and add feature F to obtain feature F of the detection branch c_det ;

[0070] The second residual link subunit is used to convert C id Multiply it with feature F and add feature F to obtain feature F of the re-identification branch c_id .

[0071] Furthermore, the channel decoupling unit is used to decouple the feature F in the channel dimension, and obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id , specifically:

[0072] F c_det =F*C det +F

[0073] F c_id =F*C id +F

[0074] C det =σ(Conv1(AvgPool(F))+Conv1(MaxPool(F)))

[0075] C id =σ(Conv2(AvgPool(F))+Conv2(MaxPool(F)))

[0076] Among them, C det is the detection branch channel attention weight, C id is the re-identification branch channel attention weight, F c_det is the feature after channel decoupling that will be used to detect branches, F c_id It is the feature of the channel decoupling that will be used for the re-identification branch, σ is the Sigmoid activation function, Conv1 is the first convolution subunit, Conv2 is the second convolution subunit, AvgPool() is the average pooling operation, and MaxPool() is the maximum pooling operation.

[0077] Furthermore, the spatial decoupling unit includes: a third convolution subunit Conv3, a fourth convolution subunit Conv4, a fifth convolution subunit, a sixth convolution subunit, a third element summation subunit, a fourth element summation subunit, a third residual link subunit, and a fourth residual link subunit;

[0078] The third convolution subunit Conv3 includes: a fifth convolution layer with a 1×1 convolution kernel and a GELU activation function; the input of the third convolution layer subunit is the feature F c_det ;

[0079] The fourth convolution subunit Conv4 includes: a sixth convolution layer with a 1×1 convolution kernel and a GELU activation function; the input of the fourth convolution layer subunit is the feature F c_id ;

[0080] The fifth convolution subunit includes: a first convolution block Conv2d0, a second convolution block Conv2d1, a third convolution block Conv2d2, and a fourth convolution block Conv2d3; the input of the fifth convolution subunit is the output of the third convolution subunit;

[0081] The first convolution block Conv2d0 is the seventh convolution layer of a 5×5 convolution kernel; the input of the first convolution block is the output of the third convolution subunit;

[0082] The second convolution block Conv2d1 includes: an eighth convolution layer with a 1×7 convolution kernel and a ninth convolution layer with a 7×1 convolution kernel; the output of the eighth convolution layer is the input of the ninth convolution layer;

[0083] The third convolution block Conv2d2 includes: a tenth convolution layer with a 1×11 convolution kernel and an eleventh convolution layer with an 11×1 convolution kernel; the output of the tenth convolution layer is the input of the eleventh convolution layer;

[0084] The fourth convolution block Conv2d3 includes: a twelfth convolution layer with a 1×21 convolution kernel and a thirteenth convolution layer with a 21×1 convolution kernel; the output of the twelfth convolution layer is the input of the thirteenth convolution layer;

[0085] The inputs of the second convolution block Conv2d1, the third convolution block Conv2d2, and the fourth convolution block Conv2d3 are all the outputs of the first convolution block Conv2d0;

[0086] The sixth convolution subunit includes: a fifth convolution block Conv2d'0, a sixth convolution block Conv2d'1, a seventh convolution block Conv2d'2, and an eighth convolution block Conv2d'3; the input of the sixth convolution subunit is the output of the fourth convolution subunit;

[0087] The fifth convolution block Conv2d'0 is the fourteenth convolution layer of the 5×5 convolution kernel; the input of the fifth convolution block is the output of the fourth convolution subunit;

[0088] The sixth convolution block Conv2d'1 includes: a fifteenth convolution layer with a 1×3 convolution kernel and a sixteenth convolution layer with a 3×1 convolution kernel; the output of the fifteenth convolution layer is the input of the sixteenth convolution layer;

[0089] The seventh convolution block Conv2d'2 includes: a seventeenth convolution layer with a 1×5 convolution kernel and an eighteenth convolution layer with a 5×1 convolution kernel; the output of the seventeenth convolution layer is the input of the eighteenth convolution layer;

[0090] The eighth convolution block Conv2d'3 includes: a nineteenth convolution layer with a 1×9 convolution kernel and a twentieth convolution layer with a 9×1 convolution kernel; the output of the nineteenth convolution layer is the input of the twentieth convolution layer;

[0091] The inputs of the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 are all the output of the fifth convolution block Conv2d'0;

[0092] The third element summation subunit is used to sum the output features of the first convolution block Conv2d0, the second convolution block Conv2d1, the third convolution block Conv2d2 and the fourth convolution block Conv2d3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling detection branch. det ;

[0093] The fourth element summation subunit is used to sum the output features of the fifth convolution block Conv2d'0, the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling re-identification branch. id ;

[0094] The third residual link subunit is used to convert S det With feature F c_det Multiply and add feature F c_det , obtain the feature F for detecting branches after spatial decoupling det ;

[0095] The fourth residual link subunit is used to convert S id With feature F c_id Multiply and add feature F c_id , obtain the feature F for re-identification branch after spatial decoupling id .

[0096] Furthermore, the spatial decoupling unit is used to c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branchesdet and the spatially decoupled feature F for the re-identification branch id , specifically:

[0097] F det =F c_det *S det +F c_det

[0098] F id =F c_id *S id +F c_id

[0099]

[0100]

[0101] Among them, F c_det is the feature used to detect branches after channel decoupling, F c_id is the feature for re-identification branch after channel decoupling, S det is the spatial attention weight of the spatially disentangled detection branch, S id is the spatial attention weight of the spatial disentangled re-ID branch, F det is the feature used to detect branches after spatial decoupling, F id It is the feature used for re-identification branch after spatial decoupling. Conv2d0() is the fifth convolution block. Both l and l' are integers from 1 to 3. Conv2d l () is the second convolution block, the third convolution block or the fourth convolution block; Conv2d'0() is the fifth convolution block, Conv2d' l' () is the sixth convolution block, the seventh convolution block or the eighth convolution block.

[0102] Furthermore, the overall loss function L used in the training of the detection branch module and the re-identification branch module is total , specifically:

[0103]

[0104]

[0105]

[0106]

[0107] Among them, L heat is the heat map sub-branch unit loss function, L id is the re-identification branch module loss function, L boxis the loss function of the center point offset sub-branch unit and the bounding box size sub-branch unit, w1 and w2 are learnable parameters, k is the target label in the target image of the t-th frame, K is the total number of targets in the target image of the t-th frame, M is the total number of target identity categories in the training set, m is the label of the target identity category in the training set, and L k (m) is the k-th target identity vector in the target image of the t-th frame, p(m) is the target identity category distribution vector predicted by the re-identification branch module, is the true offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, is the predicted offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, λ s is a preset constant, is the actual size of the kth target detection bounding box of the target image in the tth frame, is the predicted size of the kth target detection bounding box of the target image in the tth frame, (x',y') is F det Points in the heat map, is the F predicted by the heatmap sub-branch unit det The heat map value corresponding to the point (x', y') in the heat map, M x'y' It's F det The heat map value corresponding to the point (x', y') in the heat map, α and β are preset hyper parameters.

[0108] Furthermore, the heat map value M x'y' Specifically:

[0109]

[0110] Among them, σ c is the target scale adaptive standard deviation, is the position of the center point of the detection bounding box of the kth target in the tth frame image on the feature map.

[0111] The beneficial effects of the present invention are:

[0112] The present invention provides a multi-target tracking method for joint detection and re-identification based on feature decoupling. This method uses a multi-target tracking framework for joint detection and re-identification, combining a neural network used in the detection phase with a re-identification network used in the tracking phase to extract target appearance features. This reduces the model's inference time and increases its reusability. The method decouples the features extracted by the backbone network from two dimensions: channel and space. This avoids conflicts in the optimization directions of the detection and re-identification tasks during training, thereby improving the accuracy of multi-target tracking. Experimental results from embodiments of the present invention demonstrate that the present invention can effectively improve the performance of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 Flowchart of the present invention;

[0114] Figure 2 This is the overall architecture diagram of the present invention;

[0115] Figure 3 This is the architecture diagram of the channel decoupling module;

[0116] Figure 4 This is the architecture diagram of the spatial decoupling module. DETAILED DESCRIPTION

[0117] Specific implementation method 1: Figure 1-Figure 2 As shown, this embodiment is a reasoning process of a joint detection and re-identification multi-target tracking method based on feature decoupling, in which the feature channel decoupling and spatial decoupling of the feature F extracted by the backbone network are the key points of the present invention, which will be specifically introduced in specific embodiments three, four, five and six. The specific process of this embodiment is:

[0118] Step 1: Get the i-th frame image of the video sequence to be tracked, and crop the image to 1088×608 size to obtain the i-th frame target image to be detected;

[0119] Step 2: Input the target image of the i-th frame to be detected into the multi-target tracking model, and obtain the center point coordinates of the k'th target detection bounding box in the target image of the i-th frame to be detected through model inference. Width and height of the object detection bounding box The offset of the center point coordinates of the target detection bounding box and the target identity vector L i (k');

[0120] Where, k'∈[1,K'], K' is the total number of targets in the target image of the i-th frame;

[0121] in, is the offset of the center point of the k'th target detection bounding box in the i-th target image to be detected on the x-axis, is the offset of the center point of the k'th target detection bounding box in the i-th target image to be detected on the y-axis;

[0122] Step 3: Use the data obtained in step 2 Get the position of the k'th target detection bounding box in the i-th target image to be detected:

[0123]

[0124] in, is the coordinate of the upper left corner of the k'th target detection bounding box in the i-th target image to be detected, is the coordinate of the lower right corner of the k'th target detection bounding box in the i-th target image to be detected;

[0125] Step 4: Determine whether i is equal to 1. If i=1, initialize all targets in the target image of the i-th frame to be detected as new tracks, and use the target identity vector of the target as the feature of the corresponding track. Then set i=i+1 and return to step 1. If i≠1, execute step 5.

[0126] Step 5: Use the Kalman filter algorithm and the target detection bounding box position in the i-1 frame target image to obtain the target prediction bounding box position in the i-1 frame target image to be detected, and calculate the Mahalanobis distance D between the target prediction bounding box in the i-1 frame target image to be detected and the target detection bounding box in the i-1 frame. m Then calculate the cosine distance D between the different target identity vectors of the i-th frame to be detected and the different trajectory vectors that already exist in the i-1 frame r , and D m 、D r The similarity matrix D between the target in the target image of the i-th frame to be detected and the existing trajectory in the target image of the i-1-th frame is obtained by weighted summation with weight λ (0.98), specifically:

[0127] D=λD r +(1-λ)D m

[0128] Among them, D r is a matrix composed of the cosine distances between the identity vectors of different targets in the target image of the i-th frame to be detected and the target identity vectors representing different trajectories in the target image of the i-1-th frame. λ is the weight parameter set to 0.98. D m is a matrix composed of the Mahalanobis distances between the different target prediction bounding box positions in the target image of the i-th frame to be detected and the different target detection bounding box positions in the target image of the i-th frame; in this step, the cosine distance similarity between the target identity vector of the first target in the target image of the i-th frame to be detected and all the target identity vectors representing different trajectories in the target image of the i-1-th frame is D r The first row of , and so on, to obtain the matrix D r ; Similarly, get the matrix D m .

[0129] Step 6. Use the similarity matrix D obtained in step 5 as the matching cost matrix, and use the Hungarian algorithm and the matching cost matrix to perform the first match between the target detection bounding box in the target image of the i-th frame to be detected and the existing track in the target image of the i-1-th frame. Then, use the IOU overlap threshold (0.5) to perform a second match (IOU matching) on ​​the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame. Finally, set the unmatched track to an unmatched mark. If an existing track has not been matched for 30 consecutive frames, it is considered that the target corresponding to the track has disappeared, and the current existing track is deleted.

[0130] The second matching is IOU matching, and the IOU overlap threshold is set to 0.5;

[0131] The IOU matching is specifically as follows: obtain the IOU value C of the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first matching and the target prediction bounding box corresponding to each unmatched track in the target image of the i-1-th frame. If C is greater than the IOU overlap threshold, it indicates a successful match. If C is less than the IOU overlap threshold, it indicates an unsuccessful match.

[0132] Step 7: Determine whether i is equal to I. If i=I, output the trajectory matched successfully in step 6. If i<I, set i=i+1 and return to step 1.

[0133] Where I is the total number of frames in the video sequence to be tracked.

[0134] Specific implementation method 2: The multi-target tracking model is obtained by the following method:

[0135] S1. Preprocess the MOT17 dataset:

[0136] S101. Preprocess the images in the MOT17 dataset:

[0137] Each target image in the MOT17 dataset is cropped to a uniform size of 1088×608, and then random rotation, scaling, and flipping operations are used to perform data augmentation to obtain the preprocessed target image;

[0138] The MOT17 dataset is an existing dataset, which includes a training set and a test set;

[0139] The MOT17 dataset includes: each frame of target image and image label in the target tracking video;

[0140] The image labels in MOT17 include: the identity ID of each target in each frame of the target image and the coordinates of the upper left corner and lower right corner of each target detection bounding box;

[0141] S102. Preprocess the labels in the MOT17 dataset to obtain preprocessed image labels:

[0142] First, use the coordinates of the upper left corner and lower right corner of each target detection bounding box to obtain the coordinates of the center point of the target detection bounding box

[0143] in, is the coordinate of the upper left corner of the k-th target detection bounding box in the t-th frame target image, is the coordinate of the lower right corner of the k-th target detection bounding box in the t-th frame target image;

[0144] Then, get the width and height of the target detection bounding box

[0145] The position of the center point of the target detection bounding box on the feature map

[0146] in, is the width of the k-th target detection bounding box in the t-th frame target image, is the height of the k-th target detection bounding box in the target image of the t-th frame;

[0147] Then, the identity id of each target in the target image of the tth frame is encoded using a one-hot vector to obtain the target identity vector L t (k);

[0148] Then, get the offset of the center point of the target detection bounding box on the feature map

[0149] in, is the offset of the center point of the k-th target detection bounding box in the t-th frame target image;

[0150] Finally, the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, the target identity vector, and the offset of the center point of the target detection bounding box on the feature map are used as the preprocessed image label;

[0151] S103, using the preprocessed labels and preprocessed images to form a preprocessed MOT17 dataset;

[0152] The preprocessed MOT17 dataset includes: a preprocessed training set and a preprocessed test set.

[0153] S2. Use the preprocessed MOT17 dataset to train and test the multi-target tracking network to obtain the multi-target tracking model:

[0154] The multi-target tracking network includes: a backbone network module, a feature decoupling module, a target detection branch module, and a re-identification branch module;

[0155] The backbone network module is a DLA-34 network, which is used to fully extract the fusion information of the target high level and low level to obtain the feature F of the target image of the tth frame;

[0156] The dimensions of F are 272×152×64;

[0157] The feature decoupling module includes: a channel decoupling unit and a spatial decoupling unit;

[0158] The channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id ,like Figure 3 As shown;

[0159] The spatial decoupling unit is used to generate the feature F generated by the channel decoupling unit. c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branches det and the feature F used for re-identification branch after spatial decoupling id ,like Figure 4 As shown;

[0160] The target detection branch module includes a heat map sub-branch unit, a bounding box size sub-branch unit, and a center point offset sub-branch unit; the target detection branch module uses the feature F after spatial decoupling to detect the branch. det Get the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, and the offset of the center point of the target detection bounding box;

[0161] The re-identification branch module uses the feature F id Get the identity vector for each object in the target image.

[0162] Specific implementation method three: Figure 3 As shown, the channel decoupling unit includes: a maximum pooling layer subunit, an average pooling layer subunit, a first convolution subunit Conv1, a second convolution subunit Conv2, a first element summation subunit, a second element summation subunit, a first residual link subunit, and a second residual link subunit;

[0163] The inputs of the maximum pooling layer subunit and the average pooling layer subunit are both feature F, the output of the maximum pooling layer subunit is the input of the first convolution subunit Conv1 and the second convolution subunit Conv2, the output of the average pooling layer subunit is the input of the first convolution subunit Conv1 and the second convolution subunit Conv2, the output of the first convolution subunit Conv1 is the input of the first element summation subunit, the output of the second convolution subunit Conv2 is the input of the second element summation subunit, and the first element summation subunit outputs the detection branch channel attention weight C after the Sigmoid activation function. det , the first element summation subunit outputs the re-identification branch channel attention weight C through the Sigmoid activation function id , the input of the first residual link subunit is the detection branch channel attention weight C det And feature F, the output is the feature F after channel decoupling that will be used to detect the branch c_det ; The input of the second residual link subunit is the re-identification branch channel attention weight C id and feature F, the output is the feature F after channel decoupling that will be used for the re-identification branch c_id ;

[0164] The maximum pooling layer subunit is used to perform a maximum pooling operation on the feature F;

[0165] The average pooling layer subunit is used to perform an average pooling operation on the feature F;

[0166] The first convolution subunit Conv1 includes: a first convolution layer, a second convolution layer, and a ReLU activation function;

[0167] The first convolution subunit is used to aggregate the channel information of the feature F detection task after maximum pooling and average pooling;

[0168] The second convolution subunit Conv2 includes: a third convolution layer, a fourth convolution layer, and a ReLU activation function;

[0169] The second convolutional layer subunit Conv2 is used to aggregate the channel information of the feature F after maximum pooling and average pooling for the re-identification task;

[0170] The first element summation subunit is used to sum the features output by the first convolutional layer subunit element by element and obtain the detection branch channel attention weight C through the Sigmoid activation function det ;

[0171] The second element summation subunit is used to sum the features output by the second convolutional layer subunit element by element and obtain the re-identification branch channel attention weight C through the Sigmoid activation function. id ;

[0172] The first residual link subunit is used to convert C det Multiply it with feature F and add feature F to obtain feature F of the detection branch c_det ;

[0173] The second residual link subunit is used to convert C id Multiply it with feature F and add feature F to obtain feature F of the re-identification branch c_id .

[0174] Specific embodiment 4: The channel decoupling unit is used to decouple the feature F in the channel dimension, and obtain the channel decoupled feature F to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id , specifically:

[0175] F c_det =F*C det +F

[0176] F c_id =F*C id +F

[0177] C det =σ(Conv1(AvgPool(F))+Conv1(MaxPool(F)))

[0178] C id =σ(Conv2(AvgPool(F))+Conv2(MaxPool(F)))

[0179] Among them, F is the feature extracted by the backbone network, C det is the detection branch channel attention weight, C id is the re-identification branch channel attention weight, F c_det is the feature after channel decoupling that will be used to detect branches, F c_id It is the feature of the channel decoupling that will be used for the re-identification branch, σ is the Sigmoid activation function, Conv1 is the first convolution subunit, Conv2 is the second convolution subunit, AvgPool() is the average pooling operation, and MaxPool() is the maximum pooling operation.

[0180] The channel decoupling unit in this embodiment is to alleviate the problem of misalignment in the channel dimension when extracting features for detection and re-identification tasks using the same backbone network. For feature F, first use maximum pooling and average pooling to aggregate channel information to enhance the representation ability of the model. Then, the aggregated channel information is sent to Conv1 and Conv2, and then the channel attention weights C of the detection branch and the re-identification branch are obtained by element-by-element summation. det and C idFinally, using C det and C id As a weight, it is fused with the feature F through residual connection to obtain the feature F of the detection branch after channel decoupling. c_det and the feature F of the re-identification branch after channel decoupling c_id ,After the above operations, the feature F extracted by the backbone network is decoupled in the channel dimension.

[0181] Specific implementation method five: Figure 4 As shown, the spatial decoupling unit includes: a third convolution subunit Conv3, a fourth convolution subunit Conv4, a fifth convolution subunit, a sixth convolution subunit, a third element summation subunit, a fourth element summation subunit, a third residual link subunit, and a fourth residual link subunit;

[0182] The third convolution subunit Conv3 includes: a fifth convolution layer with a 1×1 convolution kernel and a GELU activation function; the input of the third convolution layer subunit is the feature F used to detect the branch after channel decoupling. c_det ;

[0183] The fourth convolution subunit Conv4 includes: a sixth convolution layer with a 1×1 convolution kernel and a GELU activation function; the input of the fourth convolution layer subunit is the feature F used for the re-identification branch after channel decoupling. c_id ;

[0184] The fifth convolution subunit includes: a first convolution block Conv2d0, a second convolution block Conv2d1, a third convolution block Conv2d2, and a fourth convolution block Conv2d3; the input of the fifth convolution subunit is the output of the third convolution subunit;

[0185] The first convolution block Conv2d0 is the seventh convolution layer of a 5×5 convolution kernel; the input of the first convolution block is the output of the third convolution subunit;

[0186] The second convolution block Conv2d1 includes: an eighth convolution layer with a 1×7 convolution kernel and a ninth convolution layer with a 7×1 convolution kernel; the output of the eighth convolution layer is the input of the ninth convolution layer;

[0187] The third convolution block Conv2d2 includes: a tenth convolution layer with a 1×11 convolution kernel and an eleventh convolution layer with an 11×1 convolution kernel; the output of the tenth convolution layer is the input of the eleventh convolution layer;

[0188] The fourth convolution block Conv2d3 includes: a twelfth convolution layer with a 1×21 convolution kernel and a thirteenth convolution layer with a 21×1 convolution kernel; the output of the twelfth convolution layer is the input of the thirteenth convolution layer;

[0189] The inputs of the second convolution block Conv2d1, the third convolution block Conv2d2, and the fourth convolution block Conv2d3 are all the outputs of the first convolution block Conv2d0;

[0190] The sixth convolution subunit includes: a fifth convolution block Conv2d'0, a sixth convolution block Conv2d'1, a seventh convolution block Conv2d'2, and an eighth convolution block Conv2d'3; the input of the sixth convolution subunit is the output of the fourth convolution subunit;

[0191] The fifth convolution block Conv2d'0 is the fourteenth convolution layer of the 5×5 convolution kernel; the input of the fifth convolution block is the output of the fourth convolution subunit;

[0192] The sixth convolution block Conv2d'1 includes: a fifteenth convolution layer with a 1×3 convolution kernel and a sixteenth convolution layer with a 3×1 convolution kernel; the output of the fifteenth convolution layer is the input of the sixteenth convolution layer;

[0193] The seventh convolution block Conv2d'2 includes: a seventeenth convolution layer with a 1×5 convolution kernel and an eighteenth convolution layer with a 5×1 convolution kernel; the output of the seventeenth convolution layer is the input of the eighteenth convolution layer;

[0194] The eighth convolution block Conv2d'3 includes: a nineteenth convolution layer with a 1×9 convolution kernel and a twentieth convolution layer with a 9×1 convolution kernel; the output of the nineteenth convolution layer is the input of the twentieth convolution layer;

[0195] The inputs of the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 are all the output of the fifth convolution block Conv2d'0;

[0196] The third element summation subunit is used to sum the output features of the first convolution block Conv2d0, the second convolution block Conv2d1, the third convolution block Conv2d2 and the fourth convolution block Conv2d3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling detection branch. det , S det As the input of the third residual link subunit;

[0197] The fourth element summation subunit is used to sum the output features of the fifth convolution block Conv2d'0, the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling re-identification branch. id , S id As the input of the fourth residual link subunit;

[0198] The third residual link subunit is used to convert S det With feature F c_det Multiply and add feature F c_det , obtain the feature F for detecting branches after spatial decoupling det ;

[0199] The fourth residual link subunit is used to convert S id With feature F c_id Multiply and add feature F c_id , obtain the feature F for re-identification branch after spatial decoupling id .

[0200] Specific embodiment 6: The spatial decoupling unit is used to generate the feature F generated by the channel decoupling unit. c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the feature F used to detect the branch after decoupling the spatial decoupling unit det The feature F used for re-identification branch after decoupling from the spatial decoupling unit id , specifically:

[0201] F det =F c_det *S det +F c_det

[0202] F id =F c_id *S id +F c_id

[0203]

[0204]

[0205] Among them, F c_det is the feature used to detect branches after channel decoupling, F c_id is the feature for re-identification branch after channel decoupling, S det is the spatial attention weight of the spatially disentangled detection branch, S id is the spatial attention weight of the spatial disentangled re-ID branch, F det is the feature used to detect branches after spatial decoupling, F id It is the feature used for re-identification branch after spatial decoupling. Conv2d0() is the fifth convolution block. Both l and l' are integers from 1 to 3. Conv2d l () is the second convolution block, the third convolution block or the fourth convolution block; Conv2d'0() is the fifth convolution block, Conv2d' l' () is the sixth convolution block, the seventh convolution block or the eighth convolution block.

[0206] In this embodiment, the characteristic F after channel decoupling c_det and F c_id By aggregating the regional features in the feature map, we generate the regional weight matrices suitable for the detection branch and the re-identification branch respectively. First, we use Conv3 and Conv4 to adjust F c_det and F c_id , then use multi-scale asymmetric convolution kernels to enhance the module's ability to extract features of targets of different scales. Next, the features extracted by convolution kernels of different scales are summed element by element, and the Sigmoid activation function is used to obtain the spatial attention weight S of the detection branch. det and the spatial attention weight S of the re-ID branch id Finally, S det and S id As the spatial attention weight of the detection branch and the re-identification branch, it is connected with the feature F through the residual connection. c_det and F c_id After the above operations, the information of the two branches is spatially decoupled.

[0207] Specific embodiment seven: The target detection branch module includes a heat map sub-branch unit, a center point offset sub-branch unit and a bounding box size sub-branch unit, specifically:

[0208] The heat map sub-branch sub-unit includes a seventh convolution sub-unit Conv5;

[0209] The Conv5 is used to adjust the decoupled detection branch feature F det Dimensions to generate the center point coordinates of the target detection bounding box;

[0210] The Conv5 includes: a twenty-first convolution layer with a 3×3 convolution kernel, a first ReLU activation function layer, and a twenty-second convolution layer with a 1×1 convolution kernel;

[0211] The loss function L of the heat map sub-branch sub-unit heat as follows:

[0212]

[0213]

[0214] Among them, L heat is the loss function of the heat map sub-branch unit, K is the total number of targets in the target image of the t-th frame, k is the target label in the target image of the t-th frame, M x'y' It's F det The heat map value corresponding to the point (x', y') in the heat map, is the F of the heat map branch predictiondet The heat map value corresponding to the point (x', y') in the heat map, α and β are preset hyper parameters. In this invention, α=2,β=4,σ c is the target scale adaptive standard deviation, is the position of the center point of the detection bounding box of the kth target in the tth frame image on the feature map;

[0215] The center point offset sub-branch unit includes an eighth convolution sub-unit Conv6;

[0216] The eighth convolution subunit Conv6 is used to adjust the feature F that will be used for the detection branch after decoupling by the spatial decoupling unit det Dimensions to generate the offset of the center point of the target detection bounding box;

[0217] The eighth convolution subunit Conv6 includes: a twenty-third convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-fourth convolution layer with a 1×1 convolution kernel;

[0218] The bounding box size sub-branch unit includes a ninth convolution sub-unit Conv7;

[0219] The ninth convolution subunit Conv7 is used to adjust the feature F that will be used for the detection branch after decoupling by the spatial decoupling unit det Dimensions to generate the width and height of the target detection bounding box;

[0220] The ninth convolution subunit Conv7 includes: a twenty-fifth convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-sixth convolution layer with a 1×1 convolution kernel;

[0221] The center point offset sub-branch unit and the bounding box size sub-branch unit loss function L box as follows:

[0222]

[0223] Among them, K is the total number of targets in the target image of the tth frame, is the true offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, is the predicted offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, is the actual size of the kth target detection bounding box of the target image in the tth frame, is the predicted size of the kth target detection bounding box of the target image in the tth frame, λ s The preset constant is 0.1;

[0224] The re-identification branch module includes a tenth convolution subunit Conv8;

[0225] The re-identification branch is used to extract the identity ID of each target one-hot vector encoding;

[0226] The Conv8 is used to adjust the re-identification branch feature F after decoupling id dimensionality to generate the target identity vector;

[0227] The tenth convolution subunit Conv8 includes: a twenty-seventh convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-eighth convolution layer with a 1×1 convolution kernel;

[0228] The re-identification branch module loss function L id Obtained through:

[0229]

[0230] Where K is the total number of targets in the target image of the t-th frame, M is the total number of target identities in the entire training set, p(m) is the target identity category distribution vector predicted by the re-identification branch module, and L k (m) is the identity vector label of the k-th target in the target image of the t-th frame after encoding using a one-hot vector;

[0231] The overall loss function L of the detection branch module and the re-identification branch module total Obtained through:

[0232]

[0233] Among them, w1 and w2 are two learnable parameters.

[0234] Example: In order to verify the beneficial effects of the present invention, the following experiments were conducted in this example:

[0235] This example conducts a series of ablation experiments on the MOT17 dataset, including experiments based on channel decoupling and spatial decoupling:

[0236] The evaluation indicators used in this embodiment include multiple objection tracking accuracy (MOTA) and the target identity precision recall F1 score (IDF1). The former focuses on measuring the performance of the multiple objection tracking detection stage; the latter focuses on measuring the performance of the multiple objection tracking matching stage. Their calculation formulas are as follows:

[0237]

[0238]

[0239] Where FN is the number of true targets that were not correctly detected, or the number of missed detections; FP is the number of incorrect detections, or the number of false alarms; IDSW is the number of target identity switches during tracking; and GT is the number of true bounding boxes. IDTP, IDFP, and IDFN are the number of correct matches, false alarms, and missed detections of the target ID, respectively.

[0240] Because the MOT challenge official website does not publish the identity labels of the MOT17 test dataset, we evenly divided the MOT17 training set into two parts: the first part is used for training, and the second part is used for testing. This design aims to ensure the fairness and verifiability of the experimental results. The final results are shown in Table 1.

[0241] Table 1 Comparison of channel and spatial decoupling ablation experiments

[0242]

[0243] According to the first and second rows of Table 1, we can see that the baseline algorithm FairMOT has higher FP and IDSW. According to the first and second rows of Table 1, after adding the channel dimension decoupling unit to the baseline algorithm, MOTA increased by 0.3 (67.4-67.7), IDF1 increased by 0.7 (71.3-72.0), and IDSW decreased by about 10.3% (438-393); according to the first and third rows of Table 1, after adding the spatial dimension decoupling unit to the baseline algorithm, MOTA increased by 0.4 (67.4-67.8), IDF1 increased by 0.4 (71.3-71.7); according to the first and fourth rows of Table 1, after adding the channel and spatial dimension decoupling unit to the baseline algorithm, MOTA increased by 0.6 (67.4-68.0), IDF1 increased by 1.5 (71.3-72.8), and IDSW decreased by about 15.5% (438-370). Therefore, it is concluded that adding channel and spatial dimension decoupling units to the baseline algorithm can alleviate the problem of feature misalignment in the channel and spatial dimensions of detection and re-identification tasks, respectively, and effectively improve the performance of the joint detection and re-identification multi-target tracking algorithm.

[0244] In addition, we also trained on multiple datasets such as CrowdHuman, Caltech, CUHK-SYSU, PRW, and MOT17. We then used the weights of the training results to predict the tracking results of MOT17. Finally, we submitted the prediction results to the MOTchallenge website and compared them with the results of the commonly used multi-target tracking algorithms on MOT17. The final results are shown in Table 2.

[0245] Table 2 Comparison of evaluation results of different methods on MOT17

[0246]

[0247]

[0248] The above ablation experiments and comparison results with various methods show that the decoupling method proposed in the present invention can effectively alleviate the competition between the detection and re-identification tasks during the training process of the joint detection and re-identification algorithm, and improve the performance of multi-target tracking; compared with other multi-target tracking algorithms, the multi-target tracking accuracy and IDF1 of the present invention have certain advantages.

Claims

1. A joint detection and re-identification multi-target tracking method based on feature decoupling, characterized by The specific process of the method is: Step 1: Obtain the i-th frame image of the video sequence to be tracked, and crop the image to a preset size to obtain the i-th frame target image to be detected; Step 2: Input the target image of the i-th frame to be detected into the multi-target tracking model, and obtain the center point coordinates of the k'th target detection bounding box in the target image of the i-th frame to be detected Width and height of the object detection bounding box The offset of the center point coordinates of the target detection bounding box and the target identity vector L i (k'); Where, k'∈[1,K'], K' is the total number of targets in the target image of the i-th frame; Step 3: Use the data obtained in step 2 Get the position of the k'th target detection bounding box in the i-th target image to be detected: Step 4: Determine whether i is equal to 1. If i=1, initialize all targets in the target image of the i-th frame to be detected as new tracks, and use the target identity vector of the target as the feature of the corresponding track. Then set i=i+1 and return to step 1. If i≠1, execute step 5. Step 5: Use the Kalman filter algorithm and the target detection bounding box position in the i-1th frame target image to obtain the target prediction bounding box position in the i-th frame target image to be detected, and use the target prediction bounding box position in the i-th frame target image to be detected, the target detection bounding box position in the i-th frame target image to be detected, the target identity vector of each target in the i-th frame target image to be detected, and the target identity vectors representing different trajectories in the i-1th frame to obtain the similarity matrix D between the target in the i-th frame target image to be detected and the existing trajectories in the i-1th frame target image; Step 6. Use D as the matching cost matrix, and use the Hungarian algorithm and the matching cost matrix to perform a first match between the target detection bounding box in the target image of the i-th frame to be detected and the existing track in the target image of the i-1-th frame. Then, use the IOU overlap threshold to perform a second match between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame. Finally, set the unmatched track to an unmatched mark. If an existing track is not matched for z consecutive frames, it is considered that the target corresponding to the track has disappeared, and the current existing track is deleted. Wherein, z is a positive integer; The IOU overlap threshold is used to perform a second match on the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame, specifically: Get the IOU value C between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first matching and the target prediction bounding box corresponding to each unmatched track in the target image of the i-1-th frame. If C is greater than the IOU overlap threshold, it means the match is successful. If C is less than the IOU overlap threshold, it means the match is unsuccessful. Step 7: Determine whether i is equal to I. If i=I, output the trajectory matched successfully in step 6; if i<I, set i=i+1 and return to step 1; Where I is the total number of frames in the video sequence to be tracked.

2. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 1, characterized in that: In step 5, the similarity matrix D of the target in the target image of the i-th frame to be detected and the existing trajectory in the target image of the i-th frame is obtained by using the target prediction bounding box position in the target image of the i-th frame to be detected, the target detection bounding box position in the target image of the i-th frame to be detected, the target identity vector of each target in the target image of the i-th frame to be detected, and the target identity vectors representing different trajectories in the i-1 frame. Specifically, it is: D=λD r +(1-λ)D m Among them, D r It is a matrix composed of the cosine distances between the identity vectors of different targets in the target image of the i-th frame to be detected and the target identity vectors representing different trajectories in the target image of the i-1th frame, λ is the weight parameter, D m It is a matrix composed of the Mahalanobis distances between the different target prediction bounding box positions in the target image of the i-th frame to be detected and the different target detection bounding box positions in the target image of the i-th frame.

3. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 2, characterized in that: The multi-target tracking model in step 2 is obtained by: S1. Preprocess the MOT17 dataset to obtain the preprocessed MOT17 dataset, specifically: S101. Preprocess the images in the MOT17 dataset: Each frame of the target image in the MOT17 dataset is cropped to a uniform size, and then data augmentation is performed on the target image to obtain the preprocessed target image; The data enhancement processing includes: random rotation, scaling and flipping operations; The MOT17 dataset includes a training set and a test set; The MOT17 dataset includes: each frame of target image and image label in the target tracking video; The image labels in MOT17 include: the identity ID of each target in each frame of the target image and the coordinates of the upper left corner and lower right corner of each target detection bounding box; S102: Preprocess the image labels in the MOT17 dataset to obtain preprocessed image labels: First, use the coordinates of the upper left corner and lower right corner of each target detection bounding box to obtain the coordinates of the center point of the target detection bounding box in, is the coordinate of the upper left corner of the k-th target detection bounding box in the t-th frame target image, is the coordinate of the lower right corner of the k-th target detection bounding box in the t-th frame target image; Then, get the width and height of the target detection bounding box in, is the width of the k-th target detection bounding box in the t-th frame target image, is the height of the k-th target detection bounding box in the target image of the t-th frame; Then, the identity id of the kth target in the tth frame target image is encoded using a one-hot vector to obtain the target identity vector L t (k); Then, get the offset of the center point of the k-th target detection bounding box in the t-th frame target image on the feature map Finally, the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, the target identity vector, and the offset of the center point of the target detection bounding box on the feature map are used as the preprocessed image label; S103, using the preprocessed labels and preprocessed images to form a preprocessed MOT17 dataset; The preprocessed MOT17 dataset includes: a preprocessed training set and a preprocessed test set; S2. Use the preprocessed MOT17 dataset to train and test the multi-target tracking network to obtain the multi-target tracking model.

4. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 3, characterized in that: The multi-target tracking network in S2 includes: a backbone network module, a feature decoupling module, a target detection branch module, and a re-identification branch module; The backbone network module is a DLA-34 network, which is used to obtain the feature F of the target image in the tth frame; The feature decoupling module includes: a channel decoupling unit and a spatial decoupling unit; The channel decoupling unit is used to decouple the feature F in the channel dimension, and obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id ; The spatial decoupling unit is used to respectively c_det and F c_id Decouple in the spatial dimension to obtain the feature F that will be used to detect the branch after spatial decoupling det and the spatially decoupled feature F for the re-identification branch id ; The target detection branch module includes: a heat map sub-branch unit, a bounding box size sub-branch unit, and a center point offset sub-branch unit; The heat map sub-branch unit includes a seventh convolution sub-unit Conv5; The Conv5 is used to adjust the feature F det Dimension, generate the center point coordinates of the target detection bounding box; The Conv5 includes: a twenty-first convolution layer with a 3×3 convolution kernel, a first ReLU activation function layer, and a twenty-second convolution layer with a 1×1 convolution kernel; The center point offset sub-branch unit includes an eighth convolution sub-unit Conv6; The eighth convolution subunit Conv6 is used to adjust the feature F det Dimensions to generate the offset of the center point of the target detection bounding box; The eighth convolution subunit Conv6 includes: a twenty-third convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-fourth convolution layer with a 1×1 convolution kernel; The bounding box size sub-branch unit includes a ninth convolution sub-unit Conv7; The ninth convolution subunit Conv7 is used to adjust the feature F det Dimensions to generate the width and height of the target detection bounding box; The ninth convolution subunit Conv7 includes: a twenty-fifth convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-sixth convolution layer with a 1×1 convolution kernel; The re-identification branch module uses the feature F id Get each target identity vector in the target image; The re-identification branch module includes a tenth convolution subunit Conv8; The Conv8 is used to adjust F id dimensionality to generate the target identity vector; The tenth convolution subunit Conv8 includes: a twenty-seventh convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-eighth convolution layer with a 1×1 convolution kernel.

5. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 4, characterized in that: The channel decoupling unit includes: a maximum pooling layer subunit, an average pooling layer subunit, a first convolution subunit Conv1, a second convolution subunit Conv2, a first element summation subunit, a second element summation subunit, a first residual link subunit, and a second residual link subunit; The maximum pooling layer subunit is used to perform a maximum pooling operation on the feature F; The average pooling layer subunit is used to perform an average pooling operation on the feature F; The first convolution subunit Conv1 includes: a first convolution layer, a second convolution layer, and a ReLU activation function; The first convolution subunit Conv1 is used to aggregate the channel information of the feature F detection task after maximum pooling and average pooling; The second convolution subunit Conv2 includes: a third convolution layer, a fourth convolution layer, and a ReLU activation function; The second convolutional layer subunit Conv2 is used to aggregate the channel information of the feature F after maximum pooling and average pooling for the re-identification task; The first element summation subunit is used to sum the features output by the first convolutional layer subunit element by element and obtain the detection branch channel attention weight C through the Sigmoid activation function det ; The second element summation subunit is used to sum the features output by the second convolutional layer subunit element by element and obtain the re-identification branch channel attention weight C through the Sigmoid activation function. id ; The first residual link subunit is used to convert C det Multiply it with feature F and add feature F to obtain feature F of the detection branch c_det ; The second residual link subunit is used to convert C id Multiply it with feature F and add feature F to obtain feature F of the re-identification branch c_id .

6. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 5, characterized in that: The channel decoupling unit is used to decouple the feature F in the channel dimension, and obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id , specifically: F c_det =F*C det +F F c_id =F*C id +F C det =σ(Conv1(AvgPool(F))+Conv1(MaxPool(F))) C id =σ(Conv2(AvgPool(F))+Conv2(MaxPool(F))) Among them, C det is the detection branch channel attention weight, C id is the re-identification branch channel attention weight, F c_det is the feature after channel decoupling that will be used to detect branches, F c_id It is the feature of the channel decoupling that will be used for the re-identification branch, σ is the Sigmoid activation function, Conv1 is the first convolution subunit, Conv2 is the second convolution subunit, AvgPool() is the average pooling operation, and MaxPool() is the maximum pooling operation.

7. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 6, characterized in that: The spatial decoupling unit includes: a third convolution subunit Conv3, a fourth convolution subunit Conv4, a fifth convolution subunit, a sixth convolution subunit, a third element summation subunit, a fourth element summation subunit, a third residual link subunit, and a fourth residual link subunit; The third convolution subunit Conv3 includes: a fifth convolution layer with a 1×1 convolution kernel and a GELU activation function; the input of the third convolution layer subunit is the feature F c_det ; The fourth convolution subunit Conv4 includes: a sixth convolution layer with a 1×1 convolution kernel and a GELU activation function; the input of the fourth convolution layer subunit is the feature F c_id ; The fifth convolution subunit includes: a first convolution block Conv2d0, a second convolution block Conv2d1, a third convolution block Conv2d2, and a fourth convolution block Conv2d3; the input of the fifth convolution subunit is the output of the third convolution subunit; The first convolution block Conv2d0 is the seventh convolution layer of a 5×5 convolution kernel; the input of the first convolution block is the output of the third convolution subunit; The second convolution block Conv2d1 includes: an eighth convolution layer with a 1×7 convolution kernel and a ninth convolution layer with a 7×1 convolution kernel; the output of the eighth convolution layer is the input of the ninth convolution layer; The third convolution block Conv2d2 includes: a tenth convolution layer with a 1×11 convolution kernel and an eleventh convolution layer with an 11×1 convolution kernel; the output of the tenth convolution layer is the input of the eleventh convolution layer; The fourth convolution block Conv2d3 includes: a twelfth convolution layer with a 1×21 convolution kernel and a thirteenth convolution layer with a 21×1 convolution kernel; the output of the twelfth convolution layer is the input of the thirteenth convolution layer; The inputs of the second convolution block Conv2d1, the third convolution block Conv2d2, and the fourth convolution block Conv2d3 are all the outputs of the first convolution block Conv2d0; The sixth convolution subunit includes: a fifth convolution block Conv2d'0, a sixth convolution block Conv2d'1, a seventh convolution block Conv2d'2, and an eighth convolution block Conv2d'3; the input of the sixth convolution subunit is the output of the fourth convolution subunit; The fifth convolution block Conv2d'0 is the fourteenth convolution layer of the 5×5 convolution kernel; the input of the fifth convolution block is the output of the fourth convolution subunit; The sixth convolution block Conv2d'1 includes: a fifteenth convolution layer with a 1×3 convolution kernel and a sixteenth convolution layer with a 3×1 convolution kernel; the output of the fifteenth convolution layer is the input of the sixteenth convolution layer; The seventh convolution block Conv2d'2 includes: a seventeenth convolution layer with a 1×5 convolution kernel and an eighteenth convolution layer with a 5×1 convolution kernel; the output of the seventeenth convolution layer is the input of the eighteenth convolution layer; The eighth convolution block Conv2d'3 includes: a nineteenth convolution layer with a 1×9 convolution kernel and a twentieth convolution layer with a 9×1 convolution kernel; the output of the nineteenth convolution layer is the input of the twentieth convolution layer; The inputs of the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 are all the output of the fifth convolution block Conv2d'0; The third element summation subunit is used to sum the output features of the first convolution block Conv2d0, the second convolution block Conv2d1, the third convolution block Conv2d2 and the fourth convolution block Conv2d3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling detection branch. det ; The fourth element summation subunit is used to sum the output features of the fifth convolution block Conv2d'0, the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling re-identification branch. id ; The third residual link subunit is used to convert S det With feature F c_det Multiply and add feature F c_det , obtain the feature F for detecting branches after spatial decoupling det ; The fourth residual link subunit is used to convert S id With feature F c_id Multiply and add feature F c_id , obtain the feature F for re-identification branch after spatial decoupling id .

8. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 7, characterized in that: The spatial decoupling unit is used to c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the decoupled features F that will be used to detect the branches det and the spatially decoupled feature F for the re-identification branch id , specifically: F det =F c_det *S det +F c_det F id =F c_id *S id +F c_id Among them, F c_det is the feature used to detect branches after channel decoupling, F c_id is the feature for re-identification branch after channel decoupling, S det is the spatial attention weight of the spatially disentangled detection branch, S id is the spatial attention weight of the spatial disentangled re-ID branch, F det is the feature used to detect branches after spatial decoupling, F id It is the feature used for re-identification branch after spatial decoupling. Conv2d0() is the fifth convolution block. Both l and l' are integers from 1 to 3. Conv2d l () is the second convolution block, the third convolution block or the fourth convolution block; Conv2d'0() is the fifth convolution block, Conv2d' l' () is the sixth convolution block, the seventh convolution block or the eighth convolution block.

9. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 8, characterized in that: The overall loss function L used in the training of the detection branch module and the re-identification branch module is total , specifically: Among them, L heat is the heat map sub-branch unit loss function, L id is the re-identification branch module loss function, L box is the loss function of the center point offset sub-branch unit and the bounding box size sub-branch unit, w1 and w2 are learnable parameters, k is the target label in the target image of the t-th frame, K is the total number of targets in the target image of the t-th frame, M is the total number of target identity categories in the training set, m is the label of the target identity category in the training set, and L k (m) is the k-th target identity vector in the target image of the t-th frame, p(m) is the target identity category distribution vector predicted by the re-identification branch module, is the true offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, is the predicted offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, λ s is a preset constant, is the actual size of the kth target detection bounding box of the target image in the tth frame, is the predicted size of the kth target detection bounding box of the target image in the tth frame, (x',y') is F det Points in the heat map, is the F predicted by the heatmap sub-branch unit det The heat map value corresponding to the point (x', y') in the heat map, M x'y' It's F det The heat map value corresponding to the point (x', y') in the heat map, α and β are preset hyper parameters.

10. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 9, characterized in that: The heat map value M x'y' Specifically: Among them, σ c is the target scale adaptive standard deviation, is the position of the center point of the detection bounding box of the kth target in the tth frame image on the feature map.

Images