Feature decoupling-based joint detection and re-identification multi-target tracking method

Abstract

The invention discloses a joint detection and re-identification multi-target tracking method based on feature decoupling, and relates to the technical field of multi-target tracking. The objective of the invention is to solve the problem of low tracking precision caused by conflicts in optimization directions in a detection and re-identification task joint training process in an existing joint detection and re-identification multi-target tracking method. The method comprises the following steps: preprocessing a multi-target tracking data set; sending the processed image and label to a backbone network to extract a feature F; decoupling the feature F in the channel dimension and the spatial dimension in sequence to obtain a feature Fdet suitable for a detection task and a feature Fid suitable for a re-identification task; obtaining the position of a target detection bounding box in the image according to the Fdet subdivision detection branch; calculating a similarity matrix D according to the Fid and IOU overlapping degree; and matching the target detection bounding box in the image with the existing trajectory by using the D and a Hungary algorithm to finally obtain a tracking result. Experiments prove that when the method is used for multi-target tracking, the tracking performance can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-target tracking, and in particular to a joint detection and re-identification multi-target tracking method based on feature decoupling. Background Art

[0002] Multi-target tracking is an important research direction in the field of computer vision. Its main task is to detect the positions of multiple targets in real time and accurately in a continuous video sequence, and then assign a unique number to each target. Targets with the same number in consecutive video frames are connected to form a track. Multi-target tracking can serve as the basis for a variety of downstream tasks, including but not limited to video surveillance analysis, autonomous driving, human-computer interaction, etc., providing these tasks with information such as target position and track for higher-level visual understanding and decision-making.

[0003] Multi-target tracking technology is widely used, but there are many challenges in achieving accurate multi-target tracking. First, the number of targets is uncertain and may change over time. Second, targets may be poorly detected due to mutual occlusion, background interference, and lighting changes, which in turn affects tracking performance. Finally, the complexity of target motion patterns and the similarity between targets also increase the difficulty of multi-target tracking.

[0004] In order to solve the above challenges, researchers proposed to apply deep learning technology to multi-target tracking. The advantage of deep learning technology is that it can automatically learn useful features from a large amount of data without manual design and selection. This enables deep learning technology to overcome the above challenges to a certain extent when dealing with multi-target tracking problems. In multi-target tracking based on deep learning, multi-target tracking algorithms are divided into two categories: detection-based tracking and joint detection tracking, depending on whether the detection and tracking stages are independent. Detection-based tracking first uses a detector to mark the target of interest in each image with a rectangular box, and then the tracker uses the output of the detector to extract the target's appearance, motion and other features again, and calculates the similarity between the existing trajectory in the previous frame to associate the detected target with the trajectory. However, this method requires two computationally intensive components, the detector and the tracker. In order to reduce the complexity of the calculation, some researchers have tried to fuse and optimize the neural network in the detection stage and the neural network used to extract the target appearance features in the tracking stage. The target tracking method based on JDE innovatively integrates the target detection link and the appearance feature extraction link into one network, and introduces the feature pyramid network in the detection stage, which greatly improves the reasoning speed of the multi-target tracking method. However, target detection and re-identification are essentially two different tasks. The detection task pays more attention to the positioning of the target in the image and the category of the target, and pays more attention to global features and deeper semantic information. The re-identification task does not require object positioning, and mainly distinguishes different objects of the same category based on the appearance features of the target. This process pays more attention to the fine-grained features of the object. Therefore, there is a conflict in the optimization direction of the target detection task and the re-identification task during the training process. Directly using the features extracted by the same backbone network for target detection and re-identification will reduce the overall tracking accuracy. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of low tracking accuracy caused by conflict in optimization directions of detection task and re-identification task during training in the existing joint detection and re-identification multi-target tracking methods, and propose a joint detection and re-identification multi-target tracking method based on feature decoupling.

[0006] A joint detection and re-identification multi-target tracking method based on feature decoupling includes the following steps:

[0007] Step 1: obtain the i-th frame image of the video sequence to be tracked, and crop the image to a preset size to obtain the i-th frame target image to be detected;

[0008] Step 2: Input the target image of the i-th frame to be detected into the multi-target tracking model, and obtain the center point coordinates of the k'th target detection bounding box in the i-th frame target image to be detected. Width and height of the object detection bounding box The offset of the center point coordinates of the object detection bounding box and the target identity vector L i (k');

[0009] Where, k'∈[1,K'], K' is the total number of targets in the target image of the i-th frame;

[0010] Step 3: Use the data obtained in step 2 Get the position of the k'th target detection bounding box in the i-th target image to be detected:

[0011]

[0012] Step 4: Determine whether i is equal to 1. If i=1, initialize all targets in the target image of the i-th frame to be detected as new trajectories, and use the target identity vector of the target as the feature of the corresponding trajectory, then set i=i+1 and return to step 1; if i≠1, execute step 5;

[0013] Step 5: Use the Kalman filter algorithm and the target detection bounding box position in the i-1th frame target image to obtain the target prediction bounding box position in the i-th frame target image to be detected, and use the target prediction bounding box position in the i-th frame target image to be detected, the target detection bounding box position in the i-th frame target image to be detected, the target identity vector of each target in the i-th frame target image to be detected, and the target identity vectors representing different trajectories in the i-1 frame to obtain the similarity matrix D of the target in the i-th frame target image to be detected and the existing trajectory in the i-1th frame target image;

[0014] Step 6. Use D as the matching cost matrix, and use the Hungarian algorithm and the matching cost matrix to perform the first match between the target detection bounding box in the target image of the i-th frame to be detected and the existing track in the target image of the i-1th frame. Then, use the IOU overlap threshold to perform a second match between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1th frame. Finally, set the unmatched track to an unmatched mark. If the existing track is not matched for z consecutive frames, it is considered that the target corresponding to the track disappears, and the existing track is deleted.

[0015] Wherein, z is a positive integer;

[0016] The IOU overlap threshold is used to perform a second match on the target detection bounding box in the target image of the i-th frame to be detected that is not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame, specifically:

[0017] Get the IOU value C of the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first matching and the target prediction bounding box corresponding to each unmatched track in the target image of the i-1-th frame. If C is greater than the IOU overlap threshold, it means the match is successful. If C is less than the IOU overlap threshold, it means the match is unsuccessful.

[0018] Step 7: Determine whether i is equal to I. If i=I, output the successfully matched trajectory in step 6. If i<I, set i=i+1 and return to step 1.

[0019] Where I is the total number of frames in the video sequence to be tracked.

[0020] Furthermore, the similarity matrix D of the target in the i-th frame target image to be detected and the existing trajectory in the i-th frame target image is obtained by using the target prediction bounding box position in the i-th frame target image to be detected, the target detection bounding box position in the i-th frame target image to be detected, the target identity vector of each target in the i-th frame target image to be detected, and the target identity vector representing different trajectories in the i-1 frame, which is specifically:

[0021] D=λD r +(1-λ)D m

[0022] Among them, D r is a matrix composed of the cosine distances between the identity vectors of different targets in the target image of the i-th frame to be detected and the target identity vectors representing different trajectories in the target image of the i-1th frame, λ is a weight parameter, and D m It is a matrix composed of the Mahalanobis distances between different target prediction bounding box positions in the i-th target image to be detected and different target detection bounding box positions in the i-th target image.

[0023] Furthermore, the multi-target tracking model in step 2 is obtained by:

[0024] S1. Preprocess the MOT17 dataset to obtain the preprocessed MOT17 dataset, specifically:

[0025] S101, preprocess the images in the MOT17 dataset:

[0026] Each frame of the target image in the MOT17 dataset is cropped to a uniform size, and then the target image is subjected to data enhancement processing to obtain the preprocessed target image.

[0027] The data enhancement processing includes: random rotation, scaling and flipping operations;

[0028] The MOT17 data set includes a training set and a test set;

[0029] The MOT17 dataset includes: each frame of target image and image label in the tracking target video;

[0030] The image labels in MOT17 include: the identity id of each target in each frame of the target image and the upper left corner coordinates and lower right corner coordinates of each target detection bounding box;

[0031] S102, preprocess the image labels in the MOT17 dataset to obtain the preprocessed image labels:

[0032] First, use the coordinates of the upper left corner and lower right corner of each target detection bounding box to obtain the coordinates of the center point of the target detection bounding box

[0033] in, is the coordinate of the upper left corner of the kth target detection bounding box in the tth frame target image, is the coordinate of the lower right corner of the kth target detection bounding box in the tth frame target image;

[0034] Then, get the width and height of the target detection bounding box

[0035] in, is the width of the kth target detection bounding box in the tth frame target image, is the height of the kth target detection bounding box in the target image of the tth frame;

[0036] Then, the identity id of the kth target in the target image of the tth frame is encoded using a one-hot vector to obtain the target identity vector L t (k);

[0037] Then, get the offset of the center point of the kth target detection bounding box in the tth frame target image on the feature map

[0038] Finally, the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, the target identity vector, and the offset of the center point of the target detection bounding box on the feature map are used as the preprocessed image label;

[0039] S103, using the preprocessed labels and the preprocessed images to form a preprocessed MOT17 data set;

[0040] The preprocessed MOT17 data set includes: a preprocessed training set and a preprocessed test set;

[0041] S2. Use the preprocessed MOT17 dataset to train and test the multi-target tracking network to obtain the multi-target tracking model.

[0042] Furthermore, the multi-target tracking network in S2 includes: a backbone network module, a feature decoupling module, a target detection branch module, and a re-identification branch module;

[0043] The backbone network module is a DLA-34 network, which is used to obtain the feature F of the target image of the tth frame;

[0044] The feature decoupling module includes: a channel decoupling unit and a space decoupling unit;

[0045] The channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id ;

[0046] The spatial decoupling unit is used to c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branch det and the spatially decoupled feature F for the re-identification branch id ;

[0047] The target detection branch module includes: a heat map sub-branch unit, a bounding box size sub-branch unit, and a center point offset sub-branch unit;

[0048] The heat map sub-branch unit includes a seventh convolution sub-unit Conv 5 ;

[0049] The Conv 5 To adjust the feature F det Dimensions, generate the center point coordinates of the target detection bounding box;

[0050] The Conv 5 Including: the 21st convolution layer with 3×3 convolution kernel, the first ReLU activation function layer, and the 22nd convolution layer with 1×1 convolution kernel;

[0051] The center point offset sub-branch unit includes an eighth convolution sub-unit Conv 6 ;

[0052] The eighth convolution subunit Conv 6 To adjust the feature F det Dimensions to generate the offset of the center point of the target detection bounding box;

[0053] The eighth convolution subunit Conv 6 Including: the 23rd convolution layer with 3×3 convolution kernel, the ReLU activation function layer, and the 24th convolution layer with 1×1 convolution kernel;

[0054] The bounding box size sub-branch unit includes a ninth convolution sub-unit Conv 7 ;

[0055] The ninth convolution subunit Conv 7 To adjust the feature F det Dimensions to generate the width and height of the target detection bounding box;

[0056] The ninth convolution subunit Conv 7 It includes: the twenty-fifth convolution layer of 3×3 convolution kernel, the ReLU activation function layer, and the twenty-sixth convolution layer of 1×1 convolution kernel; the re-identification branch module uses the feature F id Get each target identity vector in the target image;

[0057] The re-identification branch module includes an eighth convolution subunit Conv 8 ;

[0058] The Conv 8 To adjust F id dimensionality to generate the target identity vector;

[0059] The tenth convolution subunit Conv 8 It includes: the 27th convolution layer with 3×3 convolution kernel, the ReLU activation function layer, and the 28th convolution layer with 1×1 convolution kernel.

[0060] Furthermore, the channel decoupling unit includes: a maximum pooling layer subunit, an average pooling layer subunit, a first convolution subunit Conv 1 , the second convolution subunit Conv 2 , a first element summing subunit, a second element summing subunit, a first residual linking subunit, and a second residual linking subunit;

[0061] The maximum pooling layer subunit is used to perform a maximum pooling operation on the feature F;

[0062] The average pooling layer subunit is used to perform an average pooling operation on the feature F;

[0063] The first convolution subunit Conv 1 Includes: the first convolution layer, the second convolution layer, and the ReLU activation function;

[0064] The first convolution subunit is used to aggregate the channel information of the feature F detection task after maximum pooling and average pooling;

[0065] The second convolution subunit Conv 2 Including: the third convolution layer, the fourth convolution layer, and the ReLU activation function;

[0066] The second convolutional layer subunit Conv 2 Used to aggregate the channel information of feature F for re-identification tasks after maximum pooling and average pooling;

[0067] The first element summation subunit is used to sum the features output by the first convolutional layer subunit element by element and obtain the detection branch channel attention weight C through the Sigmoid activation function det ;

[0068] The second element summation subunit is used to sum the features output by the second convolutional layer subunit element by element and obtain the re-identification branch channel attention weight C through the Sigmoid activation function. id ;

[0069] The first residual link subunit is used to convert C det Multiply it with feature F and add it to get feature F of the detection branch c_det ;

[0070] The second residual link subunit is used to connect C id Multiply it with feature F and add it to get feature F of the re-identification branch c_id .

[0071] Furthermore, the channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id , specifically:

[0072] F c_det =F*C det +F

[0073] F c_id =F*C id +F

[0074] C det =σ(Conv 1 (AvgPool(F))+Conv 1 (MaxPool(F)))

[0075] C id =σ(Conv 2 (AvgPool(F))+Conv 2 (MaxPool(F)))

[0076] Among them, C det is the detection branch channel attention weight, C id is the re-identification branch channel attention weight, F c_detis the feature after channel decoupling that will be used to detect the branch, F c_id is the feature after channel decoupling that will be used for the re-identification branch, σ is the Sigmoid activation function, Conv 1 is the first convolutional subunit, Conv 2 It is the second convolution subunit, AvgPool() is the average pooling operation, and MaxPool() is the maximum pooling operation.

[0077] Furthermore, the spatial decoupling unit includes: a third convolution subunit Conv 3 , the fourth convolution subunit Conv 4 , a fifth convolution subunit, a sixth convolution subunit, a third element summation subunit, a fourth element summation subunit, a third residual link subunit, and a fourth residual link subunit;

[0078] The third convolution subunit Conv 3 Includes: the fifth convolution layer with 1×1 convolution kernel, GELU activation function; the input of the third convolution layer subunit is feature F c_det ;

[0079] The fourth convolution subunit Conv 4 Includes: the sixth convolutional layer with a 1×1 convolution kernel, GELU activation function; the input of the fourth convolutional layer subunit is feature F c_id ;

[0080] The fifth convolution subunit includes: a first convolution block Conv2d 0 , the second convolution block Conv2d 1 , the third convolution block Conv2d 2 , the fourth convolution block Conv2d 3 ; The input of the fifth convolution subunit is the output of the third convolution subunit;

[0081] The first convolution block Conv2d 0 It is the seventh convolution layer with a 5×5 convolution kernel; the input of the first convolution block is the output of the third convolution subunit;

[0082] The second convolution block Conv2d 1 It includes: the eighth convolution layer with 1×7 convolution kernel and the ninth convolution layer with 7×1 convolution kernel; the output of the eighth convolution layer is the input of the ninth convolution layer;

[0083] The third convolution block Conv2d 2 It includes: the tenth convolution layer with a 1×11 convolution kernel and the eleventh convolution layer with a 11×1 convolution kernel; the output of the tenth convolution layer is the input of the eleventh convolution layer;

[0084] The fourth convolution block Conv2d3 It includes: the 12th convolution layer with 1×21 convolution kernel and the 13th convolution layer with 21×1 convolution kernel; the output of the 12th convolution layer is the input of the 13th convolution layer;

[0085] The second convolution block Conv2d 1 , the third convolution block Conv2d 2 , the fourth convolution block Conv2d 3 The input of the first convolution block Conv2d 0 Output:

[0086] The sixth convolution subunit includes: a fifth convolution block Conv2d' 0 , the sixth convolution block Conv2d' 1 , the seventh convolution block Conv2d' 2 , the eighth convolution block Conv2d' 3 ; The input of the sixth convolution subunit is the output of the fourth convolution subunit;

[0087] The fifth convolution block Conv2d' 0 It is the fourteenth convolution layer with a 5×5 convolution kernel; the input of the fifth convolution block is the output of the fourth convolution subunit;

[0088] The sixth convolution block Conv2d' 1 Including: the fifteenth convolution layer with a 1×3 convolution kernel and the sixteenth convolution layer with a 3×1 convolution kernel; the output of the fifteenth convolution layer is the input of the sixteenth convolution layer;

[0089] The seventh convolution block Conv2d' 2 It includes: the seventeenth convolution layer with a 1×5 convolution kernel and the eighteenth convolution layer with a 5×1 convolution kernel; the output of the seventeenth convolution layer is the input of the eighteenth convolution layer;

[0090] The eighth convolution block Conv2d' 3 It includes: the 19th convolution layer with 1×9 convolution kernel and the 20th convolution layer with 9×1 convolution kernel; the output of the 19th convolution layer is the input of the 20th convolution layer;

[0091] The sixth convolution block Conv2d' 1 , the seventh convolution block Conv2d' 2 , the eighth convolution block Conv2d' 3 The input is the fifth convolution block Conv2d' 0 Output:

[0092] The third element summation subunit is used to sum the first convolution block Conv2d 0 , the second convolution block Conv2d 1 , the third convolution block Conv2d2 And the fourth convolution block Conv2d 3 The output features of are summed element by element, and the summation result is passed through the Sigmoid activation function to obtain the spatial attention weight S of the spatial decoupling detection branch. det ;

[0093] The fourth element summation subunit is used to perform the fifth convolution block Conv2d' 0 , the sixth convolution block Conv2d' 1 , the seventh convolution block Conv2d' 2 , the eighth convolution block Conv2d' 3 The output features are summed element by element, and the summation result is passed through the Sigmoid activation function to obtain the spatial attention weight S of the spatial decoupled re-identification branch. id ;

[0094] The third residual link subunit is used to convert S det With feature F c_det Multiply and add the characteristic F c_det , obtain the spatially decoupled feature F for detecting branches det ;

[0095] The fourth residual link subunit is used to convert S id With feature F c_id Multiply and add the characteristic F c_id , obtain the feature F for the re-identification branch after spatial decoupling id .

[0096] Furthermore, the spatial decoupling unit is used to c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branch det and the spatially decoupled feature F for the re-identification branch id , specifically:

[0097] F det =F c_det *S det +F c_det

[0098] F id =F c_id *S id +F c_id

[0099]

[0100]

[0101] Among them, F c_detis the feature used to detect branches after channel decoupling, F c_id is the feature for re-identification branch after channel decoupling, S det is the spatial attention weight of the spatially disentangled detection branch, S id is the spatial attention weight of the spatial disentangled re-ID branch, F det is the feature used to detect branches after spatial decoupling, F id It is the feature used for re-identification branch after spatial decoupling, Conv2d 0 () is the fifth convolutional block, l and l' are integers from 1 to 3, Conv2d l () is the second convolution block, the third convolution block or the fourth convolution block; Conv2d' 0 () is the fifth convolutional block, Conv2d' l' () is the sixth convolution block, the seventh convolution block or the eighth convolution block.

[0102] Furthermore, the overall loss function L used in the training of the detection branch module and the re-identification branch module is total , specifically:

[0103]

[0104]

[0105]

[0106]

[0107] Among them, L heat is the heat map sub-branch unit loss function, L id is the re-identification branch module loss function, L box is the loss function of the center point offset sub-branch unit and the bounding box size sub-branch unit, w 1 、w 2 is a learnable parameter, k is the target label in the target image of the tth frame, K is the total number of targets in the target image of the tth frame, M is the total number of target identity categories in the training set, m is the label of the target identity category in the training set, and L k (m) is the kth target identity vector in the target image of the tth frame, p(m) is the target identity category distribution vector predicted by the re-identification branch module, is the true offset of the center point of the kth target detection bounding box of the target image in the tth frame, is the predicted offset of the center point of the kth target detection bounding box of the target image in the tth frame, λ s is a preset constant, is the actual size of the kth target detection bounding box of the target image in the tth frame, is the predicted size of the kth target detection bounding box of the target image in the tth frame, (x', y') is F det Points in the heat map, is the F predicted by the heatmap sub-branch unit det The heat map value corresponding to the point (x', y') in the heat map, M x'y' Yes F det The heat map value corresponding to the point (x', y') in the heat map, α and β are preset hyper parameters.

[0108] Furthermore, the heat map value M x'y' Specifically:

[0109]

[0110] Among them, σ c is the target scale adaptive standard deviation, It is the position of the center point of the detection bounding box of the kth target in the tth frame image on the feature map.

[0111] The beneficial effects of the present invention are:

[0112] The present invention provides a joint detection and re-identification multi-target tracking method based on feature decoupling. The present invention adopts a joint detection and re-identification multi-target tracking framework, combines the neural network in the detection phase and the re-identification network for extracting target appearance features in the tracking phase, can reduce the inference time of the model, and increase the reusability of the model; the present invention decouples the features extracted by the backbone network from the two dimensions of channel and space, avoids the problem of conflicting optimization directions between the detection task and the re-identification task during the training process, and thus improves the accuracy of multi-target tracking. The experimental results of the embodiments of the present invention show that the present invention can effectively improve the performance of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 It is a flow chart of the present invention;

[0114] Figure 2 This is the overall architecture diagram of the present invention;

[0115] Figure 3 This is the architecture diagram of the channel decoupling module;

[0116] Figure 4 This is the architecture diagram of the spatial decoupling module. DETAILED DESCRIPTION

[0117] Specific implementation method 1: Figure 1-Figure 2As shown, this embodiment is a reasoning process of a joint detection and re-identification multi-target tracking method based on feature decoupling, in which the feature channel decoupling and spatial decoupling of the feature F extracted by the backbone network are the key points of the present invention, which will be specifically introduced in specific embodiments three, four, five and six. The specific process of this embodiment is:

[0118] Step 1: Get the i-th frame image of the video sequence to be tracked, and crop the image to a size of 1088×608 to obtain the i-th frame target image to be detected;

[0119] Step 2: Input the target image of the i-th frame to be detected into the multi-target tracking model, and obtain the center point coordinates of the k'th target detection bounding box in the i-th frame target image to be detected through model reasoning. Width and height of the object detection bounding box The offset of the center point coordinates of the object detection bounding box and the target identity vector L i (k');

[0120] Where, k'∈[1,K'], K' is the total number of targets in the target image of the i-th frame;

[0121] in, is the offset of the center point of the k'th target detection bounding box in the i-th frame of the target image to be detected on the x-axis, is the offset of the center point of the k'th target detection bounding box in the i-th target image to be detected on the y-axis;

[0122] Step 3: Use the data obtained in step 2 Get the position of the k'th target detection bounding box in the i-th target image to be detected:

[0123]

[0124] in, is the coordinate of the upper left corner of the k'th target detection bounding box in the i-th target image to be detected, is the coordinate of the lower right corner of the k'th target detection bounding box in the i-th frame of the target image to be detected;

[0125] Step 4: Determine whether i is equal to 1. If i=1, initialize all targets in the target image of the i-th frame to be detected as new trajectories, and use the target identity vector of the target as the feature of the corresponding trajectory, then set i=i+1 and return to step 1; if i≠1, execute step 5;

[0126] Step 5: Use the Kalman filter algorithm and the position of the target detection bounding box in the i-1th frame target image to obtain the position of the target prediction bounding box in the i-th frame target image to be detected, and calculate the Mahalanobis distance D between the target prediction bounding box in the i-th frame target image to be detected and the target detection bounding box in the i-th frame m Then calculate the cosine distance D between the different target identity vectors of the i-th frame to be detected and the different trajectory vectors existing in the i-1th frame r , and D m , D r The similarity matrix D between the target in the i-th frame target image to be detected and the existing trajectory in the i-1-th frame target image is obtained by weighted summation with weight λ (0.98), which is specifically:

[0127] D=λD r +(1-λ)D m

[0128] Among them, D r is a matrix composed of the cosine distances between the identity vectors of different targets in the target image of the i-th frame to be detected and the target identity vectors representing different trajectories in the target image of the i-1-th frame. λ is the weight parameter set to 0.98. D m is a matrix composed of the Mahalanobis distances between the different target prediction bounding box positions in the i-th target image to be detected and the different target detection bounding box positions in the i-th target image; in this step, the cosine distance similarity between the target identity vector of the first target in the i-th target image to be detected and all the target identity vectors representing different trajectories in the i-1-th target image is D r The first row of , and so on, to obtain the matrix D r ; Similarly, we get the matrix D m .

[0129] Step 6. Use the similarity matrix D obtained in step 5 as the matching cost matrix, and use the Hungarian algorithm and the matching cost matrix to perform the first match between the target detection bounding box in the target image of the i-th frame to be detected and the existing track in the target image of the i-1th frame. Then use the IOU overlap threshold (0.5) to perform a second match (IOU matching) between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1th frame. Finally, set the unmatched track to an unmatched mark. If the existing track has not been matched for 30 consecutive frames, it is considered that the target corresponding to the track has disappeared, and the current existing track is deleted.

[0130] The second matching is IOU matching, and the IOU overlap threshold is set to 0.5;

[0131] The IOU matching is specifically as follows: obtaining the IOU value C of the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first matching and the target prediction bounding box corresponding to each unmatched track in the target image of the i-1-th frame. If C is greater than the IOU overlap threshold, it indicates a successful match; if C is less than the IOU overlap threshold, it indicates an unsuccessful match;

[0132] Step 7: Determine whether i is equal to I. If i=I, output the successfully matched trajectory in step 6. If i<I, set i=i+1 and return to step 1.

[0133] Where I is the total number of frames in the video sequence to be tracked.

[0134] Specific implementation method 2: The multi-target tracking model is obtained by the following method:

[0135] S1. Preprocess the MOT17 dataset:

[0136] S101, preprocess the images in the MOT17 dataset:

[0137] Each frame of the target image in the MOT17 dataset is cropped to a uniform size of 1088×608, and then random rotation, scaling, and flipping operations are used to perform data augmentation to obtain the preprocessed target image;

[0138] The MOT17 dataset is an existing dataset, which includes a training set and a test set;

[0139] The MOT17 dataset includes: each frame of target image and image label in the tracking target video;

[0140] The image labels in MOT17 include: the identity id of each target in each frame of the target image and the upper left corner coordinates and lower right corner coordinates of each target detection bounding box;

[0141] S102, preprocess the labels in the MOT17 dataset to obtain preprocessed image labels:

[0142] First, use the coordinates of the upper left corner and lower right corner of each target detection bounding box to obtain the coordinates of the center point of the target detection bounding box

[0143] in, is the coordinate of the upper left corner of the kth target detection bounding box in the tth frame target image, is the coordinate of the lower right corner of the kth target detection bounding box in the tth frame target image;

[0144] Then, get the width and height of the target detection bounding box

[0145] The center point of the target detection bounding box is located on the feature map

[0146] in, is the width of the kth target detection bounding box in the tth frame target image, is the height of the kth target detection bounding box in the target image of the tth frame;

[0147] Then, the identity id of each target in the target image of the tth frame is encoded using a one-hot vector to obtain the target identity vector L t (k);

[0148] Then, get the offset of the center point of the target detection bounding box on the feature map

[0149] in, is the offset of the center point of the k-th target detection bounding box in the t-th frame target image;

[0150] Finally, the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, the target identity vector, and the offset of the center point of the target detection bounding box on the feature map are used as the preprocessed image label;

[0151] S103, using the preprocessed labels and the preprocessed images to form a preprocessed MOT17 data set;

[0152] The preprocessed MOT17 data set includes: a preprocessed training set and a preprocessed test set.

[0153] S2. Use the preprocessed MOT17 dataset to train and test the multi-target tracking network to obtain the multi-target tracking model:

[0154] The multi-target tracking network includes: a backbone network module, a feature decoupling module, a target detection branch module, and a re-identification branch module;

[0155] The backbone network module is a DLA-34 network, which is used to fully extract the high-level and low-level fusion information of the target to obtain the feature F of the target image of the tth frame;

[0156] The dimensions of F are 272×152×64;

[0157] The feature decoupling module includes: a channel decoupling unit and a space decoupling unit;

[0158] The channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the channel decoupled feature F to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id ,like Figure 3 As shown;

[0159] The spatial decoupling unit is used to generate the feature F generated by the channel decoupling unit. c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branch det and the feature F used for the re-identification branch after spatial decoupling id ,like Figure 4 As shown;

[0160] The target detection branch module includes a heat map sub-branch unit, a bounding box size sub-branch unit, and a center point offset sub-branch unit; the target detection branch module uses the feature F used for the detection branch after spatial decoupling det Get the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, and the offset of the center point of the target detection bounding box;

[0161] The re-identification branch module uses the feature F id Get the identity vector for each object in the target image.

[0162] Specific implementation method three: Figure 3 As shown, the channel decoupling unit includes: a maximum pooling layer subunit, an average pooling layer subunit, a first convolution subunit Conv 1 , the second convolution subunit Conv 2 , a first element summing subunit, a second element summing subunit, a first residual linking subunit, and a second residual linking subunit;

[0163] The inputs of the maximum pooling layer subunit and the average pooling layer subunit are both feature F, and the output of the maximum pooling layer subunit is the first convolution subunit Conv 1 And the second convolution subunit Conv 2 The input of the average pooling layer subunit is the output of the first convolution subunit Conv 1 And the second convolution subunit Conv 2 Input, the first convolution subunit Conv 1 The output of is the input of the first element summation subunit, and the second convolution subunit Conv 2 The output of is the input of the second element summation subunit, and the first element summation subunit outputs the detection branch channel attention weight C after the Sigmoid activation function det , the first element summation subunit outputs the re-identification branch channel attention weight C through the Sigmoid activation function id , the input of the first residual link subunit is the detection branch channel attention weight C detand feature F, the output is the feature F after channel decoupling that will be used to detect the branch c_det ; The input of the second residual link subunit is the re-identification branch channel attention weight C id and feature F, the output is the channel decoupled feature F that will be used for the re-identification branch c_id ;

[0164] The maximum pooling layer subunit is used to perform a maximum pooling operation on the feature F;

[0165] The average pooling layer subunit is used to perform an average pooling operation on the feature F;

[0166] The first convolution subunit Conv 1 Includes: the first convolution layer, the second convolution layer, and the ReLU activation function;

[0167] The first convolution subunit is used to aggregate the channel information of the feature F detection task after maximum pooling and average pooling;

[0168] The second convolution subunit Conv 2 Including: the third convolution layer, the fourth convolution layer, and the ReLU activation function;

[0169] The second convolutional layer subunit Conv 2 Used to aggregate the channel information of feature F for re-identification tasks after maximum pooling and average pooling;

[0170] The first element summation subunit is used to sum the features output by the first convolutional layer subunit element by element and obtain the detection branch channel attention weight C through the Sigmoid activation function det ;

[0171] The second element summation subunit is used to sum the features output by the second convolutional layer subunit element by element and obtain the re-identification branch channel attention weight C through the Sigmoid activation function. id ;

[0172] The first residual link subunit is used to convert C det Multiply it with feature F and add it to get feature F of the detection branch c_det ;

[0173] The second residual link subunit is used to connect C id Multiply it with feature F and add it to get feature F of the re-identification branch c_id .

[0174] Specific implementation method 4: The channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the channel decoupled feature F to be used for detecting the branch c_detThe feature F that will be used for the re-identification branch after channel decoupling c_id , specifically:

[0175] F c_det =F*C det +F

[0176] F c_id =F*C id +F

[0177] C det =σ(Conv 1 (AvgPool(F))+Conv 1 (MaxPool(F)))

[0178] C id =σ(Conv 2 (AvgPool(F))+Conv 2 (MaxPool(F)))

[0179] Among them, F is the feature extracted by the backbone network, C det is the detection branch channel attention weight, C id is the re-identification branch channel attention weight, F c_det is the feature after channel decoupling that will be used to detect the branch, F c_id is the feature after channel decoupling that will be used for the re-identification branch, σ is the Sigmoid activation function, Conv 1 is the first convolutional subunit, Conv 2 It is the second convolution subunit, AvgPool() is the average pooling operation, and MaxPool() is the maximum pooling operation.

[0180] The channel decoupling unit in this embodiment is to alleviate the problem of misalignment in the channel dimension when extracting features for detection tasks and re-identification tasks using the same backbone network. For feature F, first use maximum pooling and average pooling to operate the aggregated channel information to enhance the representation ability of the model, and then send the aggregated channel information to Conv 1 and Conv 2 Then, we sum each element to get the channel attention weights C of the detection branch and the re-identification branch. det and C id Finally, use C det and C id As a weight, it is fused with feature F through residual connection to obtain feature F of the detection branch after channel decoupling. c_det and the feature F of the re-identification branch after channel decoupling c_id ,After the above operations, the feature F extracted by the backbone network is decoupled in the channel dimension.

[0181] Specific implementation method five: Figure 4 As shown, the spatial decoupling unit includes: a third convolution subunit Conv 3 , the fourth convolution subunit Conv 4 , a fifth convolution subunit, a sixth convolution subunit, a third element summation subunit, a fourth element summation subunit, a third residual link subunit, and a fourth residual link subunit;

[0182] The third convolution subunit Conv 3 It includes: the fifth convolution layer with 1×1 convolution kernel and GELU activation function; the input of the third convolution layer subunit is the feature F used to detect the branch after channel decoupling c_det ;

[0183] The fourth convolution subunit Conv 4 It includes: the sixth convolutional layer with 1×1 convolution kernel and GELU activation function; the input of the fourth convolutional layer subunit is the feature F used for the re-identification branch after channel decoupling c_id ;

[0184] The fifth convolution subunit includes: a first convolution block Conv2d 0 , the second convolution block Conv2d 1 , the third convolution block Conv2d 2 , the fourth convolution block Conv2d 3 ; The input of the fifth convolution subunit is the output of the third convolution subunit;

[0185] The first convolution block Conv2d 0 It is the seventh convolution layer with a 5×5 convolution kernel; the input of the first convolution block is the output of the third convolution subunit;

[0186] The second convolution block Conv2d 1 It includes: the eighth convolution layer with 1×7 convolution kernel and the ninth convolution layer with 7×1 convolution kernel; the output of the eighth convolution layer is the input of the ninth convolution layer;

[0187] The third convolution block Conv2d 2 It includes: the tenth convolution layer with a 1×11 convolution kernel and the eleventh convolution layer with a 11×1 convolution kernel; the output of the tenth convolution layer is the input of the eleventh convolution layer;

[0188] The fourth convolution block Conv2d 3 It includes: the 12th convolution layer with 1×21 convolution kernel and the 13th convolution layer with 21×1 convolution kernel; the output of the 12th convolution layer is the input of the 13th convolution layer;

[0189] The second convolution block Conv2d 1, the third convolution block Conv2d 2 , the fourth convolution block Conv2d 3 The input of the first convolution block Conv2d 0 Output:

[0190] The sixth convolution subunit includes: a fifth convolution block Conv2d' 0 , the sixth convolution block Conv2d' 1 , the seventh convolution block Conv2d' 2 , the eighth convolution block Conv2d' 3 ; The input of the sixth convolution subunit is the output of the fourth convolution subunit;

[0191] The fifth convolution block Conv2d' 0 It is the fourteenth convolution layer with a 5×5 convolution kernel; the input of the fifth convolution block is the output of the fourth convolution subunit;

[0192] The sixth convolution block Conv2d' 1 Including: the fifteenth convolution layer with a 1×3 convolution kernel and the sixteenth convolution layer with a 3×1 convolution kernel; the output of the fifteenth convolution layer is the input of the sixteenth convolution layer;

[0193] The seventh convolution block Conv2d' 2 It includes: the seventeenth convolution layer with a 1×5 convolution kernel and the eighteenth convolution layer with a 5×1 convolution kernel; the output of the seventeenth convolution layer is the input of the eighteenth convolution layer;

[0194] The eighth convolution block Conv2d' 3 It includes: the 19th convolution layer with 1×9 convolution kernel and the 20th convolution layer with 9×1 convolution kernel; the output of the 19th convolution layer is the input of the 20th convolution layer;

[0195] The sixth convolution block Conv2d' 1 , the seventh convolution block Conv2d' 2 , the eighth convolution block Conv2d' 3 The input is the fifth convolution block Conv2d' 0 Output:

[0196] The third element summation subunit is used to sum the first convolution block Conv2d 0 , the second convolution block Conv2d 1 , the third convolution block Conv2d 2 And the fourth convolutional block Conv2d 3 The output features of are summed element by element, and the summation result is passed through the Sigmoid activation function to obtain the spatial attention weight S of the spatial decoupling detection branch. det , S detas the input of the third residual link subunit;

[0197] The fourth element summation subunit is used to perform the fifth convolution block Conv2d' 0 , the sixth convolution block Conv2d' 1 , the seventh convolution block Conv2d' 2 , the eighth convolution block Conv2d' 3 The output features are summed element by element, and the summation result is passed through the Sigmoid activation function to obtain the spatial attention weight S of the spatial decoupled re-identification branch. id , S id as the input of the fourth residual link subunit;

[0198] The third residual link subunit is used to convert S det With feature F c_det Multiply and add the characteristic F c_det , obtain the feature F used to detect the branch after spatial decoupling det ;

[0199] The fourth residual link subunit is used to convert S id With feature F c_id Multiply and add the characteristic F c_id , obtain the feature F for the re-identification branch after spatial decoupling id .

[0200] Specific implementation method 6: The spatial decoupling unit is used to generate the feature F generated by the channel decoupling unit c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the feature F used to detect the branch after the spatial decoupling unit is decoupled det The feature F used for the re-identification branch after decoupling from the spatial decoupling unit id , specifically:

[0201] F det =F c_det *S det +F c_det

[0202] F id =F c_id *S id +F c_id

[0203]

[0204]

[0205] Among them, F c_det is the feature used to detect branches after channel decoupling, F c_idis the feature for re-identification branch after channel decoupling, S det is the spatial attention weight of the spatially disentangled detection branch, S id is the spatial attention weight of the spatial disentangled re-ID branch, F det is the feature used to detect branches after spatial decoupling, F id It is the feature used for re-identification branch after spatial decoupling, Conv2d 0 () is the fifth convolutional block, l and l' are integers from 1 to 3, Conv2d l () is the second convolution block, the third convolution block or the fourth convolution block; Conv2d' 0 () is the fifth convolutional block, Conv2d' l' () is the sixth convolution block, the seventh convolution block or the eighth convolution block.

[0206] In this embodiment, for the feature F after channel decoupling, c_det and F c_id , by aggregating the regional features in the feature map, we generate the regional weight matrices suitable for the detection branch and the re-identification branch respectively. First, we use Conv 3 and Conv 4 To adjust F c_det and F c_id , and then use multi-scale asymmetric convolution kernels to enhance the module's ability to extract features of targets of different scales. Next, the features extracted by convolution kernels of different scales are summed element by element, and the sigmoid activation function is used to obtain the spatial attention weight S of the detection branch. det and the spatial attention weight S of the re-ID branch id Finally, S det and S id As the spatial attention weights of the detection branch and the re-identification branch, the residual connection is connected with the feature F c_det and F c_id After the above operations, the information of the two branches is decoupled in space.

[0207] Specific implementation method seven: The target detection branch module includes a heat map sub-branch unit, a center point offset sub-branch unit and a bounding box size sub-branch unit, specifically:

[0208] The heat map sub-branch sub-unit includes a seventh convolution sub-unit Conv 5 ;

[0209] The Conv 5 Used to adjust the decoupled detection branch feature F det Dimensions to generate the center point coordinates of the target detection bounding box;

[0210] The Conv5 Including: the 21st convolution layer with 3×3 convolution kernel, the first ReLU activation function layer, and the 22nd convolution layer with 1×1 convolution kernel;

[0211] The loss function L of the heat map sub-branch sub-unit heat as follows:

[0212]

[0213]

[0214] Among them, L heat is the loss function of the heat map sub-branch unit, K is the total number of targets in the target image of the t-th frame, k is the target label in the target image of the t-th frame, M x'y' Yes F det The heat map value corresponding to the point (x', y') in the heat map, is the F of the heat map branch prediction det The heat map value corresponding to the point (x', y') in the heat map, α and β are preset hyper parameters. In the present invention, α=2, β=4, σ c is the target scale adaptive standard deviation, is the position of the center point of the detection bounding box of the kth target in the tth frame image on the feature map;

[0215] The center point offset sub-branch unit includes the eighth convolution sub-unit Conv 6 ;

[0216] The eighth convolution subunit Conv 6 It is used to adjust the feature F that will be used to detect the branch after the spatial decoupling unit is decoupled. det Dimensions to generate the offset of the center point of the target detection bounding box;

[0217] The eighth convolution subunit Conv 6 Including: the 23rd convolution layer with 3×3 convolution kernel, the ReLU activation function layer, and the 24th convolution layer with 1×1 convolution kernel;

[0218] The bounding box size sub-branch unit includes a ninth convolution sub-unit Conv 7 ;

[0219] The ninth convolution subunit Conv 7 It is used to adjust the feature F that will be used to detect the branch after the spatial decoupling unit is decoupled. det Dimensions to generate the width and height of the target detection bounding box;

[0220] The ninth convolution subunit Conv 7Including: the 25th convolution layer with 3×3 convolution kernel, the ReLU activation function layer, and the 26th convolution layer with 1×1 convolution kernel;

[0221] The center point offset sub-branch unit and the bounding box size sub-branch unit loss function L box as follows:

[0222]

[0223] Among them, K is the total number of targets in the target image of the tth frame, is the true offset of the center point of the kth target detection bounding box of the target image in the tth frame, is the predicted offset of the center point of the k-th target detection bounding box of the target image in the t-th frame, is the actual size of the kth target detection bounding box of the target image in the tth frame, is the predicted size of the kth target detection bounding box of the target image in the tth frame, λ s The preset constant is 0.1;

[0224] The re-identification branch module includes a tenth convolution subunit Conv 8 ;

[0225] The re-identification branch is used to extract the identity ID of each target one-hot vector encoding;

[0226] The Conv 8 Used to adjust the decoupled re-identification branch feature F id dimensionality to generate the target identity vector;

[0227] The tenth convolution subunit Conv 8 Including: the 27th convolution layer with 3×3 convolution kernel, the ReLU activation function layer, and the 28th convolution layer with 1×1 convolution kernel;

[0228] The re-identification branch module loss function L id Obtained through:

[0229]

[0230] Where K is the total number of targets in the target image of the tth frame, M is the total number of target identities in the entire training set, p(m) is the target identity category distribution vector predicted by the re-identification branch module, and L k (m) is the identity vector label of the kth target in the target image of the tth frame after being encoded using a one-hot vector;

[0231] The overall loss function L of the detection branch module and the re-identification branch module total Obtained through:

[0232]

[0233] Among them, w 1 and w 2 are two learnable parameters.

[0234] Embodiment: In order to verify the beneficial effects of the present invention, the following experiments were carried out in this embodiment:

[0235] This example conducts a series of ablation experiments on the MOT17 dataset, including experiments based on channel decoupling and spatial decoupling:

[0236] The evaluation indicators used in the embodiment include two types: multiple objection tracking accuracy (MOTA) and target identity precision recall F1 score IDF1. The former focuses on measuring the performance of the multiple objection tracking detection stage; the latter focuses on measuring the performance of the multiple objection tracking matching stage. Their calculation formulas are as follows:

[0237]

[0238]

[0239] Among them, FN is the number of real targets that are not detected correctly, that is, the number of missed detections; FP is the number of wrong detections, that is, the number of false alarms; IDSW is the number of target identity switches during tracking; GT is the number of real frames. IDTP, IDFP and IDFN are the correct matches, false alarms and missed detections of the target ID respectively.

[0240] Since the MOT challenge official website did not publish the identity labels of the MOT17 test dataset, the MOT17 training set was evenly divided into two parts, the first part was used for training, and the second part was used for testing. This design is intended to ensure that the experimental results are fair and verifiable. The final results are shown in Table 1.

[0241] Table 1 Comparison of ablation experiments based on channel and spatial decoupling

[0242]

[0243] According to the first and second rows of Table 1, we can see that the baseline algorithm FairMOT has higher FP and IDSW. According to the first and second rows of Table 1, after adding the channel dimension decoupling unit to the baseline algorithm, MOTA increased by 0.3 (67.4-67.7), IDF1 increased by 0.7 (71.3-72.0), and IDSW decreased by about 10.3% (438-393); According to the first and third rows of Table 1, after adding the spatial dimension decoupling unit to the baseline algorithm, MOTA increased by 0.4 (67.4-67.8), and IDF1 increased by 0.4 (71.3-71.7); According to the first and fourth rows of Table 1, after adding the channel and spatial decoupling dimension units to the baseline algorithm, MOTA increased by 0.6 (67.4-68.0), IDF1 increased by 1.5 (71.3-72.8), and IDSW decreased by about 15.5% (438-370). Therefore, it is concluded that adding channel and spatial dimension decoupling units to the baseline algorithm can alleviate the problem of feature misalignment in the channel and spatial dimensions of detection and re-identification tasks respectively, and effectively improve the performance of the joint detection and re-identification multi-target tracking algorithm.

[0244] In addition, we also trained on multiple datasets such as CrowdHuman, Caltech, CUHK-SYSU, PRW, and MOT17. We then used the weights of the training results to predict the tracking results of MOT17. Finally, we submitted the prediction results to the MOTchallenge website and compared them with the results of the commonly used multi-target tracking algorithms on MOT17. The final results are shown in Table 2.

[0245] Table 2 Comparison of evaluation results of different methods on MOT17

[0246]

[0247]

[0248] The above ablation experiments and comparison results with various methods show that the decoupling method proposed in the present invention can effectively alleviate the competition between the detection and re-identification tasks of the joint detection and re-identification algorithm during the training process, and improve the performance of multi-target tracking; compared with other multi-target tracking algorithms, the multi-target tracking accuracy and IDF1 of the present invention have certain advantages.

Claims

1. A joint detection and re-identification multi-target tracking method based on feature decoupling, characterized in that The specific process of the method is: Step 1: obtain the i-th frame image of the video sequence to be tracked, and crop the image to a preset size to obtain the i-th frame target image to be detected; Step 2: Input the target image of the i-th frame to be detected into the multi-target tracking model, and obtain the center point coordinates of the k'th target detection bounding box in the i-th frame target image to be detected. Width and height of the object detection bounding box The offset of the center point coordinates of the object detection bounding box and the target identity vector L i (k'); Where, k'∈[1,K'], K' is the total number of targets in the target image of the i-th frame; Step 3: Use the data obtained in step 2 Get the position of the k'th target detection bounding box in the i-th target image to be detected: Step 4: Determine whether i is equal to 1. If i=1, initialize all targets in the target image of the i-th frame to be detected as new trajectories, and use the target identity vector of the target as the feature of the corresponding trajectory, then set i=i+1 and return to step 1; if i≠1, execute step 5; Step 5: Use the Kalman filter algorithm and the target detection bounding box position in the i-1th frame target image to obtain the target prediction bounding box position in the i-th frame target image to be detected, and use the target prediction bounding box position in the i-th frame target image to be detected, the target detection bounding box position in the i-th frame target image to be detected, the target identity vector of each target in the i-th frame target image to be detected, and the target identity vectors representing different trajectories in the i-1 frame to obtain the similarity matrix D of the target in the i-th frame target image to be detected and the existing trajectory in the i-1th frame target image; Step 6. Use D as the matching cost matrix, and use the Hungarian algorithm and the matching cost matrix to perform the first match between the target detection bounding box in the target image of the i-th frame to be detected and the existing track in the target image of the i-1th frame. Then, use the IOU overlap threshold to perform a second match between the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1th frame. Finally, set the unmatched track to an unmatched mark. If the existing track is not matched for z consecutive frames, it is considered that the target corresponding to the track disappears, and the existing track is deleted. Wherein, z is a positive integer; The IOU overlap threshold is used to perform a second match on the target detection bounding box in the target image of the i-th frame to be detected that is not matched in the first match and the target prediction bounding box corresponding to each unmatched track in the i-1-th frame, specifically: Get the IOU value C of the target detection bounding box in the target image of the i-th frame to be detected that was not matched in the first matching and the target prediction bounding box corresponding to each unmatched track in the target image of the i-1-th frame. If C is greater than the IOU overlap threshold, it means the match is successful. If C is less than the IOU overlap threshold, it means the match is unsuccessful. Step 7: Determine whether i is equal to I. If i=I, output the successfully matched trajectory in step 6; if i<I, set i=i+1 and return to step 1; Where I is the total number of frames in the video sequence to be tracked.

2. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 1, characterized in that: In the step 5, the target prediction bounding box position in the target image of the i-th frame to be detected, the target detection bounding box position in the target image of the i-th frame to be detected, the target identity vector of each target in the target image of the i-th frame to be detected, and the target identity vectors representing different trajectories existing in the i-1 frame are used to obtain the similarity matrix D of the target in the target image of the i-th frame to be detected and the existing trajectory in the target image of the i-1 frame, which is specifically: D=λD r +(1-λ)D m Among them, D r is a matrix composed of the cosine distances between the identity vectors of different targets in the target image of the i-th frame to be detected and the target identity vectors representing different trajectories in the target image of the i-1th frame, λ is a weight parameter, and D m It is a matrix composed of the Mahalanobis distances between different target prediction bounding box positions in the i-th target image to be detected and different target detection bounding box positions in the i-th target image.

3. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 2, characterized in that: The multi-target tracking model in step 2 is obtained by: S1. Preprocess the MOT17 dataset to obtain the preprocessed MOT17 dataset, specifically: S101, preprocess the images in the MOT17 dataset: Each frame of the target image in the MOT17 dataset is cropped to a uniform size, and then the target image is subjected to data enhancement processing to obtain the preprocessed target image; The data enhancement processing includes: random rotation, scaling and flipping operations; The MOT17 data set includes a training set and a test set; The MOT17 dataset includes: each frame of target image and image label in the tracking target video; The image labels in MOT17 include: the identity id of each target in each frame of the target image and the upper left corner coordinates and lower right corner coordinates of each target detection bounding box; S102, preprocess the image labels in the MOT17 dataset to obtain the preprocessed image labels: First, use the coordinates of the upper left corner and lower right corner of each target detection bounding box to obtain the coordinates of the center point of the target detection bounding box in, is the coordinate of the upper left corner of the kth target detection bounding box in the tth frame target image, is the coordinate of the lower right corner of the kth target detection bounding box in the tth frame target image; Then, get the width and height of the target detection bounding box in, is the width of the kth target detection bounding box in the tth frame target image, is the height of the kth target detection bounding box in the target image of the tth frame; Then, the identity id of the kth target in the target image of the tth frame is encoded using a one-hot vector to obtain the target identity vector L t (k); Then, get the offset of the center point of the kth target detection bounding box in the tth frame target image on the feature map Finally, the coordinates of the center point of the target detection bounding box, the width and height of the target detection bounding box, the target identity vector, and the offset of the center point of the target detection bounding box on the feature map are used as the preprocessed image label; S103, using the preprocessed labels and the preprocessed images to form a preprocessed MOT17 data set; The preprocessed MOT17 data set includes: a preprocessed training set and a preprocessed test set; S2. Use the preprocessed MOT17 dataset to train and test the multi-target tracking network to obtain the multi-target tracking model.

4. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 3, characterized in that: The multi-target tracking network in S2 includes: a backbone network module, a feature decoupling module, a target detection branch module, and a re-identification branch module; The backbone network module is a DLA-34 network, which is used to obtain the feature F of the target image of the tth frame; The feature decoupling module includes: a channel decoupling unit and a space decoupling unit; The channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id ; The spatial decoupling unit is used to respectively c_det and F c_id Decouple in the spatial dimension to obtain the spatially decoupled feature F that will be used to detect the branch det and the spatially decoupled feature F for the re-identification branch id ; The target detection branch module includes: a heat map sub-branch unit, a bounding box size sub-branch unit, and a center point offset sub-branch unit; The heat map sub-branch unit includes a seventh convolution sub-unit Conv5; The Conv5 is used to adjust the feature F det Dimensions, generate the center point coordinates of the target detection bounding box; The Conv5 includes: a twenty-first convolution layer with a 3×3 convolution kernel, a first ReLU activation function layer, and a twenty-second convolution layer with a 1×1 convolution kernel; The center point offset sub-branch unit includes an eighth convolution sub-unit Conv6; The eighth convolution subunit Conv6 is used to adjust the feature F det Dimensions to generate the offset of the center point of the target detection bounding box; The eighth convolution subunit Conv6 includes: a twenty-third convolution layer of a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-fourth convolution layer of a 1×1 convolution kernel; The bounding box size sub-branch unit includes a ninth convolution sub-unit Conv7; The ninth convolution subunit Conv7 is used to adjust the feature F det Dimensions to generate the width and height of the target detection bounding box; The ninth convolution subunit Conv7 includes: a twenty-fifth convolution layer of a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-sixth convolution layer of a 1×1 convolution kernel; The re-identification branch module uses the feature F id Get each target identity vector in the target image; The re-identification branch module includes an eighth convolution subunit Conv8; The Conv8 is used to adjust F id dimensionality to generate the target identity vector; The tenth convolution subunit Conv8 includes: a twenty-seventh convolution layer with a 3×3 convolution kernel, a ReLU activation function layer, and a twenty-eighth convolution layer with a 1×1 convolution kernel.

5. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 4, characterized in that: The channel decoupling unit includes: a maximum pooling layer subunit, an average pooling layer subunit, a first convolution subunit Conv1, a second convolution subunit Conv2, a first element summation subunit, a second element summation subunit, a first residual link subunit, and a second residual link subunit; The maximum pooling layer subunit is used to perform a maximum pooling operation on the feature F; The average pooling layer subunit is used to perform an average pooling operation on the feature F; The first convolution subunit Conv1 includes: a first convolution layer, a second convolution layer, and a ReLU activation function; The first convolution subunit Conv1 is used to aggregate the channel information of the feature F after maximum pooling and average pooling for the detection task; The second convolution subunit Conv2 includes: a third convolution layer, a fourth convolution layer, and a ReLU activation function; The second convolutional layer subunit Conv2 is used to aggregate the channel information of the feature F after maximum pooling and average pooling for the re-identification task; The first element summation subunit is used to sum the features output by the first convolutional layer subunit element by element and obtain the detection branch channel attention weight C through the Sigmoid activation function. det ; The second element summation subunit is used to sum the features output by the second convolutional layer subunit element by element and obtain the re-identification branch channel attention weight C through the Sigmoid activation function. id ; The first residual link subunit is used to convert C det Multiply it with feature F and add it to get feature F of the detection branch c_det ; The second residual link subunit is used to convert C id Multiply it with feature F and add it to get feature F of the re-identification branch c_id .

6. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 5, characterized in that: The channel decoupling unit is used to decouple the feature F in the channel dimension to obtain the feature F after channel decoupling to be used for detecting the branch c_det The feature F that will be used for the re-identification branch after channel decoupling c_id , specifically: F c_det =F*C det +F F c_id =F*C id +F C det =σ(Conv1(AvgPool(F))+Conv1(MaxPool(F))) C id =σ(Conv2(AvgPool(F))+Conv2(MaxPool(F))) Among them, C det is the detection branch channel attention weight, C id is the re-identification branch channel attention weight, F c_det is the feature after channel decoupling that will be used to detect the branch, F c_id is the feature after channel decoupling that will be used for the re-identification branch, σ is the Sigmoid activation function, Conv1 is the first convolution subunit, Conv2 is the second convolution subunit, AvgPool() is the average pooling operation, and MaxPool() is the maximum pooling operation.

7. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 6, characterized in that: The spatial decoupling unit includes: a third convolution subunit Conv3, a fourth convolution subunit Conv4, a fifth convolution subunit, a sixth convolution subunit, a third element summation subunit, a fourth element summation subunit, a third residual link subunit, and a fourth residual link subunit; The third convolution subunit Conv3 includes: a fifth convolution layer of a 1×1 convolution kernel, a GELU activation function; the input of the third convolution layer subunit is the feature F c_det ; The fourth convolution subunit Conv4 includes: a sixth convolution layer of a 1×1 convolution kernel, a GELU activation function; the input of the fourth convolution layer subunit is the feature F c_id ; The fifth convolution subunit includes: a first convolution block Conv2d0, a second convolution block Conv2d1, a third convolution block Conv2d2, and a fourth convolution block Conv2d3; the input of the fifth convolution subunit is the output of the third convolution subunit; The first convolution block Conv2d0 is the seventh convolution layer of a 5×5 convolution kernel; the input of the first convolution block is the output of the third convolution subunit; The second convolution block Conv2d1 includes: an eighth convolution layer with a 1×7 convolution kernel and a ninth convolution layer with a 7×1 convolution kernel; the output of the eighth convolution layer is the input of the ninth convolution layer; The third convolution block Conv2d2 includes: a tenth convolution layer of a 1×11 convolution kernel and an eleventh convolution layer of a 11×1 convolution kernel; the output of the tenth convolution layer is the input of the eleventh convolution layer; The fourth convolution block Conv2d3 includes: a twelfth convolution layer with a 1×21 convolution kernel and a thirteenth convolution layer with a 21×1 convolution kernel; the output of the twelfth convolution layer is the input of the thirteenth convolution layer; The inputs of the second convolution block Conv2d1, the third convolution block Conv2d2, and the fourth convolution block Conv2d3 are all the outputs of the first convolution block Conv2d0; The sixth convolution subunit includes: a fifth convolution block Conv2d'0, a sixth convolution block Conv2d'1, a seventh convolution block Conv2d'2, and an eighth convolution block Conv2d'3; the input of the sixth convolution subunit is the output of the fourth convolution subunit; The fifth convolution block Conv2d'0 is the fourteenth convolution layer of the 5×5 convolution kernel; the input of the fifth convolution block is the output of the fourth convolution subunit; The sixth convolution block Conv2d'1 includes: a fifteenth convolution layer with a 1×3 convolution kernel and a sixteenth convolution layer with a 3×1 convolution kernel; the output of the fifteenth convolution layer is the input of the sixteenth convolution layer; The seventh convolution block Conv2d'2 includes: a seventeenth convolution layer with a 1×5 convolution kernel and an eighteenth convolution layer with a 5×1 convolution kernel; the output of the seventeenth convolution layer is the input of the eighteenth convolution layer; The eighth convolution block Conv2d'3 includes: a nineteenth convolution layer of a 1×9 convolution kernel and a twentieth convolution layer of a 9×1 convolution kernel; the output of the nineteenth convolution layer is the input of the twentieth convolution layer; The inputs of the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 are all the outputs of the fifth convolution block Conv2d'0; The third element summation subunit is used to sum the output features of the first convolution block Conv2d0, the second convolution block Conv2d1, the third convolution block Conv2d2 and the fourth convolution block Conv2d3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupling detection branch. det ; The fourth element summation subunit is used to sum the output features of the fifth convolution block Conv2d'0, the sixth convolution block Conv2d'1, the seventh convolution block Conv2d'2, and the eighth convolution block Conv2d'3 element by element. The summation result is activated by the Sigmoid function to obtain the spatial attention weight S of the spatial decoupled re-identification branch. id ; The third residual link subunit is used to convert S det With feature F c_det Multiply and add the characteristic F c_det , obtain the spatially decoupled feature F for detecting branches det ; The fourth residual link subunit is used to convert S id With feature F c_id Multiply and add the characteristic F c_id , obtain the feature F for the re-identification branch after spatial decoupling id .

8. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 7, characterized in that: The spatial decoupling unit is used to c_det and F c_id Decouple them in the spatial dimension respectively, and obtain the spatially decoupled features F that will be used to detect the branch det and the spatially decoupled feature F for the re-identification branch id , specifically: F det =F c_det *S det +F c_det F id =F c_id *S id +F c_id Among them, F c_det is the feature used to detect branches after channel decoupling, F c_id is the feature for re-identification branch after channel decoupling, S det is the spatial attention weight of the spatially disentangled detection branch, S id is the spatial attention weight of the spatial disentangled re-ID branch, F det is the feature used to detect branches after spatial decoupling, F id It is the feature used for re-identification branch after spatial decoupling. Conv2d0() is the fifth convolution block. Both l and l' are integers from 1 to 3. Conv2d l () is the second convolution block, the third convolution block or the fourth convolution block; Conv2d'0() is the fifth convolution block, Conv2d' l' () is the sixth convolution block, the seventh convolution block or the eighth convolution block.

9. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 8, characterized in that: The overall loss function L used in the training of the detection branch module and the re-identification branch module is total , specifically: Among them, L heat is the heat map sub-branch unit loss function, L id is the re-identification branch module loss function, L box is the loss function of the center point offset sub-branch unit and the bounding box size sub-branch unit, w1 and w2 are learnable parameters, k is the target label in the target image of the tth frame, K is the total number of targets in the target image of the tth frame, M is the total number of target identity categories in the training set, m is the label of the target identity category in the training set, and L k (m) is the kth target identity vector in the target image of the tth frame, p(m) is the target identity category distribution vector predicted by the re-identification branch module, is the true offset of the center point of the kth target detection bounding box of the target image in the tth frame, is the predicted offset of the center point of the kth target detection bounding box of the target image in the tth frame, λ s is a preset constant, is the actual size of the kth target detection bounding box of the target image in the tth frame, is the predicted size of the kth target detection bounding box of the target image in the tth frame, (x', y') is F det Points in the heat map, is the F predicted by the heatmap sub-branch unit det The heat map value corresponding to the point (x', y') in the heat map, M x'y' Yes F det The heat map value corresponding to the point (x', y') in the heat map, α and β are preset hyper parameters.

10. The method for joint detection and re-identification of multiple targets based on feature decoupling according to claim 9, characterized in that: The heat map value M x'y' Specifically: Among them, σ c is the target scale adaptive standard deviation, It is the position of the center point of the detection bounding box of the kth target in the tth frame image on the feature map.

Images