Transformer-based feature association multi-target tracking method

Abstract

The present invention discloses a transformer-based feature association multi-target tracking method, comprising the following steps: obtaining a video frame of a video to be tracked; inputting the video frame into a preset feature extraction network and performing target detection to obtain multiple detection frames and their corresponding tracking targets; assigning a target tracker to each tracking target, and the target tracker obtains the tracking frame of the tracking target in the next frame based on the detection frame; wherein, in response to the overlapping of the tracking frames of multiple tracking targets in the current video frame, feature enhancement and suppression are performed based on feature encoding of a convolutional neural network and a transformer network, and feature decoding of a mutual attention mechanism to obtain an updated tracking frame. The present invention can more accurately track targets in long video frames, especially in the case of multiple targets and severe occlusion between targets, and mark the tracking targets with tracking frames in the video frames.

Description

Technical Field

[0001] The present invention relates to a transformer-based feature association multi-target tracking method, and belongs to the technical field of target tracking. Background Art

[0002] With the continuous advancement of technology, video cameras have become increasingly ubiquitous and closely integrated with human activities, such as video surveillance and autonomous driving. This has resulted in massive amounts of video data that require tracking of various objects within the video. These videos often contain the movement trajectories of multiple objects. Object tracking is a technology that tracks each object in the video and maps their trajectory.

[0003] Real-world scenes often contain multiple target trajectories. Neither performing detection on a single frame in a sequence nor tracking a single target across all targets in a frame can achieve the desired results. Because target tracking requires establishing a connection between the target in the previous and next images, using detection algorithms within a single frame only enables recognition, not correlation. Using single-target tracking algorithms across adjacent frames only enables correlation, not recognition. Furthermore, existing methods are prone to tracking box drift when targets occlude each other, and require high computing power, making them difficult to implement in practice.

[0004] The information disclosed in this background section is only intended to enhance understanding of the overall background of the invention and should not be considered as an admission or any form of suggestion that the information constitutes the prior art already known to a person of ordinary skill in the art. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a transformer-based feature association multi-target tracking method. The present invention can more accurately track targets in long video frames, especially in the case of multiple targets and severe occlusion between targets, and mark the tracked targets with tracking boxes in the video frames.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] The present invention discloses a transformer-based feature association multi-target tracking method, comprising the following steps:

[0008] Get the video frame of the video to be tracked;

[0009] Inputting the video frame into a preset feature extraction network to obtain semantic features of the video frame;

[0010] Performing target detection on the video frame according to the semantic features of the video frame to obtain multiple detection frames and their corresponding tracking targets;

[0011] Assigning a target tracker to each tracking target, wherein the target tracker obtains the tracking frame of the tracking target in the next video frame according to the detection frame;

[0012] Among them, in response to the overlapping of multiple tracking targets in the tracking frames of the current video frame, feature enhancement and suppression are performed based on the feature encoding of the convolutional neural network and the transformer network, and the feature decoding of the mutual attention mechanism, to obtain the updated tracking frames in the current video frame corresponding to the multiple overlapping tracking targets.

[0013] Furthermore, the video frame is input into a preset feature extraction network to obtain semantic features, including the following steps:

[0014] Based on the convolution kernel and the maximum pooling layer, downsampling the video frame to obtain downsampled features;

[0015] Based on a depth-wise separable convolution kernel and a grouped fully connected layer, performing multiple iterative feature extraction on the downsampled features to obtain multiple groups of extracted features;

[0016] The multiple sets of extracted features are fused based on a region candidate network to obtain semantic features of the video frame.

[0017] Furthermore, the target detection includes the following steps:

[0018] Generate an anchor frame based on the semantic features of the video frame, and input the features in the anchor frame into a classification convolutional neural network and a regression convolutional neural network respectively; wherein the classification convolutional neural network is used to calculate the classification score of the anchor frame, and the regression convolutional neural network is used to calculate the center point and width and height of each anchor frame;

[0019] According to the classification score of the anchor box and the center point, width and height of the anchor box, multiple candidate boxes are obtained based on the maximum suppression network;

[0020] According to the multiple candidate frames, a target classification network based on the softmax function is used to obtain multiple detection frames and their corresponding tracking targets.

[0021] Furthermore, the target tracker is provided with a target feature detection module and a Kalman filter; the tracking method of the target tracker is as follows:

[0022] Through the target feature detection module, the target semantic features of the detection frame of the current frame are extracted;

[0023] According to the target semantic features of the detection frame of the current frame, a prediction frame of the next frame corresponding to the tracking target is obtained through a Kalman filter;

[0024] Calculate the IOU values ​​of the detection frame and the prediction frame of the next frame, and use the detection frame whose IOU value is greater than a preset first threshold as a candidate tracking frame to obtain one or more candidate tracking frames;

[0025] A tracking frame for the next video frame is obtained based on the candidate tracking frames; wherein, in response to the presence of multiple candidate tracking frames, the candidate tracking frame with the largest cosine similarity is calculated between identity features extracted from the candidate tracking frames and identity features extracted from the prediction frame, and is used as the tracking frame for the next video frame of the tracking target.

[0026] Furthermore, the Kalman filter includes an identity feature extraction module and an identity feature memory module;

[0027] The identity feature extraction module performs a pooling and fixing operation on the target semantic features of the detection frame to obtain pooled target semantic features; and extracts the identity features of the tracking target based on the convolutional neural network according to the pooled target semantic features;

[0028] The identity feature memory module calculates update weights and reset weights based on a convolutional neural network; obtains the target tracker identity feature of the current frame based on the identity feature of the tracked target, the update weights, and the reset weights; and obtains a prediction frame for the next frame based on the target tracker identity feature of the current frame;

[0029] Among them, the target tracker identity feature of the current frame is represented by the product of the first feature and the updated weight plus the product of the second feature and the reset weight; the first feature is the identity feature of the tracked target and the identity reset feature spliced ​​through a convolutional neural network; the identity reset feature is represented by the product of the second feature and the reset weight; the second feature is the target tracker identity feature of the previous frame or the target tracker identity feature of the set initial frame.

[0030] Furthermore, the calculation formula of the IOU value of the detection box and the prediction box is as follows:

[0031]

[0032] Among them, iou is the quotient of the area of ​​the intersection and the area of ​​the union of the detection box and the prediction box; A represents the detection box; B represents the prediction box.

[0033] Furthermore, in response to the overlapping of multiple tracking targets in the tracking frames of the current video frame,

[0034] Any target tracker among the target trackers of multiple overlapping tracking targets is selected as the occluding target tracker, and the target tracker with the largest IOU value with the tracking frame of the occluding target tracker is calculated as the interfering target tracker;

[0035] Extracting target tracker identity features of the obscured target tracker and the interfering target tracker, performing feature enhancement and suppression based on feature encoding of a convolutional neural network and a transformer network, and feature decoding of a mutual attention mechanism, to obtain enhanced and suppressed features;

[0036] Obtaining an updated target tracker identity feature of the occluded target tracker based on the enhanced and suppressed features and the target tracker identity feature of the occluded target tracker;

[0037] According to the updated target tracker identity feature of the occluded target tracker, an updated tracking frame of the occluded target tracker in the current video frame is obtained.

[0038] Furthermore, the feature encoding includes the following steps:

[0039] The target tracker identity features of the masked target tracker and the interference target tracker are respectively subjected to a deep separable convolutional neural network to extract local features, and are respectively subjected to a transformer network with a self-attention mechanism to extract global features;

[0040] According to the local features and the global features, based on normalized splicing and a fully connected layer, a fusion feature of an occluded target tracker and a fusion feature of an interfered target tracker are obtained.

[0041] Furthermore, the feature decoding includes the following steps:

[0042] Obtaining obscured features and unobscured features of the obscured target tracker based on the mutual attention mechanism according to the fused features of the obscured target tracker and the fused features of the interfering target tracker;

[0043] Calculating the IOU values ​​of the tracking frame of the obstructing target tracker and the tracking frame of the interfering target tracker; and obtaining a weight value according to the IOU values;

[0044] Obtaining enhanced features based on the weight value, the target tracker identity feature of the obscured target tracker, and the unobscured feature;

[0045] A suppressed feature is obtained according to the weight value, the target tracker identity feature of the obstructing target tracker, and the obscured feature.

[0046] Furthermore, the expression of the weight value is as follows:

[0047]

[0048] Among them, weights represents the weight value; is the tracking frame of the tracking target ta on the t-th frame, where t represents the t-th frame and ta represents the tracking target; is the tracking box of the interference target di on the t-th frame, where t represents the t-th frame and di represents the interference target; min is the minimum masking threshold to be set; max represents the maximum value function; sin represents the sine function.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] The transformer-based feature association multi-target tracking method of the present invention can more accurately track targets in long video frames, especially in situations where there are multiple targets and severe occlusion between them, and mark the tracked targets with tracking boxes in the video frames. It combines the feature encoding of convolutional neural networks and transformer networks, extracting local features based on the convolutional neural network and global features based on the transformer network. This alleviates the drawback of the transformer network's lack of inductive bias and the resulting large amount of data required, greatly reduces the number of model parameters, and effectively improves target tracking accuracy in extreme occlusion situations.

[0051] The present invention also introduces new weight values, so that features of mutually obscured objects can be enhanced and suppressed more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flowchart of the transformer-based feature association multi-target tracking method;

[0053] Figure 2 It is a schematic diagram of the feature extraction network;

[0054] Figure 3 is a schematic diagram of the target tracker;

[0055] Figure 4 It is a schematic diagram of the feature encoding module;

[0056] Figure 5 It is a schematic diagram of the feature decoding module. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0058] This embodiment discloses a transformer-based feature association multi-target tracking method, comprising the following steps:

[0059] Get the video frame of the video to be tracked;

[0060] Input the video frame into the preset feature extraction network to obtain the semantic features of the video frame;

[0061] According to the semantic features of the video frame, the target is detected on the video frame to obtain multiple detection frames and their corresponding tracking targets;

[0062] A target tracker is assigned to each tracking target. The target tracker obtains the tracking frame of the tracking target in the next video frame based on the detection frame.

[0063] Among them, in response to the overlapping of multiple tracking targets in the tracking frames of the current video frame, feature enhancement and suppression are performed based on the feature encoding of the convolutional neural network and the transformer network, and the feature decoding of the mutual attention mechanism, to obtain the updated tracking frames in the current video frame corresponding to the multiple overlapping tracking targets.

[0064] like Figure 1 The specific steps are as follows:

[0065] Step 1: Obtain the video frame of the video to be tracked; input the video frame into the preset feature extraction network to obtain the semantic features of the video frame; the feature extraction network is as follows: Figure 2 shown.

[0066] The specific feature extraction steps are:

[0067] 1.1 Use convolution kernel and maximum pooling layer to downsample the input video frame image features by 4 times;

[0068] 1.2 Input the downsampled features into the feature extraction network and use depthwise separable convolution to extract features;

[0069] 1.3 Transpose the extracted image features in the following way:

[0070] (N,C,H,W)->(N,H,W,C)

[0071] Among them, N is the number of samples for one training, C is the number of channels of the feature, H is the height of the feature, and W is the width of the feature.

[0072] 1.4 Split the transposed features into g groups along the second feature dimension. Pass each group of features through a grouped fully connected layer, which consists of two fully connected layers and an activation layer. The features first pass through a fully connected layer whose output channels are four times the number of input channels, are then activated by the activation layer, and finally pass through another fully connected layer whose input channels are four times the number of output channels. Finally, concatenate the g groups of features along the second feature dimension.

[0073] 1.5 Transpose the output features in the following way:

[0074] (N,H,W,C)->(N,C,H,W)

[0075] Among them, N is the number of samples for one training, C is the number of channels of the feature, H is the height of the feature, and W is the width of the feature.

[0076] 1.6 Downsample the output features of this set of feature extraction networks by a factor of 2 and input them into the next set of feature extraction networks.

[0077] 1.7 Repeat steps (2) to (6) 4 times to obtain the output features of 4 sets of feature extraction networks, and input the output features into the region candidate network. The output features of the first set of feature extraction networks are downsampled by 2 times and fused with the output features of the second set of feature extraction networks using a convolutional neural network; the fused output features are downsampled by 2 times and fused with the output features of the third set of feature extraction networks using a convolutional neural network; the fused output features are downsampled by 2 times and fused with the output features of the fourth set of feature extraction networks using a convolutional neural network to obtain the final semantic features.

[0078] Step 2: Perform object detection on the semantic features extracted from the video frames;

[0079] The specific steps are:

[0080] 2.1 Generate an anchor box for each pixel of a semantic feature. Input the features within the anchor box corresponding to each pixel into two convolutional neural networks: a classification convolutional neural network and a regression convolutional neural network. The classification convolutional neural network calculates the classification result and classification score for each anchor box; the regression network calculates the center point, width, and height of each anchor box.

[0081] 2.2 The obtained classification results and the center, width and height of the anchor box are input into the maximum suppression network. The maximum suppression network sorts the different candidate boxes in descending order according to the classification scores calculated by the classification convolutional neural network in step (1), selects M areas as candidate boxes, and M represents the number of candidate areas finally input into the classification and regression network. Then, the remaining candidate boxes are traversed and the boxes whose overlap rate with the candidate box is greater than the threshold are deleted.

[0082] The features of the areas corresponding to these M boxes are input into the target classification network and combined with the softmax function to obtain the final tracking target detection box;

[0083] Repeat the above steps for the remaining anchor boxes so that each tracked target retains an object detection box.

[0084] Step 3: Assign a target tracker to the target for long-term tracking. The target tracker structure is as follows: Figure 3 As shown;

[0085] The specific steps are:

[0086] 3.1 In the first frame, a target tracker is assigned to each detected target. The target feature detection module of the target tracker extracts the target semantic features of the detection frame of the current frame. The identity feature extraction module performs a pooling operation based on the extracted target semantic features to fix the size of the target semantic features, and then inputs them into the convolutional neural network to extract the identity features of the tracked target.

[0087] 3.2 The identity feature memory module includes a gating unit, which uses a convolutional neural network to extract candidate features from the input features and calculate the update weights and reset weights.

[0088] 3.3 According to the identity characteristics of the tracked target, update weights and reset weights, the target tracker identity characteristics of the current frame are obtained. The specific steps are as follows:

[0089] Obtaining a second feature, where the second feature is the target tracker identity feature of the previous frame or the target tracker identity feature of the set initial frame;

[0090] Multiply the second feature by the reset weight to obtain the identity reset feature; concatenate the identity reset feature and the identity feature of the tracked target obtained in 3.1 through a convolutional neural network to obtain the first feature; multiply the first feature by the updated weight, and multiply the second feature by the reset weight, and add the two products to obtain the target tracker identity feature of the current frame;

[0091] Based on the target tracker identity features of the current frame, the predicted box of the next frame is obtained.

[0092] 3.4 In the subsequent video frames, the target area is detected based on the input video frame. The target tracker then uses the built-in Kalman filter to predict the possible location of the target in the current frame and obtain the predicted box for the next frame. The IOU value is calculated based on the position, length, and width of the predicted box and the position, length, and width of the detection box in the next frame. The detection box with an IOU value greater than the threshold is set as the candidate tracking box for the next frame.

[0093] 3.5 If only one detection frame has an IOU value greater than the preset first threshold with the predicted frame, it will be used as the next tracking frame of the target tracker. If there are multiple candidate tracking frames, the identity features are extracted from the image of the candidate tracking frame and the cosine similarity is calculated with the identity features extracted from the predicted frame. The target frame with the largest cosine similarity is used as its own tracking frame in the next frame. The IOU value calculation formula is as follows:

[0094]

[0095] Among them, iou is the quotient of the area of ​​the intersection and the area of ​​the union of the detection box and the prediction box; A represents the detection box; B represents the prediction box.

[0096] 3.6 The target tracker updates the parameters of the Kalman filter based on the target tracker identity features and detection box in the next frame.

[0097] Step 4: Suppress and enhance the features of the obscured target;

[0098] The specific steps are:

[0099] 4.1 For all target trackers in each frame, if multiple tracking targets overlap in the tracking frame of the current video frame, that is, the IOU value between a target tracker and the tracking frame of other target trackers is greater than the set second threshold, any target tracker is taken as the occluded target tracker, and the target tracker with the largest IOU value with the tracking frame of the occluded target tracker is calculated as the interfering target tracker of the occluded target tracker.

[0100] Extract the target tracker identity features of the occluded target tracker and the interference target tracker and input them into the feature encoding module, cross-space correlation features, the feature encoding module structure is as follows Figure 4 :

[0101] The specific steps are:

[0102] 4.1.1 The target tracker identity features of the occluded target tracker and the interference target tracker are extracted into corresponding local features through a deep separable convolutional neural network;

[0103] 4.1.2 The target tracker identity features of the obscured target tracker and the interfering target tracker are used to extract global features based on the self-attention mechanism through the transformer network. The attention mechanism calculation process is as follows:

[0104]

[0105] The image features are multiplied by matrices to obtain Q (Query), K (Key), and V (Value). Q is the query feature, K is the key feature, and V is the environment feature. is the dimension of the Key vector.

[0106] The feature decoding module uses the target features of the occluded target tracker as the Q query features, the target features of the interfering target tracker as the K key features and the V environmental features, performs attention mechanism calculations, and maps the occluded features in the target features to the interfering target features to obtain the global features.

[0107] 4.1.3 After normalization, the local features and global features are concatenated in the second feature dimension and input into the fully connected layer to fuse the local features with the global features; the fused features of the occluded target tracker and the fused features of the interference target tracker are obtained.

[0108] 4.2 The fusion features obtained by the masked target tracker after passing through the feature encoding module and the fusion features obtained by the interference target tracker after passing through the feature encoding module are passed through the feature decoding module to calculate the unmasked features. The structure of the feature decoding module is as follows: Figure 5 ;

[0109] The specific steps are:

[0110] 4.2.1 Input the fused features of the occluded target tracker and the interfering target tracker into the feature decoding module;

[0111] 4.2.2 The fused features of the occluded target tracker and the fused features of the interference target tracker are mapped to each other through the mutual attention mechanism to find the unobstructed features and obscured features of the occluded target tracker. The calculation principle of the mutual attention mechanism is similar to that of the self-attention mechanism. The difference is that the Q, K, and V of the self-attention mechanism are all extracted from the identity features of the current target through matrix multiplication, while the Q of the mutual attention mechanism is extracted from the identity features of the interference target tracker, and K and V are extracted from the identity features of the current target tracker.

[0112] 4.2.3 Combine the iou values ​​of the tracking frame of the occluded target tracker and the tracking frame of the interfering target tracker to obtain a weight value weights.

[0113] The target tracker identity feature of the occluded target tracker is added to the weight value weights and multiplied by the unoccluded feature to obtain the enhanced feature to enhance the unoccluded feature of the tracked target.

[0114] The target tracker identity feature of the occluded target tracker is subtracted from the weight value weights and multiplied by the occluded feature to obtain the suppressed feature to suppress the occluded feature of the tracked target.

[0115] The calculation formula for weights is as follows:

[0116]

[0117] Among them, weights is the weight value for feature enhancement and suppression. is the tracking box of the tracking target ta on the t-th frame, where t represents the t-th frame and ta represents the tracking target. is the tracking box of the interference target di on the t-th frame, where t represents the t-th frame and di represents the interference target. minis the minimum occlusion threshold set;

[0118] 4.2.4 Further, based on the enhanced and suppressed features and the target tracker identity features of the occluded target tracker, an updated target tracker identity feature of the occluded target tracker is obtained;

[0119] According to the target tracker identity feature updated by the occluded target tracker, an updated tracking frame of the occluded target tracker in the current video frame is obtained.

[0120] It should be noted that the parameters of all networks and modules involved in this embodiment have been trained to the optimal level, and the training sets used are mot16 and mot17 data sets and preprocessed. mot16 and mot17 are the most authoritative data sets for multi-target research. Among them, the mot16 data set contains 14 short videos, and the mot17 data set contains 42 short videos. Based on the training set, steps 1-4 are performed using the error back propagation algorithm to adjust the parameters of each network and module to the optimal parameters, including feature extraction network parameters, classification convolutional neural network parameters, regression convolutional neural network parameters, target tracker network parameters, feature encoding module parameters, and feature decoding module parameters.

[0121] This embodiment uses lightweight multi-target tracking based on transformer-based cross-spatial feature association to more accurately track targets in long video frames, especially when there is severe occlusion between targets, and mark the tracked targets with tracking boxes in the video frames.

[0122] This embodiment proposes a feature extraction method based on the fusion of transformer network and convolutional neural network. Experiments have verified that it has better results in multi-target tracking and has certain promotion significance.

[0123] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0127] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. The transformer-based feature association multi-target tracking method is characterized by: The steps include: Get the video frame of the video to be tracked; Inputting the video frame into a preset feature extraction network to obtain semantic features of the video frame; Performing target detection on the video frame according to the semantic features of the video frame to obtain multiple detection frames and their corresponding tracking targets; Assigning a target tracker to each tracking target, wherein the target tracker obtains the tracking frame of the tracking target in the next video frame according to the detection frame; In response to the overlapping tracking frames of multiple tracking targets in the current video frame, feature enhancement and suppression are performed based on feature encoding of convolutional neural networks and transformer networks, as well as feature decoding of mutual attention mechanisms, to obtain updated tracking frames in the current video frame corresponding to the multiple overlapping tracking targets. In response to the overlapping of multiple tracking targets in the tracking frame of the current video frame, Any target tracker among the target trackers of multiple overlapping tracking targets is selected as the occluding target tracker, and the target tracker with the largest IOU value with the tracking frame of the occluding target tracker is calculated as the interfering target tracker; Extracting target tracker identity features of the obscured target tracker and the interfering target tracker, performing feature enhancement and suppression based on feature encoding of a convolutional neural network and a transformer network, and feature decoding of a mutual attention mechanism, to obtain enhanced and suppressed features; Obtaining an updated target tracker identity feature of the occluded target tracker based on the enhanced and suppressed features and the target tracker identity feature of the occluded target tracker; Obtaining an updated tracking frame of the occluded target tracker in the current video frame according to the target tracker identity feature updated by the occluded target tracker; The feature encoding comprises the following steps: The target tracker identity features of the masked target tracker and the interference target tracker are respectively subjected to a deep separable convolutional neural network to extract local features, and are respectively subjected to a transformer network with a self-attention mechanism to extract global features; According to the local features and the global features, based on normalized splicing and a fully connected layer, a fusion feature of the occluded target tracker and a fusion feature of the interference target tracker are obtained; The feature decoding comprises the following steps: Obtaining obscured features and unobscured features of the obscured target tracker based on the mutual attention mechanism according to the fused features of the obscured target tracker and the fused features of the interfering target tracker; Calculating the IOU values ​​of the tracking frame of the obstructing target tracker and the tracking frame of the interfering target tracker; and obtaining a weight value according to the IOU values; Obtaining enhanced features based on the weight value, the target tracker identity feature of the obscured target tracker, and the unobscured feature; A suppressed feature is obtained according to the weight value, the target tracker identity feature of the obstructing target tracker, and the obscured feature.

2. The transformer-based feature association multi-target tracking method according to claim 1, characterized in that: Inputting the video frame into a preset feature extraction network to obtain semantic features includes the following steps: Based on the convolution kernel and the maximum pooling layer, downsampling the video frame to obtain downsampled features; Based on a depth-wise separable convolution kernel and a grouped fully connected layer, performing multiple iterative feature extraction on the downsampled features to obtain multiple groups of extracted features; The multiple sets of extracted features are fused based on a region candidate network to obtain semantic features of the video frame.

3. The transformer-based feature association multi-target tracking method according to claim 1, characterized in that: The target detection comprises the following steps: Generate an anchor frame based on the semantic features of the video frame, and input the features in the anchor frame into a classification convolutional neural network and a regression convolutional neural network respectively; wherein the classification convolutional neural network is used to calculate the classification score of the anchor frame, and the regression convolutional neural network is used to calculate the center point and width and height of each anchor frame; According to the classification score of the anchor box and the center point, width and height of the anchor box, multiple candidate boxes are obtained based on the maximum suppression network; According to the multiple candidate frames, a target classification network based on the softmax function is used to obtain multiple detection frames and their corresponding tracking targets.

4. The transformer-based feature association multi-target tracking method according to claim 1, wherein: The target tracker is equipped with a target feature detection module and a Kalman filter; the tracking method of the target tracker is as follows: Through the target feature detection module, the target semantic features of the detection frame of the current frame are extracted; According to the target semantic features of the detection frame of the current frame, a prediction frame of the next frame corresponding to the tracking target is obtained through a Kalman filter; Calculate the IOU values ​​of the detection frame and the prediction frame of the next frame, and use the detection frame whose IOU value is greater than a preset first threshold as a candidate tracking frame to obtain one or more candidate tracking frames; A tracking frame for the next video frame is obtained based on the candidate tracking frames; wherein, in response to the presence of multiple candidate tracking frames, the candidate tracking frame with the largest cosine similarity is calculated between identity features extracted from the candidate tracking frames and identity features extracted from the prediction frame, and is used as the tracking frame for the next video frame of the tracking target.

5. The transformer-based feature association multi-target tracking method according to claim 4, characterized in that: The Kalman filter includes an identity feature extraction module and an identity feature memory module; The identity feature extraction module performs a pooling and fixing operation on the target semantic features of the detection frame to obtain pooled target semantic features; and extracts the identity features of the tracking target based on the convolutional neural network according to the pooled target semantic features; The identity feature memory module calculates update weights and reset weights based on a convolutional neural network; Obtaining the target tracker identity feature of the current frame according to the identity feature of the tracked target, the updated weight, and the reset weight; Obtaining a prediction frame for the next frame based on the target tracker identity feature of the current frame; Among them, the target tracker identity feature of the current frame is represented by the product of the first feature and the updated weight plus the product of the second feature and the reset weight; the first feature is the identity feature of the tracked target and the identity reset feature spliced ​​through a convolutional neural network; the identity reset feature is represented by the product of the second feature and the reset weight; the second feature is the target tracker identity feature of the previous frame or the target tracker identity feature of the set initial frame.

6. The transformer-based feature association multi-target tracking method according to claim 4, characterized in that: The calculation formula of the IOU value of the detection box and the prediction box is as follows: ； Among them, iou is the quotient of the area of ​​the intersection and the area of ​​the union of the detection box and the prediction box; A represents the detection box; B represents the prediction box.

7. The transformer-based feature association multi-target tracking method according to claim 1, characterized in that: The expression of the weight value is as follows: ； Among them, weights represents the weight value; is the tracking frame of the tracking target ta on the t-th frame, where t represents the t-th frame and ta represents the tracking target; is the tracking box of the interference target di on the t-th frame, where t represents the t-th frame and di represents the interference target; is the minimum masking threshold to be set; max represents the maximum value function; sin represents the sine function.

Images