A lightweight multi-target tracking method based on attention mechanism

Abstract

The present invention discloses a lightweight multi-target tracking method based on attention mechanism, and proposes a joint attention module to enhance the ShuffleNet V2 network. The module includes a spatiotemporal pyramid module STPM and a convolutional block attention module CBAM. STPM integrates multi-scale features to capture information on different spatial and temporal scales, and CBAM aggregates channel and spatial dimension information to enhance the representation ability of the model. Then, a position encoding generator module PEGM and a dynamic template update strategy DTUS are proposed to solve the occlusion problem. PEGM uses group convolution in the input sequence, and each convolution group is responsible for processing the relative position relationship in a specific range. In order to improve the reliability of the template, DTUS is used to update the template at an appropriate time. A lightweight design is achieved, the number of parameters and calculation requirements of the tracker are reduced, and the accuracy of the target tracking method is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a lightweight multi-target tracking method based on an attention mechanism. Background Art

[0002] Object tracking is an important direction in the field of computer vision. It is widely used in practical applications such as intelligent monitoring, behavior recognition, and robot navigation. Accurately tracking arbitrary objects in real-world scenes is very challenging due to the changes in the scale, appearance, and texture of the objects. The present invention focuses on multi-target tracking, which includes establishing consistent trajectories by identifying and locating multiple targets in a continuous video sequence. The multi-target tracking algorithm mainly involves detecting the target object in each video frame and determining its position in the image. Subsequently, a unique ID is assigned to each target, which remains unchanged throughout the movement of the target. Existing target tracking methods can be roughly divided into two categories: traditional tracking methods and deep learning-based target tracking methods. Notable examples of traditional tracking algorithms include optical flow methods, Kalman filters, and kernel correlation filters. Among them, the Kalman filter iteratively and dynamically estimates the state at subsequent moments while considering the historical state and observation data. In order to meet the challenges of real-world target tracking, the Kalman filter models the motion of the target, enabling it to estimate the position of the target in subsequent frames. Nevertheless, simply learning simple features cannot fully address the complexity of real scenes.

[0003] In the field of deep learning-based object tracking, the most popular approach is "tracking by detection" (TBD), which involves extracting a set of detection results from video frames to guide the tracking process. Tracking errors caused by indistinguishable similar objects can be minimized by using Siamese network-based tracking methods with high tracking accuracy and speed. The classical algorithm fully convolutional Siamese network (SiamFC) algorithm is a notable example. The SiamFC algorithm formulates the tracking task as a similarity matching problem between template and search branches, resulting in commendable tracking performance. However, SiamFC mainly relies on convolutional layers to capture local features of the target, and its ability to model long-term dependencies of the target is limited.

[0004] To establish a robust global temporal context, a lightweight multi-object tracking method based on attention mechanism is proposed. Summary of the invention

[0005] In response to the above technical problems, the present invention provides a lightweight multi-target tracking method based on an attention mechanism.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A lightweight multi-target tracking method based on an attention mechanism, the method comprising the following steps:

[0008] S100: Acquire an image containing multiple targets, and input the images into a template branch and a search branch for preprocessing, respectively, to obtain a first image and a second image;

[0009] S200: Input the first image and the second image into the corresponding ShuffleNet V2 lightweight network to extract the attention-rich target features, generate a feature map of a specific size, and obtain a first feature map and a second feature map;

[0010] S300: Inputting the first feature map and the second feature map into the corresponding joint attention module to capture the long-distance spatiotemporal relationship, and obtaining the first aggregate feature and the second aggregate feature;

[0011] S400: The first aggregate feature and the second aggregate feature are fused and encoded by element multiplication to obtain an encoded feature map, which is used as a key for cross attention in the parallel first decoder and the second decoder; the first decoder decodes the object feature according to the encoded feature map and the learned target query, and combines the confidence branch to update the target tracking model in real time based on the dynamic template update strategy; the second decoder decodes the encoded feature map and the template feature query to obtain the tracking feature;

[0012] S500: The object features obtained by the first decoder are processed by a feedforward neural network to generate a detection frame, and the tracking features obtained by the second encoder are processed by a feedforward neural network to generate a tracking frame. The detection frame is associated with the tracking frame using a frame-IoU matching method to obtain a final tracking result.

[0013] Preferably, the ShuffleNet V2 lightweight network in S200 includes a convolutional layer, a maximum pooling layer, a first extraction stage, a second extraction stage, and a third extraction stage, each extraction stage includes a first block and a second block, the first extraction stage includes a stacked first block and three second blocks, the second extraction stage includes a stacked first block and seven second blocks, and the third extraction stage includes a stacked first block and three second blocks;

[0014] In the first block, the input features are subjected to channel segmentation operation, and the feature channels are divided into two branches. The number of channels in each branch is half of the original number of channels. The right branch undergoes 1×1 convolution, 3×3 depth convolution, and 1×1 convolution, while the left branch remains unchanged through identity mapping. After cascading and channel shuffling, the characteristics of the branches are effectively fused together.

[0015] In the second block, the input feature map is fed into two branches. The left branch includes a 3×3DWConv with a stride of 2 and a conventional 1×1 convolution operation, while the right branch undergoes a 1×1 convolution, followed by a 3×3 depthwise convolution, and then a 1×1 convolution. After channel concatenation and channel shuffling, the channel count of the output feature map is doubled; when the input feature is the first image, the output feature map is the first feature map, and when the input feature is the second image, the output feature map is the second feature map.

[0016] Preferably, the joint attention module in S300 includes a CBAM module and an STPM module, the CBAM module includes a channel attention module and a spatial attention module, the channel attention module is used to perform channel attention enhancement on the input feature to obtain a channel attention feature, and the channel attention feature is multiplied with the input feature as the input of the spatial attention module, the spatial attention module is used to perform spatial attention enhancement on the input feature to obtain a spatial attention feature, and the spatial attention feature is multiplied with the input feature of the spatial attention module to obtain an output feature map F c The STPM module consists of four parallel branches, each of which is embedded with a CBAM module, which seamlessly integrates features of four different pyramid scales and generates multi-scale attention features F by fusing spatiotemporal attention contexts at different scales. s ; Finally, the features processed by CBAM and STPM are combined and output to obtain the aggregated features.

[0017] Preferably, the channel attention module compresses the global spatial information contained in the input feature map F through global maximum pooling and global average pooling, generates two different one-dimensional feature maps F1 and F2, and derives them through a shared multi-layer perceptron MLP, which includes a fully connected layer and a ReLU nonlinear activation function. After the element-wise summation operation across channels, the two one-dimensional feature maps are then normalized using a sigmoid function. The normalization process generates a weighted statistic for each channel represented as Mc, and the output feature map F′ is obtained by multiplying Mc and the input feature map F;

[0018] The specific process for calculating the channel attention feature Mc is:

[0019]

[0020] Among them, AvgPool refers to average pooling; MaxPool refers to maximum pooling; MLP is multi-layer perceptron calculation; F is the input feature map; H1 and W1 represent the height and width of the feature map respectively; Represents the value at position (i0, j0) in the feature map.

[0021] Preferably, the feature map F′ output by the channel attention module is used as the input of the spatial attention module, based on the channel-based global maximum pooling and global feature pooling to obtain the feature map F3∈R 1×H1×W1 and F4∈R 1×1×W1 , then, these two feature maps are concatenated to generate an effective feature descriptor F5∈R 2×H1×W1 , perform a 7×7 convolution operation on F5 to reduce it to one channel, and generate a spatial attention feature Ms through an S-type operation, and multiply the spatial attention feature Ms with the input feature map F′ of the channel attention module to obtain the final generated aggregate feature Fc;

[0022] The specific process for calculating the spatial attention feature Ms is:

[0023] M s (F′)=σ{f 7×7 [AvgPool(F′);MaxPool(F′)]}

[0024] Among them, f 7×7 Represents a convolution layer with a convolution kernel of 7×7.

[0025] Preferably, in the STPM module, the input feature map F∈R C1×H1×W1 Processing is performed in 4 parallel branches; each branch divides the input feature map into s×s sub-regions, and s∈S, S={1,2,3,4} represents four pyramid scales;

[0026] In a branch of scale s, each region is defined as A CBAM is added to each branch. In each pyramid branch, the CBAM is applied to all sub-regions to generate an updated residual feature map. as well as

[0027] The updated residual feature maps obtained from four different scales are upsampled to obtain features with the same spatial dimensions as the initial feature map F

[0028] The four feature maps Zs are concatenated and input into the convolutional layer (C, 1×1 / 1) to generate the final residual feature map Y, which is summed element-wise with the input feature map F to obtain the updated feature map F s .

[0029] Preferably, the encoder includes a self-attention layer, a residual and normalization layer, a position encoding generation module, a self-attention layer, and a residual and normalization layer connected in sequence; in S400, the first aggregate feature and the second aggregate feature are fused and encoded by element-by-element multiplication to obtain an encoded feature map, specifically:

[0030] The encoder will be composed of the combined feature map F cf The linearly transformed Q, K, and V are used as input, and a set of attention weights are obtained through Softmax operation. The weights are multiplied by V to obtain a vector with attention weights. The specific calculation process is as follows:

[0031]

[0032] Among them, Q, K, and V represent query vector, keyword vector, and value vector respectively. is the key vector dimension, A K→Q is the similarity matrix between Q and K;

[0033] Using a single-head attention mechanism, Q, K, and V are input into the self-attention layer, and the relevance of each query to all keys is calculated through linear transformation and attention mechanism, and the relevance is used as the weight to perform weighted summation of the values ​​to generate the final output representation H1, specifically:

[0034] H1=Ins.Norm(Attention(Q1,K1,V1)+F cf ) (7)

[0035] Among them, F cf is the first aggregate feature F jt and the second aggregate feature F js The fusion feature is obtained by element-wise multiplication of , Q1, K1, V1 come from the linear transformation of the fusion feature, Attention(.) is the attention calculation, and the calculation process is shown in equation (6), Ins.Norm(·) represents the L2 normalization of all embedded image blocks;

[0036] H1 is input into the positional encoding generation module, and after a series of linear transformations, Q2, K2, and V2 are calculated and then input into the self-attention layer for residual connection and normalization to obtain H2:

[0037] H2=Ins.Norm(Attention(Q2,K2,V2)+PEGT) (8)

[0038] Preferably, the first decoder includes a self-attention layer, a first residual and normalization layer, a cross attention layer, a second residual and normalization layer, a feedforward neural network layer, and a third residual and normalization layer; in S400, the first decoder decodes the object feature according to the encoded feature map and the learned target query, specifically:

[0039] The first decoder feeds the learned target query into the self-attention layer for self-attention calculation and performs residual connection and normalization to obtain H3:

[0040] H3=Ins.Norm(Attention(Q3,K3,V3)+q lo ) (9)

[0041] Among them, Q3, K3 and V3 are the target query q lo Obtained after linear transformation;

[0042] Q4, K4, and V4 are jointly fed into the criss-cross attention module to calculate the temporal criss-cross attention matrix and then residually connected and normalized to obtain H c1 :

[0043] H c1 =Ins.Norm(Attention(Q4,K4,V4)+H3) (10)

[0044] Among them, Q4 is obtained by linear transformation of H3, V4 and K4 are obtained by linear transformation of H2, and finally, H c1 After inputting into the feedforward network, residual connection and normalization are used to obtain F d , the calculation formula (11) is as follows:

[0045] F d =Ins.Norm(FFN(H c1 )+H c1 ) (11)

[0046] Preferably, the second decoder includes a self-attention layer, a first residual and normalization layer, a cross attention layer, a second residual and normalization layer, a feedforward neural network layer, and a third residual and normalization layer; in S400, the second decoder decodes according to the encoded feature map and the template feature query to obtain the tracking feature, specifically:

[0047] The second decoder converts the template feature query q tt As input, it obtains H4 through residual connection and layer normalization after self-attention calculation:

[0048] H4=Ins.Norm(Attention(Q5,K5,V5)+q tt ) (12)

[0049] Among them, Q5, K5 and V5 are in the template feature query q tt Obtained after linear transformation;

[0050] Q6 obtained by mapping H4 is cross-calculated with K6 and V6 mapped by the encoding feature H2, and then connected and normalized with the template feature residual after self-attention to obtain H c2 :

[0051] H c2 =Ins.Norm(Attention(Q6,K6,V6)+H4) (13)

[0052] Among them, K6 and V6 obtained by mapping from the encoder feature H2 are cross-attended with Q6 obtained from the mapping of the feature H4 in the second decoder. This process produces H c2 ;

[0053] Finally, the feature H c2 Input into the feedforward network and normalized to obtain F t , the calculation process is as follows:

[0054] F t =Ins.Norm(FFN(H c3 )+H c2 ) (14)

[0055] Among them, FFN(.) refers to the calculation of feedforward neural network, which involves linear transformation and activation function.

[0056] Preferably, the confidence branch includes a three-layer perceptron and an S-type function group. In S400, the first decoder combines the confidence branch and updates the target tracking model in real time based on the dynamic template update strategy, specifically:

[0057] The confidence score c of each target generated detection box is obtained through the confidence branch j , where j = 1, 2, ..., n, the individual confidence of each target is fused by weighted average to obtain the global confidence:

[0058]

[0059] When the global confidence score is higher than the set threshold δ, it indicates that there is no occlusion and the target tracking model starts to be updated. Otherwise, the target tracking model is updated again after τ frames.

[0060] The above-mentioned lightweight multi-target tracking method based on the attention mechanism proposes a joint attention module, which is combined with the ShuffleNet V2 network to achieve a lightweight design while accurately and efficiently capturing the key features of the target object. SFTrans is designed based on the original transformer structure. SFTrans includes an encoder and two decoders, which reduces the number of parameters and computational requirements of the tracker; a dynamic template update strategy is proposed to improve the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1It is a flowchart of a lightweight multi-target tracking method based on an attention mechanism in one embodiment of the present invention;

[0062] Figure 2 Schematic diagram of the network structure of a lightweight multi-target tracking model based on an attention mechanism in one embodiment of the present invention;

[0063] Figure 3 Schematic diagram of the structure of the building blocks of the ShuffleNetV2 network in one embodiment of the present invention; wherein (a) is a schematic diagram of the structure of the first block, and (b) is a schematic diagram of the structure of the second block;

[0064] Figure 4 It is a schematic diagram of the structure of CBAM in one embodiment of the present invention;

[0065] Figure 5 A schematic diagram of the structure of STPM in one embodiment of the present invention;

[0066] Figure 6 Schematic diagram of the structure of SFtrans in one embodiment of the present invention;

[0067] Figure 7 This is a schematic diagram of the structure of PEGM in one embodiment of the present invention;

[0068] Figure 8 A flowchart of a dynamic template update strategy in one embodiment of the present invention;

[0069] Fig. 9 This is a schematic diagram of the loss results of the model training of the present invention;

[0070] Fig.10 : It is a visualization diagram of the joint attention heat map in one embodiment of the present invention; (a) represents the original image input to the network, (b) represents the superposition result of the region of interest after adding CBAM and the original image, (c) represents the superposition result of the region of interest obtained from the original image after STPM and the original image, and (d) represents the superposition result of the region of interest after adding the joint attention module and the original image;

[0071] Fig.11 The target tracking algorithm and occlusion experiment diagram of an embodiment of the present invention; (a) shows the tracking result of the algorithm of the present invention without the position coding generator module PEGM and the dynamic template update strategy DTUS, and (b) shows the tracking result of the algorithm of the present invention with the position coding generator module PEGM and the dynamic template update strategy DTUS added;

[0072] Fig.12Schematic diagram of experimental comparison results of the target tracking algorithm of an embodiment of the present invention and the FairMOT algorithm in real scenes; (a) shows the comparison of tracking effects of the algorithm of the present invention and the FairMOT algorithm in the teaching building corridor scene, and (b) shows the comparison of tracking effects of the algorithm of the present invention and the FairMOT algorithm in the school gate scene. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0074] In one embodiment, Figure 1 As shown, a lightweight multi-target tracking method based on an attention mechanism comprises the following steps:

[0075] S100: Acquire an image containing multiple targets, and input the images into a template branch and a search branch for preprocessing, respectively, to obtain a first image and a second image;

[0076] S200: Input the first image and the second image into the corresponding ShuffleNet V2 lightweight network to extract the attention-rich target features, generate a feature map of a specific size, and obtain a first feature map and a second feature map;

[0077] S300: Inputting the first feature map and the second feature map into the corresponding joint attention module to capture the long-distance spatiotemporal relationship, and obtaining the first aggregate feature and the second aggregate feature;

[0078] S400: The first aggregate feature and the second aggregate feature are fused and encoded by element multiplication to obtain an encoded feature map, which is used as a key for cross attention in the parallel first decoder and the second decoder; the first decoder decodes the object feature according to the encoded feature map and the learned target query, and combines the confidence branch to update the target tracking model in real time based on the dynamic template update strategy; the second decoder decodes the encoded feature map and the template feature query to obtain the tracking feature;

[0079] S500: The object features obtained by the first decoder are processed by a feedforward neural network to generate a detection frame, and the tracking features obtained by the second encoder are processed by a feedforward neural network to generate a tracking frame. The detection frame is associated with the tracking frame using a frame-IoU matching method to obtain a final tracking result.

[0080] Specifically, Figure 2As shown, the overall architecture follows a structure similar to a twin network, dividing the algorithm into two branches: a template branch and a search branch. To simplify the calculation, the present invention uses the ShuffleNet V2 lightweight network as the backbone. An important supplement is the joint attention module, including the convolutional block attention module CBAM and the spatiotemporal pyramid module STPM, which helps capture long-range spatiotemporal relationships. After the joint attention module, the features of the two branches are fused by element multiplication and used as the key for cross attention in the parallel decoder. The second decoder (corresponding to the decoder 2 in the accompanying figure) uses the target feature extracted from the template frame as the input query, while the first decoder (corresponding to the decoder 1 in the accompanying figure) uses the learned target query as input, which is a set of trainable parameters trained together with all other parameters in the network. In the encoder, the present invention introduces PEGM to enhance the anti-occlusion ability of the algorithm. In addition, decoder 1 combines the confidence branch, and the branch uses a dynamic template update strategy to update the network parameters in the architecture in real time to obtain more accurate information. In order to effectively manage the computational load, single-head self-attention is used in the present invention. Decoder 1 can obtain the detection box and decoder 2 can obtain the tracking box. In the association matching step, IoU matching is used to associate the detection box with the tracking box.

[0081] The present invention introduces the Transformer architecture and the ShuffleNet V2 lightweight network, and combines them with the joint attention module and the free position encoding module. The method aims to enhance the ability of the target tracking algorithm to perceive and utilize spatiotemporal context information. By introducing the joint attention mechanism, the algorithm can selectively focus on key intra-frame and inter-frame information, thereby improving its ability to capture target motion patterns and environmental conditions. The free position encoding module provides a large amount of position information for each target, which helps to model the position and motion state of the target with high accuracy.

[0082] like Figure 2As shown, the template branch first preprocesses the previous frame image of the current frame into an image with a channel number of C, a height and a width of H×W, and then inputs it into the ShuffleNetV2 network. Subsequently, the image undergoes convolution and maximum pooling operations to obtain a feature map of size 8C×(H / 4)×(W / 4). Next, after three stages, each stage contains block 1 and block 2 units, which are stacked 1 and 3 times, 1 and 7 times, and 1 and 3 times, respectively. These stages extract attention-rich target features and produce a feature map of a specific size (464 / C)×(H / 32)×(W / 33). Similarly, the search branch is preprocessed and input into the ShuffleNetV2 network. It undergoes the same processing as the template branch and obtains a size of (464 / C)×(H / 32)×(W / 32. In order to simplify the calculation, 464 / C is represented as C1, H / 32 is represented as H1, and W / 32 is represented as W1 in the present invention.

[0083] In one embodiment, the ShuffleNet V2 lightweight network in S200 includes a convolutional layer, a maximum pooling layer, a first extraction stage, a second extraction stage, and a third extraction stage, each extraction stage includes a first block and a second block, the first extraction stage includes a stacked 1 first block and 3 second blocks, the second extraction stage includes a stacked 1 first block and 7 second blocks, and the third extraction stage includes a stacked 1 first block and 3 second blocks;

[0084] In the first block, the input features are subjected to channel segmentation operation, and the feature channels are divided into two branches. The number of channels in each branch is half of the original number of channels. The right branch undergoes 1×1 convolution, 3×3 depth convolution, and 1×1 convolution, while the left branch remains unchanged through identity mapping. After cascading and channel shuffling, the characteristics of the branches are effectively fused together.

[0085] In the second block, the feature map output by the previous block is fed into two branches. The left branch includes a 3x3 DWConv with a stride of 2 and a conventional 1×1 convolution operation, while the right branch undergoes a 1×1 convolution, followed by a 3×3 depthwise convolution, and then a 1x1 convolution. After channel concatenation and channel shuffling, the channel count of the output feature map is doubled; when the input feature is the first image, the output feature map is the first feature map, and when the input feature is the second image, the output feature map is the second feature map.

[0086] Specifically, the ShuffleNet network is a high-performance lightweight convolutional neural network. It achieves the best balance between speed and accuracy through two core operations: group convolution and channel shuffle. These operations effectively maintain accuracy while significantly reducing computational complexity. Based on the ShuffleNetV1 network, guidelines for designing efficient and lightweight networks were introduced, resulting in the ShuffleNetV2 network. The ShuffleNetV2 network architecture includes the first block Block1 and the second block Block2, as shown in Figure 3 (a) shown and Figure 3 (b) as shown.

[0087] In one embodiment, the joint attention module in S300 includes a CBAM module and an STPM module, the CBAM module includes a channel attention module and a spatial attention module, the channel attention module is used to perform channel attention enhancement on the input feature to obtain a channel attention feature, and the channel attention feature is multiplied with the input feature as the input of the spatial attention module, the spatial attention module is used to perform spatial attention enhancement on the input feature to obtain a spatial attention feature, and the spatial attention feature is multiplied with the input feature of the spatial attention module to obtain an output feature map F c The STPM module consists of four parallel branches, each of which is embedded with a CBAM module, which seamlessly integrates features from four different pyramid scales and generates multi-scale attention features F by fusing spatiotemporal attention contexts at different scales. Finally, the features processed by CBAM and STPM are combined to output the aggregated features.

[0088] Specifically, CBAM is a lightweight convolutional attention module, which consists of two submodules: channel attention module and spatial attention module. The overall architecture of CBAM is as follows: Figure 4 From this architecture, ShuffleNet v2 derives a feature map, denoted as F∈R C1×H1×W1 , where C1 represents the number of channels in the input feature map, H1 and W1 represent its height and width, respectively. CBAM compresses the global spatial information contained in the feature map F through global maximum pooling and global average pooling. Therefore, two different feature maps are generated: F1, with a size of 1×1×C1, and F2, with a size of 1×1 times C1. Incorporating location information from each channel feature map over the entire dataset helps alleviate network bias, which is usually caused by the limited receptive field range of the convolution kernel during feature extraction.

[0089] In one embodiment, the channel attention module compresses the global spatial information contained in the input feature map F through global maximum pooling and global average pooling, generates two different one-dimensional feature maps F1 and F2, and derives them through a shared multi-layer perceptron MLP, which includes a fully connected layer and a ReLU nonlinear activation function. After the element summation operation across channels, the two one-dimensional feature maps are then normalized using a sigmoid function. The normalization process generates a weight statistic for each channel represented as Mc, and the output feature map F′ is obtained by multiplying Mc and the input feature map F.

[0090] The specific process for calculating the channel attention feature Mc is:

[0091]

[0092] Among them, AvgPool refers to average pooling; MaxPool refers to maximum pooling; MLP is multi-layer perceptron calculation; F is the input feature map; H1 and W1 represent the height and width of the feature map respectively; Represents the value at position (i0, j0) in the feature map.

[0093] Specifically, in order to make full use of the feature information extracted by the compression operation, the correlation between channels is obtained through the evaluation operation.

[0094] In one embodiment, the feature map F′ output by the channel attention module is used as the input of the spatial attention module, and the channel-based global maximum pooling and global feature pooling are performed to obtain the feature map F3∈R 1×H1×W1 and F4∈R 1×1×W1 , then, these two feature maps are concatenated to generate an effective feature descriptor F5∈R 2×H1×W1 , perform a 7×7 convolution operation on F5 to reduce it to one channel, and generate a spatial attention feature Ms through an S-type operation, and multiply the spatial attention feature Ms with the input feature map F′ of the channel attention module to obtain the final generated aggregate feature Fc;

[0095] The specific process for calculating the spatial attention feature Ms is:

[0096] M s (F′)=σ{f 7×7 [AvgPool(F′);MaxPool(F′)]}

[0097] Among them, f 7×7 Represents a convolution layer with a convolution kernel of 7×7.

[0098] In one embodiment, Figure 5 As shown, in the STPM module, the input feature map F∈RC1×H1×W1 Processing is performed in 4 parallel branches; each branch divides the input feature map into s×s sub-regions, and s∈S, S={1,2,3,4} represents four pyramid scales;

[0099] In a branch of scale s, each region is defined as A CBAM is added to each branch. In each pyramid branch, the CBAM is applied to all sub-regions to generate an updated residual feature map. as well as

[0100] The updated residual feature maps obtained from four different scales are upsampled to obtain features with the same spatial dimensions as the initial feature map F

[0101] The four feature maps Zs are concatenated and input into the convolutional layer (C, 1×1 / 1) to generate the final residual feature map Y, which is summed element-wise with the input feature map F to obtain the updated feature map F s .

[0102] Specifically, the present invention develops STPM, which aggregates different regions to obtain global information. STPM improves the model's ability to recognize small objects by aggregating spatiotemporal context information at multiple scales. The module seamlessly integrates features from four different pyramid scales and produces multi-scale attention features by fusing spatiotemporal attention contexts at different scales. The present invention adds a CBAM to each branch, enabling STPM to detect high-resolution feature maps to obtain finer details while capturing a wider range of backgrounds on low-resolution feature maps to better understand the environmental context of the target.

[0103] Furthermore, in order to effectively utilize global information and local information so that the network can quickly and accurately focus on key areas and targets, the present invention combines the features processed by CBAM and STPM. Therefore, the final output of the entire joint attention module is expressed as F j These aggregate features are calculated as shown in equation (4):

[0104] F j =λ1(α(CBAM(F c ))+β(STPM(F s ))) (4)

[0105] Among them, F c and F sRepresent the outputs of CBAM and STPM respectively. Both α1 and β involve 3×3 convolution, batch normalization and ReLU activation, while λ1 involves batch normalization and 1×1 convolution. CBAM(.) represents the calculation of the convolutional block attention module, and STPM(.) is the calculation of the spatiotemporal pyramid module. These aggregated features are then used in subsequent operations, enabling the network to dynamically prioritize important areas and target objects. The present invention obtains the aggregated feature F by joint attention calculation in the template branch. jt , and then the aggregated features F are obtained by common attention calculation in the search branch js .

[0106] like Figure 6 As shown, F jt and F js The element-by-element multiplication of cf , the encoder of the present invention combines the combined feature map F of two adjacent frames cf As input. To prevent repeated calculation, the features of the current frame extracted are temporarily stored and reused in the next frame. The decoder is parallelized into two branches. Template feature query q tt is input into the second decoder of the transformer to generate a tracking box. At the same time, the learnable feature query q lo Input into the Transformer first decoder to generate a detection box. The learnable feature query is a set of learnable parameters that are trained together with all other parameters in the network.

[0107] In one embodiment, the encoder includes a self-attention layer, a residual and normalization layer, a position encoding generation module, a self-attention layer, and a residual and normalization layer connected in sequence; in S400, the first aggregate feature and the second aggregate feature are fused and encoded by element-by-element multiplication to obtain an encoded feature map, specifically:

[0108] The encoder will be composed of the combined feature map F cf The linearly transformed Q, K, and V are used as input, and a set of attention weights are obtained through Softmax operation. The weights are multiplied by V to obtain a vector with attention weights. The specific calculation process is as follows:

[0109]

[0110] Among them, Q, K, and V represent query vector, keyword vector, and value vector respectively. is the key vector dimension, A K→Q is the similarity matrix between Q and K;

[0111] Using a single-head attention mechanism, Q, K, and V are input into the self-attention layer, and the relevance of each query to all keys is calculated through linear transformation and attention mechanism, and the relevance is used as the weight to perform weighted summation of the values ​​to generate the final output representation H1, specifically:

[0112] H1=Ins.Norm(Attention(Q1,K1,V1)+F cf ) (7)

[0113] Among them, F cf is the first aggregate feature F jt and the second aggregate feature F js The fusion feature is obtained by element-wise multiplication of , Q1, K1, V1 come from the linear transformation of the fusion feature, Attention(.) is the attention calculation, and the calculation process is shown in equation (6), Ins.Norm(·) represents the L2 normalization of all embedded image blocks;

[0114] H1 is input into the positional encoding generation module, and after a series of linear transformations, Q2, K2, and V2 are calculated and then input into the self-attention layer for residual connection and normalization to obtain H2:

[0115] H2=Ins.Norm(Attention(Q2,K2,V2)+PEGT) (8)

[0116] like Figure 7 As shown, the input sequence In the three-dimensional image space, it is reshaped into Then, the present invention iteratively uses grouped convolution on the local segment in S1′ to generate the position code P and S1′ are added together to form a small residual block, which is activated by the GeLU function and reassembled into the form of the input sequence. Equation 5 represents the calculation process of PEGT:

[0117]

[0118] Among them, λ2 represents the conversion of a two-dimensional sequence into a three-dimensional tensor, Representing the conversion of a 3D tensor into a 2D sequence, PEGT can be efficiently implemented through zero-padded 2D convolution with kernel size k, padding value (k-1) / 2, and k≥3.

[0119] In the face of occlusion, the use of fixed-length positional encodings may affect the generalization ability of the model. To address this issue, the present invention adopts a simple and effective method to use zero padding to increase the flexibility of capturing relative position information between markers. Grouped convolution is introduced to handle variable-sized input data. Specifically, the present invention applies grouped convolution to the input sequence, ensuring that each convolution group is responsible for processing a specific range of relative positional relationships, thereby more efficiently handling variable-length input and occlusion scenarios. This not only helps manage the uncertainty introduced by occlusion, but is also more efficient in terms of computation and parameter count compared to traditional methods. Then, residual connections are introduced to retain key details. Specifically, residual connections are applied between S1′ and P to enable information to be passed more directly within the network. This helps improve gradient flow during the training process of deep networks. Retaining low-level features when handling occlusion and residual connections enables the model to better capture the local structure of the target and enhance robustness to occlusion. In addition, the introduction of the nonlinear GeLU activation function contributes to the model's ability to learn more complex content patterns, especially when occlusion is involved. This activation function enables the model to capture nonlinear relationships in the data and enhances its adaptability to complex occlusion scenes.

[0120] In one embodiment, the first decoder includes a self-attention layer, a first residual and normalization layer, a cross attention layer, a second residual and normalization layer, a feedforward neural network layer, and a third residual and normalization layer; in S400, the first decoder decodes the object feature according to the encoded feature map and the learned target query, specifically:

[0121] The first decoder feeds the learned target query into the self-attention layer for self-attention calculation and performs residual connection and normalization to obtain H3:

[0122] H3=Ins.Norm(Attention(Q3,K3,V3)+q lo ) (9)

[0123] Among them, Q3, K3 and V3 are the target query q lo Obtained after linear transformation;

[0124] Q4, K4, and V4 are jointly fed into the criss-cross attention module to calculate the temporal criss-cross attention matrix and then normalized to obtain Hc1:

[0125] H c1 =Ins.Norm(Attention(Q4,K4,V4)+H3) (10)

[0126] Among them, Q4 is obtained by linear transformation of H3, V4 and K4 are obtained by linear transformation of H2, and finally, Hc1 After inputting into the feedforward network, residual connection and normalization are used to obtain F d , the calculation formula (11) is as follows:

[0127] F d =Ins.Norm(FFN(H c1 )+H c1 ) (11)

[0128] In one embodiment, the second decoder includes a self-attention layer, a first residual and normalization layer, a cross attention layer, a second residual and normalization layer, a feedforward neural network layer, and a third residual and normalization layer; in S400, the second decoder decodes according to the encoded feature map and the template feature query to obtain the tracking feature, specifically:

[0129] The second decoder converts the template feature query q tt As input, it obtains H4 through residual connection and layer normalization after self-attention calculation:

[0130] H4=Ins.Norm(Attention(Q5,K5,V5)+q tt ) (12)

[0131] Among them, Q5, K5 and V5 are in the template feature query q tt Obtained after linear transformation;

[0132] Q6 obtained by mapping H4 is cross-calculated with K6 and V6 mapped by the encoding feature H2, and then connected and normalized with the template feature residual after self-attention to obtain H c2 :

[0133] H c2 =Ins.Norm(Attention(Q6,K6,V6)+H4) (13)

[0134] Among them, K6 and V6 obtained by mapping from the encoder feature H2 are cross-attended with Q6 obtained from the mapping of the feature H4 in the second decoder. This process produces H c2 ;

[0135] Finally, the feature H c2 Input into the feedforward network and normalized to obtain F t , the calculation process is as follows:

[0136] F t =Ins.Norm(FFN(H c3 )+H c2 ) (14)

[0137] Among them, FFN(.) refers to the calculation of feedforward neural network, which involves linear transformation and activation function.

[0138] Specifically, the object feature F obtained by decoder 1 d The detection box is generated through feedforward neural network processing. At the same time, the tracking feature F obtained by encoder 2 t The detection box is processed by a feed-forward neural network to generate a tracking box. Subsequently, the detection box is associated with the tracking box using the box-IoU matching method to obtain the final tracking result. The KuhnMunkres (KM) algorithm is used to evaluate the IoU similarity between the detection box and the tracking box, which helps in the matching process.

[0139] In one embodiment, the confidence branch includes a three-layer perceptron and an S-type function group. In S400, the first decoder combines the confidence branch and updates the target tracking model in real time based on the dynamic template update strategy, specifically:

[0140] The confidence score c of each target generated detection box is obtained through the confidence branch j , where j = 1, 2, ..., n, the individual confidence of each target is fused by weighted average to obtain the global confidence:

[0141]

[0142] When the global confidence score is higher than the set threshold δ, it indicates that there is no occlusion and the target tracking model starts to be updated. Otherwise, the target tracking model is updated again after τ frames.

[0143] Specifically, in actual complex scenes, the planned template update may not always be reasonable. For example, performing a template update when occlusion occurs may result in a decrease in tracking accuracy. Therefore, the present invention proposes a dynamic template update strategy (DTUS) for template update, which is to ensure that under low confidence scores, the algorithm minimizes updates and reduces inaccuracies in the template. DTUS is to balance the requirements of real-time performance and stability to ensure effective tracking in occluded scenes. The entire process of the strategy is as follows Figure 8 shown.

[0144] The experimental environment of the present invention is carried out on a deep learning platform. The GPU of the deep learning platform is NVIDIA Geforce RTX3090. The built-in software environment is Ubuntu20.04 LTS system. The Python environment is Python3.7, CUDA11.3 and PyTorch1.10. For feature extraction, the present invention uses a lightweight ShuffleNet V2 network that can extract multi-scale feature maps. As part of preprocessing, the present invention resizes the image to 224×224. The Adam optimizer is used to train 100 epochs with a learning rate of 1e-4 and a batch size of 16. Fig. 9 The loss convergence curve of the trained model is shown.

[0145] Table 1 Backbone network ablation experiment results

[0146]

[0147] The present invention conducts comparative experiments on the MOT17 dataset to evaluate the effectiveness of the improved ShuffleNet V2 network, and the experimental results are shown in Table 1. Here, the ShuffleNet V2 network combined with the joint attention module is called ShuffleNetX, aiming to confirm the effectiveness and real-time performance of ShufflenetX. As shown in Table 2, compared with ResNet-50, the ShuffleNet X network shows excellent performance in terms of success rate and lightweight design. This improvement is attributed to the joint attention module, which provides a more robust feature representation for the target tracking task.

[0148] Table 2 Comparison of the effects of each module

[0149]

[0150] A detailed performance analysis is performed on each model, with a focus on ResNet-50, ShuffleNetX, DTUS, EC-DC (traditional encoder-decoder structure), and SFTrans structure. The specific ablation experiment results are shown in Table 2, where √' indicates that the current strategy is used, and no √' indicates that the current strategy is not used. It can be seen that the lightweight design of ShuffleNetX improves the speed of the model, and the template update module enhances the real-time adaptability of the model, especially in dealing with spatiotemporal changes. The introduction of the SFTrans structure enhances the model's ability to model spatiotemporal relationships. Model5 integrates all innovative modules, achieves the best performance, and highlights the key role of these modules in multi-target tracking.

[0151] The present invention conducts a series of experiments to more intuitively understand the effectiveness of the joint attention module in improving detection accuracy. Fig.10 In (d), the joint attention mechanism not only accurately focuses but also generates a specific region of interest for each target, represented by the orange area. The warmth of the colors on the heat map indicates the level of attention in the region of interest. The deepening of these colors shows that the joint attention mechanism effectively focuses on the relevant regions, enhancing its applicability to target tracking models.

[0152] In order to strike a balance between speed and accuracy, and to further study how changes in the update interval τ and the global confidence parameter δ affect the robustness of the algorithm, the present invention conducts a template update experiment on the MOT17 dataset, and the results are shown in Table 3. After testing, the present invention finds that update intervals τ of 30 frames, 40 frames, and 50 frames produce satisfactory results. And the parameter settings of τ equal to 40 frames and δ equal to 0.5 bring greater improvements in algorithm accuracy. This shows that this configuration is conducive to the balance between accuracy and real-time performance in target tracking tasks. Therefore, the combination of τ equal to 40 frames and δ equal to 0.5 is finally selected as the optimal configuration.

[0153] Table 3 Results of different hyperparameters in template updating

[0154]

[0155] The present invention conducted a shielding experiment on the sidewalk next to the garden, and the results are as follows: Fig.11 As shown. This verifies the effectiveness of DTUS and PEGM proposed in the present invention in dealing with occlusion. The algorithm without dynamic template update strategy and PEGM in the present invention is called LTJPA-, while the algorithm with these strategies is called LTJPA-. Fig.11 In (a), after occlusion, the pedestrian with ID 43 in frame 366 becomes ID 39, and the pedestrian with ID 22 becomes ID 37. This shows that without the dynamic template update strategy and PEGM, occlusion can lead to misjudgment and change of target identity. Fig.11 As shown in (b), pedestrians with IDs 43 and 22 retain their original identities after being occluded in frames 366 and 452, respectively. This shows that the LTJPA algorithm, which includes a dynamic template update strategy and PEGM, exhibits stronger robustness when dealing with occlusion and more effectively maintains the consistency of the target identity.

[0156] The present invention compares the tracker of the present invention with the most advanced algorithms on the MOT16, MOT17 and MOT20 test datasets, and presents the tracking results. In the MOT16, 17 and 20 datasets, FairMOT achieves better results in HOTA and MOTA than other algorithms except the present invention. The method of the present invention not only achieves the highest accuracy, but also shows excellent performance in terms of frames per second, highlighting its relative real-time efficiency.

[0157] In order to comprehensively evaluate the performance of the proposed algorithm, the present invention compares it with the FairMOT algorithm in real scenarios. The experimental results are as follows: Fig.12 This comparison highlights the robustness and effectiveness of the proposed algorithm in real scenes, especially in scenes with small objects and occlusions.

[0158] Compared with the prior art, the present invention has the following beneficial effects:

[0159] 1) A joint attention module is proposed and combined with the ShuffleNet V2 network to achieve a lightweight design while accurately and efficiently capturing the key features of the target object. A STPM is proposed to capture feature information at different time scales and spatial resolutions. Its combination with CBAM further enhances the adaptability of the ShuffleNet V2 network to various visual tasks.

[0160] 2) SFTrans is designed based on the original transformer structure. In order to improve the object recognition ability, a position encoding generator module is proposed. When occlusion occurs, zero padding and grouped convolution are used to capture the relative position information between markers. This approach reduces the number of parameters and computational requirements of the tracker.

[0161] 3) A dynamic template update strategy is proposed. During occlusion, the confidence branch calculates a global confidence score. If the global confidence score is higher than the threshold, the target tracking model is updated immediately; otherwise, the target tracking model waits for τ frames before updating, which improves the accuracy of the model.

[0162] The above is a detailed introduction to a lightweight multi-target tracking method based on an attention mechanism provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A lightweight multi-target tracking method based on attention mechanism, characterized in that: The method comprises the following steps: S100: Acquire an image containing multiple targets, and input the images into a template branch and a search branch for preprocessing, respectively, to obtain a first image and a second image; S200: Input the first image and the second image into the corresponding ShuffleNet V2 lightweight network to extract the attention-rich target features, generate a feature map of a specific size, and obtain a first feature map and a second feature map; S300: Inputting the first feature map and the second feature map into the corresponding joint attention module to capture the long-distance spatiotemporal relationship, and obtaining the first aggregate feature and the second aggregate feature; S400: The first aggregate feature and the second aggregate feature are fused and encoded by element multiplication to obtain an encoded feature map, which is used as a key for cross attention in the parallel first decoder and the second decoder; the first decoder decodes the object feature according to the encoded feature map and the learned target query, and combines the confidence branch to update the target tracking model in real time based on the dynamic template update strategy; the second decoder decodes the encoded feature map and the template feature query to obtain the tracking feature; S500: The object features obtained by the first decoder are processed by a feedforward neural network to generate a detection frame, the tracking features obtained by the second encoder are processed by a feedforward neural network to generate a tracking frame, and the detection frame is associated with the tracking frame using a frame-IoU matching method to obtain a final tracking result; The ShuffleNet V2 lightweight network in S200 includes a convolutional layer, a maximum pooling layer, a first extraction stage, a second extraction stage, and a third extraction stage. Each extraction stage includes a first block and a second block. The first extraction stage includes a stack of 1 first block and 3 second blocks. The second extraction stage includes a stack of 1 first block and 7 second blocks. The third extraction stage includes a stack of 1 first block and 3 second blocks. In the first block, the input features are subjected to channel segmentation operation, and the feature channels are divided into two branches. The number of channels in each branch is half of the original number of channels. The right branch undergoes 1×1 convolution, 3×3 depth convolution, and 1×1 convolution, while the left branch remains unchanged through identity mapping. After cascading and channel shuffling, the characteristics of the branches are effectively fused together. In the second block, the input feature map is fed into two branches. The left branch includes a 3×3DWConv with a stride of 2 and a conventional 1×1 convolution operation, while the right branch undergoes a 1×1 convolution, followed by a 3×3 depthwise convolution, and then a 1×1 convolution. After channel concatenation and channel shuffling, the channel count of the output feature map is doubled. When the input feature is the first image, the output feature map is the first feature map, and when the input feature is the second image, the output feature map is the second feature map. The joint attention module in S300 includes a CBAM module and an STPM module. The CBAM module includes a channel attention module and a spatial attention module. The channel attention module is used to perform channel attention enhancement on the input features to obtain channel attention features. The channel attention features are multiplied with the input features as the input of the spatial attention module. The spatial attention module is used to perform spatial attention enhancement on the input features to obtain spatial attention features. The spatial attention features are multiplied with the input features of the spatial attention module to obtain the output feature map F. c The STPM module consists of four parallel branches, each of which is embedded with a CBAM module, which seamlessly integrates features of four different pyramid scales and generates multi-scale attention features F by fusing spatiotemporal attention contexts at different scales. s ;Finally, the features processed by CBAM and STPM are combined and the aggregated features are output; The channel attention module compresses the global spatial information contained in the input feature map F through global maximum pooling and global average pooling, generating two different one-dimensional feature maps F1 and F2, which are derived through a shared multi-layer perceptron MLP. The multi-layer perceptron MLP includes a fully connected layer and a ReLU nonlinear activation function. After the element summation operation across channels, the two one-dimensional feature maps are then normalized using the sigmoid function. The normalization process generates a weighted statistic for each channel represented as Mc. The output feature map F′ is obtained by multiplying Mc and the input feature map F. The specific process for calculating the channel attention feature Mc is: Among them, AvgPool refers to average pooling; MaxPool refers to maximum pooling; MLP is multi-layer perceptron calculation; F is the input feature map; H1 and W1 represent the height and width of the feature map respectively; Represents the value at position (i0, j0) in the feature map.

2. The method according to claim 1, characterized in that The feature map F′ output by the channel attention module is used as the input of the spatial attention module, and the channel-based global maximum pooling and global feature pooling are performed to obtain the feature map F3∈R 1 ×H1×W1 and F4∈R 1×1×W1 , then, these two feature maps are concatenated to generate an effective feature descriptor F5∈R 2×H1×W1 , perform a 7×7 convolution operation on F5 to reduce it to one channel, and generate a spatial attention feature Ms through an S-type operation, and multiply the spatial attention feature Ms with the input feature map F′ of the channel attention module to obtain the final generated aggregate feature Fc; The specific process for calculating the spatial attention feature Ms is: M s (F′)=σ{f 7×7 [AvgPool(F′);MaxPool(F′)]} Among them, f 7×7 Represents a convolution layer with a convolution kernel of 7×7.

3. The method according to claim 2, characterized in that In the STPM module, the input feature map F∈R C1×H1×W1 Processing is performed in 4 parallel branches; each branch divides the input feature map into s×s sub-regions, and s∈S, S={1,2,3,4} represents four pyramid scales; In a branch of scale s, each region is defined as A CBAM is added to each branch. In each pyramid branch, the CBAM is applied to all sub-regions to generate an updated residual feature map. as well as The updated residual feature maps obtained from four different scales are upsampled to obtain features with the same spatial dimensions as the initial feature map F The four feature maps Zs are concatenated and input into the convolutional layer (C, 1×1 / 1) to generate the final residual feature map Y, which is summed element-wise with the input feature map F to obtain the updated feature map F s .

4. The method according to claim 3, characterized in that The encoder includes a self-attention layer, a residual and normalization layer, a position encoding generation module, a self-attention layer, and a residual and normalization layer connected in sequence; in S400, the first aggregate feature and the second aggregate feature are fused and encoded by element-by-element multiplication to obtain an encoded feature map, specifically: The encoder will be composed of the combined feature map F cf The linearly transformed Q, K, and V are used as input, and a set of attention weights are obtained through Softmax operation. The weights are multiplied by V to obtain a vector with attention weights. The specific calculation process is as follows: Among them, Q, K, and V represent query vector, keyword vector, and value vector respectively. is the key vector dimension, A K→Q is the similarity matrix between Q and K; Using a single-head attention mechanism, Q, K, and V are input into the self-attention layer, and the relevance of each query to all keys is calculated through linear transformation and attention mechanism, and the relevance is used as the weight to perform weighted summation of the values ​​to generate the final output representation H1, specifically: H1=Ins.Norm(Attention(Q1,K1,V1)+F cf )(7) Among them, F cf is the first aggregate feature F jt and the second aggregate feature F js The fusion feature is obtained by element-wise multiplication of , Q1, K1, V1 come from the linear transformation of the fusion feature, Attention(.) is the attention calculation, and the calculation process is shown in equation (6), Ins.Norm(·) represents the L2 normalization of all embedded image blocks; H1 is input into the positional encoding generation module, and after a series of linear transformations, Q2, K2, and V2 are calculated and then input into the self-attention layer for residual connection and normalization to obtain H2: H2=Ins.Norm(Attention(Q2,K2,V2)+PEGT) (8).

5. The method according to claim 4, characterized in that The first decoder includes a self-attention layer, a first residual and normalization layer, a cross attention layer, a second residual and normalization layer, a feedforward neural network layer, and a third residual and normalization layer; in S400, the first decoder decodes the object features according to the encoded feature map and the learned target query, specifically: The first decoder feeds the learned target query into the self-attention layer for self-attention calculation and performs residual connection and normalization to obtain H3: H3=Ins.Norm(Attention(Q3,K3,V3)+q lo ) (9) Among them, Q3, K3 and V3 are the target query q lo Obtained after linear transformation; Q4, K4, and V4 are jointly fed into the criss-cross attention module to calculate the temporal criss-cross attention matrix and then residually connected and normalized to obtain H c1 : H c1 =Ins.Norm(Attention(Q4,K4,V4)+H3) (10) Among them, Q4 is obtained by linear transformation of H3, V4 and K4 are obtained by linear transformation of H2, and finally, H c1 After inputting into the feedforward network, residual connection and normalization are used to obtain F d , the calculation formula (11) is as follows: F d =Ins.Norm(FFN(H c1 )+H c1 ) (11)。 6. The method according to claim 5, characterized in that The second decoder includes a self-attention layer, a first residual and normalization layer, a cross attention layer, a second residual and normalization layer, a feedforward neural network layer, and a third residual and normalization layer; in S400, the second decoder decodes according to the encoded feature map and the template feature query to obtain the tracking feature, specifically: The second decoder converts the template feature query q tt As input, it obtains H4 through residual connection and layer normalization after self-attention calculation: H4=Ins.Norm(Attention(Q5,K5,V5)+q tt )(12) Among them, Q5, K5 and V5 are in the template feature query q tt Obtained after linear transformation; Q6 obtained by mapping H4 is cross-calculated with K6 and V6 mapped by the encoding feature H2, and then connected and normalized with the template feature residual after self-attention to obtain H c2 : H c2 =Ins.Norm(Attention(Q6,K6,V6)+H4)(13) Among them, K6 and V6 obtained by mapping from the encoder feature H2 are cross-attended with Q6 obtained from the mapping of the feature H4 in the second decoder. This process produces H c2 ; Finally, the feature H c2 Input into the feedforward network and normalized to obtain F t , the calculation process is as follows: F t =Ins.Norm(FFN(H c3 )+H c2 )(14) Among them, FFN(.) refers to the calculation of feedforward neural network, which involves linear transformation and activation function.

7. The method according to claim 6, characterized in that The confidence branch includes a three-layer perceptron and an S-type function. The first decoder in S400 combines the confidence branch and updates the target tracking model in real time based on the dynamic template update strategy, specifically: The confidence score c of each target generated detection box is obtained through the confidence branch j , where j = 1, 2, ..., n, the individual confidence of each target is fused by weighted average to obtain the global confidence: When the global confidence score is higher than the set threshold δ, it indicates that there is no occlusion and the target tracking model starts to be updated. Otherwise, the target tracking model is updated again after τ frames.

Images