An adaptive Transformer target tracking system and method based on sparse representation

Abstract

A sparse representation-based adaptive Transformer target tracking system and method, the system comprising: a feature extraction module, the feature extraction module comprising a block embedding layer, a Transformer block, a sparse spatial attention module, a sparse adaptive spatial pruning module, a candidate region reconstruction module, and a prediction head module; the method comprising: simultaneously inputting a search region image and a template set image into the feature extraction module, performing feature extraction and outputting a search region feature map and a class token; sending the output search region feature map and class token into the prediction head module, calculating a predicted target bounding box using a probability head, a scale head, and a deviation head in the prediction head module; calculating the search region quality using a quality head; cropping the search region image of the current frame with the predicted target bounding box as the center, and updating the dynamic template that has not been updated the longest in the template set image; the present invention has the advantages of low computational complexity, fast data processing speed, and high tracking accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to an adaptive Transformer target tracking system and method based on sparse representation. Background Art

[0002] As one of the hot tasks in the field of computer vision, target tracking has not only been widely studied in theory, but also has wide applications in security monitoring, autonomous driving, virtual reality and other fields. Target tracking is divided into single target tracking and multi-target tracking. Target tracking refers to predicting the position of any target in subsequent video frames after a given initial frame.

[0003] In the Transformer-based target tracking method, due to the multiple use of self-attention, the target area is easily disturbed by similar targets in the non-target area during the attention calculation process, resulting in insufficient focus on the target area, blurring the boundary between the target and the background, and thus causing inaccurate target positioning. In addition, when performing spatial pruning on image features, existing tracking methods do not consider the different amount of discriminant information required to locate the target in images with different degrees of complex backgrounds, and all prune at a fixed ratio. This will lead to the loss of effective discriminant information in images with complex backgrounds, or the problem of excessive retention of background redundant information in simple background images. Existing tracking methods lack the utilization of temporal information and global spatial features, resulting in serious impact on tracking accuracy when processing complex scenes, especially when the target appearance changes drastically.

[0004] Invention patent CN202310172562.3 provides a target tracking method and system based on a dual attention feature fusion network, which constructs a multi-scale feature fusion network based on Transformer; the features in the template feature map are learned through the encoder to obtain a high-confidence target proposal box; the target proposal box is input into the decoder, and the search area features are learned and fused to obtain the target proposal box with the highest confidence value; the attention is quickly focused on the region of interest and the structured spatial information and local information are captured, and the structured spatial information in the encoder is used to explore the global context information; the features after the fusion of the template features and the search area features are sent to the prediction head to obtain the maximum response position of the tracking target in the search area for tracking. However, due to the multiple use of self-attention, the target area is easily interfered with by similar targets in the non-target area during the attention calculation process, resulting in insufficient focus on the target area, blurring the boundary between the target and the background, and thus resulting in inaccurate target positioning. In addition, the lack of utilization of temporal information and global spatial features leads to a serious impact on the tracking accuracy when processing complex scenes, especially when the target appearance changes dramatically. Summary of the Invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide an adaptive Transformer target tracking system and method based on sparse representation. In the process of feature extraction, a sparse spatial attention module is established to enhance target features and suppress background features, and sparse adaptive spatial pruning is performed on non-target areas to adaptively reduce the impact of complex background on the target according to the complexity of the image. The template set is updated based on the dynamic template of global quality assessment. It has the advantages of high tracking accuracy, low computational complexity, fast data processing speed and wide range of applications.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] An adaptive Transformer target tracking system based on sparse representation, comprising:

[0008] The feature extraction module, including the block embedding layer, the Transformer block, the sparse spatial attention module, the sparse adaptive spatial pruning module, and the candidate region reconstruction module, is used to extract features from the input search region image and the template set image, and output the extracted class token and search region image features to the prediction head module; wherein:

[0009] The block embedding layer is used to convert the input search area image and template image into tokens;

[0010] Transformer block, used to extract features from the tokens converted by the block embedding layer;

[0011] The sparse spatial attention module is used to extract sparse features related to the foreground using the sparse feature representation method, and use the extracted sparse features to weight some of the original features extracted by the Transformer block, thereby enhancing the features of the target area and suppressing the features of the non-target area;

[0012] Sparse adaptive spatial pruning module, which is used to adaptively discard redundant background information according to the complexity of the image in the search area;

[0013] Candidate region reconstruction module, used to reconstruct candidate regions to fill missing locations;

[0014] The prediction head module includes a probability head, a deviation head, a scale head, and a quality head. The probability head, the deviation head, and the scale head are used to calculate the predicted target bounding box. The quality head is used to calculate the quality of the search area and update the dynamic template that has not been updated for the longest time in the template set image.

[0015] The sparse spatial attention module first performs sparse feature representation on the search area feature s2 to obtain sparse coding Then the sparse coding is average pooled along the channel to obtain the spatial statistical information of the search area features Then perform the maximum and minimum normalization operation on m1 to obtain the attention weight Finally, m2 is used to weight the input search area feature s2 and output the sparse attention search area feature map s′2, that is, s′2=m2s2, where m2 represents the attention weight;

[0016] The specific implementation steps of the sparse feature representation are as follows: First, use the linear layer W sr Process the search area feature s2 to obtain y sr1 , for y sr1 Use the soft threshold activation function θ to get z1, and then the linear layer Q sr Process z1 and then add it to y sr1 After addition, the activation function θ is executed to obtain z2, and finally z2 passes through the linear layer Q sr Process and then sr1 Add and execute the activation function θ to obtain the sparse coding α of feature s1. The calculation formula of the soft threshold activation function θ is as follows:

[0017]

[0018] Where λ represents the threshold of the soft threshold activation function θ.

[0019] The sparse adaptive spatial pruning module first sorts the weights m2 obtained by the sparse spatial attention module from large to small to obtain the confidence score. Then, the scores are summed to obtain the total energy E of the candidate region. ρ%×E is taken as the energy threshold, where ρ is the proportion of retained energy. Finally, the top K tokens with the largest energy in the candidate region are retained to satisfy And discard the remaining tokens.

[0020] An adaptive Transformer target tracking method based on sparse representation includes the following steps:

[0021] Step 1. Input the search area image and the template set image into the feature extraction module at the same time, perform feature extraction and output the search area feature map s′3 and class token;

[0022] Step 2. The search area feature map s′3 and the class token output by the feature extraction module obtained in step 1 are fed into the prediction head module. The predicted target bounding box is calculated using the probability head, scale head, and bias head in the prediction head module. The quality of the search area is calculated using the quality head.

[0023] Step 3. If the quality of the search area predicted for the current frame calculated in step 2 is higher than the quality of the historical search area, the search area image of the current frame is first cropped with the predicted target bounding box as the center, and the size of the cropped search area image is the template size. Then, the dynamic template that has not been updated for the longest time in the template set image is updated, and the next frame prediction is performed based on the updated template set image. If the quality of the search area predicted for the current frame calculated in step 2 is equal to or lower than the quality of the historical search area, it is determined that the search area image predicted for the current frame is too complex and the target cannot be accurately located. Therefore, the template set image is not modified and the next frame prediction is directly performed.

[0024] The specific steps of the feature extraction module in step 1 extracting features from the search area image and the template set image and outputting the search area feature map s′3 and the class token are as follows:

[0025] 1.1 Input template set image and search area image, where the template set image includes an initial frame template and T-1 dynamic templates. The size of each template in the template set image is 3×H t ×W t , the search area size is 3×H s ×W s , where H t Indicates the height of the template, W t Indicates the width of the template, H s , represents the height of the search area, W s Indicates the width of the search area;

[0026] 1.2 Input each template in the template set image and the search area image into the block embedding layer respectively to obtain the template feature map set s1 and the search area feature map t1, with sizes of T×N respectively. t ×C、N s ×C; then add the template feature map set s1 and the search area feature map t1 to the position code respectively, and then splice all feature maps and learnable vector class token together to obtain the spliced feature map x1, where the size of the class token is 1×C and the size of the spliced feature map x1 is N x ×C, where N x =T×N t +N s +1, N t Indicates the number of tokens in the template feature map set, C indicates the number of channels in the feature map, N s Indicates the number of tokens in the search area feature map, N x Indicates the number of tokens of the concatenated feature map x1;

[0027] 1.3 Send the spliced feature map x1 to the N1 layer Transformer block for feature extraction, and output the extracted spliced feature map x2;

[0028] 1.4 Split the search region feature map s2 from the extracted spliced feature map x2, input the search region feature map s2 into the sparse spatial attention module for processing, and obtain the sparse attention search region feature map s′2, and then input the sparse attention search region feature map s′2 into the sparse adaptive spatial pruning module to obtain the pruned search region feature map s′2′;

[0029] 1.5 Re-join the pruned search region feature map s′2′ and the remaining features of the extracted spliced feature map x2 to obtain a new spliced feature map x′2;

[0030] 1.6 Input the new spliced feature map x′2 into the N2-layer Transformer block for feature extraction, output the extracted spliced feature map x3, split it into class token, search area feature map s3, and template feature map set t3, input the search area feature map s3 into the candidate region reconstruction module, reconstruct its spatial features to obtain the candidate region reconstruction feature map s3′;

[0031] 1.7 The output of the feature extraction module is the final class token and the candidate region reconstruction feature s3′.

[0032] The specific method of calculating the predicted target bounding box using the probability head, scale head, and deviation head in the prediction head module in step 2, and calculating the search area quality using the quality head is as follows:

[0033] 2.1 For the probability head, first, the candidate region reconstruction feature s3′ and the class token are subjected to mutual attention calculation to obtain the weighted candidate region reconstruction feature s4. The calculation formula is as follows:

[0034] s4=s3′*(s3′×classToken Τ )

[0035] Among them, × represents matrix multiplication, * represents element multiplication;

[0036] Then, the weighted candidate region reconstruction feature s4 is input into the probability head, and the center position probability score is output after calculation by the three-layer feedforward neural network: Among them, H s 、W s Represent the height and width of the search area respectively, and d represents the network step size;

[0037] 2.2 For the deviation head and scale head, the candidate region reconstruction feature s3′ is input and the deviation prediction is obtained after calculation by the three-layer feedforward neural network Scale prediction

[0038] 2.3 Then, the corresponding position of the maximum value in the center position probability score P is regarded as the target center position, that is, (x d ,y d )=argmax (x,y) P xy ,Finally, the target bounding box is calculated by the following formula:

[0039] (x,y,w,h)=(x d +O(0,x d ,y d ),y d +O(1,x d ,y d ),S(0,x d ,y d ),S(1,x d ,y d ))

[0040] Where x represents the horizontal coordinate of the center position of the target bounding box, y represents the vertical coordinate of the center position of the target bounding box, w represents the width of the target bounding box, and h represents the length of the target bounding box;

[0041] 2.4 For the quality head, its input is the class token obtained by the feature extraction module. After processing by the quality head, a quality assessment of the current search area is obtained, which is used to update the dynamic template that has not been updated for the longest time in the template set.

[0042] The specific steps of updating the dynamic template that has not been updated the longest in the template set image in step 3 are as follows:

[0043] 3.1 After completing the target bounding box prediction of the current search area, use the quality head to predict the quality of the search area image. If the search area quality q now Higher than the optimal historical search area quality q before , then set the dynamic template candidate as the current search area, and let the optimal historical search area quality q before The quality q of the current search area now ;

[0044] 3.2 Every M frames, perform template cropping on the dynamic template candidate to obtain a new dynamic template. The cropping is centered on the corresponding target bounding box, and then the new dynamic template is used to replace the dynamic template with the longest interval between the template set and the current frame, and the optimal historical search area quality q before Set to 0.

[0045] The training of the feature extraction module and the prediction head module is divided into two stages. The specific training steps are as follows:

[0046] (1) The first stage is to use the total loss function The probability head, scale head and deviation head in the feature extraction module and the prediction head module are comprehensively trained, and the total loss function It includes three types of loss functions: classification loss function, regression loss function, and sparse coding loss function; among them, the classification loss function uses the Gaussian weighted focal loss loss function L cls ; The regression loss function is derived from the regression L1 loss function And GIOU loss function L GIOU Composition; sparse coding loss function is composed of sparse L1 loss function And the fidelity loss function L FBI The total loss function is The formula is as follows:

[0047]

[0048] Among them, the classification loss function formula L cls as follows:

[0049]

[0050] Among them, α and β represent the hyperparameters that control the distribution of each point, and p xy is the probability that the predicted position is the target center, λ cls =1, λ GIOU =5, λ FBI =1, both are regularization parameters;

[0051] For each ground-truth target bounding box, the target center and its low-resolution variant Generate Gaussian ground truth heatmap using Gaussian kernel σ represents the standard deviation of the target scale adaptation;

[0052] The regression L1 loss function The formula is as follows:

[0053]

[0054] Among them, y b is the coordinate of the real target bounding box, x b is the predicted target bounding box coordinate;

[0055] The GIOU loss function L GIOU The formula is:

[0056]

[0057]

[0058] Where A is the area of the true target bounding box, B is the area of the predicted target bounding box, and C is the minimum convex closed box containing A and B;

[0059] The sparse L1 loss function The formula is as follows:

[0060]

[0061] Among them, α is sparse coding;

[0062] The fidelity loss function formula L FBI as follows:

[0063]

[0064] Among them, T is the sparse coding dictionary, s2 is the search area feature;

[0065] (2) The second stage is to train the quality head, freeze the rest of the parts, and use the quality evaluation loss function L quality For training, the total loss function is The formula is as follows:

[0066]

[0067] Among them, λ quality =1, is the regularization parameter;

[0068] For the quality evaluation function L in the second stage training quality The formula is as follows:

[0069] L quality = -plog(q)-(1-p)log(1-q)

[0070] Where p is the label, if the search area contains the target, p = 1, otherwise p = 0; q is the predicted search area quality of the quality head.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. By setting up a sparse spatial attention module to enhance target features and suppress background features, the present invention has the advantage of high tracking accuracy.

[0073] 2. By setting up a sparse adaptive spatial pruning module, unimportant background areas can be adaptively deleted according to the complexity of the image, thereby reducing the amount of calculation while ensuring tracking accuracy, so that the present invention has the advantage of fast data processing speed.

[0074] 3. By setting up a prediction head module to perform template updates for global quality assessment, the present invention can adapt to changes in target appearance in practical applications and has the advantage of a wide range of applications.

[0075] In summary, compared with the existing technology, the present invention sets a sparse spatial attention module to enhance target features and suppress background features, sets a sparse adaptive spatial pruning module to adaptively delete unimportant background areas according to the complexity of the image, and sets a prediction head module to perform template updates for global quality assessment. As a result, the present invention has the advantages of high tracking accuracy, low computational complexity, fast data processing speed and wide applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Flow chart of the method of the present invention.

[0077] Figure 2 This is a structural diagram of the feature extraction module of the present invention.

[0078] Figure 3 This is a structural diagram of the sparse spatial attention module of the present invention.

[0079] Figure 4 FIG. 1 is a schematic diagram of the tracking effect obtained through simulation in an embodiment of the present invention, wherein: Figure 4 -a is the template image of the first frame, Figure 4 -b is the search area image tracking effect diagram of the 12th frame, Figure 4 -c is the tracking effect of the search area image in the 24th frame. DETAILED DESCRIPTION

[0080] The present invention will be further described below in conjunction with the accompanying drawings.

[0081] The present invention provides an embodiment, referring to Figure 1 as well as Figure 2 , an adaptive Transformer target tracking system based on sparse representation, including:

[0082] The feature extraction module, including the block embedding layer, the Transformer block, the sparse spatial attention module, the sparse adaptive spatial pruning module, and the candidate region reconstruction module, is used to extract features from the input search region image and the template set image, and output the extracted class token and search region image features to the prediction head module; wherein:

[0083] The block embedding layer is used to convert the input search area image and template image into tokens;

[0084] Transformer block, used to extract features from the tokens converted by the block embedding layer;

[0085] The sparse spatial attention module is used to extract sparse features related to the foreground using a sparse feature representation method, and use the extracted sparse features to weight some of the original features extracted by the Transformer block, thereby enhancing the target area features and suppressing the non-target area features. By setting up the sparse spatial attention module, the present invention has the advantage of high tracking accuracy;

[0086] The sparse adaptive spatial pruning module is used to adaptively discard redundant background information according to the complexity of the search area image, thereby reducing the amount of calculation while ensuring tracking accuracy, giving the present invention the advantage of fast data processing speed;

[0087] A candidate region reconstruction module, used for reconstructing candidate regions to fill in missing positions;

[0088] The prediction head module includes a probability head, a deviation head, a scale head and a quality head. The probability head, the deviation head and the scale head are used to calculate the predicted target bounding box, and the quality head is used to calculate the quality of the search area. The dynamic template that has not been updated for the longest time in the template set image is updated, so that the present invention can adapt to changes in target appearance in practical applications and has the advantage of a wide range of uses.

[0089] refer to Figure 3 , the specific working method of the sparse spatial attention module is as follows:

[0090] First, the search area feature s2 is represented by sparse features to obtain sparse coding Then the sparse coding is average pooled along the channel to obtain the spatial statistical information of the search area features Then perform the maximum and minimum normalization operation on m1 to obtain the attention weight Finally, m2 is used to weight the input search area feature s2 and output the sparse attention search area feature map s′2, that is, s′2=m2s2, where m2 represents the attention weight;

[0091] The specific implementation steps of the sparse feature representation are as follows: First, use the linear layer W sr Process the search area feature s2 to obtain y sr1 , for y sr1 Use the soft threshold activation function θ to get z1, and then the linear layer Q sr Process z1 and then add it to ysr1 After addition, the activation function θ is executed to obtain z2, and finally z2 passes through the linear layer Q sr Process and then sr1 Add and execute the activation function θ to obtain the sparse coding α of feature s1. The calculation formula of the soft threshold activation function θ is as follows:

[0092]

[0093] Where λ represents the threshold of the soft threshold activation function θ.

[0094] The specific working method of the sparse adaptive spatial pruning module is as follows:

[0095] First, the weights m2 obtained by the sparse spatial attention module are sorted from large to small to obtain the confidence score. Then, the scores are summed to obtain the total energy E of the candidate region. ρ%×E is taken as the energy threshold, where ρ is the proportion of retained energy. Finally, the top K tokens with the largest energy in the candidate region are retained to satisfy And discard the remaining tokens.

[0096] If the image background is simple, the energy sum of a few candidate regions can reach the energy threshold, so more candidate regions will be discarded, greatly reducing the amount of calculation; on the contrary, for complex background images, more candidate regions are needed to reach the energy threshold, so fewer candidate regions are discarded. The retention of effective information can ensure that the model has high discriminability.

[0097] An adaptive Transformer target tracking method based on sparse representation includes the following steps:

[0098] Step 1. Input the search area image and the template set image into the feature extraction module at the same time, perform feature extraction and output the search area feature map s′3 and class token;

[0099] Step 2. The search area feature map s′3 and the class token output by the feature extraction module obtained in step 1 are fed into the prediction head module. The predicted target bounding box is calculated using the probability head, scale head, and bias head in the prediction head module. The quality of the search area is calculated using the quality head.

[0100] Step 3. If the quality of the search area predicted for the current frame calculated in step 2 is higher than the quality of the historical search area, the search area image of the current frame is first cropped with the predicted target bounding box as the center, and the size of the cropped search area image is the template size. Then, the dynamic template that has not been updated for the longest time in the template set image is updated, and the next frame prediction is performed based on the updated template set image. If the quality of the search area predicted for the current frame calculated in step 2 is equal to or lower than the quality of the historical search area, it is determined that the search area image predicted for the current frame is too complex and the target cannot be accurately located. Therefore, the template set image is not modified and the next frame prediction is directly performed.

[0101] The specific steps of the feature extraction module in step 1 extracting features from the search area image and the template set image and outputting the search area feature map s′3 and the class token are as follows:

[0102] 1.1 Input template set image and search area image, where the template set image includes an initial frame template and T-1 dynamic templates. The size of each template in the template set image is 3×H t ×W t , the search area size is 3×H s ×W s , where H t Indicates the height of the template, W t Indicates the width of the template, H s , represents the height of the search area, W s Indicates the width of the search area;

[0103] 1.2 Input each template in the template set image and the search area image into the block embedding layer respectively to obtain the template feature map set s1 and the search area feature map t1, with sizes of T×N respectively. t ×C、N s ×C; then add the template feature map set s1 and the search area feature map t1 to the position code respectively, and then splice all feature maps and learnable vector class token together to obtain the spliced feature map x1, where the size of the class token is 1×C and the size of the spliced feature map x1 is N x ×C, where N x =T×N t +N s +1, N t Indicates the number of tokens in the template feature map set, C indicates the number of channels in the feature map, N s Indicates the number of tokens in the search area feature map, N x Indicates the number of tokens of the concatenated feature map x1;

[0104] 1.3 Send the spliced feature map x1 to the N1 layer Transformer block for feature extraction, and output the extracted spliced feature map x2. In this embodiment, N1=4;

[0105] 1.4 Split the search region feature map s2 from the extracted spliced feature map x2, input the search region feature map s2 into the sparse spatial attention module for processing, and obtain the sparse attention search region feature map s′2, and then input the sparse attention search region feature map s′2 into the sparse adaptive spatial pruning module to obtain the pruned search region feature map s″2;

[0106] 1.5 Re-join the pruned search region feature map s″2 and the remaining features of the extracted spliced feature map x2 to obtain a new spliced feature map x′2;

[0107] 1.6 Input the new spliced feature map x′2 into the N2-layer Transformer block for feature extraction, output the extracted spliced feature map x3, split it into class token, search area feature map s3, and template feature map set t3, input the search area feature map s3 into the candidate region reconstruction module, reconstruct its spatial features to obtain the candidate region reconstruction feature map s′3. In this embodiment, N2 = 8;

[0108] 1.7 The output of the feature extraction module is the final class token and the candidate region reconstruction feature s′3.

[0109] The specific method of calculating the predicted target bounding box using the probability head, scale head, and deviation head in the prediction head module in step 2 and calculating the search area quality using the quality head is as follows:

[0110] 2.1 For the probability head, first, the candidate region reconstruction feature s′3 and the class token are subjected to mutual attention calculation to obtain the weighted candidate region reconstruction feature s4. The calculation formula is as follows:

[0111] s4=s′3*(s′3×classToken Τ )

[0112] Among them, × represents matrix multiplication, * represents element multiplication;

[0113] Then, the weighted candidate region reconstruction feature s4 is input into the probability head, and the center position probability score is output after calculation by the three-layer feedforward neural network: Among them, H s 、W s Represent the height and width of the search area respectively, and d represents the network step size;

[0114] 2.2 For the deviation head and scale head, the candidate region reconstruction feature s′3 is input and the deviation prediction is obtained after calculation by the three-layer feedforward neural network Scale prediction

[0115] 2.3 Then, the corresponding position of the maximum value in the center position probability score P is regarded as the target center position, that is, (x d ,y d )=argmax (x,y) P xy ,Finally, the target bounding box is calculated by the following formula:

[0116] (x,y,w,h)=(x d +O(0,x d ,y d ),y d +O(1,x d ,y d ),S(0,x d ,y d ),S(1,x d ,y d ))

[0117] Among them, x represents the horizontal coordinate of the center position of the target bounding box, y represents the vertical coordinate of the center position of the target bounding box, w represents the width of the target bounding box, and h represents the length of the target bounding box.

[0118] 2.4 For the quality head, its input is the class token obtained by the feature extraction module. After processing by the quality head, a quality assessment of the current search area is obtained, which is used to update the dynamic template that has not been updated for the longest time in the template set.

[0119] The specific steps of updating the dynamic template that has not been updated the longest in the template set image in step 3 are as follows:

[0120] 3.1 After completing the target bounding box prediction of the current search area, use the quality head to predict the quality of the search area image. If the search area quality q now Higher than the optimal historical search area quality q before , then set the dynamic template candidate as the current search area, and let the optimal historical search area quality q before The quality q of the current search area now ;

[0121] 3.2 Every M frames, perform template cropping on the dynamic template candidate to obtain a new dynamic template. The cropping is centered on the corresponding target bounding box, and then the new dynamic template is used to replace the dynamic template with the longest interval between the template set and the current frame, and the optimal historical search area quality q before Set to 0.

[0122] The training of the feature extraction module and the prediction head module is divided into two stages. The specific training steps are as follows:

[0123] (1) The first stage is to use the total loss function The probability head, scale head and deviation head in the feature extraction module and the prediction head module are comprehensively trained, and the total loss function It includes three types of loss functions: classification loss function, regression loss function, and sparse coding loss function; among them, the classification loss function uses the Gaussian weighted focal loss loss function L cls ; The regression loss function is derived from the regression L1 loss function And GIOU loss function L GIOU Composition; sparse coding loss function is composed of sparse L1 loss function And the fidelity loss function L FBI The total loss function is The formula is as follows:

[0124]

[0125] Classification loss function formula L cls as follows:

[0126]

[0127] Among them, α and β represent the hyperparameters that control the distribution of each point. In this embodiment, α = 2, β = 4, p xy is the probability that the predicted position is the target center, λ cls =1, λ GIOU =5, λ FBI =1, both are regularization parameters;

[0128] For each ground-truth target bounding box, the target center and its low-resolution variant Generate Gaussian ground truth heatmap using Gaussian kernel σ represents the standard deviation of the target scale adaptation;

[0129] The regression L1 loss function The formula is as follows:

[0130]

[0131] Among them, y b is the coordinate of the real target bounding box, x b is the predicted target bounding box coordinate;

[0132] The GIOU loss function L GIOU The formula is:

[0133]

[0134]

[0135] Where A is the area of the true target bounding box, B is the area of the predicted target bounding box, and C is the minimum convex closed box containing A and B;

[0136] The sparse L1 loss function The formula is as follows:

[0137]

[0138] Where α is sparse coding;

[0139] The fidelity loss function formula L FBI as follows:

[0140]

[0141] Where T is the sparse coding dictionary and s2 is the search area feature;

[0142] (2) The second stage is to train the quality head, freeze the rest of the parts, and use the quality evaluation loss function L quality For training, the total loss function is The formula is as follows:

[0143]

[0144] where λ quality =1, is the regularization parameter;

[0145] For the quality evaluation function L in the second stage training quality The formula is as follows:

[0146] L quality = -plog(q)-(1-p)log(1-q)

[0147] Where p is the label, if the search area contains the target, p = 1, otherwise p = 0; q is the predicted search area quality of the quality head.

[0148] like Figure 4 FIG. 1 is a schematic diagram of the tracking effect obtained through simulation in an embodiment of the present invention, wherein: Figure 4 -a is the template image of the first frame, Figure 4 -b is the search area image tracking effect diagram of the 12th frame, Figure 4-c is the image tracking effect diagram of the search area in the 24th frame. The red box in the figure is the calculated target bounding box, that is, the location of the tracked target. It can be seen from the figure that the adaptive Transformer target tracking method based on sparse representation provided by the present invention can adaptively reduce the impact of complex background on the tracked target according to the complexity of the image. Compared with the existing technology, the tracking accuracy of the present invention is higher.

Claims

1. An adaptive Transformer target tracking system based on sparse representation, characterized in that: include: The feature extraction module, including the block embedding layer, the Transformer block, the sparse spatial attention module, the sparse adaptive spatial pruning module, and the candidate region reconstruction module, is used to extract features from the input search region image and the template set image, and output the extracted class token and search region image features to the prediction head module; wherein: The block embedding layer is used to convert the input search area image and template image into tokens; Transformer block, used to extract features from the tokens converted by the block embedding layer; The sparse spatial attention module is used to extract sparse features related to the foreground using the sparse feature representation method, and use the extracted sparse features to weight some of the original features extracted by the Transformer block, thereby enhancing the features of the target area and suppressing the features of the non-target area; Sparse adaptive spatial pruning module, which is used to adaptively discard redundant background information according to the complexity of the image in the search area; Candidate region reconstruction module, used to reconstruct candidate regions to fill missing locations; The prediction head module includes a probability head, a deviation head, a scale head, and a quality head. The probability head, the deviation head, and the scale head are used to calculate the predicted target bounding box. The quality head is used to calculate the quality of the search area and update the dynamic template that has not been updated for the longest time in the template set image.

2. The adaptive Transformer target tracking system based on sparse representation according to claim 1, characterized in that The sparse spatial attention module first performs sparse feature representation on the search area feature s2 to obtain sparse coding Then the sparse coding is average pooled along the channel to obtain the spatial statistical information of the search area features Then perform the maximum and minimum normalization operation on m1 to obtain the attention weight Finally, m2 is used to weight the input search area feature s2 and output the sparse attention search area feature map s′2, that is, s2 = m2s2, where m2 represents the attention weight; The specific implementation steps of the sparse feature representation are as follows: First, use the linear layer W sr Process the search area feature s2 to obtain y sr1 , for y sr1 Use the soft threshold activation function θ to get z1, and then the linear layer Q sr Process z1 and then add it to y sr1 After addition, the activation function θ is executed to obtain z2, and finally z2 passes through the linear layer Q sr Process and then sr1 Add and execute the activation function θ to obtain the sparse coding α of feature s1. The calculation formula of the soft threshold activation function θ is as follows: Where λ represents the threshold of the soft threshold activation function θ.

3. The adaptive Transformer target tracking system based on sparse representation according to claim 1, characterized in that The sparse adaptive spatial pruning module first sorts the weights m2 obtained by the sparse spatial attention module from large to small to obtain the confidence score. Then, the scores are summed to obtain the total energy E of the candidate region. ρ%×E is taken as the energy threshold, where ρ is the proportion of retained energy. Finally, the top K tokens with the largest energy in the candidate region are retained to satisfy And discard the remaining tokens.

4. An adaptive Transformer target tracking method based on sparse representation, characterized in that: The following steps are involved: Step 1. Input the search area image and the template set image into the feature extraction module at the same time, perform feature extraction and output the search area feature map s′3 and class token; Step 2. The search area feature map s′3 and the class token output by the feature extraction module obtained in step 1 are fed into the prediction head module. The predicted target bounding box is calculated using the probability head, scale head, and bias head in the prediction head module. The quality of the search area is calculated using the quality head. Step 3. If the quality of the search area predicted for the current frame calculated in step 2 is higher than the quality of the historical search area, the search area image of the current frame is first cropped with the predicted target bounding box as the center, and the size of the cropped search area image is the template size. Then, the dynamic template that has not been updated for the longest time in the template set image is updated, and the next frame prediction is performed based on the updated template set image. If the quality of the search area predicted for the current frame calculated in step 2 is equal to or lower than the quality of the historical search area, it is determined that the search area image predicted for the current frame is too complex and the target cannot be accurately located. Therefore, the template set image is not modified and the next frame prediction is directly performed.

5. The adaptive Transformer target tracking method based on sparse representation according to claim 4, characterized in that: The specific steps of the feature extraction module in step 1 extracting features from the search area image and the template set image and outputting the search area feature map s′3 and the class token are as follows: 1.1 Input template set image and search area image, where the template set image includes an initial frame template and T-1 dynamic templates. The size of each template in the template set image is 3×H t ×W t , the search area size is 3×H s ×W s , where H t Indicates the height of the template, W t Indicates the width of the template, H s , represents the height of the search area, W s Indicates the width of the search area; 1.2 Input each template in the template set image and the search area image into the block embedding layer respectively to obtain the template feature map set s1 and the search area feature map t1, with sizes of T×N respectively. t ×C、N s ×C; then add the template feature map set s1 and the search area feature map t1 to the position code respectively, and then splice all feature maps and learnable vector class token together to obtain the spliced feature map x1, where the size of the class token is 1×C and the size of the spliced feature map x1 is N x ×C, where N x =T×N t +N s +1, N t Indicates the number of tokens in the template feature map set, C indicates the number of channels in the feature map, N s Indicates the number of tokens in the search area feature map, N x Indicates the number of tokens of the concatenated feature map x1; 1.3 Send the spliced feature map x1 to the N1 layer Transformer block for feature extraction, and output the extracted spliced feature map x2; 1.4 Split the search region feature map s2 from the extracted spliced feature map x2, input the search region feature map s2 into the sparse spatial attention module for processing, and obtain the sparse attention search region feature map s′2, and then input the sparse attention search region feature map s′2 into the sparse adaptive spatial pruning module to obtain the pruned search region feature map s″2; 1.5 Re-join the pruned search region feature map s″2 and the remaining features of the extracted spliced feature map x2 to obtain a new spliced feature map x′2; 1.6 Input the new spliced feature map x′2 into the N2-layer Transformer block for feature extraction, output the extracted spliced feature map x3, split it into class token, search area feature map s3, and template feature map set t3, input the search area feature map s3 into the candidate region reconstruction module, reconstruct its spatial features to obtain the candidate region reconstruction feature map s′3; 1.7 The output of the feature extraction module is the final class token and the candidate region reconstruction feature s′3.

6. The adaptive Transformer target tracking method based on sparse representation according to claim 4, characterized in that The specific method of calculating the predicted target bounding box using the probability head, scale head, and deviation head in the prediction head module in step 2, and calculating the search area quality using the quality head is as follows: 2.1 For the probability head, first, the candidate region reconstruction feature s′3 and the class token are subjected to mutual attention calculation to obtain the weighted candidate region reconstruction feature s4. The calculation formula is as follows: s4=s′3*(s′3×classToken T ) Among them, × represents matrix multiplication, * represents element multiplication; Then, the weighted candidate region reconstruction feature s4 is input into the probability head, and the center position probability score is output after calculation by the three-layer feedforward neural network: Among them, H s 、W s Represent the height and width of the search area respectively, and d represents the network step size; 2.2 For the deviation head and scale head, the candidate region reconstruction feature s′3 is input and the deviation prediction is obtained after calculation by the three-layer feedforward neural network Scale prediction 2.3 Then, the corresponding position of the maximum value in the center position probability score P is regarded as the target center position, that is, (x d ,y d )=argmax (x,y) P xy ,Finally, the target bounding box is calculated by the following formula: (x,y,w,h)=(x d +O(0,x d ,y d ),y d +O(1,x d ,y d ),S(0,x d ,y d ),S(1,x d ,y d )) Where x represents the horizontal coordinate of the center position of the target bounding box, y represents the vertical coordinate of the center position of the target bounding box, w represents the width of the target bounding box, and h represents the length of the target bounding box; 2.4 For the quality head, its input is the class token obtained by the feature extraction module. The quality evaluation of the current search area is calculated by the quality head and is used to update the dynamic template that has not been updated for the longest time in the template set.

7. The adaptive Transformer target tracking method based on sparse representation according to claim 4, characterized in that: The specific steps of updating the dynamic template that has not been updated the longest in the template set image in step 3 are as follows: 3.1 After completing the target bounding box prediction of the current search area, use the quality head to predict the quality of the search area image. If the search area quality q now Higher than the optimal historical search area quality q before , then set the dynamic template candidate as the current search area, and let the optimal historical search area quality q before The quality q of the current search area now ; 3.2 Every M frames, perform template cropping on the dynamic template candidate to obtain a new dynamic template. The cropping is centered on the corresponding target bounding box, and then the new dynamic template is used to replace the dynamic template with the longest interval between the template set and the current frame, and the optimal historical search area quality q before Set to 0.

8. The method for adaptive Transformer target tracking based on sparse representation according to claim 4, 5 or 6, characterized in that: The training of the feature extraction module and the prediction head module is divided into two stages. The specific training steps are as follows: (1) The first stage is to use the total loss function The probability head, scale head and deviation head in the feature extraction module and the prediction head module are comprehensively trained, and the total loss function It includes three types of loss functions: classification loss function, regression loss function, and sparse coding loss function; wherein, the classification loss function uses the Gaussian weighted focalloss loss function L cls ; The regression loss function is derived from the regression L1 loss function And GIOU loss function L GIOU Composition; sparse coding loss function is composed of sparse L1 loss function And the fidelity loss function L FBI The total loss function is The formula is as follows: Among them, the classification loss function formula L cls as follows: Among them, α and β represent the hyperparameters that control the distribution of each point, and p xy is the probability that the predicted position is the target center, λ cls =1, λ GIOU =5, λ FBI =1, both are regularization parameters; For each ground-truth target bounding box, the target center and its low-resolution variant Generate Gaussian ground truth heatmap using Gaussian kernel σ represents the standard deviation of the target scale adaptation; The regression L1 loss function The formula is as follows: Among them, y b is the coordinate of the real target bounding box, x b is the predicted target bounding box coordinate; The GIOU loss function L GIOU The formula is: Among them, A is the area of the true target bounding box, B is the area of the predicted target bounding box, and C is the minimum convex closed box containing A and B; The sparse L1 loss function The formula is as follows: Among them, α is sparse coding; The fidelity loss function formula L FBI as follows: Among them, T is the sparse coding dictionary, s2 is the search area feature; (2) The second stage is to train the quality head, freeze the rest of the parts, and use the quality evaluation loss function L quality For training, the total loss function is The formula is as follows: Among them, λ quality =1, is the regularization parameter; For the quality evaluation function L in the second stage training quality The formula is as follows: L quality =-plog(q)-(1-p)log(1-q) Where p is the label, if the search area contains the target, p = 1, otherwise p = 0; q is the predicted search area quality of the quality head.

Images