Target tracking method based on adaptive template updating and lightweight Transformer

Abstract

The invention relates to the technical field of computer vision, in particular to a target tracking method based on adaptive template updating and lightweight Transform. The invention provides an efficient and robust target tracking algorithm for solving the problems that a traditional twin network is insufficient in robustness in a complex scene and a Transform architecture is highly dependent on computing resources. Firstly, a supervision feedback module is designed, supervision information related to a task is introduced in a feature extraction stage, and a network is guided to be more focused on a target area, so that the feature discrimination capability is improved, and background interference is effectively suppressed; secondly, a lightweight Transform structure is constructed, the calculation complexity and the parameter scale are remarkably reduced while the global modeling capability is maintained, and the balance between the model performance and the calculation efficiency is achieved; and finally, designing a self-adaptive template updating mechanism, and dynamically updating the template content in combination with the state information of the current frame and the environment change, thereby enhancing the adaptability of the model to the target appearance change and reducing the tracking drift risk.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a visual single-target tracking method based on Siamese networks. Background Technology

[0002] Single-object tracking is a fundamental and practical research area in computer vision, widely applied in various real-world scenarios such as public safety monitoring, autonomous driving systems, intelligent robot navigation, and human-computer interaction. Its core task is to manually specify the target's position in the initial frame of a given video sequence, and then use a tracking algorithm to continuously predict the target's spatial location in subsequent frames, thereby achieving continuous tracking and localization. However, single-object tracking faces numerous challenges, such as the arbitrariness of target categories, dynamic changes in appearance, and interference factors commonly present in real-world environments, including motion blur, complex backgrounds, partial or complete occlusion, and changes in lighting. These significantly increase the difficulty of accurately locating the target and place higher demands on the robustness and adaptability of the algorithm.

[0003] To address the aforementioned issues, this invention provides a target tracking method based on adaptive template updating and a lightweight Transformer, which can accurately track targets in highly disruptive environments such as cluttered backgrounds, target pose changes, and interference from similar objects. Summary of the Invention

[0004] (1) ResNet-50 is used as the backbone network for feature extraction to extract high-dimensional semantic features from the input template image and the search image. The traditional ResNet structure is optimized for the task to better adapt to the target representation requirements in target tracking and improve the discriminativeness and compactness of the features;

[0005] (2) Introduce a supervision feedback module to dynamically inject task-related guidance information during the feature extraction stage. By supervising the previous features, the current network is guided to pay more attention to the target area, which significantly enhances the target perception ability and suppresses redundant background information, thereby improving the stability of the model in interference scenarios.

[0006] (3) Design a lightweight Transformer structure that retains the global modeling capability of the original Transformer, while reducing computational complexity and parameter scale through structural compression, weight sharing and other methods, thereby significantly improving running efficiency while ensuring modeling effect and meeting actual deployment needs;

[0007] (4) An adaptive template update mechanism is proposed. Based on the current frame's state information, tracking stability, and changes in the target's appearance, the template content is dynamically adjusted to achieve continuous matching between template features and target appearance, thereby effectively reducing the risk of tracking drift caused by appearance changes or occlusion.

[0008] (5) By calculating the similarity map between the output features of the template branch and the search branch, the position with the highest similarity is extracted as the target estimation position. Combined with the interpolation method, it is mapped to the original image coordinates and the final tracking result is output to realize the end-to-end target tracking process.

[0009] Specifically, step (1) involves preprocessing and feature extraction of the input image, and the specific steps are as follows:

[0010] (1.1) The template branch selects the first frame image in the target tracking video sequence as the initial target template, and the final template image has a resolution of 127×127 and 3 channels. The search image is also processed using the same strategy, and the final resolution is 255×255 with 3 channels;

[0011] (1.2) ResNet-50 is used as the backbone network for feature extraction to extract features from the input image;

[0012] (1.3) The preprocessed template image and search image are input into ResNet-50 to extract multi-level deep semantic features. The output features are as follows: the template feature map size is 14×14, the search feature map size is 31×31, and the number of channels is 256 for both.

[0013] Specifically, the template features and search features obtained in step (1.3) are input into the supervision feedback module in step (2), and the specific processing steps of the supervision feedback module for the input features are as follows:

[0014] (2.1) A 3×3 convolutional layer is used to model the local context of the input high-dimensional feature map, extracting spatial structure and local contextual relationships, and providing a more compact feature representation for subsequent processing without changing the number of channels. Then, a batch normalization layer is used to normalize the feature distribution, alleviate gradient instability, and accelerate network convergence. Furthermore, the ReLU activation function is used to provide the network with non-linear modeling capabilities, as shown in the following formula:

[0015] F = ReLU(BN(Conv(f)))

[0016] To further compress the spatial dimension and highlight global information, adaptive average pooling is used to aggregate the feature maps into a 1×1 global vector, thereby simplifying the classification decision process. Finally, a fully connected layer and a softmax activation function are used to map the features to a binary classification probability distribution of the source and target domains, forming an auxiliary supervision branch, as shown in the following formula:

[0017] y i =Softmax(FC(Pool(F)))

[0018] Among them, y i Let f represent the prediction result of the i-th sample on the auxiliary supervision branch, and f be the input feature map, which can be the template or the output feature of the search region. The entire structure not only improves the discriminativeness and stability of the features, but also ensures that it plays a regularizing role on the backbone network during the training phase without introducing additional computational overhead during the inference phase.

[0019] (2.2) To enable the supervised output layer to learn more discriminative high-level representations, a cross-entropy loss is added to the output of the auxiliary supervised branch. The loss function of the main task itself includes target classification loss and bounding box regression loss. The target classification loss uses cross-entropy loss to measure the difference between the predicted and true labels of the target category. The bounding box regression loss uses a smoothed L1 loss function to measure the error between the predicted and true boxes. Finally, the total loss function of the entire network is formulated as follows:

[0020]

[0021] Among them, y i ∈{0,1} represents the label of the input sample in the auxiliary classification task.

[0022] Specifically, the input in step (2.2) into the lightweight Transformer structure, and the specific steps of executing step (3) to lightweight Transformer structure are as follows:

[0023] (3.1) The template features and search features after the fusion of supervised feedback are used as inputs to the Transformer, and a simplified multi-head self-attention mechanism is used to model the global feature interaction between the two.

[0024] (3.2) The encoder consists of N stacked encoder layers, each typically including a multi-head self-attention module and a feedforward network. To reduce computational burden, the feedforward module is omitted in this invention, retaining only the multi-head self-attention part. This design not only significantly reduces computational complexity but also helps maintain the integrity of spatial structural information. First, the input tensor is flattened along the spatial dimension and transformed into a one-dimensional sequence form, as shown in the following formula:

[0025] X flat =Reshape(X)∈R L×B×C

[0026] To enhance the model's spatial modeling capabilities, this invention introduces relative position encoding, directly incorporating positional information into the attention weights, thereby improving the model's sensitivity to local structure and positional information. This is then added to the sequence features, as shown in the following formula:

[0027] X seq =X flat +P

[0028] For each encoder layer, the input features first enter the multi-head self-attention module. The multi-head attention mechanism maps the input to queries, keys, and values, as shown in the following formula:

[0029]

[0030] in, This is a scaling factor to avoid excessively large inner product values. To avoid the vanishing gradient problem and accelerate convergence, a residual connection is added after each multi-head self-attention mechanism. To preserve the spatial structure of the image and reduce spatial information loss, this invention introduces an L2Norm normalization operation. This operation normalizes each channel by calculating the L2 norm of each instance, thus standardizing the feature magnitude of each pixel, as shown in the following formula:

[0031] Y = L2Norm(X) seq +Attention(T))

[0032] (3.3) The decoder consists of M stacked decoder layers, each containing a sparse multi-head self-attention module and a cross-multi-head self-attention module. The goal of this structure is to combine the encoder output information to achieve more efficient feature reconstruction and target localization.

[0033] A sparse multi-head self-attention module is introduced. Similar to standard multi-head attention, this module incorporates sparsity constraints in attention computation, thereby reducing computational complexity and improving local modeling capabilities. Its attention mechanism formula is as follows:

[0034]

[0035] Where Q = XW Q K = XW K V = XW V M is a sparse mask matrix. After residual concatenation between the module output and input, L2 normalization is applied to maintain numerical stability, as shown in the following formula:

[0036] Y′=L2Norm(X′ seq +S-Attention(T))

[0037] Next, the output feature Y′ will be used as the query input to the cross-attention module. Simultaneously, the sequence features from the encoder output will participate in the cross-attention calculation as keys and values. This module is used to incorporate global contextual information from the encoder to guide accurate regression or classification of the target location, as shown in the following formula:

[0038]

[0039] Finally, residual connections are used again, followed by L2 normalization, to obtain the output of this layer's decoder, as shown in the following formula:

[0040] Y″=L2Norm(Y′+C-Attention(T))

[0041] Specifically, step (4) includes the following steps:

[0042] (4.1) To quantify the matching degree between the current frame and the template, this invention designs a lightweight IoU prediction module. This module effectively predicts the IoU score between the current detection box and the historical template by learning the spatial and appearance relationship of the target. It combines the image features of the current frame with the positional structure information of the prediction result of the previous frame. This module effectively captures the relationship between the target's appearance and spatial layout through multimodal fusion and an MLP regression network, achieving accurate and stable IoU estimation, as shown in the following formula:

[0043]

[0044] Among them, F t F t-1 T represents the features of the candidate region in the current frame and the candidate region in the previous frame, respectively. old This represents the normalized feature encoding of the candidate region in the previous frame. This vector integrates the appearance of the target in the current frame with its historical spatial dynamic information, providing contextual support for IoU estimation.

[0045] (4.2) Based on the two predicted IoU values, construct the difference index, as shown in the following formula:

[0046] ΔIoU=IoU new -IoU old

[0047] When this value exceeds the set threshold δ, it indicates that the current frame features are more accurate and closer to the target state than the old template, and the template update process can be considered. This IoU prediction mechanism, by introducing the spatial information of the previous frame, realizes the joint modeling of appearance features and target dynamics, making the template update strategy more temporally consistent and predictively stable.

[0048] Specifically, step (5) includes the following steps: performing cross-correlation operation on the feature outputs of the template branch and the search branch to obtain a similarity score map, and mapping the highest value in the score map to the original image to obtain the tracking result. Attached Figure Description

[0049] Appendix Figure 1 This is a flowchart of the present invention;

[0050] Appendix Figure 2 This is a schematic diagram of the invention;

[0051] Appendix Figure 3 for Figure 2 Block diagram of the supervisory feedback module;

[0052] Appendix Figure 4 for Figure 2 Block diagram of the encoder principle;

[0053] Appendix Figure 5 for Figure 2 Block diagram of the decoder principle; Detailed Implementation

[0054] As attached Figure 1 The diagram illustrates a target tracking method that combines 3D attention and a pyramid decoder. The steps are explained in detail below.

[0055] Step (1) involves preprocessing and feature extraction of the input image. The specific steps are as follows:

[0056] (1.1) The template branch selects the first frame image in the target tracking video sequence as the initial target template, and the final template image has a resolution of 127×127 and 3 channels. The search image is also processed using the same strategy, and the final resolution is 255×255 with 3 channels;

[0057] (1.2) ResNet-50 is used as the backbone network for feature extraction to extract features from the input image;

[0058] (1.3) The preprocessed template image and search image are input into ResNet-50 to extract multi-level deep semantic features. The output features are as follows: the template feature map size is 14×14, the search feature map size is 31×31, and the number of channels is 256 for both.

[0059] Specifically, the template features and search features obtained in step (1.3) are input into the supervision feedback module in step (2), and the specific processing steps of the supervision feedback module for the input features are as follows:

[0060] (2.1) A 3×3 convolutional layer is used to model the local context of the input high-dimensional feature map, extracting spatial structure and local contextual relationships, and providing a more compact feature representation for subsequent processing without changing the number of channels. Then, a batch normalization layer is used to normalize the feature distribution, alleviate gradient instability, and accelerate network convergence. Furthermore, the ReLU activation function is used to provide the network with non-linear modeling capabilities, as shown in the following formula:

[0061] F = ReLU(BN(Conv(f)))

[0062] To further compress the spatial dimension and highlight global information, adaptive average pooling is used to aggregate the feature maps into a 1×1 global vector, thereby simplifying the classification decision process. Finally, a fully connected layer and a softmax activation function are used to map the features to a binary classification probability distribution of the source and target domains, forming an auxiliary supervision branch, as shown in the following formula:

[0063] y i =Softmax(FC(Pool(F)))

[0064] Among them, y i Let f represent the prediction result of the i-th sample on the auxiliary supervision branch, and f be the input feature map, which can be the template or the output feature of the search region. The entire structure not only improves the discriminativeness and stability of the features, but also ensures that it plays a regularizing role on the backbone network during the training phase without introducing additional computational overhead during the inference phase.

[0065] (2.2) To enable the supervised output layer to learn more discriminative high-level representations, a cross-entropy loss is added to the output of the auxiliary supervised branch. The loss function of the main task itself includes target classification loss and bounding box regression loss. The target classification loss uses cross-entropy loss to measure the difference between the predicted and true labels of the target category. The bounding box regression loss uses a smoothed L1 loss function to measure the error between the predicted and true boxes. Finally, the total loss function of the entire network is formulated as follows:

[0066]

[0067] Among them, y i ∈{0,1} represents the label of the input sample in the auxiliary classification task.

[0068] Specifically, the input in step (2.2) into the lightweight Transformer structure, and the specific steps of executing step (3) to lightweight Transformer structure are as follows:

[0069] (3.1) The template features and search features after the fusion of supervised feedback are used as inputs to the Transformer, and a simplified multi-head self-attention mechanism is used to model the global feature interaction between the two.

[0070] (3.2) The encoder consists of N stacked encoder layers, each typically including a multi-head self-attention module and a feedforward network. To reduce computational burden, the feedforward module is omitted in this invention, retaining only the multi-head self-attention part. This design not only significantly reduces computational complexity but also helps maintain the integrity of spatial structural information. First, the input tensor is flattened along the spatial dimension and transformed into a one-dimensional sequence form, as shown in the following formula:

[0071] X flat=Reshape(X)∈R L×B×C

[0072] To enhance the model's spatial modeling capabilities, this invention introduces relative position encoding, directly incorporating positional information into the attention weights, thereby improving the model's sensitivity to local structure and positional information. This is then added to the sequence features, as shown in the following formula:

[0073] X seq =X flat +P

[0074] For each encoder layer, the input features first enter the multi-head self-attention module. The multi-head attention mechanism maps the input to queries, keys, and values, as shown in the following formula:

[0075]

[0076] in, This is a scaling factor to avoid excessively large inner product values. To avoid the vanishing gradient problem and accelerate convergence, a residual connection is added after each multi-head self-attention mechanism. To preserve the spatial structure of the image and reduce spatial information loss, this invention introduces an L2Norm normalization operation. This operation normalizes each channel by calculating the L2 norm of each instance, thus standardizing the feature magnitude of each pixel, as shown in the following formula:

[0077] Y = L2Norm(X) seq +Attention(T))

[0078] (3.3) The decoder consists of M stacked decoder layers, each containing a sparse multi-head self-attention module and a cross-multi-head self-attention module. The goal of this structure is to combine the encoder output information to achieve more efficient feature reconstruction and target localization.

[0079] A sparse multi-head self-attention module is introduced. Similar to standard multi-head attention, this module incorporates sparsity constraints in attention computation, thereby reducing computational complexity and improving local modeling capabilities. Its attention mechanism formula is as follows:

[0080]

[0081] Where Q = XW Q K = XW K V = XW V M is a sparse mask matrix. After residual concatenation between the module output and input, L2 normalization is applied to maintain numerical stability, as shown in the following formula:

[0082] Y′=L2Norm(X′ seq+S-Attention(T))

[0083] Next, the output feature Y′ will be used as the query input to the cross-attention module. Simultaneously, the sequence features from the encoder output will participate in the cross-attention calculation as keys and values. This module is used to incorporate global contextual information from the encoder to guide accurate regression or classification of the target location, as shown in the following formula:

[0084]

[0085] Finally, residual connections are used again, followed by L2 normalization, to obtain the output of this layer's decoder, as shown in the following formula:

[0086] Y″=L2Norm(Y′+C-Attention(T))

[0087] Specifically, step (4) includes the following steps:

[0088] (4.1) To quantify the matching degree between the current frame and the template, this invention designs a lightweight IoU prediction module. This module effectively predicts the IoU score between the current detection box and the historical template by learning the spatial and appearance relationship of the target. It combines the image features of the current frame with the positional structure information of the prediction result of the previous frame. This module effectively captures the relationship between the target's appearance and spatial layout through multimodal fusion and an MLP regression network, achieving accurate and stable IoU estimation, as shown in the following formula:

[0089]

[0090] Among them, F t F t-1 T represents the features of the candidate region in the current frame and the candidate region in the previous frame, respectively. old This represents the normalized feature encoding of the candidate region in the previous frame. This vector integrates the appearance of the target in the current frame with its historical spatial dynamic information, providing contextual support for IoU estimation.

[0091] (4.2) Based on the two predicted IoU values, construct the difference index, as shown in the following formula:

[0092] ΔIoU=IoU new -IoU old

[0093] When this value exceeds the set threshold δ, it indicates that the current frame features are more accurate and closer to the target state than the old template, and the template update process can be considered. This IoU prediction mechanism, by introducing the spatial information of the previous frame, realizes the joint modeling of appearance features and target dynamics, making the template update strategy more temporally consistent and predictively stable.

[0094] Specifically, step (5) includes the following steps: performing cross-correlation operation on the feature outputs of the template branch and the search branch to obtain a similarity score map, and mapping the highest value in the score map to the original image to obtain the tracking result.

Claims

1. A target tracking method based on adaptive template update and lightweight Transformer, characterized in that, Includes the following steps: Step 1: ResNet-50 is used as the backbone network for feature extraction to extract high-dimensional semantic features from the input template image and the search image. The traditional ResNet structure is optimized for the task, making it more suitable for the target representation requirements in target tracking and improving the discriminativeness and compactness of the features. Step 2: Introduce a supervised feedback module to dynamically inject task-related guidance information during the feature extraction stage. By supervising the previous features, the current network is guided to pay more attention to the target region, which significantly enhances the target perception ability and suppresses redundant background information, thereby improving the stability of the model in interference scenarios. Step 3: Design a lightweight Transformer structure that retains the global modeling capabilities of the original Transformer while reducing computational complexity and parameter size through structural compression and weight sharing. This significantly improves operational efficiency while ensuring modeling effectiveness and meeting actual deployment requirements. Step 4 proposes an adaptive template update mechanism, which dynamically adjusts the template content based on the current frame's state information, tracking stability, and changes in the target's appearance, thereby achieving continuous matching between template features and the target's appearance and effectively reducing the risk of tracking drift caused by appearance changes or occlusion. Step 5: By calculating the similarity map between the output features of the template branch and the search branch, the position with the highest similarity is extracted as the estimated position of the target. Combined with the interpolation method, it is mapped to the coordinates of the original image, and the final tracking result is output, realizing the end-to-end target tracking process.

2. The target tracking method based on adaptive template update and lightweight Transformer according to claim 1, characterized in that: Step 1 includes the following specific methods: Step 2-1: The template branch selects the first frame image in the target tracking video sequence as the initial target template, resulting in a template image with a resolution of 127×127 and 3 channels. The search image is processed using the same strategy, resulting in a final resolution of 255×255 and 3 channels. Step 2-2: Use ResNet-50 as the backbone network for feature extraction to extract features from the input image; Steps 2-3 involve inputting the preprocessed template image and search image into ResNet-50 to extract multi-level deep semantic features. The output features are as follows: the template feature map size is 14×14, the search feature map size is 31×31, and both have 256 channels.

3. The specific processing steps of the supervised feedback module for the input features are as follows: Step 3-1 employs an auxiliary supervision branch to assist the main task and simplify the classification decision process. This entire structure not only improves the discriminative power and stability of the features but also ensures regularization of the backbone network during training without introducing additional computational overhead during inference. Step 3-2: Input the features after the supervision feedback and main task fusion processing into the subsequent Transformer structure.

4. Input the values ​​from step 3-2 into the Transformer structure, and perform step 4 to lightweight the Transformer structure. The specific steps are as follows: Step 4-1: The template features and search features after fusing supervised feedback are used as input to the Transformer, and a simplified multi-head self-attention mechanism is used to model the global feature interaction between the two. Step 4-2: The encoder consists of N stacked encoder layers, each typically including a multi-head self-attention module and a feedforward network. To reduce computational burden, the feedforward module is omitted in this invention, retaining only the multi-head self-attention part. This design not only significantly reduces computational complexity but also helps maintain the integrity of spatial structural information. Step 4-3: The decoder consists of M stacked decoder layers, each containing a sparse multi-head self-attention module and a cross-multi-head self-attention module. The goal of this structure is to combine the encoder output information to achieve more efficient feature reconstruction and target localization.

5. The specific method of the adaptive template update mechanism described in step 5 is as follows: Step 5-1: To quantify the matching degree between the current frame and the template, this invention designs a lightweight IoU prediction module. This module effectively predicts the IoU score between the current detection box and the historical template by learning the spatial and appearance relationships of the target. It combines the image features of the current frame with the positional and structural information of the prediction results from the previous frame. Step 5-2: This module effectively captures the relationship between the target's appearance and spatial layout through multimodal fusion and an MLP regression network, achieving accurate and stable IoU estimation. Finally, based on the two predicted IoU values, a difference index is constructed.

6. Perform cross-correlation on the feature outputs of the template branch and the search branch to obtain a similarity score map. The highest value in the score map can be mapped to the original image to obtain the tracking result.

Images