Adaptive background perception Transform visual target tracking method and device and medium
Through the adaptive background perception mechanism and pyramid-type corner point prediction head, dynamically optimize the interaction between target and background, the problem of difficult distinction between target and background in ViT single-stream tracking method is solved, and the tracking accuracy and stability are improved, which is suitable for actual needs such as autonomous driving and drone monitoring.
Patent Information
- Application Number
- CN202510460386.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-14
- Publication Date
- 2025-08-15
AI Technical Summary
The existing single-stream tracking method based on ViT is difficult to effectively distinguish between targets and background information, resulting in a decrease in tracking accuracy, and background complexity and occlusion problems reduce the stability and robustness of the model.
Adaptive background perception mechanism and pyramid-type corner point prediction head are introduced. By dividing search area tokens, dynamically optimizing the interaction process between the target and the background, multi-scale feature fusion and end-to-end Transformer structure are adopted, and target prediction is optimized by combining multiple loss functions.
It improves the accuracy and robustness of target tracking, can effectively reduce background interference, adapt to target scale changes in complex scenarios, and realize real-time tracking applications.
Smart Images
Figure CN120495945A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of computer vision and deep learning, and in particular relates to a Transformer visual target tracking method, device and medium with adaptive background perception. Background Art
[0002] Target tracking is an important research direction in computer vision and has wide applications in autonomous driving, drone monitoring, intelligent security and other fields. In recent years, the application of Transforme architecture in visual tasks has made significant progress, and the ViT (Vision Transformer) framework has been widely used in target detection and tracking. However, due to the limitations of the global interaction mode, the existing ViT-based single-stream tracking method has difficulty in effectively distinguishing the target from the background information, resulting in a decrease in tracking accuracy. In addition, the background complexity and occlusion problems further reduce the stability of the tracking model, making the robustness in real application scenarios challenging. To address these problems, the present invention proposes an adaptive background-aware Transformer target tracking method, which dynamically optimizes the interaction process between the target and the background by introducing an adaptive background perception mechanism and a pyramid corner prediction head, thereby improving tracking accuracy and robustness. Summary of the Invention
[0003] Purpose of the invention: The purpose of the present invention is to provide a Transformer visual target tracking method, device and medium with adaptive background perception, which can effectively distinguish the target from the background and improve the accuracy of target tracking.
[0004] Technical solution: The adaptive background-aware Transformer visual target tracking method of the present invention includes the following steps:
[0005] S1: A pair of images in a given video sequence is processed and input into multiple encoder layers of the Transformer network to learn the relationship between the target and background areas;
[0006] S2: Before token interaction in the search area, an adaptive background-aware mechanism is used to partition the search area. The input token sequence is divided into tokens T1 generated by the template image and its central target block, tokens T2 in the search image that are highly correlated with the target, and tokens T3 remaining in the search image. Background interference is filtered out and the processing is focused on the target-related area. The search tokens are accurately classified through a trainable prediction module and Laplace distribution probability.
[0007] S3: The encoder receives three token sequences (T1, T2, T3) and calculates the correlation between the template and the strong target search area and the template and the weak target search area through two attention blocks, respectively, to obtain two fused sequences. The two fused sequences are summed and normalized, and the fused sequence after the interaction between the target and the background is output;
[0008] S4: The search area fusion sequence is used as the input of the prediction head and reinterpreted as a two-dimensional spatial feature map before being fed into the prediction head. A multi-scale feature fusion method is used to generate feature maps of different resolutions through a pyramid structure. The final corner prediction result is obtained by fusing the multi-scale feature maps.
[0009] Furthermore, the pair of images in the video sequence in step S1 includes an initial target template image and a search area image.
[0010] Furthermore, the implementation process of step S1 is as follows:
[0011] S1.1: Segment a pair of images in the input video sequence into multiple fixed-size image blocks; then flatten these image blocks and convert them into one-dimensional vectors;
[0012] S1.2: For the flattened image block sequence, map each image block to C-dimensional space by linear projection; let each image block be x i , the mapped image block is represented as z i , which is expressed as follows:
[0013] z i =Wx i +b
[0014] Where W is the linear projection matrix, b is the offset, and C is the feature dimension after projection;
[0015] S1.3: Add a learnable position embedding tag to each image block so that the Transformer can capture the position relationship of the image blocks in space. Add the position embedding tag to the linearly projected image block representation to obtain a feature sequence with position information.
[0016] Furthermore, the implementation process of step S2 is as follows:
[0017] S2.1: The input sequence needs to be divided into three different categories of tokens, T1, T2, and T3; T1 is all tokens generated by the template image and its central target block, representing the initial information of the target; T2 is the tokens in the search image that are highly correlated with the target and are mainly used for target perception modeling; T3 is the remaining token sequence in the search image containing irrelevant or weakly correlated information;
[0018] S2.2: Aggregate template tokens through a global max pooling operation, so that the search token is infused with target-related information to generate a target-aware feature representation; concatenate the feature representation with each search token; the concatenated vector is input into a lightweight multi-layer perceptron (MLP) to predict the probability of each search token belonging to the T2 and T3 categories;
[0019] And use Laplace distribution to optimize the category discrimination of search tokens; assuming that the feature difference f of the search token Δ It obeys the Laplace distribution, and its probability density function is:
[0020]
[0021] Among them, b is the scale parameter used to control the width of the distribution, f Δ For characteristic differences;
[0022] The final probability calculation formula combines the Softmax function with the feature representation output by the MLP:
[0023] P=Softmax(Contact(MLP(T2,T3)))·p(f Δ )
[0024] Among them, the Softmax function is used to normalize the output category probability to ensure that the sum is 1, P (i,0) and P (i,1) denote the probability of assigning the i-th search token to T2 and T3 respectively;
[0025] S2.3: For the three token sequences, the attention mechanism inputs in the encoder are analogous to: template input query vector Q1, key vector K1, value vector V1, strong target search inputs are Q2, K2, V2, and weak target search inputs are Q3, K3, V3;
[0026] S2.4: The target’s center block information is additionally embedded in the template features to further enhance the target information representation capability. The features of the target’s center block are extracted from the target area in the initial frame and dynamically updated during the subsequent tracking process to ensure the target’s feature consistency.
[0027] Furthermore, the implementation process of step S3 is as follows:
[0028] S3.1: The target encoder receives three sets of token sequences as input and implements cross-correlation modeling through an attention block. For the template token group T1 and the strong target search area token T2, the correlation between the template and the search area is calculated:
[0029]
[0030] Where D is the feature dimension in the attention mechanism, V is the value vector used for weighting, and W is a learnable weight;
[0031] S3.2: For the template token group T1 and the weak target search area token T3, calculate the correlation between the template and the search area:
[0032]
[0033] S3.3: After the first two steps of calculation, the fusion sequence of the template token T1 interacting with the strong target search area token T2 and the weak target search area token T3 is obtained. These two sets of fusion sequences are added and normalized, and the fusion sequence after the interaction between the target and the background is output.
[0034] Furthermore, the implementation process of step S4 is as follows:
[0035] S4.1: Remap the search region fusion sequence to two-dimensional space to obtain the feature map F m , and standardize the features:
[0036] F m =reshape(F s ,H,W,C)
[0037] Among them, F s is the fusion token sequence, H, W, C are width, height, and number of channels respectively;
[0038] S4.2: Construct a multi-scale feature pyramid by applying a series of convolutional layers and upsampling interpolation layers to the input feature map to generate three feature maps of different resolutions. The three feature maps of different resolutions are restored to the same size as the original input and then added and fused to obtain the final multi-scale feature representation.
[0039] S4.3: The corner regression head is used to predict the coordinates of the four corner points of the target box and the probability of each corner point. During the training process, the regression loss and confidence loss are used for optimization, and the L1 loss is used to measure the deviation between the predicted corner points and the actual corner points:
[0040]
[0041] Among them, L reg is the regression loss, (x i ,y i ) is the predicted coordinate, is the target's true coordinate; binary cross entropy loss is used for corner confidence loss to measure the deviation between the predicted corner probability and the true probability:
[0042]
[0043] Among them, P ij is the predicted probability of corner points, is the binary label of the true corner point; the final loss function is:
[0044] L total =λ1L1+λ2L conf
[0045] Among them, λ1 and λ2 are the weight hyperparameters of the loss, which are used to balance the regression error and the confidence error.
[0046] An electronic device according to the present invention includes a memory and a processor, wherein:
[0047] a memory for storing computer programs capable of running on the processor;
[0048] A processor is configured to execute the steps of the above-described adaptive background-aware Transformer visual target tracking method when running the computer program.
[0049] The present invention provides a storage medium having a computer program stored thereon, which, when executed by at least one processor, implements the steps of the above-mentioned adaptive background-aware Transformer visual target tracking method.
[0050] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: the present invention uses an adaptive background perception mechanism to dynamically screen target tokens and suppress background tokens, so that the model can more accurately capture target features, thereby effectively reducing background interference; the present invention introduces a pyramid corner prediction head, and multi-scale feature fusion significantly improves the prediction accuracy of the target bounding box, which is particularly suitable for complex scenes with large changes in target scale; the present invention adopts an end-to-end Transformer structure and combines multiple loss functions to optimize target prediction, so that the model can still maintain stable tracking performance under conditions of occlusion and complex background; the present invention uses ViT to unify target feature extraction and information interaction, simplifies the network structure, reduces computational overhead, and helps to realize real-time tracking applications, adapting to actual needs such as autonomous driving and drone monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 Flowchart of the Transformer visual target tracking method with adaptive background perception;
[0052] Figure 2 This is the architecture diagram of the Transformer visual object tracking method with adaptive background perception;
[0053] Figure 3The target encoder workflow diagram proposed by the present invention;
[0054] Figure 4 This is the workflow diagram of the pyramid corner prediction head module proposed in the present invention. DETAILED DESCRIPTION
[0055] The present invention will be further described in detail below with reference to the accompanying drawings.
[0056] like Figure 1 、 Figure 2 As shown, the present invention proposes an attention-enhanced spatiotemporal Transformer single target tracking method, comprising the following steps:
[0057] Step 1: Given a pair of images in a video sequence, including an initial target template image and a search region image, the images are first segmented and flattened to obtain a sequence of image patches, which are then mapped into a C-dimensional space via linear projection. Simultaneously, a learnable position embedding tag is added to each image patch. The resulting tag sequence is concatenated and input into multiple encoder layers of a Transformer network to learn the relationship between the target and background regions.
[0058] The input template image and the search area image are segmented to obtain multiple fixed-size image blocks. These image blocks are then flattened and converted into one-dimensional vectors for subsequent processing.
[0059] For the flattened image block sequence, each image block is mapped to a C-dimensional space by linear projection. Let each image block be x i , the mapped image block is represented as z i , which is expressed as follows:
[0060] z i =Wx i +b
[0061] Among them, W is the linear projection matrix, b is the offset, and C is the feature dimension after projection.
[0062] Adding a learnable position embedding tag to each image patch allows the Transformer to capture the positional relationship of the image patches in space. Adding the position embedding tag to the linearly projected image patch representation yields a feature sequence with position information.
[0063] Step 2: Before token interaction in the search area, an adaptive background-aware mechanism is used to partition the search area. The input token sequence is classified into three categories: T1 (tokens generated from the template image and its central target patch), T2 (tokens highly correlated with the target in the search image), and T3 (tokens remaining in the search image). Background interference is filtered out and processing is focused on target-related regions. A trainable prediction module and Laplace distribution probabilities are used to accurately classify search tokens.
[0064] The input sequence needs to be divided into three different categories of tokens: T1, T2, and T3. T1 is all tokens generated from the template image and its central target block, representing the initial information of the target. T2 is the tokens in the search image that are highly correlated with the target and are mainly used for target perception modeling. T3 is the remaining token sequence in the search image, containing irrelevant or weakly correlated information.
[0065] The template tokens are aggregated through a global maximum pooling operation, so that the search token is injected with target-related information to generate target-aware feature representations. Subsequently, these target-aware feature representations are concatenated with each search token. The concatenated vectors are input into a lightweight multi-layer perceptron (MLP), which is responsible for predicting the probability of each search token belonging to the T2 and T3 categories. The Laplace distribution is used to optimize the category discrimination of the search token. Assuming that the feature difference f of the search token Δ It obeys the Laplace distribution, and its probability density function is:
[0066]
[0067] Among them, b is the scale parameter used to control the width of the distribution, f Δ is the feature difference. The final probability calculation formula combines the Softmax function with the feature representation of the MLP output and can be expressed as:
[0068] P=Softmax(Contact(MLP(T2,T3)))·p(f Δ )
[0069] Among them, the Softmax function is used to normalize the output category probability to ensure that the sum is 1, P (i,0) and P (i,1) denote the probability that the i-th search token is assigned to T2 and T3, respectively.
[0070] For these three sets of token sequences, the attention mechanism inputs in the encoder can be respectively analogized as: template input query vector Q1, key vector K1, value vector V1, strong target search inputs Q2, K2, V2, and weak target search inputs Q3, K3, V3.
[0071] The target's center block information is additionally embedded in the template features to further enhance the target information representation capability. The target center block features are extracted from the target region in the initial frame and dynamically updated during the subsequent tracking process to ensure target feature consistency.
[0072] Step 3: The encoder receives three sets of token sequences (T1, T2, T3) and uses two attention blocks to calculate the correlation between the template and the strong target search area and the template and the weak target search area. During the calculation process, the attention mechanism is weighted according to the feature dimension D, and the value vector V and the learnable weight W are combined to enhance the adaptability of the model. Finally, the two sets of fused sequences are summed and normalized to output the fused sequence after the interaction between the target and the background, as shown in the figure. Figure 3 shown.
[0073] The target encoder receives three sets of token sequences as input and implements cross-correlation modeling through an attention block. For the template token group T1 and the strong target search area token T2, the correlation between the template and the search area is calculated as follows:
[0074]
[0075] Among them, D is the feature dimension in the attention mechanism, V is the value vector used for weighting, and W is a learnable weight designed to further enhance the adaptability of the model.
[0076] For the template token group T1 and the weak target search area token T3, the same calculation method is used, and the formula is as follows:
[0077]
[0078] Through calculation, the model can adaptively adjust the interaction between the template and the search area to fully explore the target-related information.
[0079] After the first two steps of calculation, the fusion sequence of the template token T1 interacting with the strong target search area token T2 and the weak target search area token T3 will be obtained. These two sets of interaction sequences are added and standardized, and the fusion sequence after the interaction between the target and the background is output.
[0080] Step 4: The search area fusion sequence is used as the input of the prediction head and reinterpreted as a two-dimensional spatial feature map before being fed into the prediction head. A multi-scale feature fusion method is used to generate feature maps of different resolutions through a pyramid structure. After fusing the multi-scale feature maps, the final corner prediction result is obtained, as shown in the following example: Figure 4 shown.
[0081] Remap the search area fusion sequence to two-dimensional space to obtain the feature map F m , and standardize the features. The formula is as follows:
[0082] F m =reshape(F s ,H,W,C)
[0083] Among them, F s is the fusion token sequence, H, W, C are the width, height and number of channels respectively.
[0084] A multi-scale feature pyramid is constructed by applying a series of convolutional layers and upsampling interpolation layers to the input feature map to generate three feature maps of different resolutions. The three feature maps of different resolutions are restored to the same size as the original input and then added and fused to obtain the final multi-scale feature representation.
[0085] The corner regression head predicts the coordinates of the four corner points of the target box and predicts the probability of each corner point. During the training process, the regression loss and confidence loss are used for optimization. The L1 loss is used to measure the deviation between the predicted corner points and the actual corner points. The formula is as follows:
[0086]
[0087] Among them, L reg is the regression loss, (x i ,y i ) is the predicted coordinate, is the target's true coordinate. Binary Cross Entropy (BCE) loss is used for corner confidence loss to measure the deviation between the predicted corner probability and the true probability; the formula is as follows:
[0088]
[0089] Among them, P ij is the predicted probability of corner points, is the binary label of the real corner point (1 indicates the existence of the corner point, 0 indicates the background); the final loss function can be expressed as:
[0090] L total =λ1L1+λ2L conf
[0091] Among them, λ1 and λ2 are the weight hyperparameters of the loss, which are used to balance the regression error and the confidence error.
[0092] The present invention also provides an electronic device comprising a memory and a processor, wherein: the memory is used to store a computer program that can be run on the processor; the processor is used to execute the steps of the adaptive background-aware Transformer visual target tracking method described above when running the computer program.
[0093] The present invention also provides a storage medium having a computer program stored thereon, which, when executed by at least one processor, implements the steps of the above-mentioned adaptive background-aware Transformer visual target tracking method.
[0094] To further validate the effectiveness of this object tracking method, simulations were conducted on the accuracy and success rate of the object tracking method in this embodiment. The results are shown in Tables 1, 2, and 3. This experiment used only the GOT-10K, COCO, TrackingNet, and LaSOT datasets for 300 training rounds, and the model was evaluated on an online evaluation system. Overall, the experimental results demonstrate that the proposed method performs well on multiple large-scale and standard datasets, demonstrating strong robustness and generalization capabilities.
[0095] Table 1 Statistics of GOT-10K experimental results
[0096] AO <![CDATA[SR 0.50 ]]> <![CDATA[SR 0.75 ]]> Proposed method 0.735 0.835 0.712
[0097] Table 2 Statistics of LaSOT experimental results
[0098] AUC PNorm P Proposed method 0.705 0.799 0.768
[0099] Table 3 Statistics of TNLK experimental results
[0100] AUC P Proposed method 0.567 0.575
[0101] Where AO is the average overlap, which represents the average of the overlaps between all ground truth and estimated bounding boxes; SR is the percentage of successfully tracked frames with measured overlap exceeding a threshold (e.g., 0.5); P is the accuracy, PNorm is the normalized accuracy, and AUC is the area under the curve.
[0102] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.
Claims
1. A background-aware adaptive Transformer visual object tracking method, characterized by: The following steps are involved: S1: A pair of images in a given video sequence is processed and input into multiple encoder layers of the Transformer network to learn the relationship between the target and background areas; S2: Before token interaction in the search area, the search area is divided using an adaptive background perception mechanism, which divides the input token sequence into tokens T1 generated by the template image and its central target block, tokens T2 in the search image that are highly correlated with the target, and tokens T3 remaining in the search image; Filter out background interference and focus on target-related areas; accurately classify search tokens through trainable prediction modules and Laplace distribution probability; S3: The encoder receives three token sequences (T1, T2, T3) and calculates the correlation between the template and the strong target search area and the template and the weak target search area through two attention blocks, respectively, to obtain two fused sequences. The two fused sequences are summed and normalized, and the fused sequence after the interaction between the target and the background is output; S4: The search area fusion sequence is used as the input of the prediction head and reinterpreted as a two-dimensional spatial feature map before being fed into the prediction head. A multi-scale feature fusion method is used to generate feature maps of different resolutions through a pyramid structure. The final corner prediction result is obtained by fusing the multi-scale feature maps.
2. The adaptive background-aware Transformer visual object tracking method according to claim 1, characterized in that: In step S1, a pair of images in the video sequence includes an initial target template image and a search area image.
3. The adaptive background-aware Transformer visual object tracking method according to claim 1, characterized in that: The implementation process of step S1 is as follows: S1.1: Segment a pair of images in the input video sequence into multiple fixed-size image blocks; then flatten these image blocks and convert them into one-dimensional vectors; S1.2: For the flattened image block sequence, map each image block to C-dimensional space by linear projection; let each image block be x i , the mapped image block is represented as z i , which is expressed as follows: z i =Wx i +b Where W is the linear projection matrix, b is the offset, and C is the feature dimension after projection; S1.3: Add a learnable position embedding tag to each image block so that the Transformer can capture the position relationship of the image blocks in space. Add the position embedding tag to the linearly projected image block representation to obtain a feature sequence with position information.
4. The adaptive background-aware Transformer visual object tracking method according to claim 1, characterized in that The implementation process of step S2 is as follows: S2.1: The input sequence needs to be divided into three different categories of tokens, T1, T2, and T3; T1 is all tokens generated by the template image and its central target block, representing the initial information of the target; T2 is the tokens in the search image that are highly correlated with the target and are mainly used for target perception modeling; T3 is the remaining token sequence in the search image containing irrelevant or weakly correlated information; S2.2: Aggregate template tokens through a global max pooling operation, so that search tokens are injected with target-related information to generate target-aware feature representations; concatenate the feature representations with each search token; The concatenated vectors are fed into a lightweight multi-layer perceptron (MLP) to predict the probability of each search token belonging to the T2 and T3 categories. And use Laplace distribution to optimize the category discrimination of search tokens; assuming that the feature difference f of the search token Δ It obeys the Laplace distribution, and its probability density function is: Among them, b is the scale parameter used to control the width of the distribution, f Δ For characteristic differences; The final probability calculation formula combines the Softmax function with the feature representation output by the MLP: P=Softmax(Contact(MLP(T2,T3)))·p(f Δ ) Among them, the Softmax function is used to normalize the output category probability to ensure that the sum is 1, P (i,0) and P (i,1) denote the probability of assigning the i-th search token to T2 and T3 respectively; S2.3: For the three token sequences, the attention mechanism inputs in the encoder are analogous to: template input query vector Q1, key vector K1, value vector V1, strong target search inputs are Q2, K2, V2, and weak target search inputs are Q3, K3, V3; S2.4: The target’s center block information is additionally embedded in the template features to further enhance the target information representation capability. The features of the target’s center block are extracted from the target area in the initial frame and dynamically updated during the subsequent tracking process to ensure the target’s feature consistency.
5. The adaptive background-aware Transformer visual object tracking method according to claim 1, characterized in that: The implementation process of step S3 is as follows: S3.1: The target encoder receives three sets of token sequences as input and implements cross-correlation modeling through an attention block. For the template token group T1 and the strong target search area token T2, the correlation between the template and the search area is calculated: Where D is the feature dimension in the attention mechanism, V is the value vector used for weighting, and W is a learnable weight; S3.2: For the template token group T1 and the weak target search area token T3, calculate the correlation between the template and the search area: S3.3: After the first two steps of calculation, the fusion sequence of the template token T1 interacting with the strong target search area token T2 and the weak target search area token T3 is obtained. These two sets of fusion sequences are added and normalized, and the fusion sequence after the interaction between the target and the background is output.
6. The adaptive background-aware Transformer visual object tracking method according to claim 1, characterized in that: The implementation process of step S4 is as follows: S4.1: Remap the search region fusion sequence to two-dimensional space to obtain the feature map F m , and standardize the features: F m =reshape(F s ,H,W,C) Among them, F s is the fusion token sequence, H, W, C are width, height, and number of channels respectively; S4.2: Construct a multi-scale feature pyramid by applying a series of convolutional layers and upsampling interpolation layers to the input feature map to generate three feature maps of different resolutions. The three feature maps of different resolutions are restored to the same size as the original input and then added and fused to obtain the final multi-scale feature representation. S4.3: The corner regression head is used to predict the coordinates of the four corner points of the target box and the probability of each corner point. During the training process, the regression loss and confidence loss are used for optimization, and the L1 loss is used to measure the deviation between the predicted corner points and the actual corner points: Among them, L reg is the regression loss, (x i ,y i ) is the predicted coordinate, is the target's true coordinate; binary cross entropy loss is used for corner confidence loss to measure the deviation between the predicted corner probability and the true probability: Among them, P ij is the predicted probability of corner points, is the binary label of the true corner point; the final loss function is: L total =λ1L1+λ2L conf Among them, λ1 and λ2 are the weight hyperparameters of the loss, which are used to balance the regression error and the confidence error.
7. An electronic device, characterized in that: comprising a memory and a processor, wherein: a memory for storing computer programs capable of running on the processor; A processor, configured to execute the steps of the adaptive background-aware Transformer visual target tracking method according to any one of claims 1 to 6 when running the computer program.
8. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the adaptive background-aware Transformer visual target tracking method according to any one of claims 1 to 6.