A token convolution-based target tracking method, device and electronic equipment
Patent Information
- Application Number
- CN202610683671.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-18
- Publication Date
- 2026-08-18
AI Technical Summary
然而，这种方式会丢失大量的目标局部结构与细节信息，难以捕捉目标在复杂场景下的细粒度变化
Smart Images
Figure CN122597458A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and pattern recognition technology, and specifically relates to a target tracking method, device and electronic device based on token convolution. Background Technology
[0002] Object tracking is a core task in computer vision, with wide applications in scenarios such as video surveillance and autonomous driving. In recent years, with the widespread use of the Transformer architecture in vision tasks, Transformer-based object tracking methods have gradually become the mainstream tracking framework due to their powerful global modeling and feature interaction capabilities, significantly improving tracking robustness in complex scenarios.
[0003] Current mainstream Transformer tracking methods typically use tokens to globally represent template image features and fuse features through attention mechanisms. However, this approach loses a significant amount of local target structure and detail information, making it difficult to capture fine-grained changes in the target in complex scenes.
[0004] Therefore, how to enhance the expressive power of multi-dimensional features of targets and improve the accuracy of feature interaction are key issues that urgently need to be addressed in the current field of target tracking. Summary of the Invention
[0005] To address the aforementioned problems in the existing technology, this invention provides a target tracking method based on token convolution.
[0006] The technical problem to be solved by this invention is achieved through the following technical solution: A target tracking method based on token convolution includes: Obtain a pre-trained target tracking model; the target tracking model includes a block embedding module, a multi-token encoder, a token convolutional interaction module, and a prediction head; The template image and the search image are input into the target tracking model. The block embedding module converts the template image into the current template feature sequence and the search image into the current search feature sequence. The current template feature sequence, the current search feature sequence, and the preset multi-Token sequence are concatenated to obtain a composite input tensor. The composite input tensor is then processed by a multi-Token encoder to generate an enhanced multi-Token sequence and an enhanced search feature sequence that incorporate template feature information. Each sub-Token in the multi-Token sequence is used to describe different semantic attributes of the target. The Token Convolution Interaction Module reconstructs the enhanced multi-Token sequence into a grouped convolution kernel, and performs grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map. The multi-channel response feature map is processed by the prediction head to generate target tracking results, which include classification results and bounding box coordinates.
[0007] Optionally, the multi-token encoder includes stacked multi-layer Transformer blocks, each Transformer block including a multi-head attention layer, a shared feedforward network, and a LoRA adapter. The multi-token encoder processes the composite input tensor to generate enhanced multi-token sequences and enhanced search feature sequences, including: For each Transformer block, a multi-head attention layer is used to aggregate information from the composite input tensor to generate a multi-Token sequence after self-attention computation. After concatenating multiple token sequences to obtain a joint long input vector, it is fed in parallel into a shared feedforward network and a LoRA adapter. The shared feedforward network generates a reconstructed joint feature vector, and the LoRA adapter generates a feature offset. Based on the reconstruction of the joint feature vector and feature offset, a corrected joint long vector is generated. After all Transformer blocks have been iterated and processed, an enhanced multi-token sequence and an enhanced search feature sequence are generated.
[0008] Optionally, a reconstructed joint feature vector is generated through a shared feedforward network, including: The input joint long vector is expanded in dimension and nonlinearly activated by the first linear layer of the shared feedforward network to generate a high-dimensional activated feature vector. The high-dimensional activation feature vector is restored by using the second linear layer of the shared feedforward network to generate a reconstructed joint feature vector; the dimension of the reconstructed joint feature vector is the same as the dimension of the input joint long vector.
[0009] Optionally, after generating the feature offsets through the LoRA adapter, the modified joint feature vector is generated based on the reconstructed joint feature vector and the feature offsets, including: The input joint long vector is mapped to a low-rank space using a preset reduced-rank matrix in the LoRA adapter to obtain low-rank space features. After performing dimensionality restoration on the low-rank space features using the preset ascending-rank matrix in the LoRA adapter, feature offsets are generated based on the preset scaling factor. The reconstructed joint feature vector and feature offsets are summed element by element to generate the corrected joint long vector.
[0010] Optionally, the enhanced multi-token sequence is reconstructed into a grouped convolutional kernel through the Token convolution interaction module, and then grouped convolutional operations are performed with the enhanced search feature sequence to generate a multi-channel response feature map, including: The Token convolution interaction module maps the high-dimensional features of each sub-Token in the enhanced multi-Token sequence to the convolution kernel parameter space, and reconstructs the grouped convolution kernel; wherein, the grouped convolution kernel includes the sub-Token convolution kernel corresponding to each sub-Token; A grouped convolution strategy is adopted, in which the sub-token convolution kernel is convolved with the corresponding channel group in the enhanced search feature sequence separately to obtain the grouped convolution result; wherein, the grouped convolution result includes tensors of independent response features of a preset number of groups, the preset number being determined based on the number of sub-tokens; The tensors of each group of independent response features are spliced and reshaped along the channel dimension to generate a multi-channel response feature map.
[0011] Optionally, the method further includes: Obtain the joint loss function; where the joint loss function is constructed based on the classification loss, regression loss and intersection-union ratio loss. The classification loss is used to balance positive and negative samples, the regression loss is used to constrain the absolute error of the center point coordinates and width and height of the predicted bounding box, and the intersection-union ratio loss is used to optimize the overlap between the predicted bounding box and the true bounding box. The network parameters of the target tracking model are updated based on the joint loss function until a preset stopping condition is met, resulting in updated network parameters. The optimized target tracking model is then obtained based on the updated network parameters.
[0012] Optionally, the template image is converted into a current template feature sequence and the search image is converted into a current search feature sequence via the block embedding module, including: The template image and the search image are cropped separately by the block embedding module to obtain the cropped template image and the cropped search image. After normalizing the cropped template image and the cropped search image, their sizes are adjusted to obtain a template image of the first preset size and a search image of the second preset size. The template image of the first preset size and the search image of the second preset size are subjected to non-overlapping block segmentation and linear projection processing respectively to obtain the template feature sequence and the search feature sequence.
[0013] The present invention also provides a target tracking device based on token convolution, the device comprising: The acquisition module is used to acquire a pre-trained target tracking model; the target tracking model includes a block embedding module, a multi-token encoder, a token convolutional interaction module, and a prediction head. The conversion module is used to input the template image and the search image into the target tracking model. The block embedding module converts the template image into the current template feature sequence and the search image into the current search feature sequence. The enhancement module is used to concatenate the current template feature sequence, the current search feature sequence, and the preset multi-Token sequence to obtain a composite input tensor. The composite input tensor is then processed by the multi-Token encoder to generate an enhanced multi-Token sequence and an enhanced search feature sequence that integrate template feature information. Each sub-Token in the multi-Token sequence is used to describe different semantic attributes of the target. The grouped convolution module is used to reconstruct the enhanced multi-token sequence into a grouped convolution kernel through the token convolution interaction module, and perform grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map. The prediction module is used to process the multi-channel response feature map through the prediction head to generate target tracking results; the target tracking results include classification results and bounding box coordinates.
[0014] Optionally, the above-mentioned enhancement module is specifically used for: For each Transformer block, a multi-head attention layer is used to aggregate information from the composite input tensor to generate a multi-Token sequence after self-attention computation. After concatenating multiple token sequences to obtain a joint long input vector, it is fed in parallel into a shared feedforward network and a LoRA adapter. The shared feedforward network generates a reconstructed joint feature vector, and the LoRA adapter generates a feature offset. Based on the reconstruction of the joint feature vector and feature offset, a corrected joint long vector is generated. After all Transformer blocks have been iterated and processed, an enhanced multi-token sequence and an enhanced search feature sequence are generated.
[0015] Optionally, the above-mentioned enhancement module is also used for: The input joint long vector is expanded in dimension and nonlinearly activated by the first linear layer of the shared feedforward network to generate a high-dimensional activated feature vector. The high-dimensional activation feature vector is restored by using the second linear layer of the shared feedforward network to generate a reconstructed joint feature vector; the dimension of the reconstructed joint feature vector is the same as the dimension of the input joint long vector.
[0016] Optionally, the above-mentioned enhancement module is also used for: The input joint long vector is mapped to a low-rank space using a preset reduced-rank matrix in the LoRA adapter to obtain low-rank space features. After performing dimensionality restoration on the low-rank space features using the preset ascending-rank matrix in the LoRA adapter, feature offsets are generated based on the preset scaling factor. The reconstructed joint feature vector and feature offsets are summed element by element to generate the corrected joint long vector.
[0017] Optionally, the above-mentioned grouped convolution module is specifically used for: The Token convolution interaction module maps the high-dimensional features of each sub-Token in the enhanced multi-Token sequence to the convolution kernel parameter space, and reconstructs the grouped convolution kernel; wherein, the grouped convolution kernel includes the sub-Token convolution kernel corresponding to each sub-Token; A grouped convolution strategy is adopted, in which the sub-token convolution kernel is convolved with the corresponding channel group in the enhanced search feature sequence separately to obtain the grouped convolution result; wherein, the grouped convolution result includes tensors of independent response features of a preset number of groups, the preset number being determined based on the number of sub-tokens; The tensors of each group of independent response features are spliced and reshaped along the channel dimension to generate a multi-channel response feature map.
[0018] Optionally, the above-mentioned device is also used for: Obtain the joint loss function; where the joint loss function is constructed based on the classification loss, regression loss and intersection-union ratio loss. The classification loss is used to balance positive and negative samples, the regression loss is used to constrain the absolute error of the center point coordinates and width and height of the predicted bounding box, and the intersection-union ratio loss is used to optimize the overlap between the predicted bounding box and the true bounding box. The network parameters of the target tracking model are updated based on the joint loss function until a preset stopping condition is met, resulting in updated network parameters. The optimized target tracking model is then obtained based on the updated network parameters.
[0019] Optionally, the above conversion module is specifically used for: The template image and the search image are cropped separately by the block embedding module to obtain the cropped template image and the cropped search image. After normalizing the cropped template image and the cropped search image, their sizes are adjusted to obtain a template image of the first preset size and a search image of the second preset size. The template image of the first preset size and the search image of the second preset size are subjected to non-overlapping block segmentation and linear projection processing respectively to obtain the template feature sequence and the search feature sequence.
[0020] The present invention also provides an electronic device, the electronic device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described target tracking method based on token convolution.
[0021] The present invention also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described target tracking method based on token convolution.
[0022] This invention provides a target tracking method, apparatus, and electronic device based on token convolution. The method includes: acquiring a pre-trained target tracking model; inputting a template image and a search image into the target tracking model; converting the template image into a current template feature sequence and the search image into a current search feature sequence using a block embedding module; concatenating these with a preset multi-token sequence to obtain a composite input tensor; processing the composite input tensor using a multi-token encoder to generate an enhanced multi-token sequence and an enhanced search feature sequence that integrate template feature information; reconstructing the enhanced multi-token sequence into grouped convolution kernels using a token convolution interaction module; performing grouped convolution operations with the enhanced search feature sequence to generate a multi-channel response feature map; and processing the multi-channel response feature map using a prediction head to generate a target tracking result. This invention enhances the expressive power of multi-dimensional target features and improves the accuracy of feature interaction.
[0023] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description
[0024] Figure 1 This is a flowchart illustrating a target tracking method based on token convolution provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a process for generating template feature sequences and searching feature sequences provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a process for generating enhanced multi-token sequences and enhanced search feature sequences provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of a process for generating a reconstructed joint feature vector according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a process for generating a modified joint long vector according to an embodiment of the present invention; Figure 6 This is a schematic diagram of a process for generating a multi-channel response feature map according to an embodiment of the present invention; Figure 7 This is a flowchart illustrating an update model provided in an embodiment of the present invention; Figure 8 This is a framework diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0025] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.
[0026] To enhance the expressive power of multi-dimensional features of targets and improve the accuracy of feature interaction, this invention provides a target tracking method based on token convolution.
[0027] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.
[0028] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a target tracking method based on token convolution provided in an embodiment of the present invention, including: Step 101: Obtain the pre-trained target tracking model.
[0029] The target tracking model includes a block embedding module, a multi-token encoder, a token convolutional interaction module, and a prediction head. The block embedding module converts the input image into a one-dimensional feature sequence; the token convolutional interaction module uses the enhanced multi-token sequence as a dynamic convolutional kernel to perform local matching with the search features; and the prediction head outputs the target tracking result.
[0030] A multi-token encoder decomposes a single token into multiple parallel sub-tokens and adaptively enhances them. Each sub-token is designed to carry different semantic attributes of the target, such as geometric structure, texture features, and motion trends. These multiple parallel sub-tokens form a multi-token sequence. During model instantiation, this sequence serves as a resident state variable in subsequent feature co-computation, thereby enhancing the model's robustness to target deformation in complex backgrounds at the architectural level. Here, "token" can also be referred to as a token.
[0031] Furthermore, this token sequence is weighted using a normal distribution during initialization, rather than being pre-set manually. It also requires binding positional encoding, giving each sub-token its own positional semantic information, allowing it to distinguish the order and spatial meaning within its own sequence.
[0032] The aforementioned target tracking model needs to be pre-built. In some optional embodiments, the main body of the model can be constructed by defining a core class `VisionTransformerCE`. This core class inherits from the `VisionTransformer` base class and integrates the Candidate Elimination (CE) operator. After the model is built, it can be trained using pre-prepared training set data. During the initialization phase of model training, the resolution parameters of the input image can be explicitly defined as 256×256 pixels, the image patch size as 16×16 pixels, and the feature embedding dimension as 768 dimensions. This parameterized configuration ensures that the image can be accurately divided into a 16×16 non-overlapping image patch sequence during the input phase, providing a standardized input dimension for subsequent linear projection processing.
[0033] Step 102: Input the template image and the search image into the target tracking model. The template image is converted into the current template feature sequence and the search image is converted into the current search feature sequence through the block embedding module.
[0034] The template image is a pre-defined reference image containing the target. During target tracking, the target tracking model needs to extract the target's specific features, such as color, texture, and shape, from this template image to locate the target in the search image. The search image is the image in which the target's position needs to be located, and it is usually a complete scene.
[0035] After obtaining the template image and the search image, the template image and the search image can be input into the target tracking model. The input template image and the search image can be initialized, and the template image can be converted into the current template feature sequence and the search image can be converted into the current search feature sequence through the block embedding module.
[0036] In some alternative embodiments, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a process for generating template feature sequences and searching feature sequences provided by an embodiment of the present invention, including: Step 201: The template image and the search image are cropped using the block embedding module to obtain the cropped template image and the cropped search image.
[0037] Step 202: After normalizing the cropped template image and the cropped search image, adjust their sizes to obtain a template image of the first preset size and a search image of the second preset size.
[0038] Step 203: Perform non-overlapping block segmentation and linear projection processing on the template image of the first preset size and the search image of the second preset size respectively to obtain the template feature sequence and the search feature sequence.
[0039] Specifically, for the template image, cropping can be performed based on the center pixel coordinates of the target bounding box pre-annotated in the first frame image. For example, the cropping size can be set to 128×128 pixels to obtain the cropped template image, which is used to capture the initial appearance features of the target. For the search image, dynamic cropping can be performed according to the predicted position of the previous frame. For example, the cropping size can be set to 256×256 pixels to obtain the cropped search image, which covers the potential motion area of the target.
[0040] Next, the cropped template image and the cropped search image can be normalized. Then, the bilinear interpolation algorithm is used to adjust them to the standard input size preset by the model, thereby obtaining the template image of the first preset size and the search image of the second preset size.
[0041] For the cropped template image and the cropped search image, the nn.Conv2d operator can be used to traverse the image in the form of a non-overlapping sliding window, aggregating the spatial information of each 16×16 pixel block into a 768-dimensional feature vector. In this nn.Conv2d operator, the convolution kernel size is pre-set to 16×16, the stride is 16, and the number of output channels is 768. The kernel size corresponds to the size of the image block, with each kernel covering a 16×16 pixel block. The stride indicates that the kernels slide without overlap, thus precisely segmenting the image into non-overlapping 16×16 image blocks. The number of output channels indicates that each 16×16 pixel block is linearly projected into a 768-dimensional feature vector. After this operation, the tensor dimension of the image can be transformed from [B, 3, 256, 256] to [B, 768, 16, 16], that is, the image is discretized in space into a 16×16 grid, where B represents the batch size and 3 represents the RGB three channels.
[0042] Next, dimensional recombination can be performed. Specifically, the flatten(2) operator can be called to merge and flatten the second-dimensional height and the third-dimensional width, generating an intermediate tensor with dimensions [B, 768, 256]. Subsequently, the transpose(1,2) operator can be called to exchange the channel dimension and the sequence dimension. Through this transpose operation, the tensor shape finally evolves to [B, 256, 768], i.e., BNC format, where N=256 represents the number of image patches and C=768 represents the feature embedding dimension, ultimately yielding the corresponding template feature sequence and search feature sequence.
[0043] Step 103: Concatenate the current template feature sequence, the current search feature sequence, and the preset multi-Token sequence to obtain a composite input tensor. Process the composite input tensor through a multi-Token encoder to generate an enhanced multi-Token sequence and an enhanced search feature sequence that integrate template feature information.
[0044] In this multi-token sequence, each sub-token is used to describe different semantic attributes of the target. The composite input tensor is formed by concatenating the multi-token sequence, template feature sequence, and search feature sequence sequentially along the sequence dimension. Let the batch size be B, the number of sub-tokens be J, the feature embedding dimension be C, the template feature sequence length be L_c, and the search feature sequence length be L_s, then the shape of the composite input tensor is (B, J+L_c+L_s, C). For example, when B=1, J=3, C=768, L_c=144, and L_s=576, the total sequence length of the composite input tensor is 723, containing 3 sub-tokens, 144 template image patch features, and 576 search image patch features.
[0045] The composite input tensor is then processed by a multi-token encoder to generate an enhanced multi-token sequence and an enhanced search feature sequence that incorporate template feature information. In some optional embodiments, the multi-token encoder includes stacked multi-layer Transformer blocks, each Transformer block including a multi-head attention layer, a shared feedforward network, and a LoRA adapter. Here, a lightweight shared feedforward network and LoRA adapter can be used, thereby enabling data processing without significantly increasing the computational load.
[0046] like Figure 3 As shown, Figure 3 This is a schematic diagram of a process for generating enhanced multi-token sequences and enhanced search feature sequences according to an embodiment of the present invention, including: Step 301: For each Transformer block, perform information aggregation processing on the composite input tensor through a multi-head attention layer to generate a multi-Token sequence after self-attention computation.
[0047] Step 302: After concatenating the multiple token sequences to obtain the input joint long vector, input it in parallel into the shared feedforward network and LoRA adapter. The shared feedforward network generates the reconstructed joint feature vector, and the LoRA adapter generates the feature offset.
[0048] Step 303: Based on the reconstructed joint feature vector and feature offset, make corrections to generate the corrected joint long vector.
[0049] Step 304: After all Transformer blocks have been iterated and processed, generate the enhanced multi-Token sequence and the enhanced search feature sequence.
[0050] In this process, after the aforementioned embedding module transforms the input image, the feature vectors do not yet contain information about their interrelationships. Here, iterative context modeling of the composite input tensor is performed using stacked multi-layer Transformer blocks to achieve interaction between feature vectors, thereby generating a multi-token sequence after self-attention computation. The dimensions of this multi-token sequence are [B, 3, 768].
[0051] Within each Transformer block, a multi-head attention layer enables each image patch to aggregate information from other locations, generating context-aware features. Specifically, within the multi-head attention layer, each position in the composite input tensor generates a query vector, key vector, and value vector through a linear mapping. The dot product of each query vector and all key vectors is calculated, scaled, and normalized using Softmax to obtain attention weights. Then, the value vectors are weighted and summed to obtain the output features for the current position, thus generating a multi-token sequence after attention computation. This process is executed independently in multiple parallel attention heads, and the outputs are concatenated along the channels and fused using a linear transformation. This approach breaks down the boundary between the template image and the search image, allowing the multi-token to simultaneously absorb discriminative semantic information from both the template and search feature sequences.
[0052] The multi-token sequence is then concatenated to obtain the input joint long vector. This can be achieved by concatenating the sub-tokens along the channel dimension, transforming them into an input joint long vector of dimensions [B, 1, 2304]. This input joint long vector is then fed in parallel into a shared feedforward network and a LoRA adapter, where the shared feedforward network generates a reconstructed joint feature vector. This process enforces semantic coupling between sub-tokens in an extremely high-dimensional space, uncovering the correlation between targets across different dimensions.
[0053] In some alternative embodiments, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a process for generating a reconstructed joint feature vector according to an embodiment of the present invention, including: Step 401: The input joint long vector is expanded in dimension and nonlinearly activated through the first linear layer of the shared feedforward network to generate a high-dimensional activated feature vector.
[0054] Step 402: Perform dimensionality restoration processing on the high-dimensional activation feature vector through the second linear layer of the shared feedforward network to generate a reconstructed joint feature vector.
[0055] The shared feedforward network includes a first linear layer and a second linear layer. The first linear layer expands the dimension of the input joint long vector to 9216 dimensions, and uses the GeLU nonlinear activation function to activate features, thereby generating a high-dimensional activation feature vector.
[0056] The high-dimensional activation feature vector of 9216 dimensions is then input into the second linear layer. A linear transformation restores the dimension from 9216 to the original dimension of the input joint long vector, which is 3×C. If the dimension of each sub-token is C=1024, then the dimension of the input joint long vector is 3072. It should be noted that the second linear layer does not apply an activation function to maintain the linear mapping capability of the feature representation. The final generated joint feature vector has the same dimension as the input joint long vector.
[0057] After generating feature offsets using the LoRA adapter, the joint feature vector can be corrected based on the reconstructed joint feature vector and the feature offsets to generate a corrected joint long vector. In some optional embodiments, such as Figure 5 As shown, Figure 5 This is a schematic diagram of a process for generating a modified joint long vector according to an embodiment of the present invention, including: Step 501: Perform low-rank space mapping on the input joint long vector using the preset reduced-rank matrix in the LoRA adapter to obtain low-rank space features.
[0058] Step 502: After performing dimension restoration processing on the low-rank spatial features using the preset rising-rank matrix in the LoRA adapter, generate feature offsets based on the preset scaling factor.
[0059] Step 503: Sum the reconstructed joint feature vector and feature offset element by element to generate the corrected joint long vector.
[0060] In order to enable the model to sensitively capture subtle appearance changes during the tracking process, the input joint long vector can be input in parallel into the LoRA adapter. The LoRA adapter includes a preset descending rank matrix and a preset ascending rank matrix. Optionally, the dimension of the preset descending rank matrix is 2304×8 and the dimension of the preset ascending rank matrix is 8×2304.
[0061] Therefore, the input joint long vector can be mapped to a low-rank space using a preset reduced-rank matrix in the LoRA adapter to obtain low-rank space features. Then, the low-rank space features are restored to their dimensions using a preset increased-rank matrix in the LoRA adapter, and feature offsets are generated based on preset scaling factors.
[0062] Next, the reconstructed joint feature vector and feature offsets can be summed element-wise to generate a corrected joint long vector, achieving accurate correction of the multi-token sequence. Finally, after all Transformer blocks have been iterated, an enhanced multi-token sequence and an enhanced search feature sequence are generated. An enhanced template feature sequence can also be output here.
[0063] Optionally, a background token removal module can be embedded in specific layers of the multi-Token encoder. The specific layers can be selected by starting from the middle layer of the multi-Token encoder and selecting at even intervals, for example, embedding at layers 3, 6, and 9 respectively. This background token removal module can use automatically generated classification tokens as a benchmark to calculate the attention weight between the token and each token in the search region. Based on the magnitude of the attention weight, tokens below a preset threshold are dynamically removed. These removed tokens correspond to background image patches in the search image, thereby purifying the features of the search region, reducing background interference, and improving tracking robustness.
[0064] Step 104: Reconstruct the enhanced multi-token sequence into a grouped convolution kernel through the Token convolution interaction module, and perform grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map.
[0065] In this invention, instead of employing traditional global cross-correlation or attention fusion, the enhanced multi-token sequence is treated as a set of dynamically generated filters. In some optional embodiments, such as... Figure 6 As shown, Figure 6This is a schematic flowchart of a process for generating a multi-channel response feature map provided by an embodiment of the present invention, including: Step 601: Map the high-dimensional features of each sub-Token in the enhanced multi-Token sequence to the convolution kernel parameter space through the Token convolution interaction module, and reconstruct to obtain the grouped convolution kernel.
[0066] Step 602: Using a grouped convolution strategy, the sub-Token convolution kernel is convolved with the corresponding channel group in the enhanced search feature sequence individually to obtain the grouped convolution result.
[0067] Step 603: Concatenate and reshape the tensors of each group of independent response features along the channel dimension to generate a multi-channel response feature map.
[0068] The grouped convolutional kernel includes a sub-token convolutional kernel corresponding to each sub-token. Through the token convolution interaction module, the high-dimensional features of each sub-token in the enhanced multi-token sequence can be mapped to the convolutional kernel parameter space. For example, each sub-token is mapped to a parameter set with 768×3×3 weights, and then the reshape operator is called to reshape it into the standard convolutional kernel format of [J, 768, 3, 3], that is, the grouped convolutional kernel is reconstructed.
[0069] To maintain the independence of different sub-tokens, a grouped convolution strategy can be adopted. The enhanced search feature sequence is used as the convolution object, and the grouped convolution kernel is used as the convolution operator. The sub-token convolution kernel is convolved separately with the corresponding channel group in the enhanced search feature sequence to obtain the grouped convolution result. The grouped convolution result includes tensors of a preset number of independent response features, the preset number being determined based on the number of sub-tokens. This grouping interaction mechanism effectively avoids feature aliasing between different attributes, ensuring that each sub-token can independently find its represented response feature point in the search region.
[0070] After the grouped convolution output, a tensor containing J groups of independent response features can be obtained. The tensors of each group of independent response features are then concatenated and reshaped along the channel dimension to generate a multi-channel response feature map.
[0071] Step 105: Process the multi-channel response feature map using the prediction head to generate target tracking results.
[0072] The target tracking result includes classification results and bounding box coordinates. During the processing of the multi-channel response feature map by the prediction head, the Sigmoid activation function can be used to restrict the response value of each pixel to within [0, 1], representing the probability that the location belongs to the target, thus obtaining the classification result. The pixel with the highest response value can be used as the target's prediction center, and combined with the size prediction value output by the prediction head, the bounding box coordinates of the target in the current frame can be calculated.
[0073] If the maximum response value is lower than the preset threshold, it is determined that the target may be completely occluded or moved out of the field of view. At this time, the model enters the re-detection mode or keeps the multi-Token state of the previous frame unchanged until the target reappears.
[0074] In some alternative embodiments, the target tracking model can also be optimized end-to-end using a joint loss function to ensure that the model achieves an optimal balance between classification accuracy and bounding box localization accuracy. For example... Figure 7 As shown, Figure 7 This is a flowchart illustrating an update model provided in an embodiment of the present invention, including: Step 701: Obtain the joint loss function.
[0075] Step 702: Update the network parameters of the target tracking model based on the joint loss function until a preset stopping condition is reached to obtain the updated network parameters, and obtain the optimized target tracking model based on the updated network parameters.
[0076] The joint loss function is constructed based on classification loss, regression loss, and intersection-union ratio (IURR) loss. The classification loss is used to balance positive and negative samples, the regression loss is used to constrain the absolute error of the center point coordinates and width and height dimensions of the predicted bounding box, and the IURR loss is used to optimize the overlap between the predicted bounding box and the true bounding box.
[0077] Furthermore, weighting coefficients for the classification loss, regression loss, and intersection-union ratio (IU) loss can be pre-set. The total loss can then be calculated using these weighting coefficients. This total loss allows the model to simultaneously consider both the prediction of the target's existence probability and the regression of precise geometric attributes such as bounding box coordinates during training. For example, the weighting coefficient for the classification loss can be set to 1, the weighting coefficient for the regression loss can be set to 5, and the weighting coefficient for the IU loss can be set to 2.
[0078] This invention proposes a target tracking method based on token convolution. The method includes: acquiring a pre-trained target tracking model, wherein the target tracking model includes a block embedding module, a multi-token encoder, a token convolution interaction module, and a prediction head; inputting a template image and a search image into the target tracking model; converting the template image into a current template feature sequence and the search image into a current search feature sequence through the block embedding module; concatenating the current template feature sequence, the current search feature sequence, and a preset multi-token sequence to obtain a composite input tensor; processing the composite input tensor through the multi-token encoder to generate an enhanced multi-token sequence and an enhanced search feature sequence that integrate template feature information; reconstructing the enhanced multi-token sequence into a grouped convolution kernel through the token convolution interaction module and performing a grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map; and processing the multi-channel response feature map through the prediction head to generate a target tracking result. This invention uses a multi-token encoder to decompose a single token into multiple collaborative sub-tokens to describe different semantic attributes of the target. By leveraging the parallel representation capabilities of multiple sub-tokens, the expressive power of the target's multi-dimensional features is enhanced. Furthermore, when faced with drastic target deformation or cluttered backgrounds, the model can accurately locate the geometric center of the target in the search region through the collaborative response of multiple sub-tokens, significantly improving the accuracy of target tracking. Simultaneously, the introduction of a token convolution interaction mechanism, which explicitly uses tokens as convolution kernels for local feature detection, enhances the model's ability to capture fine-grained information about the target, thereby improving the accuracy of feature interaction.
[0079] Based on the same inventive concept, embodiments of the present invention also provide a target tracking device based on token convolution, the device comprising: The acquisition module is used to acquire the pre-trained target tracking model; the target tracking model includes a block embedding module, a multi-token encoder, a token convolution interaction module, and a prediction head.
[0080] The conversion module is used to input template images and search images into the target tracking model. The block embedding module converts the template image into the current template feature sequence and the search image into the current search feature sequence.
[0081] The enhancement module is used to concatenate the current template feature sequence, the current search feature sequence, and the preset multi-Token sequence to obtain a composite input tensor. The composite input tensor is then processed by the multi-Token encoder to generate an enhanced multi-Token sequence and an enhanced search feature sequence that integrate template feature information. Each sub-Token in the multi-Token sequence is used to describe different semantic attributes of the target.
[0082] The grouped convolution module is used to reconstruct the enhanced multi-token sequence into a grouped convolution kernel through the token convolution interaction module, and perform grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map.
[0083] The prediction module is used to process the multi-channel response feature map through the prediction head to generate target tracking results; the target tracking results include classification results and bounding box coordinates.
[0084] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804. Memory 803 is used to store computer programs; When the processor 801 executes the program stored in the memory 803, it implements the steps of the above-described target tracking method based on token convolution.
[0085] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.
[0086] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0087] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0088] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0089] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the above-described target tracking method based on token convolution.
[0090] Optionally, the computer-readable storage medium may be non-volatile memory (NVM), such as at least one disk storage device.
[0091] Optionally, the computer-readable storage medium may also be at least one storage device located remotely from the aforementioned processor.
[0092] It should be noted that, for the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple. For relevant parts, please refer to the description of the method embodiments. All embodiments of the above-described target tracking method based on token convolution are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.
[0093] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.
Claims
1. A target tracking method based on token convolution, characterized in that, The method includes: Obtain a pre-trained target tracking model; wherein the target tracking model includes a block embedding module, a multi-token encoder, a token convolutional interaction module, and a prediction head; The template image and the search image are input into the target tracking model. The block embedding module converts the template image into a current template feature sequence and the search image into a current search feature sequence. The current template feature sequence, the current search feature sequence, and the preset multi-Token sequence are concatenated to obtain a composite input tensor. The composite input tensor is then processed by the multi-Token encoder to generate an enhanced multi-Token sequence and an enhanced search feature sequence that integrate template feature information. Each sub-Token in the multi-Token sequence is used to describe different semantic attributes of the target. The enhanced multi-Token sequence is reconstructed into a grouped convolutional kernel by the Token convolution interaction module, and then grouped convolution operation is performed with the enhanced search feature sequence to generate a multi-channel response feature map. The prediction head processes the multi-channel response feature map to generate target tracking results; wherein, the target tracking results include classification results and bounding box coordinates.
2. The method according to claim 1, characterized in that, The multi-token encoder comprises stacked multi-layer Transformer blocks, each Transformer block including a multi-head attention layer, a shared feedforward network, and a LoRA adapter. The process of processing the composite input tensor through the multi-token encoder to generate an enhanced multi-token sequence and an enhanced search feature sequence incorporating template feature information includes: For each Transformer block, the multi-head attention layer performs information aggregation processing on the composite input tensor to generate a multi-Token sequence after self-attention computation; After concatenating the multi-token sequences to obtain the input joint long vector, it is input in parallel into the shared feedforward network and the LoRA adapter. The shared feedforward network generates the reconstructed joint feature vector, and the LoRA adapter generates the feature offset. Based on the reconstructed joint feature vector and the feature offset, a corrected joint long vector is generated. After all the Transformer blocks have been iterated and processed, the enhanced multi-Token sequence and the enhanced search feature sequence are generated.
3. The method according to claim 2, characterized in that, The step of generating the reconstructed joint feature vector through the shared feedforward network includes: The input joint long vector is subjected to dimensional expansion and nonlinear activation processing through the first linear layer of the shared feedforward network to generate a high-dimensional activation feature vector. The high-dimensional activation feature vector is restored by performing dimensionality recovery processing through the second linear layer of the shared feedforward network to generate the reconstructed joint feature vector; wherein the dimension of the reconstructed joint feature vector is the same as the dimension of the input joint long vector.
4. The method according to claim 2, characterized in that, After generating the feature offset through the LoRA adapter, the feature vector is corrected based on the reconstructed joint feature vector and the feature offset to generate a corrected joint long vector, including: The input joint long vector is mapped to a low-rank space using a preset reduced-rank matrix in the LoRA adapter to obtain low-rank space features. After performing dimensionality restoration on the low-rank spatial features using the preset ascending-rank matrix in the LoRA adapter, the feature offset is generated based on the preset scaling factor. The reconstructed joint feature vector and the feature offset are summed element by element to generate the corrected joint long vector.
5. The method according to any one of claims 1-4, characterized in that, The step of reconstructing the enhanced multi-token sequence into a grouped convolutional kernel through the token convolution interaction module, and performing a grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map includes: The Token convolution interaction module maps the high-dimensional features of each sub-Token in the enhanced multi-Token sequence to the convolution kernel parameter space, and reconstructs the grouped convolution kernel; wherein, the grouped convolution kernel includes the sub-Token convolution kernel corresponding to each sub-Token; A grouped convolution strategy is adopted, in which the sub-token convolution kernel is convolved with the corresponding channel group in the enhanced search feature sequence individually to obtain the grouped convolution result; wherein, the grouped convolution result includes tensors of a preset number of independent response features, the preset number being determined based on the number of sub-tokens; The tensors of each group of independent response features are concatenated and reshaped along the channel dimension to generate the multi-channel response feature map.
6. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the joint loss function; wherein the joint loss function is constructed based on classification loss, regression loss and intersection-union ratio loss, the classification loss is used to balance positive and negative samples, the regression loss is used to constrain the absolute error of the center point coordinates and width and height of the predicted bounding box, and the intersection-union ratio loss is used to optimize the overlap between the predicted bounding box and the true bounding box; The network parameters of the target tracking model are updated based on the joint loss function until a preset stopping condition is reached, resulting in updated network parameters. An optimized target tracking model is then obtained based on the updated network parameters.
7. The method according to any one of claims 1-4, characterized in that, The step of converting the template image into a current template feature sequence and the search image into a current search feature sequence through the block embedding module includes: The template image and the search image are cropped using the block embedding module to obtain the cropped template image and the cropped search image. After normalizing the cropped template image and the cropped search image, their sizes are adjusted to obtain a template image of a first preset size and a search image of a second preset size. The template image of the first preset size and the search image of the second preset size are subjected to non-overlapping block segmentation and linear projection processing, respectively, to obtain the template feature sequence and the search feature sequence.
8. A target tracking device based on token convolution, characterized in that, The device includes: An acquisition module is used to acquire a pre-trained target tracking model; wherein the target tracking model includes a block embedding module, a multi-token encoder, a token convolution interaction module, and a prediction head; The conversion module is used to input the template image and the search image into the target tracking model, and convert the template image into a current template feature sequence and the search image into a current search feature sequence through the block embedding module; The enhancement module is used to concatenate the current template feature sequence, the current search feature sequence, and a preset multi-Token sequence to obtain a composite input tensor. The multi-Token encoder processes the composite input tensor to generate an enhanced multi-Token sequence and an enhanced search feature sequence that integrate template feature information. The grouped convolution module is used to reconstruct the enhanced multi-Token sequence into a grouped convolution kernel through the Token convolution interaction module, and perform grouped convolution operation with the enhanced search feature sequence to generate a multi-channel response feature map. The prediction module is used to process the multi-channel response feature map through the prediction head to generate target tracking results; wherein, the target tracking results include classification results and bounding box coordinates.
9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the target tracking method based on token convolution as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the target tracking method based on token convolution as described in any one of claims 1-7.