A single-target tracking method based on space-frequency domain feature enhancement

Abstract

The present application relates to a kind of single target tracking methods based on space-frequency domain feature enhancement, comprising: obtaining video frame sequence, determining template image, and determining the i th search area image according to preset selection strategy;Based on template image, the i th search area image and the i th global token, generate the i th global token;The i th global token is input into improved lightweight Transform network, and the i th global token is extracted in multiple stages, and the image features of each stage and the i th global token updated are output;Search area feature corresponding to search area is extracted, to carry out the dynamic enhancement and fusion of space and frequency domain, and the search area feature of space-frequency domain joint enhancement is output;According to the i th global token and the search area feature of space-frequency domain joint enhancement, the target boundary frame of target in the i th video frame is predicted.The method can realize the good balance between tracking precision and speed.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a single-target tracking method based on spatial frequency domain feature enhancement. Background Technology

[0002] Single-object tracking is one of the core tasks in computer vision, aiming to continuously and stably locate a specified target in a video sequence. This technology has wide applications in embedded or mobile scenarios such as autonomous driving, intelligent monitoring, and drone inspection. With the development of deep learning, Transformer-based tracking methods have made significant progress due to their powerful global modeling capabilities; however, their high computational complexity and large number of parameters make it difficult to achieve real-time tracking on resource-constrained platforms.

[0003] To reduce computational overhead, existing technologies attempt to achieve efficient tracking through model pruning, quantization, or the design of lightweight network structures. While these solutions can significantly improve the real-time performance of inference in tracking tasks, the following problems still exist in practical applications: First, existing lightweight networks are insufficient in extracting target features in tracking tasks. Especially in complex scenarios such as drastic changes in target appearance, cluttered backgrounds, and low light, they are unable to simultaneously retain global contextual information and local details, resulting in a decrease in feature discrimination power and thus affecting tracking accuracy and stability.

[0004] Second, some methods attempt to introduce spatial or frequency domain feature enhancements to improve robustness, but they usually only focus on a single domain (such as enhancing only spatial details or suppressing only frequency noise), lack a collaborative enhancement mechanism for spatial and frequency domain features, and are not tightly integrated with the Transformer architecture, making it difficult to fully utilize the global interaction capabilities of the Transformer to achieve deep fusion of features from both domains.

[0005] Third, existing lightweight tracking methods often struggle to achieve an ideal balance between tracking accuracy and inference speed: high-precision methods are computationally intensive and slow, making it difficult to meet real-time requirements; while speed-oriented methods suffer significant accuracy loss, especially in complex scenarios where tracking drift and target loss are serious problems.

[0006] In summary, how to fully explore and integrate spatial and frequency domain features within a lightweight Transformer framework while maintaining low computational overhead and high tracking robustness is a technical problem that urgently needs to be solved in this field, but for which no effective solution has yet been found. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a single-target tracking method based on spatial frequency domain feature enhancement. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a single-target tracking method based on spatial frequency domain feature enhancement, comprising: Obtain the video frame sequence, use the target region in the first video frame as the template image, and determine the i-th search region image according to the preset selection strategy, where 2≤i≤I, and I is the total number of frames in the video frame sequence; Based on the template image, the i-th search region image, and the i-th global word element, generate the i-th global word element; The i-th all-terminal is input into the improved lightweight Transformer network, which includes a backbone network, an adaptive hierarchical spatial-frequency domain refinement module, and a single-target tracking head, wherein: The backbone network is used to perform multi-stage feature extraction on the i-th global word, and output the image features of each stage and the updated i-th global word. The adaptive hierarchical spatial-frequency domain refinement module is used to extract search region features corresponding to the search region from the image features of each stage, dynamically enhance and fuse the search region features in the spatial and frequency domains, and output the jointly enhanced search region features in the spatial and frequency domains. The single-target tracking head is used to predict the target bounding box in the i-th video frame based on the i-th global lexical unit and the search region features jointly enhanced by the spatial and frequency domains.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: To address the challenge of balancing tracking accuracy and computational cost in existing lightweight tracking methods, this invention provides a single-target tracking method based on spatial-frequency domain feature enhancement. This method converts the template image and search region image into tokens and concatenates them with global tokens. Then, it utilizes an improved lightweight Transformer network, including a backbone network, an adaptive hierarchical spatial-frequency domain refinement module, and a tracking head, for processing. This allows for the efficient extraction of spatial details and frequency-domain anti-interference features of the target while maintaining a lightweight structure, achieving efficient fusion of multi-scale features. This method effectively alleviates the problems of insufficient feature discrimination and susceptibility to background interference in complex scenes found in existing lightweight tracking methods. It improves tracking accuracy while maintaining low computational cost, thus achieving a good balance between tracking speed and robustness. This makes it suitable for applications with high real-time requirements, such as autonomous driving and intelligent monitoring. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the data flow in the improved lightweight Transformer network provided in this embodiment of the invention; Figure 2This is a schematic diagram of the network architecture and data flow of a single LightViT structure block provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the network architecture and data flow of the image block merging module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network structure of the spatial dynamic fusion unit provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the network structure of the frequency domain dynamic fusion unit provided in an embodiment of the present invention. Detailed Implementation

[0010] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0011] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0012] The single-target tracking method based on spatial frequency domain feature enhancement proposed in this invention will now be described in detail with reference to the accompanying drawings.

[0013] Specifically, this method includes: S1: Obtain the video frame sequence, use the target region in the first video frame as the template image, and determine the i-th search region image according to the preset selection strategy, 2≤i≤I, where I is the total number of frames in the video frame sequence.

[0014] Here, the video frame sequence can come from real-time camera capture, stored video files, or other image sequence sources. Each frame is an image at a specific moment, with frame numbers ranging from 1 to I. In the first frame, the location of the target (usually a rectangular box) is determined through manual annotation or known target detection results. The image region within this rectangular box is cropped out and used as a template image for matching and locating the same target in subsequent frames.

[0015] For the i-th frame (i starts from frame 2 and ends at frame 1), a search region is needed, which is a candidate region in that frame that may contain the target. The i-th search region image is obtained as follows: obtain the target bounding box of the (i-1)-th video frame; in the i-th video frame, using the center coordinates of the target bounding box of the (i-1)-th video frame as the center, expand the target bounding box of the (i-1)-th video frame according to a preset ratio to obtain the i-th candidate region; crop and scale the i-th candidate region to the preset search region size to obtain the i-th search region image.

[0016] It should be understood that "preset selection strategy" is a higher-level concept, representing a predetermined method. In a specific embodiment of the present invention, this strategy is as follows: based on the target position tracked in the (i-1)th frame (i.e., the output bounding box of the previous frame), in the current i-th frame, the bounding box is expanded by a certain proportion (e.g., 2 to 3 times) with the center of that position as the center, forming a rectangular region. Then, it is cropped and scaled to a fixed size (e.g., 256×256 pixels) to obtain the search region image. Using this strategy of "determining the search region of the current frame based on the tracking results of the previous frame" can significantly reduce the spatial range that needs to be processed and improve computational efficiency.

[0017] S2: Generate the i-th global lexical unit based on the template image, the i-th search region image, and the i-th global lexical unit.

[0018] Specifically, S2 includes: performing 8x downsampling, block embedding, flattening and splicing operations on the template image and the i-th search region image in sequence to obtain the i-th image word; splicing the i-th image word and the i-th global word to obtain the i-th global word.

[0019] After operation S2, the original template image and search region image have been transformed into a structured word sequence (i.e., the i-th all-word sequence). This all-word sequence simultaneously contains the target prior information of the template, the candidate region information of the current frame, and learnable global context information. However, how to efficiently extract discriminative multi-scale features from these words, how to enhance the robustness of features in complex scenes, and how to achieve a balance between accuracy and speed in a lightweight network are the core problems that this invention needs to solve. To this end, S3 will be the focus of this invention. By constructing an improved lightweight Transformer network, deep feature extraction, joint spatial-frequency domain enhancement, and target position prediction are performed on the i-th all-word sequence, thereby achieving high-precision, real-time single-target tracking.

[0020] S3: Input the i-th all-terminal into the improved lightweight Transformer network (e.g., ... Figure 1As shown), the improved lightweight Transformer network includes a backbone network, an adaptive hierarchical spatial-frequency domain refinement module, and a single-target tracking head, wherein: The backbone network is used to perform multi-stage feature extraction on the i-th global word and output the image features of each stage and the updated i-th global word. The adaptive hierarchical spatial-frequency domain refinement module is used to extract the search region features corresponding to the search region from the image features at each stage, and to dynamically enhance and fuse the search region features in the spatial and frequency domains, outputting the jointly enhanced search region features in the spatial and frequency domains. A single-target tracking head is used to predict the target bounding box in the i-th video frame based on the search region features jointly enhanced by the i-th global term and the spatial-frequency domain.

[0021] The data processing steps for the i-th all-terminal in the improved lightweight Transformer network are now explained with reference to the accompanying figures and the improved model structure.

[0022] Specifically, the backbone network is an improved LightViT backbone network; the improved LightViT backbone network includes multiple sequentially connected feature extraction stages, each feature extraction stage including at least two LightViT structural blocks; an image block merging module is set between two adjacent feature extraction stages; wherein: The image words output by each feature extraction stage constitute the image features of that stage. The global words in each feature extraction stage are passed sequentially between units, with the global words output by the last feature extraction stage serving as the updated i-th global word.

[0023] Taking a backbone network comprising three feature extraction stages as an example: The first feature extraction stage contains two cascaded LightViT blocks with 96 output channels; the second feature extraction stage contains two cascaded LightViT blocks with 192 output channels; and the third feature extraction stage contains six cascaded LightViT blocks with 384 output channels. An image block merging module is set between adjacent stages to spatially downsample and channel-adjust all words output from the previous stage before feeding them into the next stage.

[0024] Within each feature extraction stage, all LightViT blocks share the same global vocabulary: the first LightViT block in this stage receives all vocabulary from the previous stage (or the initial input), updates it through attention calculation within the block, and then passes it to the next LightViT block in this stage, and so on, until the last LightViT block in this stage outputs the updated global vocabulary. This updated vocabulary is then passed to the next stage via the image block merging module (or directly). Therefore, all vocabulary in each feature extraction stage is passed sequentially between stages, and each stage's vocabulary carries the global contextual information and local detail information accumulated up to that stage. Simultaneously, the image vocabulary output from each feature extraction stage (i.e., the image portion output by the last LightViT block in that stage) is reorganized with appropriate dimensions to constitute the image features of that stage. For example, the image vocabulary output from the first stage, after reorganization, yields a resolution equal to the original... Figure 1 8. Feature map with 96 channels; the image words output in the second stage are recombined to obtain a resolution equal to the original. Figure 1 / 16. Feature map with 192 channels; the image words output in the third stage are recombined to obtain a resolution equal to the original. Figure 1 / 32, feature maps with 384 channels. These multi-scale image features are fed into a subsequent adaptive hierarchical spatial-frequency domain refinement module to enhance feature representation. Finally, the global token output from the third feature extraction stage (i.e., the last stage) is used as the updated i-th global token, and together with the search region features jointly enhanced by the spatial-frequency domain, it is fed into the single-target tracking head to predict the target position.

[0025] Figure 2 This is a schematic diagram illustrating the network architecture and data flow of a single LightViT structure block provided in an embodiment of the present invention. Figure 2 As shown, each LightViT structure block includes: The first splitting module is used to split the input all words into first image words and first global words. The first image words are further split into first template words and first search region words. The first LightViT block in the first feature extraction stage receives the i-th all words, and the remaining LightViT blocks receive the updated all words output by the previous LightViT block or the previous image block merging module. The local self-attention module, connected to the first splitting module, is used to perform local self-attention operations on the first template word and the first search region word respectively, and to concatenate the two attention results to output the local attention. The global cross-attention module, connected to the first splitting module, is used to perform multi-head cross-attention on the first global word unit and outputs global aggregation attention and global broadcast attention respectively. The first fusion module connects the local self-attention module and the global cross-attention module respectively. It is used to add the local attention and the global broadcast attention element by element, and then concatenate the sum with the global aggregate attention by channel to output the fused attention feature. The residual connection and normalization module is connected to the input of the first splitting module and the output of the first fusion module, respectively. It is used to add the fused attention features to all the words in the input LightViT structure block and then perform layer normalization to obtain the normalized features. The bidirectional attention feedforward network module connects the residual connection and the normalization module. It is used to perform bidirectional attention feedforward processing on the normalized features and output the updated global vocabulary corresponding to the LightViT structure block. The updated global vocabulary includes the updated first image vocabulary and the updated first global vocabulary.

[0026] Here, the mathematical expression for local attention is: ; in, , , This indicates the query, key, and value corresponding to the term in the first search area. , , This represents the query, key, and value corresponding to the first template term. This indicates a splicing operation. This indicates a local self-attention operation.

[0027] Global Aggregate Attention and global broadcast attention The mathematical expressions are as follows: ; ; ; in, , , This represents the query, key, and value corresponding to the first image term. It is the query corresponding to the first global term, and , This represents the key and value corresponding to global aggregated attention. This indicates a multi-headed cross-attention operation. This indicates a linear projection.

[0028] Local attention and global broadcast attention Add them together and aggregate attention globally. Concatenate by channel. Combine the concatenated attention with the input of this module. The sums are then fed into a normalization layer for further processing to obtain the enhanced features. : ; in, This indicates normalization processing.

[0029] In conclusion, Figure 2 The single LightViT block shown achieves deep feature updates for all input words through the collaborative processing of a first splitting module, a local self-attention module, a global cross-attention module, a first fusion module, a residual connection and normalization module, and a bidirectional attention feedforward network module. It should be noted that within the same feature extraction stage of the backbone network and between different stages, all LightViT blocks adopt the same... Figure 2 The network architecture and data processing methods are exactly the same. Each structural block independently performs the aforementioned splitting, local self-attention, global cross-attention, fusion, residual normalization, and bidirectional attention feedforward operations on all the received words, and outputs the updated total words. Therefore, regardless of the structural block's location in the backbone network, its internal processing logic and parameter update rules remain consistent, thus ensuring the standardization and scalability of the feature extraction process.

[0030] The network architecture and data processing method of the image block merging module will now be introduced.

[0031] Figure 3 This is a schematic diagram of the network architecture and data flow of the image block merging module provided in an embodiment of the present invention. Figure 3 As shown, the image patch merging module includes: The second splitting unit is used to split the updated all words from the previous feature extraction stage into second image words and second global words, and the second image words are further split into second template words and second search region words. The first branch is used to feed the second template word and the second search region word into the max pooling layer, the first linear layer and the first adjustment layer respectively for processing, and output the first dimension adjustment feature; The second branch is used to sequentially feed the second template word and the second search region word into the second adjustment layer, the second linear layer and the third adjustment layer for processing, and output the second dimension adjustment feature; The addition unit is used to add the first dimension adjustment feature and the second dimension adjustment feature element by element to obtain the processed image words; The global processing unit is used to adjust the number of channels of the second global lexical unit through the fourth linear layer to obtain the processed global lexical unit; The splicing unit is used to splice the processed image words and the processed global words by channel, and output all words corresponding to the image block merging module.

[0032] It should be noted that the two branches share weights and perform a separate linear transformation on the global tokens; finally, the processed image tokens are concatenated with the global tokens for output. These features differ from conventional CNN downsampling (such as direct convolution or pooling) and image patch merging in the original LightViT (which does not involve independent processing of global tokens).

[0033] In conclusion, Figure 3 The image patch merging module shown uses a second splitting unit, a first branch, a second branch, an addition unit, a global processing unit, and a concatenation unit to downsample and merge all the updated word units output from the previous feature extraction stage and adjust the number of channels, generating all word units suitable for the next feature extraction stage. It should be noted that in the backbone network, the image patch merging module set between any two adjacent feature extraction stages adopts the same... Figure 3 The network architecture and data processing methods are exactly the same. Each image patch merging module independently performs the above-mentioned splitting, bi-branching, global word linear transformation, addition, and concatenation operations on all the words it receives, and outputs the processed words. Therefore, the processing logic of all image patch merging modules is consistent, thereby ensuring the consistency and reproducibility of the downsampling operation in the backbone network during multi-stage feature extraction.

[0034] After multi-stage feature extraction by the backbone network, image features at three different scales are obtained (corresponding to the outputs of the first, second, and third stages, respectively), along with the updated i-th global term. The image features at each stage contain multi-level semantic information from local to global perspectives, but directly using them for target localization still suffers from background interference and insufficient feature discrimination. To address this, this invention further incorporates an adaptive hierarchical spatial-frequency domain refinement module. This module takes the image features from each stage output by the backbone network as input and, through a collaborative enhancement mechanism of spatial dynamic fusion and frequency dynamic filtering, extracts more robust search region features, thereby improving tracking accuracy in complex scenes.

[0035] The adaptive hierarchical spatial frequency domain refinement module includes: The feature alignment and stitching unit is used to separate the corresponding search region features from the image features of each stage output by the backbone network, and to stitch together the search region features at different scales after upsampling or downsampling to obtain multi-scale fusion features. The spatial dynamic fusion unit connects to the feature alignment and stitching unit, which is used to sequentially pass multi-scale fused features through depthwise full-dimensional dynamic convolution and pointwise full-dimensional dynamic convolution, and then process them through batch normalization and activation functions to output spatial enhanced features. The frequency domain dynamic fusion unit, connected to the spatial domain dynamic fusion unit, is used to adaptively generate a dynamic filter bank based on the spatial domain enhancement features. After transforming the spatial domain enhancement features to the frequency domain, the dynamic filter bank filters them and then transforms them back to the spatial domain to output the frequency domain enhancement features. The dynamic weighted fusion unit, which connects the spatial dynamic fusion unit and the frequency dynamic fusion unit respectively, is used to multiply the spatial enhancement features and the frequency enhancement features by the learnable first weight coefficient and the second weight coefficient respectively, and then add them to output the search region features jointly enhanced by the spatial and frequency domains.

[0036] Here, the backbone network includes a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage; the number of LightViT blocks in the third feature extraction stage is greater than the number of LightViT blocks in the other feature extraction stages. For example, the number of LightViT blocks in the first and second feature extraction stages is 2 each, while the number of LightViT blocks in the third feature extraction stage is 6. The feature alignment and stitching unit is configured as follows: The first search region features are separated from the image features corresponding to the first feature extraction stage, and downsampled to align with the scale of the search region features corresponding to the second feature extraction stage, thus obtaining the first aligned features. The second search region features are extracted from the image features corresponding to the second feature extraction stage and used as the second alignment features. The third search region features are extracted from the image features corresponding to the third feature extraction stage and upsampled to align with the scale of the second alignment features, thus obtaining the third alignment features. ; align the first feature Second alignment feature and third alignment feature The data is stitched together along the channel dimension to output multi-scale fused features. .

[0037] This multi-scale fusion feature The mathematical expression is: ; in, Indicates the features of the search region in the first stage. Perform downsampling and compare with the second-stage features Alignment This indicates that the features of the search region in the third stage are upsampled and compared with... Alignment.

[0038] In the feature alignment and splicing unit, the search region features output from different stages of the backbone network are upsampled or downsampled and aligned, and then spliced ​​in the channel dimension. This effectively integrates multi-level semantic information from shallow to deep layers, preserving both the detailed information of the target and the high-level semantic context, providing rich basic features for subsequent enhancement.

[0039] Figure 4 This is a schematic diagram of the network structure of the spatial dynamic fusion unit provided in an embodiment of the present invention. Figure 4 As shown, the spatial dynamic fusion unit is configured to: sequentially input multi-scale fusion features into a depthwise full-dimensional dynamic convolutional layer and a pointwise full-dimensional dynamic convolutional layer to obtain a first intermediate feature; sequentially process the first intermediate feature with batch normalization and ReLU activation function to obtain a second intermediate feature; and sequentially input the second intermediate feature into a 1×1 convolutional layer and a batch normalization layer to output spatial enhancement features.

[0040] Here, spatial enhancement features The mathematical expression is: ; in, Represents the ReLU activation function. and These represent depthwise full-dimensional dynamic convolution and pointwise full-dimensional dynamic convolution, respectively. This indicates a batch normalization operation. This represents 1×1 convolution and batch normalization operations.

[0041] It should be noted that depthwise full-dimensional dynamic convolution, pointwise full-dimensional dynamic convolution, batch normalization, ReLU activation function, and 1×1 convolution are all existing data processing operations known in the art, and their specific implementations can be found in relevant prior art. This invention is not intended to protect these independent sub-operations themselves, but rather to classify the above operations according to... Figure 4 The sequence and connection relationships shown are combined to form a complete spatial dynamic fusion unit to achieve dynamic spatial enhancement of multi-scale fusion features, and work in conjunction with the frequency dynamic fusion unit to achieve joint spatial and frequency domain feature enhancement.

[0042] In the spatial dynamic fusion unit, depthwise full-dimensional dynamic convolution and pointwise full-dimensional dynamic convolution are used to adaptively capture the local spatial structure within the search area. Combined with batch normalization and ReLU activation, the robustness to spatial changes such as target deformation and partial occlusion is significantly improved, while maintaining low computational overhead.

[0043] Figure 5 This is a schematic diagram of the network structure of the frequency domain dynamic fusion unit provided in an embodiment of the present invention. Figure 5As shown, the frequency domain dynamic fusion unit includes: The weight generation subunit is used to perform average pooling on the spatial enhancement features to obtain pooled features. The multilayer perceptron subunit, connected to the weight generation subunit, is used to process the pooled features using a multilayer perceptron and output the original weights of each filter. The normalization subunit, connected to the multilayer sensing subunit, is used to perform Softmax normalization on the original weights of each filter to obtain the normalized weights of each filter. The filter coefficient storage sub-unit is used to store multiple learnable basis filter coefficients; The filter combination subunit is connected to the normalization subunit and the filter coefficient storage subunit, respectively. It is used to multiply the normalized weights of each filter with the corresponding base filter coefficients element by element and then sum them to generate a dynamic filter bank. The linear mapping subunit is used to perform linear mapping on the spatial domain enhancement features to obtain the first intermediate feature; The activation subunit is connected to the linear mapping subunit, which is used to perform StarReLU activation on the first intermediate feature to obtain the second intermediate feature; The Fourier transform subunit, connected to the activation subunit, is used to perform a 2D fast Fourier transform on the second intermediate feature to obtain the frequency domain feature. The filtering subunit is connected to the filter combination subunit and the Fourier transform subunit, respectively, and is used to filter the frequency domain features using the dynamic filter bank to obtain the filtered frequency domain features. The inverse Fourier transform subunit, connected to the filtering subunit, is used to perform a 2D inverse fast Fourier transform on the filtered frequency domain features to obtain the spatial domain features. The output sub-unit is connected to the inverse Fourier transform sub-unit and is used to output frequency domain enhancement features.

[0044] Here, the weights of each filter The mathematical expression is: ; in, satisfy , The number of filters. Indicates average pooling. This represents a multilayer perceptron. This represents the Softmax function.

[0045] Furthermore, dynamic filter banks The mathematical expression is: ; in, The first learnable number iEach filter coefficient This indicates element-wise multiplication.

[0046] spatial enhancement features Convert to frequency domain features Its mathematical expression is: ; in, Represents a linear mapping. This represents the StarReLU activation function. This indicates a 2D-FFT operation.

[0047] Using dynamic filter banks Frequency domain features Filtering is performed, and the result is converted to spatial domain processing, outputting frequency domain enhanced features. : ; in, This indicates a 2D-IFFT operation.

[0048] In the frequency domain dynamic fusion unit, features are filtered in the frequency domain through an adaptively generated dynamic filter bank: first, the spatial domain enhanced features are transformed to the frequency domain, and filter weights generated based on the input content are used to suppress background noise and periodic interference, and then the transformation is reversed back to the spatial domain. This process effectively enhances the anti-interference ability of features in complex environments such as low light and haze.

[0049] Finally, the search region features are jointly enhanced by the spatial and frequency domains. It can be represented as: ;in, and These are the learnable weight coefficients.

[0050] By introducing learnable weight coefficients, the spatial domain enhancement features and frequency domain enhancement features are adaptively weighted and summed, enabling the network to dynamically adjust the contribution ratio of spatial and frequency domain information according to the specific scene of the current frame, thus avoiding feature bias that may be caused by single-domain enhancement.

[0051] This adaptive hierarchical spatial-frequency domain refinement module achieves deep collaborative enhancement of spatial and frequency domain features through the cooperation of various modules. The output spatial-frequency jointly enhanced search region features possess both fine spatial localization capabilities and excellent frequency domain noise resistance. Compared with using only a lightweight backbone network, this module significantly improves the tracker's discriminative power and robustness in complex scenarios such as rapid target movement, occlusion, lighting changes, and cluttered backgrounds, thus effectively solving the core problem of insufficient accuracy in lightweight tracking methods.

[0052] It should be noted that the single-target tracking head adopts existing technology known in the art, specifically referring to the tracking head structure disclosed in existing literature (Yan B, Peng H, Fu J, et al. Learning Spatio-Temporal Transformer for VisualTracking[C] / / Proceedings of the IEEE / CVF International Conference on ComputerVision (ICCV), 2021:10448-10457). This tracking head receives the joint enhanced search region features in the spatial and frequency domains output by the adaptive hierarchical spatial and frequency domain refinement module, and global terms output from the last stage of the backbone network, predicting the target bounding box in the current frame. The improvement of this invention lies in the design of the aforementioned backbone network and adaptive hierarchical spatial and frequency domain refinement module, while the tracking head directly adopts the existing mature structure, achieving the tracking effect of this invention without modification.

[0053] It should be noted that for each frame (i=2,3,…,I) in the video sequence, the same processing is performed according to steps S1 to S3 above: that is, based on the tracking result of the (i-1)th video frame, the search region image of the i-th frame is determined, the i-th global word is generated, input into the improved lightweight Transformer network, and finally the target bounding box of the i-th frame is output. This process is iterated until the last frame (the i-th frame) of the video sequence is processed, thus completing the single-target tracking task for the entire video sequence. Throughout the process, each frame shares the same set of network parameters, and the global word is initialized and updated independently between frames without cross-frame transfer.

[0054] To address the challenge of balancing tracking accuracy and computational cost in existing lightweight tracking methods, this invention provides a single-target tracking method based on spatial-frequency domain feature enhancement. This method first acquires a video frame sequence, using the target region in the first frame as a template, and determines the search region for subsequent frames according to a preset strategy. Then, the template image, the search region image of the current frame, and the current global words are converted into all words and input into an improved lightweight Transformer network. The backbone network of this network, through multiple feature extraction stages and an image patch merging module, utilizes a combination of local self-attention and global cross-attention to enable deep interaction between global words and image words. This allows for the extraction of multi-scale features while maintaining lightweight design, and outputs image features at each stage and updated global words. An adaptive hierarchical spatial-frequency domain refinement module extracts search region features from the image features at each stage. Through the synergistic enhancement of spatial dynamic convolution and frequency dynamic filtering, background interference is suppressed and the essential features of the target are highlighted. Finally, dynamically weighted fusion yields the jointly enhanced search region features in the spatial and frequency domains. The single-target tracking head predicts the target bounding box based on the updated global words and the enhanced features. The entire method performs the same processing on all frames iteratively, sharing network parameters. Through the joint optimization of multi-stage feature extraction, spatial-frequency domain dynamic enhancement, and global lexical guidance of the aforementioned backbone network, this invention significantly improves the discriminative power and robustness of the lightweight tracker in complex scenes with only a small increase in computational overhead, effectively resolving the contradiction between accuracy and speed, and achieving real-time, high-precision single-target tracking.

[0055] To verify the reliability and effectiveness of this invention, the following description, in conjunction with examples, further illustrates the invention. The backbone network of this invention is initialized using LightViT pre-trained on ImageNet, employing the AdamW optimizer with a weight decay set to 10. 4. The initial learning rate is set to 4×10. 4. Training was performed using a single NVIDIA RTX 4090 GPU on a Python 3.9 and PyTorch 1.11.0 environment. A total of 300 training epochs were conducted, with a batch size of 32 and 60,000 sample pairs per epoch. The learning rate was reduced to 1 / 10 of its original value at the 240th epoch. During testing, the same environment configuration as training was used, and all tests on the dataset were performed using a single NVIDIA RTX 4090 GPU. Inference speed tests were conducted on both a single NVIDIA RTX 4090 GPU and a single Intel i9 13900K CPU.

[0056] This invention compares existing lightweight algorithms on three large-scale target tracking datasets: TrackingNet, LaSOT, and GOT-10k. The algorithms described are: "Cui Y, Song T, Wu G, et al. MixFormerV2: efficient fully transformer tracking [C] / / Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023: 58736-58751." (abbreviated as MixformerV2-S), "Gopal GY, Amer M A. Separable self and mixed attention transformers for efficient object tracking [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. 2024: 6708-6717." (abbreviated as SMAT), and "Kang B, ChenX, Wang D, et al. Exploring lightweight hierarchical vision transformers for efficient visual tracking [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: A comparison is made between this invention and other lightweight tracking algorithms on three large-scale datasets: TrackingNet, LaSOT, and GOT-10k. Table 1 compares the tracking accuracy of this invention with other lightweight tracking algorithms. Table 2 compares the model complexity of this invention with other lightweight tracking algorithms.

[0057] Table 1

[0058] Table 2

[0059] As shown in Tables 1 and 2, on the TrackingNet and LaSOT datasets, this invention achieves tracking accuracy comparable to HiT-Base, but with almost half the number of parameters, and significantly faster execution speed on GPUs and CPUs. On the GOT-10k dataset, the tracking accuracy of this invention is significantly higher than other tracking algorithms. In summary, this invention effectively balances the trade-off between tracking accuracy and inference speed, providing a novel solution for single-target tracking in complex scenarios.

[0060] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A single-target tracking method based on spatial frequency domain feature enhancement, characterized in that, include: Obtain the video frame sequence, use the target region in the first video frame as the template image, and determine the i-th search region image according to the preset selection strategy, where 2≤i≤I, and I is the total number of video frame sequences; Based on the template image, the i-th search region image, and the i-th global word element, generate the i-th global word element; The i-th all-terminal is input into the improved lightweight Transformer network, which includes a backbone network, an adaptive hierarchical spatial-frequency domain refinement module, and a single-target tracking head, wherein: The backbone network is used to perform multi-stage feature extraction on the i-th global word, and output the image features of each stage and the updated i-th global word. The adaptive hierarchical spatial-frequency domain refinement module is used to extract search region features corresponding to the search region from the image features of each stage, dynamically enhance and fuse the search region features in the spatial and frequency domains, and output the jointly enhanced search region features in the spatial and frequency domains. The single-target tracking head is used to predict the target bounding box in the i-th video frame based on the i-th global lexical unit and the search region features jointly enhanced by the spatial and frequency domains.

2. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 1, characterized in that, The backbone network is an improved LightViT backbone network; the improved LightViT backbone network includes multiple sequentially connected feature extraction stages, each feature extraction stage including at least two LightViT structural blocks; an image block merging module is provided between two adjacent feature extraction stages; wherein: The image words output by each feature extraction stage constitute the image features of that stage. The global words in each feature extraction stage are passed sequentially between units, and the global words output by the last feature extraction stage are used as the updated i-th global word. Each LightViT structure block includes: The first splitting module is used to split the input total word units into a first image word unit and a first global word unit, wherein the first image word unit is further split into a first template word unit and a first search region word unit; wherein, the total word units received by the first LightViT structure block in the first feature extraction stage are the i-th total word units, and the total word units received by the remaining LightViT structure blocks are the updated total word units output by the previous LightViT structure block or the previous image block merging module; The local self-attention module, connected to the first splitting module, is used to perform local self-attention operations on the first template word and the first search region word respectively, and to concatenate the two attention results to output the local attention. A global cross-attention module, connected to the first splitting module, is used to perform multi-head cross-attention on the first global word unit and output global aggregation attention and global broadcast attention respectively; The first fusion module is connected to the local self-attention module and the global cross-attention module respectively. It is used to add the local attention and the global broadcast attention element by element, and then concatenate the sum with the global aggregate attention by channel to output the fused attention feature. The residual connection and normalization module is connected to the input of the first splitting module and the output of the first fusion module, respectively. It is used to add the fused attention features to all the words in the input LightViT structure block and then perform layer normalization to obtain the normalized features. The bidirectional attention feedforward network module, connected to the residual connection and normalization module, is used to perform bidirectional attention feedforward processing on the normalized features and output the updated all words corresponding to the LightViT structure block. The updated all words include the updated first image words and the updated first global words.

3. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 2, characterized in that, The image block merging module includes: The second splitting unit is used to split the updated all words from the previous feature extraction stage into second image words and second global words, and the second image words are further split into second template words and second search region words. The first branch is used to sequentially feed the second template word and the second search region word into the max pooling layer, the first linear layer and the first adjustment layer for processing, and output the first dimension adjustment feature; The second branch is used to sequentially feed the second template word and the second search region word into the second adjustment layer, the second linear layer and the third adjustment layer for processing, and output the second dimension adjustment feature; The addition unit is used to add the first dimension adjustment feature and the second dimension adjustment feature element by element to obtain the processed image words; A global processing unit is used to adjust the number of channels of the second global lexical unit through the third linear layer to obtain the processed global lexical unit; The splicing unit is used to splice the processed image words and the processed global words by channel, and output all words corresponding to the image block merging module.

4. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 2, characterized in that, The adaptive hierarchical spatial frequency domain refinement module includes: The feature alignment and stitching unit is used to separate the corresponding search region features from the image features of each stage output by the backbone network, and to stitch together the search region features of different scales after upsampling or downsampling to obtain multi-scale fusion features. The spatial dynamic fusion unit, connected to the feature alignment and stitching unit, is used to sequentially pass the multi-scale fused features through depthwise full-dimensional dynamic convolution and pointwise full-dimensional dynamic convolution, and then process them through batch normalization and activation functions to output spatial enhanced features. A frequency domain dynamic fusion unit, connected to the spatial domain dynamic fusion unit, is used to adaptively generate a dynamic filter bank based on the spatial domain enhancement features. After transforming the spatial domain enhancement features to the frequency domain, the dynamic filter bank filters the features and then transforms them back to the spatial domain to output the frequency domain enhancement features. The dynamic weighted fusion unit, which is connected to the spatial dynamic fusion unit and the frequency dynamic fusion unit respectively, is used to multiply the spatial enhancement feature and the frequency enhancement feature by a learnable first weight coefficient and a second weight coefficient respectively, and then add them to output the search region feature of the joint spatial and frequency domain enhancement.

5. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 4, characterized in that, The backbone network includes a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage; the number of LightViT structural blocks in the third feature extraction stage is greater than the number of LightViT structural blocks in the other feature extraction stages. The feature alignment and splicing unit is configured as follows: The first search region features are separated from the image features corresponding to the first feature extraction stage, and downsampled to align with the scale of the search region features corresponding to the second feature extraction stage, thus obtaining the first aligned features. The second search region feature is separated from the image features corresponding to the second feature extraction stage and used as the second alignment feature; The third search region features are separated from the image features corresponding to the third feature extraction stage, and upsampled to align with the scale of the second alignment features to obtain the third alignment features; The first alignment feature, the second alignment feature, and the third alignment feature are concatenated along the channel dimension to output the multi-scale fusion feature.

6. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 4, characterized in that, The spatial dynamic fusion unit is configured as follows: The multi-scale fused features are sequentially input into a depthwise full-dimensional dynamic convolutional layer and a pointwise full-dimensional dynamic convolutional layer to obtain the first intermediate feature; The first intermediate feature is then subjected to batch normalization and ReLU activation function sequentially to obtain the second intermediate feature; The second intermediate feature is sequentially input into a 1×1 convolutional layer and a batch normalization layer to output the spatial enhancement feature.

7. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 4, characterized in that, The frequency domain dynamic fusion unit includes: A weight generation subunit is used to perform average pooling on the spatial enhancement features to obtain pooled features. A multilayer perceptron subunit, connected to the weight generation subunit, is used to process the pooling features using a multilayer perceptron and output the original weights of each filter. The normalization subunit, connected to the multilayer sensing subunit, is used to perform Softmax normalization on the original weights of each filter to obtain the normalized weights of each filter. The filter coefficient storage subunit is used to store multiple learnable basis filter coefficients; The filter combination subunit is connected to the normalization subunit and the filter coefficient storage subunit, respectively, and is used to multiply the normalized weights of each filter with the corresponding base filter coefficients element by element and then sum them to generate the dynamic filter bank. A linear mapping subunit is used to perform a linear mapping on the spatial domain enhancement feature to obtain a first intermediate feature; An activation subunit is connected to the linear mapping subunit to perform StarReLU activation on the first intermediate feature to obtain a second intermediate feature. The Fourier transform subunit, connected to the activation subunit, is used to perform a 2D fast Fourier transform on the second intermediate feature to obtain the frequency domain feature. A filtering subunit, connected to the filter combination subunit and the Fourier transform subunit respectively, is used to filter the frequency domain features using the dynamic filter bank to obtain filtered frequency domain features; The inverse Fourier transform subunit, connected to the filtering subunit, is used to perform a 2D inverse fast Fourier transform on the filtered frequency domain features to obtain spatial domain features. The output subunit, connected to the inverse Fourier transform subunit, is used to output the frequency domain enhancement feature.

8. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 1, characterized in that, The image of the i-th search region is obtained in the following way: Obtain the target bounding box of the (i-1)th video frame; In the i-th video frame, the target bounding box of the (i-1)-th video frame is expanded according to a preset ratio, with the center coordinates of the target bounding box of the (i-1)-th video frame as the center, to obtain the i-th candidate region; The image of the i-th search region is obtained by cropping and scaling the i-th candidate region to a preset search region size.

9. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 1, characterized in that, The step of generating the i-th global lexical unit based on the template image, the i-th search region image, and the i-th global lexical unit includes: The template image and the i-th search region image are sequentially downsampled by 8 times, segmented and embedded, flattened and stitched together to obtain the i-th image word; The i-th image word and the i-th global word are concatenated to obtain the i-th global word.

10. The single-target tracking method based on spatial frequency domain feature enhancement according to claim 5, characterized in that, The number of LightViT structural blocks in the first feature extraction stage and the second feature extraction stage is 2, and the number of LightViT structural blocks in the third feature extraction stage is 6.

Images