Mask-guided frequency domain-spatial domain hybrid network target tracking method

By introducing a mask-guided frequency domain-space hybrid network in target tracking, combining the template mask encoder and the frequency domain-space feature fusion module, the problem of insufficient target tracking performance in complex scenarios in the prior art is solved, and the problem of insufficient target tracking performance in complex scenarios is realized. Excellent tracking effect with fewer model parameters and calculation amounts.

CN119006979APending Publication Date: 2024-11-22UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411152423.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing target tracking methods are difficult to effectively track in complex scenarios such as occlusion, scale changes, similar interferers and motion blur, and the deep semantic features extracted by the backbone network have a low resolution, and the shallow texture features have a high resolution but no It can be fully combined, and frequency domain operations are not fully utilized.

Method used

Using mask-guided frequency domain-space domain hybrid network, a learned template mask vector is generated through a template mask encoder, and the frequency domain-space domain-space feature fusion stage is used in the backbone network feature fusion stage and template image-search image feature interaction stage. Module and frequency domain-airspace attention interaction module to improve target tracking performance.

Benefits of technology

In complex scenarios, the performance of target tracking is improved, key semantic information is retained, model parameters and calculation amount are reduced, and excellent tracking effect is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006979A_ABST
    Figure CN119006979A_ABST
Patent Text Reader

Abstract

The invention provides a mask-guided frequency domain-spatial domain hybrid network target tracking method, which relates to the field of pattern recognition and computer vision, and comprises the following steps: carrying out visual feature extraction on an image pair comprising a search image and a template image to respectively obtain multi-layer and multi-resolution search branch features and template branch features; fusing the extracted multi-layer and multi-resolution search branch features and template branch features to obtain fused backbone network features; stacking the template mask and the template image, and encoding to generate a learnable template mask vector; jointly inputting the fused backbone network features and the generated template mask vector into a frequency domain-spatial domain attention interaction module to promote full interaction and enhancement of template branch features and search branch features, and generating final search branch features; and inputting the generated final search branch features into the tracking prediction head network to generate coordinates of the to-be-tracked target frame. The target tracking performance can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition and computer vision, and in particular to a mask-guided frequency domain-spatial domain hybrid network target tracking method. Background Art

[0002] Object tracking is widely used in autonomous driving, video surveillance, and drone tracking. Although the development of deep learning has spawned a variety of effective object tracking methods, object tracking still faces many challenges in complex scenes such as occlusion, scale changes, similar distractors, and motion blur.

[0003] In recent years, the target tracking method based on the twin network has attracted much attention. This method models target tracking as a similarity matching task, achieving a balance between accuracy and speed. Specifically, the tracking method based on the twin network uses two backbone networks with shared parameters to extract visual features from the template and the search area image respectively, and then interacts and enhances these features to obtain the final search area features, which are then input into the tracking prediction head network to finally obtain the coordinates of the target box to be tracked.

[0004] Existing tracking methods usually use a small-sized template to highlight the specific features of the target, while using a large-sized search area to include the surrounding environment information. However, a small-sized template may lead to the loss of key contextual information, such as ignoring distractors around the target. Directly increasing the size of the template image is not feasible because it will introduce too much background interference information, which may hinder the tracking model from learning discriminative features for accurately locating the target.

[0005] In addition, the deep semantic features extracted by the backbone network have good target discrimination, but the resolution is low. In contrast, the shallow texture features mainly focus on details such as color and shape, have high resolution, and are crucial for accurate positioning. Inspired by this, some tracking methods use stacking or simple convolution to fuse multi-layer backbone network features to improve tracking performance. However, although these fusion methods are simple, they fail to give full play to the unique advantages of deep and shallow features.

[0006] On the other hand, recent studies have shown that attention mechanisms focus on learning low-frequency features, while convolutional neural networks excel at learning high-frequency features. Therefore, if the features are mapped to the frequency domain and adjusted in combination with complementary spatial features, the tracking features can be adaptively optimized from a hybrid perspective of the frequency domain and the spatial domain. This approach helps alleviate the limitations of convolution and attention mechanisms in the frequency domain while maintaining spatial positioning capabilities. However, existing trackers mainly rely on convolutional backbone networks for feature extraction and feature interaction and enhancement through spatial or channel attention, ignoring the importance of frequency domain operations. Even though some tracking methods introduce frequency domain modules, they lack synergy with spatial domain operations, and there is still room for improving the design of frequency domain mechanisms. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a mask-guided frequency-domain-spatial domain hybrid network target tracking method, designs a template mask encoder to generate a mask vector used in conjunction with a large-size template image, so as to retain more surrounding information while focusing on the target discriminant features. At the same time, a frequency-domain-spatial hybrid network is designed, which consists of a frequency-domain-spatial domain feature fusion module and a frequency-domain-spatial domain attention interaction module, which respectively improves the target tracking performance in the backbone network feature fusion stage and the template image-search image feature interaction stage.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: A mask-guided frequency-domain-spatial-domain hybrid network target tracking method comprises the following steps: Step 1: Input the image pair including the search image Z and the template image X into the convolutional backbone network for visual feature extraction, and obtain multi-layer and multi-resolution search branch features and template branch features respectively; Step 2: Use the frequency domain-spatial domain feature fusion module FSF in the search branch and template branch respectively to fuse the multi-layer and multi-resolution search branch features and template branch features extracted in step 1 to obtain the fused backbone network features, which include the fused template branch features. And the fused search branch features ; Step 3: Stack the template mask B with the template image Z and pass it to the template mask encoder TME to encode and generate a learnable template mask vector ; Step 4: Combine the backbone network features fused in step 2 with the template mask vector generated in step 3 Jointly input to the frequency-spatial attention interaction module FSA to promote template branch features and search branch features Full interaction and enhancement to generate the final search branch features ; Step 5: The final search branch features generated in step 4 Input to the tracking prediction head network to generate the coordinates of the target box to be tracked.

[0009] Beneficial effects: 1) This paper proposes a template mask encoder for generating a learnable template mask vector, which can adaptively learn prior position and appearance knowledge while retaining key semantic information when used with a large-size template and a large-size search image, thereby enhancing tracking performance in complex scenes.

[0010] 2) The present invention proposes a frequency domain-spatial domain hybrid network, which consists of a frequency domain-spatial domain feature fusion module and a frequency domain-spatial domain attention interaction module, which improves the target tracking performance in the backbone network feature fusion stage and the template image-search image feature interaction stage respectively.

[0011] 3) The present invention achieves excellent tracking performance while using relatively fewer model parameters and computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of a template image and a search image; (a) is a traditional twin tracking method using a small-size template image and a large-size search image; (b) is a tracking method proposed in the present invention using a large-size template image, a large-size search image and a template mask; Figure 2 This is a comparison diagram of the tracking performance, parameter amount and calculation amount of the present invention and other representative tracking methods on the LaSOT dataset. The diameter of the circle in the figure represents the floating-point operation number of the model. Figure 3 A schematic diagram of the principle of a mask-guided frequency-space hybrid network target tracking method proposed in the present invention; Figure 4 Schematic diagram of template mask encoder; Figure 5 It is a schematic diagram of the frequency domain-spatial domain feature fusion module; Figure 6 Schematic diagram of the frequency-domain-spatial-domain attention interaction module; (a) is frequency-domain attention, (b) is frequency-domain-spatial attention, (c) is mask-guided spatial attention, and (d) is original spatial attention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0014] like Figure 1As shown, compared with the previous tracking method that only uses a small-sized template image and a large-sized search image, a key advantage of the present invention is that a large-sized template image and a large-sized search image are used at the same time, and a rectangular mask of the position of the target to be tracked in the template image is introduced. The large-sized template image can contain more surrounding environment information, and the learnable template mask vector generated by the template mask encoder can highlight the prior position and appearance knowledge of the target to be tracked. Therefore, when dealing with challenging environments such as similar interference objects and motion blur, the present invention can achieve better tracking effects. Among them, Figure 1 (a) is the traditional twin tracking method using a small-sized template image and a large-sized search image; Figure 1 (b) is the tracking method proposed by the present invention using a large-size template image, a large-size search image and a template mask.

[0015] Convolutional networks and attention mechanisms have their own strengths in learning high-frequency and low-frequency features. Based on this, the present invention adopts frequency domain operations and combines the positioning capabilities of spatial domain operations to adaptively optimize tracking features from a frequency domain-spatial domain hybrid perspective. The design of this frequency domain-spatial domain hybrid network can effectively alleviate the limitations of convolutional networks and attention mechanisms in the frequency domain while ensuring excellent spatial positioning capabilities. The collaborative learning of frequency domain and spatial domain operations in the present invention significantly improves tracking performance.

[0016] like Figure 2 As shown, compared with the single-stream tracking method that requires pre-training of visual Transformer on a large-scale dataset and leads to a significant increase in model parameters and computational floating-point operations, the present invention follows the tracking paradigm based on the twin network, which can achieve excellent tracking effects while reducing model parameters and computational complexity.

[0017] like Figure 3 As shown, a mask-guided frequency-domain-spatial-domain hybrid network target tracking method of the present invention comprises the following steps: Step 1: Input the image pair including the search image Z and the template image X into the convolutional backbone network for visual feature extraction, and obtain multi-layer and multi-resolution search branch features and template branch features respectively; Step 2: Use the frequency domain-spatial domain feature fusion module FSF in the search branch and template branch respectively to fuse the multi-layer and multi-resolution search branch features and template branch features extracted in step 1 to obtain the fused backbone network features, which include the fused template branch features. And the fused search branch features ; Step 3: Stack the template mask B with the template image Z and pass it to the template mask encoder TME to encode and generate a learnable template mask vector ; Step 4: Combine the backbone network features fused in step 2 with the template mask vector generated in step 3 Jointly input to the frequency-spatial attention interaction module FSA to promote template branch features and search branch features Full interaction and enhancement to generate the final search branch features ; Step 5: The final search branch features generated in step 4 Input to the tracking prediction head network to generate the coordinates of the target box to be tracked.

[0018] Specifically, Figure 5 As shown, there are two frequency domain-spatial domain feature fusion modules in step 2. The two frequency domain-spatial domain feature fusion modules iteratively aggregate the three-layer backbone network features. The fused features are expressed as , UpSample and DownSample represent upsampling and downsampling respectively to maintain the consistency of feature resolution of different layers of backbone networks. is the template branch feature after fusion, is the fused search branch feature, , are the height and width of the template image input to the backbone network respectively; , are the height and width of the search image input to the backbone network respectively; is the step size of the backbone network output visual features; the fusion process is expressed as: , ; Among them, X and Z are the search image and template image input to the backbone network respectively; , , They are the visual features of the 2nd, 3rd, and 4th layers of the backbone network output after the template image is input into the backbone network; , , They are the visual features of the 2nd, 3rd, and 4th layers of the backbone network output after the search image is input into the backbone network; Even with upsampling and downsampling operations, there are still significant semantic inconsistencies in the backbone features between different layers. Therefore, it may not be possible to fuse multi-level features by simply stacking or adding them to fully exploit the complementarity of features at different layers. To address this issue, the frequency-spatial feature fusion module iteratively aggregates the two features.

[0019]

[0020]

[0021] in, is the intermediate feature of the fusion process, is the final fused feature; ⊕ and ⊙ represent element-by-element addition and element-by-element multiplication, respectively. First, the two features Add, C, H, W are input features The number of channels, height, and width are calculated, and the results are then input into the global context operation G and the frequency domain attention F. The outputs of the global context operation G and the frequency domain attention F are added together, and then the Sigmoid function is applied to generate the weights , whose values ​​range from 0 to 1. The weights are obtained through a similar process , the difference is It is generated collaboratively by frequency domain attention F and local context operation L.

[0022] Get weight , After that, use ( ) and 1- (1- ) to integrate , , enabling the network to , Make a soft selection.

[0023] Specifically, the global context operation G uses global average pooling to compress features of size H×W into a scalar of size 1×1, which then passes through a two-layer 1×1 convolution. This 1×1 scalar contains the global context information of the feature. At the same time, a two-layer 1×1 convolution is used as the local context operation L (aggregator), which aggregates the context information for each pixel position in a spatial manner. The features produced by the local context operation L have the same resolution as the initial features, making it possible to preserve and emphasize detailed information. When the global context operation G and the local context operation L are combined with the frequency domain attention F, the global context operation G tends to emphasize large targets with global distribution, while the local context operation L can focus on small targets with local distribution. This combination allows for better aggregation of contextual discriminative information at different scales from a frequency domain perspective, effectively alleviating the problem of small target size and frequency deviation of the CNN-based local context operation L and global context operation G.

[0024] The input feature of frequency domain attention F is , obtained by fast Fourier transform FFT Since the FFT obtains a real signal, only the positive frequencies below the Nyquist frequency need to be retained to retain the complete information. This operation is expressed as:

[0025] in, is a complex-valued tensor, in the channel domain, The real and imaginary parts of are separated and stacked.

[0026]

[0027] Among them, among them, is the frequency domain feature after Fourier transform, , I. and R. represent the real and imaginary parts respectively, and Y is the feature after the real and imaginary parts are separated and stacked in the channel dimension. Subsequently, a convolutional layer with ReLU and layer normalization is used, and the calculation process is expressed as:

[0028] With the support of the above structure, the frequency characteristics can be adjusted while the nonlinear characteristics can be significantly enhanced.

[0029] Finally, the features are remapped back to the spatial domain using the inverse fast Fourier transform (iFFT);

[0030] where Complex. reconstructs the complex value using the real and imaginary parts.

[0031] Specifically, the template mask encoder in step 3 is as follows: Figure 4 As shown. The template mask encoder uses the template image and the binary rectangular mask of the target location in the template image As input. Where R represents the dimension of the variable, is the height of the template image, is the width of the template image. The binary rectangular mask B is generated from the ground-truth bounding box annotation of the template frame with negligible computational cost. Specifically, the assignment rule for position p is expressed as , specifically defined as:

[0032] Next, Z and B are stacked in the channel dimension and the stacked data is input into the template mask encoder. It consists of three layers of multi-layer perceptrons (MLPs). Finally, the template mask encoder generates a learnable template mask vector .in, is the template mask vector The number of channels.

[0033] The above process is expressed as:

[0034] Among them, Concat means stacking features in the channel dimension.

[0035] Specifically, the frequency domain-spatial domain attention interaction module in step 4 includes: like Figure 3 As shown, after obtaining the template mask vector generated by the template mask encoder Afterwards, the template generated by the frequency-spatial feature fusion module and the search area aggregate features and Flatten separately to obtain the mask vector , search for branch features and template branch features These vectors are then fed into an N-layer frequency-spatial attention interaction module.

[0036] like Figure 6 As shown in (b), the frequency-spatial attention interaction module combines frequency-domain attention with the original spatial attention or mask-guided spatial attention mechanism to promote bidirectional information interaction. This integration bidirectionally enhances the discriminative ability of frequency-domain and spatial-domain features. Specifically, bidirectional interaction can be achieved through the proposed frequency-domain interaction FI (Frequency-Interaction) and spatial-domain interaction SI (Spatial-Interaction).

[0037] 1) Mask-guided spatial attention and original spatial attention: like Figure 3 As shown in (c) and (d) of Figure 6, given As input, query, key, and value matrices are generated by linear projection.

[0038] The query, key, and value matrix generation process for the search branch is:

[0039] The query, key, and value matrix generation process of the template branch is:

[0040] in, is the linear projection matrix, and is the query, key, and value matrix for the search region branch ( and belongs to the template branch).

[0041] When the search branch calculates attention, the query matrix From the search image features, key matrix Sum Matrix From template image features; When calculating attention in the template branch, the query matrix From the template image features, key matrix Sum Matrix From search image features; This is to utilize the cross-attention mechanism to perform spatial interaction between the template and the search region. This approach is consistent with the essence of the Siamese network-based tracking method, which accurately locates the target by matching the similarity between the template image and the search image.

[0042] It is worth noting that in the mask-guided spatial attention, the value matrix and the mask vector Perform element-wise addition. This introduces The target location and visual information implied in .

[0043] The subsequent computations of the original spatial attention and mask-guided spatial attention are the same. The query, key, and value matrices are divided into h heads, i.e. , , , to use multi-head cross attention, where Q, K, and V are query matrices, key matrices, and value matrices respectively. The dimensions of each head are , c is the number of channels, and h is the number of attention heads. Figure 6(c) and Figure 6(d) correspond to the case of h = 1. The output of the i-th head Calculated as:

[0044] in, Represents relative position encoding; superscript T represents matrix transposition, i represents The i-th component of ;.

[0045] Softmax represents an activation function. It can convert any real number vector into a probability distribution. The formula of the Softmax function is as follows:

[0046] in, Represents the input vector No. elements, The first The index of the elements, is a vector The number of elements in the denominator. is the exponential sum of all input vector elements and is used to normalize the result to a probability distribution.

[0047] The output of the Softmax function is a set of probability values, each value is between 0 and 1, and the sum of the probabilities of all outputs is 1.

[0048] Finally, by stacking all To generate , and linearly project them To integrate.

[0049]

[0050] in, represents a linear mapping, Represents a linear projection operation.

[0051] 2) Frequency Domain Attention like Figure 6 As shown in (a), the query matrix Q of the template or search region branch is directly input into the frequency domain attention F to generate . Represents the output of frequency domain attention.

[0052]

[0053] The frequency domain attention F used is consistent with that described in the frequency domain-spatial domain feature fusion module.

[0054] 3) Frequency domain interaction FI and spatial domain interaction SI: In order to promote the bidirectional information exchange between the frequency domain attention path and the spatial attention path, frequency domain interaction FI and spatial domain interaction SI are designed.

[0055] The frequency domain interaction FI and spatial domain interaction SI can adaptively reweight the features of the two paths in the frequency domain or spatial domain. The spatial domain interaction SI generates a weight map of size H×W×1 for each unique spatial location, corresponding to the information derived from spatial attention.

[0056]

[0057] Among them, the Sigmoid function is the activation function, and Conv1×1 is the 1×1 convolutional layer.

[0058] The Sigmoid function converts the input real value into a value between 0 and 1. The formula of the Sigmoid function is as follows:

[0059] in represents the real value of the input, is the base of natural logarithms (approximately equal to 2.71828), is the negative exponent of the input value. The output value of the Sigmoid function is between 0 and 1, and when the input value tends to positive infinity, the output value tends to 1; when the input value tends to negative infinity, the output value tends to 0. This characteristic makes the Sigmoid function particularly suitable for outputting probability values ​​or for activation functions in neural networks.

[0060] The frequency domain interaction FI produces a 1×1×C weight map corresponding to the global frequency domain information obtained through the frequency domain attention path.

[0061]

[0062] in, and are two weight maps, GAP represents global average pooling. Then and Respectively and Element-wise multiplication is performed to facilitate information exchange between the spatial attention path and the frequency domain attention path. Finally, the reweighted features from the two paths are added element-wise and fused using a 1×1 convolutional layer to obtain the final output .

[0063]

[0064] in, represents linear projection, ⊙ represents element-wise multiplication, and ⊕ represents element-wise addition.

Claims

1. A mask-guided frequency-domain-spatial-domain hybrid network target tracking method, characterized in that: The steps include: Step 1: Input the image pair including the search image Z and the template image X into the convolutional backbone network for visual feature extraction, and obtain multi-layer and multi-resolution search branch features and template branch features respectively; Step 2: Use the frequency domain-spatial domain feature fusion module FSF in the search branch and template branch respectively to fuse the multi-layer and multi-resolution search branch features and template branch features extracted in step 1 to obtain the fused backbone network features, which include the fused template branch features. And the fused search branch features ; Step 3: Stack the template mask B with the template image Z and pass it to the template mask encoder TME to encode and generate a learnable template mask vector ; Step 4: Combine the backbone network features fused in step 2 with the template mask vector generated in step 3 Jointly input to the frequency-spatial attention interaction module FSA to promote template branch features and search branch features Full interaction and enhancement to generate the final search branch features ; Step 5: The final search branch features generated in step 4 Input to the tracking prediction head network to generate the coordinates of the target box to be tracked.

2. According to claim 1, a mask-guided frequency-domain-spatial-domain hybrid network target tracking method is characterized in that: There are two frequency domain-spatial domain feature fusion modules FSF in step 2. The two frequency domain-spatial domain feature fusion modules FSF iteratively aggregate the three-layer backbone network features. The fused features are expressed as , UpSample and DownSample represent upsampling and downsampling respectively, to maintain the consistency of feature resolution of different layers of backbone networks; is the template branch feature after fusion, is the fused search branch feature, , are the height and width of the template image input to the backbone network respectively; , are the height and width of the search image input to the backbone network respectively; is the step size of the backbone network output visual features; the fusion process is expressed as: , ; in, , They are the search image and template image input into the backbone network respectively; , , They are the visual features of the 2nd, 3rd, and 4th layers of the backbone network output after the template image is input into the backbone network; , , They are the visual features of the 2nd, 3rd, and 4th layers of the backbone network output after the search image is input into the backbone network; The frequency domain-spatial domain feature fusion module FSF is used to iteratively fuse two features: , , , ; in, is the intermediate feature of the fusion process, is the final fused feature; ⊕ and ⊙ represent element-by-element addition and element-by-element multiplication respectively; first, the two features Add; C, H, W are input features respectively The number of channels, height, and width are calculated, and the results are then input into the global context operation G and the frequency domain attention F; the outputs of the global context operation G and the frequency domain attention F are added, and then the Sigmoid function is applied to generate the weights , whose values ​​range from 0 to 1; the weights are obtained through a similar process , It is generated collaboratively by frequency domain attention F and local context operation L; Get weight , After that, use , and , To integrate , , enabling the network to , Make a soft selection.

3. According to claim 2, a mask-guided frequency-domain-spatial-domain hybrid network target tracking method is characterized in that: The global context operation G uses global average pooling to compress features of size H×W into a scalar of size 1×1, and then passes through a two-layer 1×1 convolution; the 1×1 scalar contains the global context information of the feature; at the same time, a two-layer 1×1 convolution is used as the local context operation L, which aggregates context information for each pixel position in a spatial manner; the features generated by the local context operation L have the same resolution as the initial features, so that detailed information can be retained and emphasized.

4. The mask-guided frequency-domain-spatial-domain hybrid network target tracking method according to claim 2, characterized in that: The input feature of the frequency domain attention F is , obtained by fast Fourier transform FFT ; Since FFT obtains a real signal, only the positive frequencies below the Nyquist frequency need to be retained to retain complete information; this operation is expressed as: ; in, is a complex-valued tensor, in the channel domain, The real and imaginary parts of are separated and stacked; ; in, is the frequency domain feature after Fourier transform, , .and . represent the real part and the imaginary part respectively, The features after the real and imaginary parts are separated and stacked in the channel dimension; then, a convolutional layer with ReLU and layer normalization is used. The calculation process is expressed as: ; Finally, the features are remapped back to the spatial domain using an inverse fast Fourier transform: ; Where Complex. reconstructs the complex value using the real and imaginary parts, and iFFT stands for inverse fast Fourier transform.

5. The mask-guided frequency-domain-spatial-domain hybrid network target tracking method according to claim 1, characterized in that: The template mask encoder TME in step 3 is based on the template image and the binary rectangular mask of the target location in the template image As input; where represents the dimension of the variable, is the height of the template image, is the width of the template image; binary rectangular mask is generated from the ground-truth bounding box annotations of the template frame; the assignment rule for position p is expressed as , specifically defined as: ; Next, Z and B are stacked in the channel dimension and the stacked data is input into the template mask encoder; it consists of three layers of multi-layer perceptrons. Finally, the template mask encoder generates a learnable template mask vector ; in, is the template mask vector The number of channels; The above process is expressed as: ; Among them, Concat means stacking features in the channel dimension.

6. The mask-guided frequency-domain-spatial-domain hybrid network target tracking method according to claim 1, characterized in that: The step 4 comprises: After obtaining the template mask vector generated by the Template Mask Encoder (TME) Afterwards, the template generated by the frequency-spatial fusion module Frequency-Spatial Fusion (FSF) and the search area aggregate features and Flatten separately to obtain the mask vector , search for branch features and template branch features ; Mask vector , the search branch feature S, the template branch feature T and the feature are input into the frequency-spatial attention interaction module FSA of N layers; The frequency-spatial attention interaction module FSA combines frequency-domain attention with the original spatial attention or mask-guided spatial attention mechanism to promote two-way information interaction; the two-way interaction is achieved through the proposed frequency-domain interaction FI and spatial-domain interaction SI.

7. A mask-guided frequency-domain-spatial-domain hybrid network target tracking method according to claim 6, characterized in that: The mask-guided spatial attention and the original spatial attention are: The search branch uses mask-guided spatial attention, and the template branch uses normal spatial attention. Both branches use multi-head cross attention to calculate the attention map in the spatial domain. Given As input, query, key, and value matrices are generated by linear projection: The query, key, and value matrix generation process for the search branch is: ; The query, key, and value matrix generation process of the template branch is: ; in, is the linear projection matrix, and is the query, key, and value matrix for the search region branch, and The query, key, and value matrices belonging to the template branch; When the search branch calculates attention, the query matrix From the search image features, key matrix Sum Matrix From template image features; When calculating attention in the template branch, the query matrix From the template image features, key matrix Sum Matrix From search image features; When the search branch uses mask-guided spatial attention, the value matrix and the mask vector Perform element-by-element addition, introducing The target location and visual information implied in The calculation of the original spatial attention is the same as above; The query, key, and value matrices are divided into h heads, i.e. , , To use multiple cross-attention; in, They are query matrix, key matrix, and value matrix respectively; the dimension of each header is , is the number of channels, is the number of attention heads; Output of the i-th head Calculated as: ; in, Represents relative position encoding; superscript T represents matrix transposition, i represents The i-th component of ; Finally, by stacking all To generate , and linearly project them To integrate: ; in, represents a linear mapping, Represents a linear projection operation.

8. The mask-guided frequency-domain-spatial-domain hybrid network target tracking method according to claim 6, characterized in that: Frequency domain attention includes: The query matrix Q of the template or search region branch is directly input into the frequency domain attention F to generate ; Represents the output of frequency domain attention; ; The frequency domain attention F used is consistent with that described in the frequency domain-spatial domain feature fusion module.

9. The mask-guided frequency-domain-spatial-domain hybrid network target tracking method according to claim 6, characterized in that: Frequency domain interaction FI and spatial domain interaction SI include: The frequency domain interaction FI and the spatial domain interaction SI can adaptively reweight the features of the two paths in the frequency domain or spatial domain; the spatial domain interaction SI generates a weight map of size H×W×1 for each unique spatial position, corresponding to the information obtained by spatial attention: ; Among them, the Sigmoid function is the activation function, It is a 1×1 convolutional layer; The frequency domain interaction FI produces a 1×1×C weight map corresponding to the global frequency domain information obtained through the frequency domain attention path: ; in, and are two weight maps, GAP represents global average pooling. Then and Respectively and Element-wise multiplication to facilitate information exchange between the spatial attention path and the frequency domain attention path; Finally, the reweighted features from both paths are added element-wise and fused using a 1×1 convolutional layer to obtain the final output : ; in, represents linear projection, ⊙ represents element-wise multiplication, and ⊕ represents element-wise addition.

Citation Information

Cited By

  • ViT time sequence-positioning enhancement-based power target equipment visual segmentation method

    CN119672039A

  • End-to-end single target tracking method based on underwater clue driving prompt learning

    CN119832025A

  • Remote sensing image directed target detection method based on double-domain feature fusion

    CN119919819A

  • Industrial flaw detection method based on frequency domain mask strategy

    CN120088193A

  • Optical communication chip image enhancement method, system and device based on direction sensitive filtering and edge enhancement coupling and medium

    CN120976061A