A Visible Light-Thermal Infrared Target Tracking Method Based on Frequency Domain Fusion

Abstract

The present invention discloses a visible light - thermal infrared target tracking method based on frequency - domain fusion, belonging to the field of computer vision technology. This method is named WTTrack and includes the following steps: Step 1, read video frame images, divide them into a search region and a template, and perform block - division and encoding processing; Step 2, use Transformer to extract image features of visible light and thermal infrared, and fuse cross - modal feature information; that is, extract and fuse the template features of the two modalities in the feature space, and perform frequency - domain fusion on the search region through wavelet transform and cross - attention, and finally generate complementary feature representations; Step 3, input the final feature representation into the tracking head for prediction and generate a tracking box to achieve target tracking. The present invention can fully exploit and fuse the frequency - domain information and spatial features of visible light and thermal infrared modalities, and has better accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly relates to a visible light-thermal infrared target tracking method based on frequency domain fusion. Background Art

[0002] RGB visible light plays an important role in the field of visual target tracking. However, target tracking that only utilizes visible light information may still fail in complex or extreme scenarios, such as fast movement, insufficient light, and motion blur.

[0003] Multi-modal tracking has received increasing attention because it can achieve more robust tracking by leveraging cross-modal complementary information. Taking visible light-thermal infrared (RGB-T) tracking as an example, it can integrate the ability of thermal infrared images to penetrate obstacles and the sensitivity in low-light environments, while visible light images can provide rich color details and texture information under good lighting conditions. Currently, many researchers have introduced a large number of new methods to enhance the robustness and accuracy of RGB-T target tracking. At present, the processing methods for RGB-T multi-modal fusion tracking mainly focus on feature processing, especially in feature fusion. Most methods enhance and fuse the original spatial features.

[0004] Although great progress has been made in the research of RGB-T target tracking in recent years, there are still deficiencies in feature fusion. Existing tracking methods hardly pay attention to the frequency domain features of images, and information in different frequency domains has different characteristics. For example, low frequencies correspond to the global content of the image, and high frequencies correspond to the texture details of the image, and these characteristics are helpful for both the classification and localization of target tracking. Therefore, how to effectively utilize and interactively fuse the frequency domain features between different modalities is a technical problem that needs to be solved currently.

[0005] Based on this, the present invention designs a visible light-thermal infrared target tracking method based on frequency domain fusion to solve the above problems. Summary of the Invention

[0006] In view of the above-mentioned drawbacks of the prior art, the present invention provides a visible light-thermal infrared target tracking method based on frequency domain fusion. By simultaneously extracting and fusing frequency domain and spatial information, the final feature representation is obtained for tracking.

[0007] To achieve the above object, the present invention is realized through the following technical solutions:

[0008] A visible light-thermal infrared target tracking method based on frequency domain fusion, comprising the following steps:

[0009] Step 1, select video frame images, divide them into a search area and a template, and perform block and encoding processing;

[0010] Step 2: Input the tokens of the visible light and thermal infrared modalities into the Transformer encoder module. After being extracted by the Transformer encoder module, the total feature tokens are divided into search region tokens and template tokens again. For the search region tokens, the discrete wavelet transform is first used to obtain the frequency domain information, which includes low-frequency components and high-frequency components. Then, the frequency domain information of the visible light and thermal infrared modalities is interacted, and then the cross-attention mechanism is used to perform information interaction and fusion of the two modalities again to obtain the final search region tokens of the visible light modality and the final search region tokens of the thermal infrared modality ; For the template tokens of the visible light and thermal infrared modalities, using the original spatial information, modality fusion is performed in the Fusion module to obtain the new template token Z f ; Then, the final search region tokens of the visible light modality and the final search region tokens of the thermal infrared modality are combined with the new template token Z f respectively, and the template and search region tokens after being fused by the wavelet transform fusion module WTFM are output. This process is repeated multiple times to obtain the final feature representation;

[0011] Step 3: Input the final feature representation into the tracking head for prediction and generate the tracking box.

[0012] Furthermore, Step 1 is specifically as follows: In the dataset, a batch of images are randomly selected as the input, divided into a search region of size 256×256 and a template image of 128×128, and then block and encoding processing are performed.

[0013] Furthermore, the LasHeR dataset is used. The LasHeR dataset contains a total of 1224 video sequences and 730000 frame pairs.

[0014] Furthermore, in Step 2, the low-frequency component is the low-frequency-low-frequency LL component, and the high-frequency component is the high-frequency H component; the calculation method is as follows:

[0015] Let the search region tokens of the visible light and thermal infrared modalities be X r and X t respectively, and calculate the low-frequency-low-frequency LL component and the high-frequency H component under the two modalities respectively;

[0016] ;

[0017] Among them, DWT is the discrete wavelet transform, Conv is the convolution operation, is the low-frequency-low-frequency LL component of the visible light modality, is the high-frequency H component of the visible light modality, is the low-frequency - low-frequency LL component of the thermal infrared modality, is the high-frequency H component of the thermal infrared modality.

[0018] Furthermore, the high-frequency H component includes three high-frequency components: low-frequency - high-frequency LH, high-frequency - low-frequency HL, and high-frequency - high-frequency HH, which respectively represent horizontal, vertical, and diagonal high-frequency information.

[0019] Furthermore, in step two, the frequency domain information of the visible light and thermal infrared modalities is interacted, and the frequency domain information includes the low-frequency - low-frequency LL component and the high-frequency - high-frequency HH component;

[0020] ;

[0021] Among them, Avg is the average pooling operation, W l is the low-frequency linear weight, W h is the high-frequency linear weight, is the high-frequency - high-frequency component of the visible light modality, is the high-frequency - high-frequency component of the thermal infrared modality, is the low-frequency - low-frequency component of the new visible light search area, is the low-frequency - low-frequency component of the new thermal infrared search area, is the high-frequency - high-frequency component of the new visible light search area, is the high-frequency - high-frequency component of the new thermal infrared search area.

[0022] Furthermore, in step two, the cross-attention mechanism is used to perform information interaction and fusion between the two modalities again, and the residual connection is used to retain a part of the original feature information to obtain the search area token of the final visible light modality and the search area token of the final thermal infrared modality ;

[0023] ;

[0024] Among them, Proj is the linear mapping operation, is the Key component of the visible light, is the Value component of the visible light, Cat is the concatenation operation, W k is the linear weight of the Key, is the Key component of the thermal infrared, is the Value component of the thermal infrared; is the low-frequency - high-frequency component of the visible light search area after interaction, is the high-frequency - low-frequency component of the visible light search area after interaction, is the low-frequency - high-frequency component of the thermal infrared search area after interaction, is the high-frequency - low-frequency component of the thermal infrared search area after interaction;

[0025] ;

[0026] Among them, is the Query component of visible light, is the search area token of visible light, W q is the linear weight of Query, is the Query component of thermal infrared, is the search area token of thermal infrared, is the transpose of the visible light Key component, N X is the feature length of the search area token, V r is the Value component of visible light, is the frequency-domain fused visible light search area token after cross-attention calculation, is the frequency-domain fused thermal infrared search area token after cross-attention calculation, is the transpose of the thermal infrared Key component, V t is the Value component of thermal infrared;

[0027] ;

[0028] Among them, Proj is the linear mapping operation, IWT is the inverse wavelet transform, is the search area token of the final visible light modality, is the search area token of the final thermal infrared modality, is the low-frequency - high-frequency component of the visible light search area, is the high-frequency - low-frequency component of the visible light search area, is the high-frequency - high-frequency component of the visible light search area, is the low-frequency - high-frequency component of the thermal infrared search area, is the high-frequency - low-frequency component of the thermal infrared search area, is the high-frequency - high-frequency component of the thermal infrared search area.

[0029] Furthermore, in step two, the search area token of the final visible light modality and the search area token of the final thermal infrared modality Combine with the new template token Z respectively f Combine them, and output the template and the search area token after being fused by the wavelet transform fusion module WTFM. Repeat this process multiple times to obtain the final feature representation;

[0030] ;

[0031] Among them, Z f is the template token after spatial feature fusion processing in the Fusion module, F r is the final visible light token, F t is the final thermal infrared token, GELU is the activation function, LN is the layer normalization operation, is the template token of visible light, is the template token of thermal infrared, is the search area token of the final visible light modality, is the search area token of the final thermal infrared modality.

[0032] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention takes the video pairs of visible light and thermal infrared as inputs, divides them into search areas and templates after block coding; performs different fusion processes on the search areas and templates respectively, uses frequency domain information for interactive processing on the search areas, and uses spatial information for interactive processing on the templates, which can effectively fuse frequency domain information and spatial information; the performance of the present invention on multiple visible light-thermal infrared tracking datasets is better than other advanced RGB-T tracking methods.

[0033] 2. The present invention can also meet the basic requirements in terms of real-time performance, without losing speed while ensuring effective tracking.

[0034] 3. The present invention uses discrete wavelet transform for frequency domain decomposition, thereby performing frequency domain information interaction. Benefiting from the effective fusion of frequency domain features, the method of the present invention achieves a more robust tracking effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1It is the framework diagram of the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention;

[0037] Figure 2 It is the comparison result of the attention maps between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods;

[0038] Figure 3 It is the first comparison result of the tracking results between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods (the first row of the zipper);

[0039] Figure 4 It is the second comparison result of the tracking results between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods (the occluded bicycle);

[0040] Figure 5 It is the third comparison result of the tracking results between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods (the boy in black clothes);

[0041] Figure 6 It is the fourth comparison result of the tracking results between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods (the item folded with the left hand);

[0042] Figure 7 It is the comparison result of the precision rates between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods on the LasHeR dataset;

[0043] Figure 8 It is the comparison result of the success rates between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods on the LasHeR dataset. Specific embodiments

[0044] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] Embodiment 1: A visible light-thermal infrared target tracking method based on frequency domain fusion, named WTTrack, includes the following steps:

[0046] Step 1: Read the video frame images, divide them into a search area and a template, and perform block and encoding processing;

[0047] Specifically, in the dataset, such as the LasHeR dataset, a batch of images is randomly selected as input, divided into a search area of size 256×256 and a template image of 128×128, and then block and encoding processing is performed. The LasHeR dataset contains a total of 1224 video sequences and 730,000 frame pairs, with rich image resources.

[0048] Step 2: Use a Transformer to extract the image features of visible light and thermal infrared, and fuse the cross-modal feature information; that is, extract and fuse the template features of the two modalities in the feature space, and perform frequency-domain fusion on the search area through wavelet transform and cross-attention, and finally generate complementary feature representations;

[0049] Input the tokens of the visible light and thermal infrared modalities into the Transformer encoder module. The total feature tokens extracted by the Transformer encoder module are divided into search area tokens and template tokens again. The frequency-domain information is obtained for the search area using discrete wavelet transform and fused with the spatial features to obtain new search area tokens; that is, first use discrete wavelet transform to divide the tokens into four high-frequency and low-frequency components, and cross the high and low frequencies of the two modalities to obtain enhanced frequency-domain information, and then fuse it with the spatial information. For the templates of the two modalities, use the original spatial information and perform modality fusion in the Fusion module to obtain new template tokens Z f ; Finally, concatenate the new search area tokens and new template tokens and continue to input them into the Transformer encoder module. Repeat this process multiple times to obtain the final feature representation.

[0050] Step 2 includes the following steps:

[0051] Step 2.1: Let the search area tokens of the visible light and thermal infrared modalities be X r and X t , and calculate the low-low (LL) component and high (H) component of the two modalities respectively;

[0052] ;

[0053] Among them, DWT is the discrete wavelet transform, Conv is the convolution operation, is the low-low (LL) component of the visible light modality, is the high (H) component of the visible light modality, is the low-low (LL) component of the thermal infrared modality, is the high (H) component of the thermal infrared modality;

[0054] Among them, the high-frequency H (High) component includes three high-frequency components: low-high LH (Low-High), high-low HL (High-Low), and high-high HH (High-High), which respectively represent horizontal, vertical, and diagonal high-frequency information;

[0055] Step 2.2: Interact the frequency-domain information of the visible light and thermal infrared modalities. The frequency-domain information includes the low-low LL component and the high-high HH component;

[0056] ;

[0057] Among them, Avg is the average pooling operation, W l is the low-frequency linear weight, W h is the high-frequency linear weight, is the high-high component of the visible light modality, is the high-high component of the thermal infrared modality, is the low-low component of the new visible light search area, is the low-low component of the new thermal infrared search area, is the high-high component of the new visible light search area, is the high-high component of the new thermal infrared search area.

[0058] Step 2.3: Then, use the cross-attention mechanism to perform information interaction and fusion between the two modalities again, and use the residual connection to retain a part of the original feature information to obtain the search area token of the final visible light modality and the search area token of the final thermal infrared modality;

[0059] ;

[0060] Among them, Proj is the linear mapping operation, is the Key component of the visible light, is the Value component of the visible light, Cat is the concatenation operation, W k is the linear weight of the Key, is the Key component of the thermal infrared, is the Value component of the thermal infrared;

[0061] is the low-high component of the visible light search area after interaction using a method similar to that in Step 2.2, The high-frequency - low-frequency components of the visible light search area after interaction using a method similar to that in Step 2.2 The low-frequency - high-frequency components of the thermal infrared search area after interaction using a method similar to that in Step 2.2 The high-frequency - low-frequency components of the thermal infrared search area after interaction using a method similar to that in Step 2.2

[0062] ;

[0063] Among them, is the Query component of visible light, is the search area token of visible light, W q is the linear weight of Query, is the Query component of thermal infrared, is the search area token of thermal infrared, is the transpose of the visible light Key component, N X is the search area token feature length, V r is the Value component of visible light, is the visible light search area token after frequency domain fusion by cross - attention calculation, is the thermal infrared search area token after frequency domain fusion by cross - attention calculation, is the transpose of the thermal infrared Key component, V t is the Value component of thermal infrared;

[0064] ;

[0065] Among them, Proj is the linear mapping operation, IWT is the inverse wavelet transform, is the search area token of the final visible light modality, is the search area token of the final thermal infrared modality, is the low-frequency - high-frequency components of the visible light search area, is the high-frequency - low-frequency components of the visible light search area, is the high-frequency - high-frequency components of the visible light search area, is the low-frequency - high-frequency components of the thermal infrared search area, is the high-frequency - low-frequency components of the thermal infrared search area, is the high-frequency - high-frequency components of the thermal infrared search area.

[0066] Step 2.4. Then, the search area token of the final visible light modality and the search area token Combine with the new template token Z respectively f Combine them, and output the template and the search region token after being fused by the Wavelet Transform Fusion Module (WTFM), which is the final feature representation;

[0067] ;

[0068] Among them, Z f is the template token after spatial feature fusion processing in the Fusion module, F r is the final visible light token, F t is the final thermal infrared token, GELU is the activation function, LN is the layer normalization operation, is the visible light template token, is the thermal infrared template token, is the search region token of the final visible light modality, is the search region token of the final thermal infrared modality;

[0069] Step 3: Input the final feature representation into the tracking head to perform prediction and generate a tracking box, realizing RGB-T object tracking;

[0070] Input the final feature representation after fusing the visible light and thermal infrared modalities into the tracking head, and the tracking head generates a tracking box for tracking; evaluate on multiple datasets, and the evaluation result is the tracking effect.

[0071] The present invention takes a video pair of visible light and thermal infrared as input, divides it into blocks and encodes it into a search region and a template; performs different fusion processes on the search region and the template respectively, uses frequency domain information for interactive processing on the search region, and uses spatial information for interactive processing on the template, and can realize the effective fusion of frequency domain information and spatial information; the performance of the present invention on multiple visible light-thermal infrared tracking datasets is better than other advanced RGB-T tracking methods.

[0072] The present invention can also meet the basic requirements in terms of real-time performance, without losing speed while ensuring effective tracking.

[0073] The present invention uses discrete wavelet transform for frequency domain decomposition to perform frequency domain information interaction. Benefiting from the effective fusion of frequency domain features, the method of the present invention achieves a more robust tracking effect.

[0074] Experimental Example 1: The present invention is compared with other RGB-T object tracking methods in terms of performance, and the results are shown in Table 1.

[0075] Table 1 Comparison results of the method of the present invention (WTTrack) and other methods on three RGBT tracking datasets

[0076]

[0077] It can be seen that the present invention has certain performance advantages over other methods on the three datasets RGBT210, RGBT234, and LasHeR. The main reason is that the present invention uses the information in the frequency domain and the original spatial information to fuse the visible light and thermal infrared modalities, and learns a more sufficient feature representation. On the RGBT210 dataset, the method of the present invention improves the precision by 3.5% compared with the latest CAT++ method and improves by 0.4% compared with the second-ranked TBSI method. On the RGBT234 dataset, the method of the present invention improves the precision by 4.5% and the success rate by 2.8% compared with OSTrack respectively. On LasHeR, the method of the present invention improves the precision by 4.6% and the success rate by 3.8% compared with SDSTrack. Based on the above experimental comparisons, it fully demonstrates the good performance of the present invention in RGBT object tracking.

[0078] Experimental Example 2: The present invention is further compared with the three current best-performing tracking methods in terms of the challenge attributes of RGBT234. The comparison results are shown in Table 2.

[0079] Table 2 Comparison results of the method of the present invention (WTTrack) and other methods on the challenge attributes of the RGBT234 dataset

[0080]

[0081] It can be seen that in terms of the challenge attributes of No Occlusion (NO), Low Illumination (LI), Hyaline Occlusion (HO), Low Resolution (LR), and Background Clutter (BC), compared with the three trackers CMPP, TBSI, and ViPT, the method of the present invention (WTTrack) achieves the best results. It can also achieve comparable performance in terms of the challenge attributes of Motion Blur (MB), Camera Motion (CM), Scale Variation (SV), Thermal Cross (TC), fast motion (FM), partial occlusion (PO), and deformation (DEF). Therefore, the experimental results show that the present invention has good tracking performance in multiple challenge scenarios, indicating that the method of the present invention (WTTrack) has good accuracy and robustness.

[0082] Experimental Example 3:

[0083] Figure 2 This is the comparison result of the attention maps between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods; it can be seen that compared with OSTrack, the method of the present invention (WTTrack) can notice the target more accurately.

[0084] Figures 3 - 6 This is the comparison result of the tracking results between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods; it can be seen that in the case where other methods lose the target, the method of the present invention (WTTrack) can still track the target well.

[0085] Figure 7 This is the comparison result of the precision between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods on the LasHeR dataset; it can be seen that compared with other methods, the method of the present invention (WTTrack) achieves the optimal precision result.

[0086] Figure 8 This is the comparison result of the success rate between the visible light-thermal infrared target tracking method based on frequency domain fusion of the present invention and other methods on the LasHeR dataset. It can be seen that the success rate of the method of the present invention (WTTrack) is improved compared with other methods.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visible light-thermal infrared target tracking method based on frequency domain fusion, characterized in that, Including the following steps: Step 1: Select video frame images, divide them into a search area and a template, and perform block and encoding processing; Step 2: Input the tokens of the visible light and thermal infrared modalities into the Transformer encoder module. The total feature tokens extracted by the Transformer encoder module are divided into search region tokens and template tokens again. For the search region tokens, first use the discrete wavelet transform to obtain the frequency domain information, which includes low-frequency components and high-frequency components. Then, interact the frequency domain information of the visible light and thermal infrared modalities. After that, use the cross-attention mechanism to perform information interaction and fusion of the two modalities again to obtain the final search region tokens of the visible light modality and the final search region tokens of the thermal infrared modality ; Use the original spatial information for the template tokens of the visible light and thermal infrared modalities, and perform modality fusion in the Fusion module to obtain the new template token Z f ; Then, combine the final search region tokens of the visible light modality and the final search region tokens of the thermal infrared modality with the new template token Z f respectively, and output the template and search region tokens fused by the wavelet transform fusion module WTFM. Repeat this process multiple times to obtain the final feature representation; Step 3: Input the final feature representation into the tracking head for prediction and generate a tracking box.

2. The visible light-thermal infrared target tracking method based on frequency domain fusion according to claim 1, wherein Specifically, Step 1 is as follows: In the dataset, randomly select a batch of images as input, divide them into a search area of size 256×256 and a template image of 128×128, and then perform block and encoding processing.

3. The visible light-thermal infrared target tracking method based on frequency domain fusion according to claim 2, wherein The LasHeR dataset is used, and the LasHeR dataset contains a total of 1224 video sequences and 730000 frame pairs.

4. The visible light-thermal infrared target tracking method based on frequency domain fusion according to claim 1, wherein In Step 2, the low-frequency component is the low-frequency - low-frequency LL component, and the high-frequency component is the high-frequency H component; the calculation method is: Let the search area tokens for the visible light and thermal infrared modalities be X r and X t , respectively calculate the low-frequency - low-frequency LL component and the high-frequency H component in the two modalities; ； Among them, DWT is the discrete wavelet transform, Conv is the convolution operation, is the low-frequency - low-frequency LL component of the visible light modality, is the high-frequency H component of the visible light modality, is the low-frequency - low-frequency LL component of the thermal infrared modality, is the high-frequency H component of the thermal infrared modality.

5. The visible light-thermal infrared target tracking method based on frequency domain fusion according to claim 4, characterized in that The high-frequency H component includes three high-frequency components: low-frequency - high-frequency LH, high-frequency - low-frequency HL, and high-frequency - high-frequency HH, which represent horizontal, vertical, and diagonal high-frequency information respectively.

6. The method for visible-light thermal-infrared target tracking based on frequency-domain fusion according to claim 5, wherein In Step 2, the frequency-domain information of the visible light and thermal infrared modalities is interacted, and the frequency-domain information includes the low-frequency - low-frequency LL component and the high-frequency - high-frequency HH component; ； Among them, Avg is the average pooling operation, W l is the low-frequency linear weight, W h is the high-frequency linear weight, is the high-frequency - high-frequency component of the visible light modality, is the high-frequency - high-frequency component of the thermal infrared modality, is the low-frequency - low-frequency component of the new visible light search area, is the low-frequency - low-frequency component of the new thermal infrared search area, is the high-frequency - high-frequency component of the new visible light search area, is the high-frequency - high-frequency component of the new thermal infrared search area.

7. The visible light-thermal infrared target tracking method based on frequency domain fusion according to claim 6, characterized in that, In step two, the cross-attention mechanism is used to perform information interaction and fusion between the two modalities again, and the residual connection is used to retain a part of the original feature information to obtain the search region tokens of the final visible light modality and the search region tokens of the final thermal infrared modality ; ； Among them, Proj is a linear mapping operation, is the Key component of visible light, is the Value component of visible light, Cat is a splicing operation, W k is the linear weight of Key, is the Key component of thermal infrared, is the Value component of thermal infrared; is the low-frequency - high-frequency component of the visible light search area after interaction, is the high-frequency - low-frequency component of the visible light search area after interaction, is the low-frequency - high-frequency component of the thermal infrared search area after interaction, is the high-frequency - low-frequency component of the thermal infrared search area after interaction; ； Among them, is the Query component of visible light, is the search area token of visible light, W q is the linear weight of Query, is the Query component of thermal infrared, is the search area token of thermal infrared, is the transpose of the visible light Key component, N X is the feature length of the search area token, V r is the Value component of visible light, is the visible light search area token of frequency domain fusion after cross-attention calculation, is the thermal infrared search area token of frequency domain fusion after cross-attention calculation, is the transpose of the thermal infrared Key component, V t is the Value component of thermal infrared; ； Among them, Proj is a linear mapping operation, IWT is an inverse wavelet transform, is the search area token of the final visible light modality, is the search area token of the final thermal infrared modality, is the low-frequency - high-frequency component of the visible light search area, is the high-frequency - low-frequency component of the visible light search area, is the high-frequency - high-frequency component of the visible light search area, is the low-frequency - high-frequency component of the thermal infrared search area, is the high-frequency - low-frequency component of the thermal infrared search area, is the high-frequency - high-frequency component of the thermal infrared search area.

8. The method for visible-light thermal-infrared target tracking based on frequency-domain fusion according to claim 7, wherein In step 2, the search area token of the final visible light modality and the search area token of the final thermal infrared modality are respectively combined with the new template token Z f to output the template and search area tokens fused by the wavelet transform fusion module WTFM. This process is repeated multiple times to obtain the final feature representation; ； Among them, Z f is the template token after spatial feature fusion processing in the Fusion module, F r is the final visible light token, F t is the final thermal infrared token, GELU is the activation function, LN is the layer normalization operation, is the template token of visible light, is the template token of thermal infrared, is the search area token of the final visible light modality, is the search area token of the final thermal infrared modality.

Images