Two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization

Abstract

The present invention relates to a two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization, which belongs to the field of computer vision technology. The specific steps are as follows: the main network framework includes a template branch and a search branch. The video frame image of the object to be tested is subjected to data processing, feature network extraction, and spatial position enhancement of the space-frequency domain equalization module to obtain two branch features. The features are integrated and processed as input to a two-stage fusion lightweight attention module. The present invention designs a space-frequency domain equalization module, combines multiple different domain information such as modal domain, spatial domain, and frequency domain, and performs feature-level information interaction to balance the features of different modal data. The two-stage fusion lightweight attention module is designed in combination with the output of the space-frequency domain equalization module, and uses feature-level information and decision-level information to readjust the distribution weight ratio of different modal and frequency domain data, balance features, and further improve feature expression capabilities.

Description

Technical Field

[0001] The present invention relates to a two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization, belonging to the technical field of computer vision. Background Art

[0002] Object tracking is an important task in the field of computer vision. For any video sequence, it is necessary to determine the target to be tracked in the first frame and the location and scale estimation of the target in each subsequent frame. With the emergence of correlation filtering and deep learning, the performance of object trackers has continued to improve. Despite significant progress, most visual tracking algorithms focus on visible light tracking. However, real-life scenes are complex and changeable. For example, in video objects with complex backgrounds, a large amount of deformation, motion, and occlusion often occur; in some scenes with changing lighting, the target is often in dim light, highly exposed, or submerged in the background. A simple visible light tracker cannot effectively solve the above problems.

[0003] In most cases, visible light images possess rich structural texture information, high contrast, and color information, but are susceptible to environmental influences. Infrared images, while lacking detailed texture and color information and exhibiting low contrast, contain the target's primary structural information and are insensitive to environmental changes. They are stable under conditions of drastic light changes or low light levels, and can penetrate rain, fog, and snow. Furthermore, thermal infrared images, which record the thermal radiation of objects, are highly effective in separating foreground and background from cluttered backgrounds for targets with distinct thermal signatures, such as pedestrians. Therefore, the introduction of infrared mode is expected to improve tracker performance, and target tracking based on the fusion of infrared and visible light images is gaining increasing attention.

[0004] The fusion of visible light and infrared modal images is a key challenge in RGBT tracking. Existing fusion tracking methods can be roughly divided into three categories: pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion fuses strictly registered image pairs pixel by pixel, then performs object tracking based on the merged image. Feature-level fusion extracts features from different modalities and fuses them according to fusion rules designed by different methods. Decision-level fusion tracks different modalities separately and then fuses the results obtained for each modality.

[0005] Each of the three methods has its pros and cons. Pixel-level fusion is sensitive to noise and places high demands on image registration. Decision-level fusion pays little attention to the feature complementarity between infrared and visible light images, leading to unreliable tracking results that rely on a single modality. Feature-level fusion is a commonly used fusion method, and how to design feature fusion methods is a question worth exploring. Direct feature fusion and content-dependent weighted fusion strategies mostly fail to consider the feature differences between the two modal images during the fusion process. While some methods take into account the differences in features between different modalities and use feature-level fusion strategies to calculate the channel weight ratio of the fused features, the high proportion of background information in the search image significantly affects the calculation of the feature weight ratio. This is because while the unique information of different modalities can complement each other, it often also introduces a large amount of negative information such as background and noise.

[0006] While the use of visible light trackers has increased in recent years, many challenges remain, such as thermal crosstalk, high exposure, and low illumination, which are difficult to resolve with a single modality. While some RGB-T tracker designs have been developed, most employ only a single-stage fusion approach, which prevents the fusion of data features from being properly processed and utilized. Some trackers employ a dual-stage fusion approach, but these approaches ignore the unique characteristics of each modality or have complex designs, which limits tracker performance and fails to effectively address these issues. Summary of the Invention

[0007] Purpose of the invention: In view of the above-mentioned existing problems and shortcomings, the purpose of the present invention is to provide a two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization, and to design a space-frequency domain equalization module. It combines multiple modal domains including modal domain, frequency domain, and spatial domain to perform feature-level information interaction to balance the features of different modal data and improve the feature expression ability of the network. Based on SiamBAN and SiamDL, feature-level and decision-level information are simultaneously fused as input features of the two-stage fusion lightweight attention module. A two-stage fusion lightweight attention module is designed to improve the original decision-level fusion method, reduce its feature loss and make it more lightweight, thereby enhancing the robustness of the tracker. Different training methods such as data augmentation, two-stage training, and end-to-end training have been used to conduct multiple tests on GTOT and LasHeR, and relatively advanced results have been achieved. In addition, in some complex scenarios, the tracker can achieve advanced results.

[0008] Technical solution: To achieve the above-mentioned purpose, the present invention adopts the following technical solution:

[0009] Step 1: The main network framework is divided into a template branch and a search branch. The whole network consists of input, feature-level fusion stage, two-level fusion stage and tracking head.

[0010] Step 2: The template branch inputs the visible light template image and the infrared template image, which are sent to the space-frequency domain equalization module for data processing to obtain a visible light high-frequency template image and an infrared low-frequency template image;

[0011] Step 3: The visible light template image, infrared template image, visible light high-frequency template image, and infrared low-frequency template image are respectively passed through the feature extraction network in the space-frequency domain equalization module to obtain visible light template features, infrared template features, visible light high-frequency template features, and infrared low-frequency template features;

[0012] Step 4: The features obtained in step 3 are subjected to spatial position enhancement in the space-frequency domain equalization module to obtain the final output features of the space-frequency domain equalization module corresponding to the template branch, namely the modulated visible light template features and infrared template features;

[0013] Step 5: The search branch inputs the visible light search image and the infrared search image, which are sent to the space-frequency domain equalization module for data processing to obtain a visible light high-frequency search image and an infrared low-frequency search image;

[0014] Step 6: The visible light search image, infrared search image, visible light high-frequency search image, and infrared low-frequency search image are respectively passed through the feature extraction network in the space-frequency domain equalization module to obtain visible light search features, infrared search features, visible light high-frequency search features, and infrared low-frequency search features;

[0015] Step 7: The features obtained in step 6 are subjected to spatial position enhancement in the space-frequency domain equalization module to obtain the final output features of the space-frequency domain equalization module corresponding to the search branch, namely the modulated visible light search features and infrared search features;

[0016] Step 8: The visible light template features, infrared template features, visible light search features, and infrared search features obtained in steps 4 and 7 are fed into the information integration. The integrated features are then used as the input of the two-stage fusion lightweight attention module. Finally, the tracking results are obtained through classification and regression of the tracking head.

[0017] Furthermore, the main network framework is divided into a template branch and a search branch, and is composed of an input, a feature-level fusion stage, a two-stage fusion stage, and a tracking head. The inputs are a visible light template image, an infrared template image, a visible light search image, and an infrared search image. The feature-level fusion stage consists of a spatial-frequency domain equalization module and the first half of the information integration process, specifically including data processing, feature network extraction, spatial position enhancement, and feature channel lightweighting. The two-stage fusion stage consists of the second half of the information integration module and a two-stage fusion lightweight attention module. The tracking head comprises classification and regression components.

[0018] Furthermore, the four input images are derived from paired dual-modal video frame images and are preprocessed in the same way as SiamFC: the input includes a template branch and a search branch. For the template branch, the input image size is 127×127×3. With the center of the original image target constraint frame as the origin, an area approximately twice the size of the target is cropped, and the part exceeding the original image is filled with the color mean, and then the cropped area is scaled to 127×127 as the template image. For the search branch, the input image size is 255×255×3. With the center of the original image target constraint frame as the origin, an area approximately four times the size of the target is cropped, and the part exceeding the original image is filled with the color mean, and then the cropped area is scaled to 255×255 as the search image. The initial preprocessing method for visible light images and infrared thermal images is the same. At this point, the input image pairs of different modalities are obtained, denoted as z r , x r , z t , x t , representing visible light template image, visible light search image, infrared template image, and infrared search image respectively.

[0019] Furthermore, in step 2 and step 5, the data processing in the space-frequency domain equalization module includes two modules, the M(R) gradient convolution module and the W(T) wavelet transform module. The specific steps of the M(R) gradient convolution module are as follows: This module is used to extract the high-frequency information of the visible light image. The specific process is as follows: the image to be processed is the visible light template z r and visible light search x r , let the visible light image be R. The gradient map of image R is obtained by calculating the difference between adjacent pixels:

[0020]

[0021] Here, M(R) represents the operation of extracting the gradient map, whose elements are the gradient lengths of the pixel at coordinate X = (x, y). In general, we implement the above process using gradient convolution and abbreviate it as M(R), also known as the gradient convolution module.

[0022] The specific steps of the W(T) wavelet transform module are as follows: This module is used to extract the low-frequency information of the infrared image. The specific process is as follows: The image to be processed is the infrared template z t and infrared search x t , let the infrared image be T. First, decompose T into three single-channel images r, g, and b, and then use Harr wavelet decomposition and transformation to remove high-frequency information, reconstruct, and recombine them into a three-channel image of infrared low frequency.

[0023]

[0024] Among them, Spilt represents the decomposition function of decomposing T into three single-channel images, wavedec2 represents the Harr wavelet transform function, T b , T g , T r is the image generated by T through Spilt and wavedec2; rhf represents the high-frequency information removal function, waverec2 represents the wavelet reconstruction function, T′ b , T′ g , T′ r It's T b , T g , T r The images generated by rhf and waverec2 are combined into a single channel T′ by the Merge function. b , T′ g , T′ r Image, get the infrared low-frequency image T'. In general, we use wavelet transform to implement the above process and abbreviate it as W(T), also known as wavelet transform module.

[0025] Furthermore, the specific steps of the feature extraction network in step 3 or step 6 are as follows: To balance tracker accuracy and speed, the first four layers of ResNet50 are used as the backbone network to extract features, and the outputs of layers 3 and 4 are used in subsequent network calculations. In the fourth layer, the double stride of the convolution is eliminated, and a dilated convolution is used instead to increase the receptive field. The input to the feature extraction network of this algorithm includes not only data from different modal domains, but also data from different frequency domains. In the modal domain, different modalities have unique characteristics, and in the frequency domain, different frequency domain data also have unique characteristics. Therefore, the backbone networks for visible light, infrared, high-frequency visible light, and low-frequency infrared do not share weights. Four feature network extraction modules are used to represent the feature extraction networks for four different types of data. Since unified feature extraction is required for the template and search area images in the time domain, the weights of the four types of data corresponding to the template and search area are shared. To balance tracker speed and parameters, this algorithm sets the first two layers of the backbone network to be shared across all domains.

[0026] The network feature extraction process is recorded as Then we have:

[0027]

[0028] where f zr ,f zt ,f xr ,f xt Represents the visible light template, infrared template, visible light search area, and infrared search area features output by the feature extraction network.

[0029]

[0030] where f zrh ,f ztl ,f xrh ,f xtl Represents the visible light high-frequency template, infrared low-frequency template, visible light high-frequency search area, and infrared low-frequency search area features output by the feature extraction network. The shallow network features of the image are closer to the input, with more detailed information such as color, texture, edge, and corners, but with lower semantics and more noise; the deep network features of the image are closer to the output, with stronger semantic information, but poorer detail perception. Therefore, the present invention simultaneously utilizes the outputs of the 3rd and 4th layers of the feature extraction network, which respectively participate in subsequent feature fusion calculations to enhance the network's expressive power by interacting with more information at different levels. The abbreviated notation used here is as shown in Equations 3.3 and 3.4.

[0031] Furthermore, the specific steps for spatial position enhancement in step 4 or step 7 are as follows: The SA spatial attention module is introduced during the spatial position enhancement stage to initially enhance the foreground and weaken the background to achieve self-enhancement of features. Then, features from different domains are adaptively fused through intelligent addition to further enhance and balance the target features to achieve mutual enhancement of features.

[0032] SA: For the input feature f, f∈C×H×W, average pooling and maximum pooling are performed on the channel dimension of f, and then the spatial position attention weight map W is generated after feature splicing, convolution integration, and activation function. f , f∈1×H×W. The specific formula is as follows:

[0033]

[0034] Here, Avgpool represents the average pooling function, Maxpool represents the maximum pooling function, concat represents the dimension concatenation function, conv represents the integrated convolution, and Sigmoid represents the activation function. The process in Equation 3.5 is abbreviated as SAM.

[0035] Then use the weight map Enhance the original feature f to obtain the re-adjusted feature F. The specific formula is as follows:

[0036]

[0037] Where γ is the network adaptive learning parameter set, which is obtained by network learning. The process of Equation 3.5 and Equation 3.6 is abbreviated as SA.

[0038] So far, for the template branch, the input feature is f zr , f zrh , f zt , f ztl, the corresponding output feature after SA modulation is recorded as F zr , F zrh , F zt , F ztl ; For the search branch, the input feature is f xr , f xrh , f xt , f xtl , the corresponding output feature after SA modulation is recorded as F xr , F xrh , F xt , F xtl Smart addition adaptive fusion: Adaptively add the visible light high-frequency branch to the infrared branch, and adaptively add the infrared low-frequency branch to the visible light branch. The specific formula is as follows:

[0039]

[0040] Among them, α1, α2, α3, and α4 are weight ratio parameters obtained by network adaptive learning. zr , F′ zt , F′ xr , F′ xt Respectively represent the visible light template, infrared template, visible light search, and infrared search features after feature equalization. They are the output features after adaptive fusion of features from different domains, and are also the final output features of the template branch and search branch after the space-frequency domain equalization module.

[0041] Furthermore, the information integration in step 8 includes two stages: lightweight feature channel and preliminary fusion of two-level information. The specific steps are as follows:

[0042] Lightweight feature channels:

[0043] The present invention utilizes the interactive information of the features of the 3rd and 4th layers of the backbone network. They have different numbers of output channels, so all features are reduced to 256 channels through a 1x1 convolution layer. This approach has two advantages. On the one hand, it reduces the number of parameters and makes the network lightweight. On the other hand, the 1×1 convolution plays a role in integrating features. For the template branch, a cropping convolution is performed after the 1x1 convolution, and the feature remains in the center of the 7×7 area. It not only retains the entire target information, but also reduces the impact of the background. For the search branch, no cropping operation is performed on it. The specific formula is as follows:

[0044]

[0045] Among them, crop represents crop convolution, conv r and conv t Represents the 1x1 convolution of the visible light branch and the infrared branch respectively.zr , F″ zt , F″ xr , F″ xt They represent the visible light template, infrared template, visible light search, and infrared search features after the feature channels are lightweight.

[0046] Initial fusion of two-level information:

[0047] The feature-level information and decision-level information are initially and simply fused as the input of the subsequent two-level fusion lightweight attention module. The present invention continues the design method of introducing decision-level information in SiamDL. For the template branch, different modal features are directly spliced as subsequent input; for the search branch, the decision-level information of visible light and infrared is used to generate a mask, the search branch features are strengthened, the weights are redistributed, and then spliced as subsequent income. The classification results of visible light and infrared are generated by the template area and search area corresponding to their respective modalities through the convolution block operation with padding, which is called a mask and is denoted by R mask , T mask , representing the visible light classification mask and the infrared classification result mask, respectively. The specific process of generating masks and stitching is similar to SiamDL. However, the difference is that the features processed in the two-level information preliminary fusion of this algorithm are features that have undergone spatial-frequency domain collaborative enhancement. The specific formula for the two-level information preliminary fusion process is as follows:

[0048]

[0049]

[0050] Among them, concat represents the channel dimension concatenation function. xr , C xt They represent the visible light search features and infrared search features after preliminary enhancement using decision-level information. x , F z Represent the search features and template features after splicing different modalities. x Representative C xr , C xt At this point, we get the input of the two-stage fusion lightweight attention module, F z , F z , F x , C x .

[0051] Furthermore, the two-stage fusion lightweight attention module in step 8 is composed of a channel attention mechanism and a spatial attention mechanism. The specific steps are as follows:

[0052] Channel Attention Mechanism:

[0053] After global average pooling of features, local cross-channel feature interactions are considered between each channel and its K neighboring channels. This can be effectively implemented using a fast 1D convolution of size K. The size of kernel K represents the coverage of local cross-channel interactions, i.e., the number of "neighbors" participating in attention prediction for a channel. To accommodate the framework of this algorithm, the template convolution kernel K in the template branch is designed to have a fixed size of 3×3; in the search branch, the search area convolution kernel K is designed to have a fixed size of 7×7.

[0054] For the input feature f, f∈C×H×W. The specific formula is as follows:

[0055]

[0056] Among them, Avgpool represents the global average pooling function, unfold represents the sliding local area expansion function, the sliding window size is k, conv represents the convolution kernel, and the convolution kernel size is also k. Sigmoid represents the activation function, It is the channel attention weight map generated by formula 3.11. The process of formula 3.11 is abbreviated as CAM.

[0057] Spatial Attention Mechanism:

[0058] The spatial attention mechanism in the two-stage fusion lightweight attention mechanism is the same as the SAM process in the SA module and will not be repeated here.

[0059] Therefore, the formula of the two-stage fusion lightweight attention module is as follows:

[0060] For template branches:

[0061] C′ z =F z +α·F z CAM(F z )+β·F z ·SAM(F z ) (12)

[0062] For searching branches:

[0063] C′ x =F x +α·F x CAM(C x )+β·F x ·SAM(C x ) (13)

[0064] Among them, α and β are network adaptive learning parameters. z , C′ x Represents the final output template and search features.

[0065] Furthermore, the specific steps of the tracking head in step 8 are as follows: The tracking head includes classification and regression, which is implemented by referring to the SiamBAN method. The tasks of the classification head and regression head are to find the position features that are most similar to the target template features in the search feature area and predict the target bounding box. Therefore, for the template feature C′ z and C′ x Perform deep cross-correlation to aggregate template features and search features. Then, use classification convolution to classify each point in the relevant layer into foreground and background, outputting two channels. Use regression convolution to predict the offset position from the four sides of the target box, outputting four channels. The specific formula is as follows:

[0066]

[0067] Among them, P cls Representative classification results, P reg Represents the regression result graph, ★ represents the convolution operation with the template feature as the convolution kernel, Conv cls Represents the classification convolution layer, Conv reg Represents the regression convolution layer.

[0068] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0069] (1) A space-frequency domain equalization module is designed. It combines multiple modal domains including modal domain, frequency domain, and spatial domain to perform feature-level information interaction to balance the features of different modal data, focusing on the complementary characteristics of different modal data. At the same time, it uses feature-level and decision-level information to more reasonably balance the fusion features and improve the feature expression ability of the network.

[0070] (2) Based on SiamBAN and SiamDL, both feature-level and decision-level information are fused as input features for a two-level fusion lightweight attention module. A two-level fusion lightweight attention module is designed to improve the original decision-level fusion method, reducing feature loss and making it more lightweight, thereby enhancing the robustness of the tracker.

[0071] (3) We used different training methods, including data augmentation, two-stage training, and end-to-end training. We conducted multiple tests on GTOT and LasHeR, achieving state-of-the-art results. Furthermore, our tracker is more advanced in some complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is a schematic diagram of the overall structure of the algorithm framework of the present invention;

[0073] Figure 2 is a schematic diagram of a space-frequency domain equalization module of the present invention;

[0074] Figure 3 is a schematic diagram of the SA spatial position enhancement module of the present invention;

[0075] Figure 4 is a schematic diagram of a two-stage fusion attention module of the present invention;

[0076] Figure 5 is a schematic diagram comparing the algorithm of an embodiment of the present invention with an existing advanced RGB algorithm on the GTOT dataset;

[0077] Figure 6 1 is a schematic diagram comparing the algorithm of an embodiment of the present invention with the existing advanced RGB-T algorithm on the GTOT dataset;

[0078] Figure 7 1. It is a schematic diagram comparing various attribute indicators of the algorithm of the embodiment of the present invention and the current advanced algorithm on the GTOT test set;

[0079] Figure 8 3 is a schematic diagram comparing the algorithm and the baseline on the LASHER dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0081] 1. The main framework is as follows Figure 1 As shown in the figure, a two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization is proposed. The main framework of the SiamSFD network is divided into a template branch and a search branch. The whole consists of input, feature-level fusion stage, two-stage fusion stage and tracking head. The input is a visible light template image, an infrared template image, a visible light search image and an infrared search image. The feature-level fusion stage consists of a space-frequency domain equalization module and the first half of information integration, which specifically includes data processing, feature network extraction, spatial position enhancement and feature channel lightweighting. The two-stage fusion stage consists of the second half of the information integration module and a two-stage fusion lightweight attention module. The tracking head consists of classification and regression. After training is completed, the tracking process is as follows:

[0082] (1) The initial frame target tight constraint frame is obtained by preliminarily preprocessing the data in the space-frequency domain equalization module to obtain the input template z of different modes r , z t Then, by extracting the frequency domain information, we can obtain the frequency domain information input template z of different modes. rh , z tl . Send it to the feature extraction network to get the template feature fzr , f zt , f zrh , f ztl .

[0083] (2) For subsequent frames, the visible light and infrared search area images x are extracted by pre-processing the data in the spatial frequency domain equalization module around the target. r , x t , and x is obtained by extracting frequency domain information rh , x tl . Send it to the feature extraction network to obtain the search area feature f xr , f xt , f xrh , f xtl .

[0084] (3)f zr , f zt , f zrh , f ztl and f xr , f xt , f xrh , f xtl The template feature F′ is obtained by spatial position enhancement in the space-frequency domain equalization module. zr , F′ zt and search feature F′ xr , F′ xt .

[0085] (4) F′ zr , F′ zt , F′ xr , F′ xt Send it to the information integration module to obtain the adjusted different modal splicing template features F z and search feature F x And the different modal splicing search features C that are initially integrated into the decision information x .

[0086] (5) F z , F z and F x , C x They are respectively sent to the two-stage fusion lightweight attention module to balance the weights of the combined bimodal features.

[0087] (6) The weighted features are sent to the classification and regression heads, and the coordinate point u with the highest classification prediction value is recorded. The output frame at the regression coordinate point u is used as the tracker's prediction frame for the target, and the coordinates of the tracking target frame are modified step by step.

[0088] (7) Repeat steps (2), (3), (4), (5), and (6) to complete the tracking.

[0089] 2. Spatial-Frequency Domain Equalization Module

[0090] like Figure 2 As shown in the figure, the space-frequency domain equalization module fully considers the unique characteristics of different modal data, combines multiple domain information including modal domain, frequency domain and spatial domain, and performs feature-level information fusion through spatial position enhancement. Figure 2 As shown in Figure 1, the spatial-frequency domain equalization module consists of three parts: data processing, feature extraction network, and spatial position enhancement. Both the template branch and the search branch are processed by the spatial-frequency domain equalization module. Therefore, i = 127, a = 15 represents the template branch, and i = 255, a = 31 represents the search branch. Data processing includes extracting high-frequency information (M(R)) and low-frequency information (W(T)).

[0091] 3. M(R) Gradient Convolution Module

[0092] M(R) is used to extract high-frequency information. The specific process is as follows: the image to be processed is the visible light template z r and visible light search x r , let the visible light image be R. The gradient map of image R is obtained by calculating the difference between adjacent pixels:

[0093]

[0094] Here, M(R) represents the operation of extracting a gradient map, whose elements are the gradient lengths of the pixel at coordinate X = (x, y). The operation of obtaining the gradient can be easily implemented using a convolutional layer with a fixed kernel. In fact, we do not consider gradient direction information because the gradient strength is sufficient to reveal the clarity of local regions in the restored image. Therefore, we use the intensity map as the gradient map. Because most areas of the gradient map are close to zero, the convolutional neural network can focus more on the spatial relationships of the contours. As a result, the network may be more likely to capture structural dependencies, thereby providing structural priors for subsequent networks. In general, we implement the above process using gradient convolution and abbreviate it as M(R), also known as the gradient convolution module.

[0095] 4. W(T) Wavelet Transform Module

[0096] W(T) is used to extract low-frequency information. The specific process is as follows: The image to be processed is the infrared template z t and infrared search x t The infrared image is denoted as T. The tracking infrared dataset used in this chapter is an extended three-channel image. In the process of extracting low-frequency information, T is first decomposed into three single-channel images r, g, and b (the r, g, and b here are pseudo r, g, and b, different from the three channels of the color image). Then, the Harr wavelet decomposition and transform are used to remove high-frequency information, reconstruct, and reassemble into a three-channel infrared low-frequency image.

[0097]

[0098] Among them, Spilt represents the decomposition function of decomposing T into three single-channel images, wavedec2 represents the Harr wavelet transform function, T b , T g , T r is the image generated by T through Spilt and wavedec2; rhf represents the high-frequency information removal function, waverec2 represents the wavelet reconstruction function, T′ b , T′ g , T′ r It's T b , T g , T r The images generated by rhf and waverec2 are combined into a single channel T′ by the Merge function. b , T′ g , T′ r Image, get the infrared low-frequency image T'. In general, we use wavelet transform to implement the above process and abbreviate it as W(T), also known as wavelet transform module.

[0099] 5. Network Feature Extraction

[0100] The network feature extraction process is recorded as Then we have:

[0101]

[0102] where f zr , f zt , f xr , f xt Represents the visible light template, infrared template, visible light search area, and infrared search area features output by the feature extraction network.

[0103]

[0104] where f zrh , f ztl , f xrh , f xtl Represents the visible light high-frequency template, infrared low-frequency template, visible light high-frequency search area, and infrared low-frequency search area features output by the feature extraction network.

[0105] 6. Spatial Position Enhancement: SA Module and Adaptive Fusion

[0106] Introducing SA in the spatial position enhancement stage (such as Figure 3The spatial attention module (shown in Figure 1) initially enhances the foreground and weakens the background to achieve self-enhancement of features. It then adaptively fuses features from different domains through intelligent addition, further enhancing and balancing the target features to achieve mutual enhancement of features.

[0107] SA: For input feature f, f∈C×H×W, average pooling and maximum pooling are performed on the channel dimension of f, and then the spatial position attention weight map W is generated after feature splicing, convolution integration, and activation function. f , f∈1×H×W. The specific formula is as follows:

[0108]

[0109] Here, Avgpool represents the average pooling function, Maxpool represents the maximum pooling function, concat represents the dimension concatenation function, conv represents the integrated convolution, and Sigmoid represents the activation function. The process in Equation 3.5 is abbreviated as SAM.

[0110] Then use the weight map Enhance the original feature f to obtain the re-adjusted feature F. The specific formula is as follows:

[0111]

[0112] Where γ is the network adaptive learning parameter set, which is obtained by network learning. The process of Equation 3.5 and Equation 3.6 is abbreviated as SA.

[0113] So far, for the template branch, the input feature is f zr , f zrh , f zt , f ztl , the corresponding output feature after SA modulation is recorded as F zr , F zrh , F zt , F ztl ; For the search branch, the input feature is f xr , f xrh , f xt , f xtl , the corresponding output feature after SA modulation is recorded as F xr , F xrh , F xt , F xtl .

[0114] Smart addition adaptive fusion: Adaptively add the visible light high-frequency branch to the infrared branch, and adaptively add the infrared low-frequency branch to the visible light branch. The specific formula is as follows:

[0115]

[0116] Among them, α1, α2, α3, and α4 are weight ratio parameters obtained by network adaptive learning. zr , F′ zt , F′ xr , F′ xt Respectively represent the visible light template, infrared template, visible light search, and infrared search features after feature equalization. They are the output features after adaptive fusion of features from different domains, and are also the final output features of the template branch and search branch after the space-frequency domain equalization module.

[0117] 7. Information Integration

[0118] Lightweight feature channels:

[0119] The present invention utilizes the interactive information of the features of the 3rd and 4th layers of the backbone network. They have different numbers of output channels, so all features are reduced to 256 channels through a 1x1 convolution layer. This approach has two advantages. On the one hand, it reduces the number of parameters and makes the network lightweight. On the other hand, the 1×1 convolution plays a role in integrating features. For the template branch, a cropping convolution is performed after the 1x1 convolution, and the feature remains in the center of the 7×7 area. It not only retains the entire target information, but also reduces the impact of the background. For the search branch, no cropping operation is performed on it. The specific formula is as follows:

[0120]

[0121] Among them, crop represents crop convolution, conv r and conv t Represents the 1x1 convolution of the visible light branch and the infrared branch respectively. zr , F″ zt , F″ xr , F″ xt They represent the visible light template, infrared template, visible light search, and infrared search features after the feature channels are lightweight.

[0122] Initial fusion of two-level information:

[0123] The feature-level information and decision-level information are initially and simply fused as the input of the subsequent two-level fusion lightweight attention module. The present invention continues the design method of introducing decision-level information in SiamDL. For the template branch, different modal features are directly spliced as subsequent input; for the search branch, the decision-level information of visible light and infrared is used to generate a mask, the search branch features are strengthened, the weights are redistributed, and then spliced as subsequent income. The classification results of visible light and infrared are generated by the template area and search area corresponding to their respective modalities through the convolution block operation with padding, which is called a mask and is denoted by R mask , T mask, representing the visible light classification mask and the infrared classification result mask, respectively. The specific process of generating masks and stitching is similar to SiamDL. However, the difference is that the features processed in the two-level information preliminary fusion of this algorithm are features that have undergone spatial-frequency domain collaborative enhancement. The specific formula for the two-level information preliminary fusion process is as follows:

[0124]

[0125]

[0126] Among them, concat represents the channel dimension concatenation function. xr , C xt They represent the visible light search features and infrared search features after preliminary enhancement using decision-level information. x , F z Represent the search features and template features after splicing different modalities. x Representative C xr , C xt At this point, we get the input of the two-stage fusion lightweight attention module, F z , F z , F x , C x .

[0127] 8. Two-level fusion lightweight attention module

[0128] like Figure 4 As shown:

[0129] Channel Attention Mechanism:

[0130] After global average pooling of features, local cross-channel feature interactions are considered between each channel and its K neighboring channels. This can be effectively implemented using a fast 1D convolution of size K. The size of kernel K represents the coverage of local cross-channel interactions, i.e., the number of "neighbors" participating in attention prediction for a channel. To accommodate the framework of this algorithm, the template convolution kernel K in the template branch is designed to have a fixed size of 3×3; in the search branch, the search area convolution kernel K is designed to have a fixed size of 7×7.

[0131] For the input feature f, f∈C×H×W. The specific formula is as follows:

[0132]

[0133] Among them, Avgpool represents the global average pooling function, unfold represents the sliding local area expansion function, the sliding window size is k, conv represents the convolution kernel, and the convolution kernel size is also k. Sigmoid represents the activation function, W fcis the channel attention weight map generated by the above formula. The process of formula 11 is abbreviated as CAM.

[0134] Spatial Attention Mechanism:

[0135] The spatial attention mechanism in the two-stage fusion lightweight attention mechanism is the same as the SAM process in the SA module and will not be repeated here.

[0136] Therefore, the formula of the two-stage fusion lightweight attention module is as follows:

[0137] For template branches:

[0138] C′ z =F z +α·F z CAM(F z )+β·F z ·SAM(F z ) (12)

[0139] For searching branches:

[0140] C′ x =F x +α·F x ·CAM(C x )+β·F x ·SAM(C x ) (13)

[0141] Among them, α and β are network adaptive learning parameters. z , C′ x Represents the final output template and search features.

[0142] 9. Tracking Head

[0143] The tracking head includes classification and regression, which is implemented by referring to SiamBAN. The tasks of the classification head and regression head are to find the location features that are most similar to the target template features in the search feature area and predict the target bounding box. Therefore, for the template feature C′ z and C′ x Perform deep cross-correlation to aggregate template features and search features. Then, use classification convolution to classify each point in the relevant layer into foreground and background, outputting two channels. Use regression convolution to predict the offset position from the four sides of the target box, outputting four channels. The specific formula is as follows:

[0144]

[0145] Among them, P cls Representative classification results, P reg Represents the regression result graph. ★ represents the convolution operation with the template feature as the convolution kernel, Convcls Represents the classification convolution layer, Conv reg Represents the regression convolution layer.

[0146] Loss Function

[0147] Labels and Loss Function: Similar to SiamBAN, this algorithm uses ellipses within the ground-truth object box (GT) to label positive and negative samples. Positive samples are used for regression prediction, where the predicted value is the distance between the four edges of the predicted box and the ground-truth object box (GT). This value is then mapped back to the object box in the original image during post-processing. The training loss for the entire network is a weighted combination of the visible light modality classification loss, the infrared modality classification loss, and the classification and regression losses of the bimodal fusion features:

[0148]

[0149] In this algorithm, we empirically set λ1 = 0.2, λ2 = 0.2, λ3 = 1, and λ4 = 1. Cross entropy loss is used for classification, and Giou loss is used for regression.

[0150] Training: The template image size was 127×127, and the search region image size was 255×255. Different training strategies were tested on different test sets to verify the effectiveness of the method. GTOT: A two-stage training approach was employed, dividing the framework into two separate training phases: feature-level fusion and decision-level fusion. In the first phase, a bimodal backbone network was initialized with the pre-trained parameters of ResNet50. All four layers were initially frozen, and then the parameters of the last two layers were fine-tuned. This network used the Adam optimization algorithm with a weight decay of 0.000l. The learning rate was warmed up from 0.001 to 0.003, and then exponentially decayed from 0.003. In the second phase, the model trained in the first phase was used as the pre-trained model for fine-tuning. The learning rate decayed starting from 0.002. The dataset is primarily RGBT234, supplemented by the Lasher dataset; no other datasets are used. LASHER: An end-to-end training method is used, using a bimodal backbone network initialized with ImageNet pre-trained parameters. All four layers are initially frozen, and then the parameters of the two layers are fine-tuned after thawing. This network uses the Adam optimization algorithm with a weight decay of 0.0001. The learning rate decays from 0.002 to 0.00001. The datasets include GOTlOK, LASOT, RGBT234, and Lasher. In addition, data augmentation strategies are used, adding exposure, low light, blur, and other strategies to influence the image quality of the two modalities to alternately deteriorate, which helps to enhance tracker performance. The specific data augmentation methods can be found in this project.

[0151] Inference: Update the target bounding box based on the classification score map and the regression prediction box. Specifically, the regression prediction box is used to penalize target shape changes, and a cosine window is used to penalize center distance. The classification score map is updated by generating two penalty masks, and the regression prediction box at the position with the highest score is interpolated to update the current target bounding box.

[0152] This algorithm was built on the PyTorch platform with the following hardware parameters: CPU: Intel i7 10400F, GPU: Nvidia RTX 3090.

[0153] Datasets and evaluation metrics

[0154] GOT10K contains over 10,000 visible light sequences encompassing 560 categories of moving objects, providing comprehensive and relatively unbiased coverage of various moving objects. Each sequence averages 150 frames, each with precise human annotations. Compared to similar tracking datasets, this dataset offers a richer variety and is well-suited for training tracking tasks.

[0155] LASOT contains 1,400 visible light sequences, encompassing 70 types of moving objects. Each sequence contains an average of 2,500 frames, but the time difference between frames is smaller than that of GOT10K, and each frame is accurately manually annotated.

[0156] GTOT contains 50 visible-infrared paired sequences, with an average of 150 frames per sequence for each modality. It has seven challenging attributes. However, the dataset has a limited number of categories, low resolution, and poor quality.

[0157] RGBT234 contains 234 visible-infrared paired sequences, with each sequence containing 500 frames on average for a single modality. It has 12 challenging attributes and a relatively small number of categories.

[0158] LasHeR contains 1,224 bimodal paired sequences, with an average of 600 frames per column for each unimodal pair. It has 19 challenging attributes and 32 dataset categories, making it the first large-scale dataset for the bimodal tracking challenge.

[0159] When testing GTOT, we used precision and success rate as evaluation metrics. PR is the percentage of frames where the distance between the inferred and ground-truth bounding boxes is within a certain distance threshold. SR is the percentage of frames where the inferred and ground-truth bounding boxes are within a certain overlap threshold. The area under the curve is calculated as the SR score. When calculating PR, for the GTOT and RGBT234 datasets, the distance thresholds used by most algorithms are 5 and 20 pixels, respectively.

[0160] When testing LasHeR, PR, NPR, and SR are used as evaluation indicators. PR and SR are the same as above, and the distance threshold for PR is set to 20.

[0161] Comparison with state-of-the-art trackers

[0162] GTOT: Figure 5 and Figure 6 Quantitative results are shown on the GTOT dataset, a short-range tracking challenge with an average frame count of 150. The tracker achieves a PR of 88.8 and a SR of 73.3. Figure 5 The results of testing our algorithm against state-of-the-art RGB algorithms on the GTOT dataset show that our algorithm surpasses MDNet by 7.6% and 10%, respectively, and surpasses DSST, ECO, CCOT, ACFN, SiamDW, and many more. This demonstrates the effectiveness of our method's introduction of the infrared modality and its clever design. Figure 6 The results of testing our algorithm against a state-of-the-art RGB-T algorithm on the GTOT dataset show that our algorithm achieves a PR close to MANet and surpasses MANet by 0.9% in SR. It also surpasses the baseline SiamBan+RGB-T algorithm by 8.5% and 5.8%, respectively. This demonstrates the effectiveness and advancement of our approach.

[0163] GTOT: Figure 7 The comparison results of trackers for various attributes on the GTOT test set are shown in more detail. The comparison results include Siamban+RGBT, SiamRPN, ATOM, DIMP, SiamFT, SGT, mfDIMP, MANet, Siamban and our algorithm. Among them, SiamBAN (RGBT) is also the tracker implemented in this article. After obtaining visible light and infrared features through the ResNet50 network, the two features are directly combined by channel, and then the connected features are sent to the tracking classification regression head. It can be seen that our algorithm maintains the top three in each attribute of OCC, LSV, FM, LI, TC, SO, and DEF. Compared with the baseline, it surpasses 10% and 6.8% in OCC, 10% and 6.6% in LSV, 6% and 5.3% in LI, etc.

[0164] Figure 8 The comparison of the LasHeR test index and the baseline of this algorithm is shown in Figure 2. Compared with the baseline SiamBAN (RGBT), the tracker of this algorithm exceeds its PR by 3.3% and SR by 3.9%, and exceeds its NPR by 3.6%.

Claims

1. A two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization, characterized by: The steps include: Step 1: The main network framework is divided into a template branch and a search branch. The whole network consists of input, feature-level fusion stage, two-level fusion stage and tracking head. The template branch input is a visible light template image and an infrared template image. Step 2: The image obtained in step 1 is sent to the spatial-frequency domain equalization module, and after data processing, a visible light high-frequency template image and an infrared low-frequency template image are obtained; Step 3: The visible light template image, infrared template image, visible light high-frequency template image, and infrared low-frequency template image are respectively passed through the feature extraction network in the space-frequency domain equalization module to obtain visible light template features, infrared template features, visible light high-frequency template features, and infrared low-frequency template features; Step 4: The features obtained in step 3 are subjected to spatial position enhancement in the space-frequency domain equalization module to obtain the final output features of the space-frequency domain equalization module corresponding to the template branch, namely the modulated visible light template features and infrared template features; Step 5: The search branch inputs the visible light search image and the infrared search image, which are sent to the space-frequency domain equalization module for data processing to obtain a visible light high-frequency search image and an infrared low-frequency search image; Step 6: The visible light search image, infrared search image, visible light high-frequency search image, and infrared low-frequency search image are respectively passed through the feature extraction network in the space-frequency domain equalization module to obtain visible light search features, infrared search features, visible light high-frequency search features, and infrared low-frequency search features; Step 7: The features obtained in step 6 are subjected to spatial position enhancement in the space-frequency domain equalization module to obtain the final output features of the space-frequency domain equalization module corresponding to the search branch, namely the modulated visible light search features and infrared search features; Step 8: The visible light template features, infrared template features, visible light search features, and infrared search features obtained in steps 4 and 7 are fed into the information integration. The integrated features are then used as the input of the two-stage fusion lightweight attention module. Finally, the tracking results are obtained through classification and regression of the tracking head.

2. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 1 is characterized by: The feature-level fusion stage consists of a space-frequency domain equalization module and the first half of information integration, which specifically includes data processing, feature network extraction, spatial position enhancement and feature channel lightweighting. The two-level fusion stage consists of the second half of the information integration module and the two-level fusion lightweight attention module. The tracking head consists of classification and regression.

3. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 2 is characterized by: The input visible light search image, infrared search image, visible light high-frequency search image and infrared low-frequency search image are derived from paired dual-modal video frame images and are preprocessed in the same way as SiamFC: the input includes a template branch and a search branch. For the template branch, the input image size is 127×127×3. The center of the original image target constraint frame is used as the origin, and an area twice the size of the target is cropped. The part exceeding the original image is filled with the color mean. Then, the cropped area is scaled to 127×127 as the template image. For the search branch, the input image size is 255×255×3. The center of the original image target constraint frame is used as the origin, and an area four times the size of the target is cropped. The part exceeding the original image is filled with the color mean. Then, the cropped area is scaled to 255×255 as the search image. The data of the visible light image and the infrared thermal image are preprocessed in the same way. At this point, the input image pairs of different modalities are obtained, denoted as z r 、x r 、z t 、x t , representing visible light template image, visible light search image, infrared template image, and infrared search image respectively.

4. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 3 is characterized by: In step 2 and step 5, the data processing in the space-frequency domain equalization module includes two modules, the M(R) gradient convolution module and the W(T) wavelet transform module. The specific steps of the M(R) gradient convolution module are as follows: This module is used to extract the high-frequency information of the visible light image. The specific process is as follows: the image to be processed is the visible light template z r and visible light search x r , let the visible light image be R, and the gradient map of image R is obtained by calculating the difference between adjacent pixels: Where M(R) represents the operation of extracting the gradient map, and its coordinate X = (x, y) is the gradient length of the pixel. The above process is implemented using gradient convolution and is abbreviated as gradient convolution module M(R). Represents the gradient operation on R(X); The specific steps of the W(T) wavelet transform module are as follows: This module is used to extract the low-frequency information of the infrared image. The specific process is as follows: The image to be processed is the infrared template z t and infrared search x t , let the infrared image be T, and the tracking infrared data set used is an extended three-channel image. In the process of extracting low-frequency information, T is first decomposed into three single-channel images of r, g, and b. Then, Harr wavelet decomposition and transformation are used to remove high-frequency information, reconstruct, and reassemble into an infrared low-frequency three-channel image. The formula is as follows: Among them, Spilt represents the decomposition function of decomposing T into three single-channel images, wavedec2 represents the Harr wavelet transform function, T b 、T g 、T r It is the image generated by Spilt and wavedec2, rhf represents the high frequency information removal function, waverec2 represents the wavelet reconstruction function, T′ b , T′ g , T′ r It's T b 、T g 、T r The images generated by rhf and waverec2 are merged into a single channel T′ by the Merge function. b , T′ g , T′ r Image, get the infrared low-frequency image T′.

5. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 4 is characterized by: The specific steps of the feature extraction network in step 3 or step 6 are: In order to balance the accuracy and speed of the tracker, the first four layers of ResNet50 are used as the backbone network to extract features. The outputs of the third and fourth layers participate in the calculation of the subsequent network. The first two layers of the backbone network are set to be shared in all domains. The network feature extraction process is recorded as Then we have: where f zr 、f zt 、f xr 、f xt Represents the visible light template, infrared template, visible light search area and infrared search area features output by the feature extraction network, then where f zrh 、f ztl 、f xrh 、f xtl Represents the visible light high-frequency template, infrared low-frequency template, visible light high-frequency search area, and infrared low-frequency search area features output by the feature extraction network, z rh 、z tl 、x rh 、x tl They respectively represent visible light high-frequency information, infrared low-frequency information, visible light high-frequency search area information, and infrared low-frequency search area information.

6. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 5 is characterized by: The specific steps of spatial position enhancement in step 4 or step 7 are: The SA spatial attention module is introduced in the spatial position enhancement stage to initially enhance the foreground and weaken the background to achieve feature self-enhancement. Then, features from different domains are adaptively fused through intelligent addition to further enhance and balance the target features to achieve mutual feature enhancement. The SA spatial attention module is as follows: for the input feature f, f∈C×H×W, average pooling and maximum pooling are performed on the f channel dimension, and then the spatial position attention weight map W is generated after feature splicing, convolution integration, and activation function. f , f∈1×H×W, the specific formula is as follows: Among them, Avgpool represents the average pooling function, Maxpool represents the maximum pooling function, concat represents the dimension splicing function, conv represents the integrated convolution, Sigmoid represents the activation function, and the process of formula (5) is recorded as SAM. Post-use weight map The original feature f is enhanced to obtain the re-adjusted feature F. The specific formula is as follows: Among them, γ is the network adaptive learning parameter set, which is obtained by network learning. The process of formula (5) and formula (6) is recorded as SA. The input feature is f zr 、f zrh 、f zt 、f ztl , the corresponding output feature after SA modulation is recorded as F zr 、F zrh 、F zt 、F ztl ; For the search branch, the input feature is f xr 、f xrh 、f xt 、f xtl , the corresponding output feature after SA modulation is recorded as F xr 、F xrh 、F xt 、F xtl , Intelligent addition adaptive fusion: The visible light high-frequency branch is adaptively added to the infrared branch, and the infrared low-frequency branch is adaptively added to the visible light branch. The formula is as follows: Among them, α1, α2, α3, and α4 are weight ratio parameters obtained by network adaptive learning. zr , F′ zt , F′ xr , F′ xt They represent the visible light template, infrared template, visible light search, and infrared search features after feature equalization, respectively. They are the output features after adaptive fusion of features from different domains, as well as the final output features of the template branch and search branch after the space-frequency domain equalization module.

7. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 6 is characterized by: The information integration in step 8 includes two stages: lightweight feature channel and preliminary fusion of two-level information. The specific steps are as follows: Lightweight feature channel reduction: All feature channels are reduced to 256 channels through a 1x1 convolution layer. The specific formula is as follows: Among them, crop represents crop convolution, conv r and conv t Represents the 1x1 convolution of the visible light branch and the infrared branch, respectively, F″ zt 、F″ zt 、F″ xr 、F″ xt Respectively represent the visible light template, infrared template, visible light search and infrared search features after the feature channel is lightweight, Initial fusion of two-level information: Initially and simply fuse feature-level information and decision-level information as the input of the subsequent two-level fusion lightweight attention module. For the template branch, directly splice the features of different modalities as the subsequent input; for the search branch, use the decision-level information of visible light and infrared to generate a mask, strengthen the search branch features, redistribute the weights, and then splice them as the subsequent income. The classification results of visible light and infrared are generated by the template area and search area corresponding to their respective modalities through the convolution block operation with padding, which is called a mask, denoted by R mask 、T mask , respectively represent the visible light classification mask and the infrared classification result mask. The specific process of generating masks and splicing is similar to SiamDL. The specific formula of the two-level information preliminary fusion process is as follows: Among them, concat represents the channel dimension splicing function, C xr 、C xt They represent the visible light search features and infrared search features after preliminary enhancement using decision-level information, respectively. x 、F z Represent the search features and template features after splicing different modalities, C x Representative C xr 、C xt Splicing features, and then get the input of the two-level fusion lightweight attention module, F z 、F z 、F x 、C x .

8. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 7 is characterized in that: The two-stage fusion lightweight attention module in step 8 is composed of a channel attention mechanism and a spatial attention mechanism. The specific steps are as follows: Channel attention mechanism: After the features are globally averaged and pooled, local cross-channel feature interactions are considered between each channel and its K adjacent channels. This can be effectively achieved through a fast 1D convolution of size K. The size of the kernel K also represents the coverage of the local cross-channel interaction, that is, how many "neighbors" participate in the attention prediction of a channel. To adapt to the framework of this algorithm, in the template branch, the template convolution kernel K is designed to be a fixed size of 3×3; in the search branch, the search area convolution kernel K is designed to be a fixed size of 7×7. For the input feature f, f∈C×H×W, the specific formula is as follows: Among them, Avgpool represents the global average pooling function, unfold represents the sliding local area expansion function, the sliding window size is k, conv represents the convolution kernel, the convolution kernel size is also k, Sigmoid represents the activation function, It is the channel attention weight map generated by formula (11), and the process of formula (11) is abbreviated as CAM. Spatial Attention Mechanism: The spatial attention mechanism in the two-stage fusion lightweight attention mechanism is the same as the SAM process in the SA module. The formula of the two-stage fusion lightweight attention module is as follows: For template branch: C′ z =F z +α·F z CAM(F z )+β·F z ·SAM(F z ) (12), For the search branch: C′ x =F x +α·F x CAM(C x )+β·F x ·SAM(C x ) (13), Among them, α and β are network adaptive learning parameters, C′ z , C′ x Represents the final output template and search features.

9. The two-stage fusion RGBT tracking algorithm based on space-frequency domain equalization according to claim 8 is characterized in that: The specific steps of the tracking head in step 8 are as follows: The tracking head includes classification and regression. Referring to the SiamBAN implementation method, the tasks of the classification head and regression head are to find the position features that are most similar to the search feature area and the target template features, and predict the target bounding box. z and C′ x Perform deep cross-correlation to gather template features and search features, then use classification convolution to perform foreground-background classification on each point in the relevant layer, and output two channels; use regression convolution to predict the offset position from the four sides of the target box, and output four channels. The specific formula is as follows: Among them, P cls Representative classification results, P reg Represents the regression result graph, ★ represents the convolution operation with the template feature as the convolution kernel, Conv cls Represents the classification convolution layer, Conv reg Represents the regression convolution layer.

Images