An RGB-T target tracking method and system based on frequency domain sparse fusion

Abstract

This invention discloses an RGB-T target tracking method and system based on frequency domain sparse fusion, belonging to the field of computer vision technology. The method includes: constructing a two-stream twin backbone network to extract multi-level features from visible light and infrared images respectively; generating query vectors from visible light features and key and value vectors from infrared features; and performing cross-modal feature fusion through a frequency domain sparse fusion module; the fused features are then output as target location and scale information via a classification regression branch. The frequency domain sparse fusion module transfers cross-attention calculation to the frequency domain, captures global cross-modal correlations through Fourier transform, and introduces a sparse sampling mechanism to select the most significant frequency points, combined with bilinear interpolation spectral recovery to reduce computational complexity. A multi-scale edge enhancement module is embedded between the second and third layers of the backbone network, strengthening edge features through stepwise pooling and residual connections. This reduces computational complexity while improving tracking accuracy and success rate.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to an RGB-T target tracking method and system based on frequency domain sparse fusion. Background Technology

[0002] Single-object tracking is an important research direction in the field of computer vision, aiming to continuously predict the scale and position information of a target given in the first frame of a video sequence in subsequent frames. However, in actual tracking, there are complex challenges such as target deformation, occlusion, illumination changes, low light, and thermal crossover. Single-modal visible light tracking is difficult to maintain stable performance in these scenarios.

[0003] Visible light images offer advantages such as high resolution and rich texture details, but suffer from severe information loss under conditions like low light and extreme weather. Thermal infrared images are insensitive to changes in illumination and can penetrate obstructions such as fog and haze, but thermal crossover occurs when the target and background temperatures are similar. Visible light and infrared images are naturally complementary in their imaging mechanisms and information representation; therefore, RGB-T (visible-infrared) target tracking has become an important technical approach to address these issues.

[0004] Existing RGB-T tracking methods mainly face the following problems: First, most methods struggle to adaptively fuse features from both modes. Simple addition or splicing cannot effectively utilize the complementary characteristics between modes, and low-quality infrared features can contaminate high-quality visible light features. Second, as the network depth increases, details such as edges and textures in shallow layers are easily lost, affecting the accurate localization of target boundaries. Third, existing cross-attention fusion methods have high computational complexity and do not fully consider the role of frequency domain information in cross-modal association modeling.

[0005] To address the aforementioned issues, there is an urgent need for an RGB-T target tracking method that can adaptively fuse dual-modal features, preserve edge detail information, and is computationally efficient. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides an RGB-T target tracking method and system based on frequency domain sparse fusion. By using visible light features as query vectors and infrared features as key and value vectors for frequency domain sparse fusion, the tracking accuracy and success rate are significantly improved while reducing computational complexity.

[0007] In a first aspect, the present invention provides an RGB-T target tracking method based on frequency domain sparse fusion, comprising the following steps: Step 1: Construct a dual-stream twin backbone network and extract features from the input visible light template image, infrared template image, visible light search image, and infrared search image to obtain visible light template features, infrared template features, visible light search features, and infrared search features. Step 2: The extracted visible light template features and visible light search features are convolved to generate query vectors, and the infrared template features and infrared search features are convolved to generate key vectors and value vectors, respectively. Cross-modal feature fusion is then performed through the frequency domain sparse fusion module. Specifically, for the template branch, the visible light template features and infrared template features are input into the frequency domain sparse fusion module to obtain fused template features; for the search branch, the visible light search features and infrared search features are input into the frequency domain sparse fusion module to obtain fused search features. Specifically, the infrared template features are convolved to generate template key vectors and template value vectors, and the infrared search features are convolved to generate search key vectors and search value vectors. Step 3: Input the fused template features and fused search features into the classification and regression branch, calculate the similarity between the fused template features and fused search features through cross-correlation operation, and output the target location and scale information.

[0008] As a preferred embodiment of the technical solution of the present invention, the frequency domain sparse fusion module performs the following operations: Perform Fourier transforms on the query vector and key vector respectively to map them to the frequency domain; The similarity weight matrix between the query vector and the key vector is calculated in the frequency domain, and the similarity weight matrix is ​​sparsified by a sparse sampling mechanism. The sparsified similarity weight matrix is ​​mapped back to the spatial domain through inverse Fourier transform to obtain the sparse similarity weight matrix. The value vector is weighted using the sparse similarity weight matrix and added to the input visible light template feature or visible light search feature to obtain the fusion feature corresponding to that branch.

[0009] By shifting the cross-attention computation process from the spatial domain to the frequency domain and leveraging the global properties of the Fourier transform, low-frequency structural information and high-frequency detail information across modes can be captured simultaneously, making cross-modal complementary fusion more stable. By introducing a sparse sampling mechanism, information with high spectral energy is selectively focused, salient regions are enhanced, and noise and meaningless information are suppressed, improving fusion performance while reducing memory usage. Residual connections are used to add the fused features to the original visible light features, avoiding the loss of visible light feature information during fusion and preventing mode drift.

[0010] As a preferred embodiment of the technical solution of the present invention, the sparse sampling mechanism specifically includes: Calculate the significance score for each frequency point in the frequency domain, wherein the significance score is the channel average of the modulus of the complex number; Based on the preset sampling ratio, select the top k frequency points with the highest significance scores; The selected frequency points are retained, while the remaining frequency points are discarded, resulting in the sparsified frequency domain features.

[0011] By averaging the channel values ​​of the complex modulus to calculate the significance score, the importance of each frequency point in cross-modal fusion can be accurately assessed. The top-k sampling mechanism retains only the k frequency points with the highest significance scores, effectively removing redundant information and noise in the frequency domain, reducing the computation of invalid spectra, and lowering the computational cost of the model.

[0012] As a preferred embodiment of the technical solution of the present invention, after sparse sampling, the frequency domain features after sparsification are subjected to spectral recovery to adapt to the input dimension of the inverse Fourier transform, specifically including: The sparsified frequency domain features are split into real and imaginary parts, and then upsampled to the complete spectrum with the same resolution as the original frequency domain using bilinear interpolation to obtain a sparsified similarity weight matrix of full size.

[0013] By splitting the sparsified spectrum into real and imaginary parts and performing interpolation recovery separately, the technical problem of the reduced spectral scale after sparse sampling failing to meet the input conditions of the inverse Fourier transform is solved, enabling seamless integration of the sparse sampling mechanism with the inverse Fourier transform. Upsampling using bilinear interpolation smoothly amplifies the sampled points in the frequency domain, reconstructing the complete spectrum while preserving amplitude and scale information as much as possible, thus ensuring the integrity of the spatial domain features after the inverse Fourier transform. This spectral recovery mechanism allows this method to maintain the integrity and accuracy of the fused features while reducing computational complexity.

[0014] As a preferred embodiment of the technical solution of the present invention, when calculating the similarity weight matrix between the query vector and the key vector in the frequency domain: After performing Fourier transforms on the query vector and key vector respectively, the conjugate transposes of the query vector and key vector are multiplied element-wise in the frequency domain, and then multiplied by a learnable frequency domain weight parameter to obtain the frequency domain similarity weight matrix.

[0015] By multiplying the query vector and the key vector's conjugate transpose element-wise in the frequency domain, the similarity between two modalities can be calculated while considering phase information, thus accurately capturing their correlation in the frequency domain and improving the accuracy of cross-modal matching. Introducing learnable frequency domain weight parameters allows for adaptive adjustment of the response intensity of each channel in the frequency domain, enabling the model to dynamically adjust the weight allocation of frequency domain features according to different scenarios and data characteristics, enhancing the model's scenario adaptability. Compared to traditional spatial domain inner product calculations, element-wise multiplication in the frequency domain is more computationally efficient while preserving global information and is not limited by the receptive field of convolution.

[0016] As a preferred embodiment of the technical solution of the present invention, the similarity weight matrix obtained by inverse Fourier transform is subjected to in-channel standardization: Calculate the mean and variance of the similarity weight matrix in the spatial dimension, and use the mean and variance to standardize the features of each channel to obtain the standardized sparse similarity weight matrix.

[0017] Standardizing each channel by calculating the mean and variance of its spatial dimension reduces scale shift and differences in attention response values ​​across channels, suppresses extreme response values ​​caused by frequency domain transformation, and improves the model's numerical stability. Intra-channel standardization makes the attention weights comparable across different channels, which is beneficial for subsequent feature weighting and fusion operations, improving the quality of the fused features. This standardization process effectively prevents numerical instability caused by frequency domain transformation and accelerates the model's convergence speed.

[0018] As a preferred embodiment of the technical solution of the present invention, the dual-stream twin backbone network adopts a structure of shallow-layer parameter sharing and deep-layer parameter independence: Shallow networks share parameters between the visible light mode and the infrared mode, while deep networks learn feature representations for their respective modes independently. The template branch and search branch in the dual-stream twin backbone network have the same structure and share weights.

[0019] Visible light template images and infrared template images exhibit high similarity in low-level visual features such as edge contours and texture characteristics. Parameter sharing in the first three layers reduces model redundancy and training difficulty, minimizing the risk of overfitting. The last two layers employ an independent branch structure, allowing each modality branch to learn its unique deep semantic features, thus improving the quality of feature representation before fusion. Shared weights between the template branch and the search branch ensure consistency in feature extraction and improve computational efficiency.

[0020] As a preferred embodiment of the technical solution of the present invention, a multi-scale edge enhancement module is embedded between the second and third layers of the dual-stream twin backbone network to enhance edge features; the multi-scale edge enhancement module performs the following operations: The input features are reconstructed to generate basic edge responses; A progressive average pooling method is used to extract edge response features at different scales. The difference between the pooled features and the original features before pooling is calculated to highlight the edge position. The base edge response and the multi-scale edge-enhanced features are added together through residual connections to output the edge-enhanced features.

[0021] Edge information is crucial for locating target boundaries in target tracking. By introducing a multi-scale edge enhancement module into the shallow network, texture information is effectively recovered, enabling the model to maintain relatively stable boundary perception capabilities even when processing images with weak textures. Successive average pooling is used to gradually expand the receptive field, maintaining computational efficiency while enhancing the hierarchical correlation between edge features at different scales. Subtracting the base edge response from the pooled features effectively suppresses responses in flat regions, highlighting target edge locations with abrupt changes in grayscale or features.

[0022] As a preferred embodiment of the technical solution of the present invention, the classification and regression branch includes a classification branch and a regression branch: The classification branch is used to output a classification response map, determining whether each location in the search area is the target area; The regression branch is used to output a regression response map, predicting the distance between the target bounding box and the target center; The classification and regression branches calculate the similarity between the fusion template features and the fusion search features through deep cross-correlation operations.

[0023] The classification and regression branches are processed in parallel, respectively responsible for target region discrimination and target bounding box regression. This clear division of tasks improves tracking accuracy and localization precision. The deep cross-correlation operation uses template features as kernels to convolve search features, effectively detecting the similarity between the fused template features and the fused search features, thus accurately locating the target.

[0024] Secondly, the present invention also provides an RGB-T target tracking system based on frequency domain sparse fusion, comprising: The dual-stream twin backbone network module is used to extract features from the input visible light template image, infrared template image, visible light search image, and infrared search image, respectively. A frequency domain sparse fusion module is used to generate a query vector from visible light features through convolution, and to generate key and value vectors from infrared features through convolution, and to perform cross-modal feature fusion through frequency domain sparse fusion. The frequency domain sparse fusion module includes: a Fourier transform unit for converting the query vector and key vector to the frequency domain; a sparse sampling unit for sparsifying the similarity weight matrix in the frequency domain and retaining the frequency points with the highest saliency scores; an inverse Fourier transform unit for mapping the sparsified frequency domain features back to the spatial domain; and a feature weighting unit for weighting the value vector using the sparse similarity weight matrix and adding it to the visible light features. The classification and regression module is used to perform cross-correlation calculations on the fused template features and fused search features, and output the target location and scale information.

[0025] As can be seen from the above technical solutions, this application has the following advantages: By generating query vectors from visible light features and key and value vectors from infrared features, cross-modal feature fusion dominated by visible light features is achieved. While maintaining the dominance of visible light features, it adaptively selects the regions in infrared features that are most relevant and complementary to visible light features for fusion, avoiding interference from low-quality infrared features to high-quality visible light features, and improving the discriminativeness and stability of the fused features. The structural design of the dual-stream twin backbone network allows the template branch and search branch to share weights, ensuring the consistency of feature extraction, while reducing the number of model parameters and computational complexity. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a diagram of the RGB-T target tracking network structure.

[0028] Figure 2 This is a diagram showing the structure of template branches and search branches.

[0029] Figure 3 This diagram illustrates the operation performed by the multi-scale edge enhancement module.

[0030] Figure 4 The diagram illustrates the operation performed by the frequency domain sparse fusion module.

[0031] Figure 5 This is a flowchart of the sparse sampling mechanism.

[0032] Figure 6 Evaluation curves for the GTOT dataset, where, Figure 6(a) in the figure is the accuracy curve for evaluating the GTOT dataset, i.e., the PR curve; Figure 6 (b) in the figure is the success rate curve for the GTOT dataset, i.e., the SR curve.

[0033] Figure 7 Evaluation curves for the RGBT234 dataset, where, Figure 7 (a) in the figure is the accuracy curve for evaluating the RGBT234 dataset, i.e., the PR curve; Figure 7 (b) in the figure is the success rate curve for evaluating the RGBT234 dataset, i.e., the SR curve.

[0034] Figure 8 This is a flowchart illustrating the method provided in an embodiment of the present invention. Detailed Implementation

[0035] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0037] This embodiment provides an RGB-T target tracking method based on frequency-domain sparse fusion. This method uses the visible light target tracking network SiamBAN as its basic architecture, extends the backbone network into a two-stream Siamese network, and proposes a frequency-domain sparse fusion RGB-T target tracking network (Siamese tracker with Frequency-domain Sparse Fusion, SiamFS) by fusing two-modal information at the feature level.

[0038] like Figure 1As shown, the RGB-T target tracking network based on frequency domain sparse fusion mainly consists of three parts: a template branch, a search branch, and a classification and regression branch. The template and search branches are two-stream twin network backbone structures, extracting features from two modalities of the image. While the two branches have the same structure, they process different images and perform different tasks within the network. The template branch takes into account visible light and infrared template images of size 127×127×3, providing the target's appearance features and outputting fused template features. The search branch takes into account visible light and infrared search images of size 255×255×3, providing the image to be tracked, which includes the target and a relatively complex background, and outputs fused search features. The multi-scale fused features output from the third, fourth, and fifth layers of the backbone network are simultaneously input into the neck network. A 1×1 convolution reduces the number of channels in the multi-scale features to 256, and then simultaneously inputs them into the classification and regression branch. The classification and regression branch first processes each layer of the fused feature map separately, that is, it detects the similarity between the fused template features and the fused search features through deep cross-correlation, obtaining three response maps. The difference between the classification and regression branches lies in the kernel size setting during the channel count adjustment process after cross-correlation. Finally, the three response maps are weighted and fused to obtain the final classification and regression responses. The classification branch outputs a classification response map, responsible for determining whether it is a target region, while the regression branch outputs a regression response map, responsible for the distance of the regression prediction box from the target center.

[0039] During the training process, the template images and search images of the two modalities are obtained in two ways: positive samples and negative samples. Positive samples are taken from two sets of RGB-T images with an interval of t frames in the same video frame sequence, where t is usually set to 100. Negative sample template images and search images are taken from two sets of RGB-T images in different sequences.

[0040] During the tracking test phase, the template image is the first frame in the video frame sequence, cropped based on known annotation information to obtain the target region. The search image is the subsequent test frames. After feature extraction, the image yields a feature map, which is then input into the Frequency-domain Sparse Cross-modal Fusion (FSCF) module to fuse features. The fused features are then input into the neck network to reduce the number of channels to 256, effectively controlling computational complexity while maintaining feature expressive power. This provides semantically consistent and distributionally stable intermediate feature representations for subsequent cross-correlation operations and classification / regression tasks. The fused template features and fused search features are cross-correlated in the classification / regression branch to complete target tracking and localization.

[0041] The dual-stream twin backbone network, frequency domain sparse fusion module, multi-scale edge enhancement module, and classification regression branch together constitute an end-to-end RGB-T target tracking network. The multi-scale edge enhancement module is embedded between the second and third layers of the dual-stream twin backbone network, and the frequency domain sparse fusion module and classification regression branch are sequentially connected after the dual-stream twin backbone network.

[0042] In the RGB target tracking network, the template branch and the search branch form a twin backbone network with identical structures and shared weights, extracting features from visible light images. In the RGB-T target tracking process, to adapt to the differences between different modal images and maximize feature extraction, the twin network is reconstructed into a two-stream twin backbone network structure in the feature extraction part. This extracts deep features from both visible light and infrared images, with weights shared within each modality. Feature-level fusion is then used before inputting into subsequent networks. In the RGB-T network, the template branch and the search branch still have identical structures and shared weights. The structures of the template branch and the search branch are as follows: Figure 2 As shown.

[0043] Visible light images and infrared images share high similarities in low-level visual features such as edge contours and texture features. Therefore, the first three layers of the dual-stream twin backbone network adopt a design of inter-modal parameter sharing to reduce model redundancy and training difficulty, while the last two layers learn deep-level two-modal features. The first layer consists of a convolutional layer, a batch normalization layer, a linear rectified function, and a pooling layer. The last four layers are all constructed by stacking multiple Bottleneck structures, including convolutional layers, BN layers, and ReLU activation functions.

[0044] During image feature extraction, shallow feature maps retain rich spatial geometric information of the target, are more sensitive to changes in the target's scale, shape, and position, and also contain low-level visual details such as texture and edges. Therefore, enhancing shallow features can not only reduce noise interference that may occur in the shallow backbone but also ensure that more key local structural and textural information participates in subsequent fusion to form more representative image features. Based on this, a multi-scale feature enhancement module (MFEM) is introduced between the second and third layers of the backbone network to more fully mine and utilize the important information in shallow features and strengthen edge features. Deep feature maps mainly represent the high-level semantic information of the target, and there are significant differences between different modalities. Therefore, an independent branch structure is adopted in the last two layers of the network, with each modality branch learning its unique semantic features, thereby improving the quality of feature representation before fusion. In order to increase the receptive field of the convolutional kernel and aggregate richer contextual information while maintaining resolution, dilated convolutions with dilation rates of 2 and 4 are used to replace standard convolutions, respectively. The two modal image features are then input into the FSCF module, which outputs fused template features and search features to achieve complementary fusion of cross-modal features.

[0045] Edge information typically contains the shape and structure of objects in an image and is crucial for locating target boundaries in target tracking. To enhance the representation of edge information, effectively recover texture information, and maintain stable boundary perception even when processing images with weak textures, a multi-scale edge enhancement module is introduced into the backbone network. This module improves the model's ability to perceive edge information by fusing edge information from different scales. The multi-scale edge enhancement module, such as... Figure 3 As shown.

[0046] The features obtained from the second layer of the backbone network The input multi-scale edge enhancement module extracts edge information from feature maps at multiple different scales. First, it... Features are reconstructed to enhance edge-related responses and provide basic features for subsequent comprehensive acquisition of edge information. The reconstruction process is shown in Equation (1).

[0047] (1) in, The convolution is represented. A three-branch structure is then constructed to extract multi-scale features. The receptive field is gradually expanded by a stepwise pooling method. While maintaining computational efficiency, this makes the edge features at different scales have stronger hierarchical correlation. The receptive field is expanded by 3×3 average pooling to obtain the response characteristics of edge features under different receptive field conditions. After the first pooling, the edges become weaker and the features are more stable. After the second pooling, local details disappear and only features at larger spatial scales are retained. After the third pooling, small targets and finer edges disappear, and global structural information is obtained. Local averaging allows the feature map to obtain greater neighborhood information. Features at different receptive fields are obtained respectively. The features obtained after three poolings are shown in Equation (2).

[0048] (2) in, This represents 3×3 average pooling.

[0049] Average pooling smooths features within a local neighborhood, and by subtracting the pooled features from the original features, it can effectively suppress responses in flat regions and highlight target edge locations with abrupt changes in grayscale or features. A convolutional unit consists of convolution, normalization, and a sigmoid activation function. Normalization constrains the distribution of edge responses, stabilizing the distribution of edge features and accelerating network convergence. As a learnable parameter, it can flexibly adapt to different network structures. The sigmoid function constrains edge responses within the [0,1] interval, and edge enhancement is applied to the original features in a weighted manner, avoiding the destruction of original semantic information due to excessive edge noise.

[0050] After being processed by the convolutional unit, the residual structure is added to the features before average pooling to form a residual structure, which is beneficial to stabilize the training process and reduce the interference of the module on the original network feature expression. While retaining the original feature information, the edge expression ability is enhanced. The edge enhancement process is shown in Equation (3).

[0051] (3) in, This represents a convolutional unit.

[0052] Enhanced multi-scale feature edge information is obtained ( Finally, the features of the four branches are integrated, that is... , , and The calculation process is shown in equation (4).

[0053] (4) It contains stronger edge information, providing more discriminative features for subsequent classification and regression.

[0054] Modal fusion is the most crucial step in extending a visible light target tracking network into an RGBT tracking network. Its core idea lies in fully leveraging the complementary advantages of each mode, focusing on more valuable modes in different channel spaces, and reducing the impact of low-quality modes on the fused features. To prevent low-quality infrared features from affecting high-quality visible light features, and to extract effective information from infrared features to enhance visible light features, a frequency-domain sparse cross-modal fusion (FSCF) module was designed. This module guides the thermal infrared modes through visible light modes, maintaining the dominance of visible light features while adaptively selecting the most relevant and complementary regions of infrared features to the visible light features, thereby achieving stable and efficient cross-modal fusion.

[0055] Cross-attention filters auxiliary modal features using spatial similarity weights, and then incorporates the filtered results into the features providing the Q vector. The spatial similarity weights are jointly determined by the Q vector and the K vector. When cross-attention is used for RGBT feature fusion, the higher the correlation between the thermal infrared feature region and the visible light feature region, the larger the spatial similarity weight, and the more significantly the corresponding V vector will be enhanced and participate in the fusion. This structure can effectively suppress irrelevant information such as infrared background noise and heat source interference from entering the fusion result, improving the discriminativeness and stability of the fused features at the module structure level. To establish correlation and complementarity between the visible light and thermal infrared modalities, a cross-modal sparse fusion module is designed based on the cross-attention module, with the structure as follows: Figure 4 As shown.

[0056] The two modal features obtained after the two images pass through the backbone network are processed by a cross-mechanism in the FSCF module. The features are transformed by query, key, and value to obtain feature vectors in three directions. Specifically, three convolutional modules are used to process the two modal features respectively. The visible light features are used as the dominant modality and are transformed by query to generate a Q vector, which is used to guide the fusion process. The infrared modality is used as the guided modality and is transformed by key and value to obtain K and V vectors. The specific calculation process is shown in Equation (5).

[0057] (5) in, , and Indicates that the three parameters are different Convolution kernel, Indicates the characteristics of visible light. The infrared features are represented by the Q vector, which characterizes the discriminative query information in the visible light features and guides selective attention to key spatial locations during the fusion process. The K vector represents the response features of each spatial location in the infrared features and is used to match with the Q vector to measure cross-modal correlation. The V vector contains the semantic and structural information in the infrared modality that actually supplements the visible light features. The Q and K vectors are fused in the frequency domain. First, FFT is applied, and then they are multiplied to generate frequency domain cross-modal attention weights. These weights determine the degree of participation of the infrared features in the fusion process. To remove redundancy and reduce computation, a sparse sampling mechanism is designed to sparsify the frequency domain cross-modal attention weights. Then, IFFT is used to transform them back to the spatial domain to obtain the sparse similarity weight matrix. Fusion features The calculation process is shown in equation (6).

[0058] (6) Since visible light features provide a wealth of information in most scenarios, residual connections are introduced to prevent the loss of visible light feature information and mode drift during fusion. These residual connections add the fused features to the original visible light features, ultimately yielding a fusion result of visible light and infrared features guided by visible light features. The FSCF module adaptively adjusts the participation of infrared features in the fusion process, structurally enabling selective information exchange between modes.

[0059] In the traditional calculation of the spatial similarity weight matrix, the similarity matrix is ​​obtained from the Q vector and the K vector, and then the spatial similarity weight matrix between the two modes is obtained. The calculation process is shown in Equation (7).

[0060] (7) in, This represents the similarity weight matrix between two modes. Let Q represent the dimension of vector K. The correlation between the two vectors is obtained by multiplying vector Q by the transpose of vector K, which has a large computational cost and a space complexity of O(k). The time complexity is ,in, The size of the feature map is represented by the following calculation process: , Indicates the height of the feature map, Indicates the width of the feature map. This represents the number of channels. During matrix multiplication, each element can be obtained through the inner product, as shown in (8).

[0061] (8) in, This represents the vectorized form of the i-th image patch in the Q vector. This represents the vectorized form of the j-th image patch in the K vector. The inner product process is equivalent to... After being flipped, it becomes a convolution kernel, and... Perform convolution, therefore The All elements in the column can be obtained through convolution. According to the convolution theorem, convolution or correlation in the spatial domain is equivalent to element-wise multiplication in the frequency domain. Therefore, element-wise multiplication in the frequency domain is used for efficient estimation of the similarity weight matrix. Transforming the Q-vector and K-vector to the frequency domain and then multiplying them achieves efficient estimation of the attention map. Furthermore, each point in the frequency domain contains global information. During point-wise multiplication in the frequency domain, each multiplication corresponds to the global convolution effect, unaffected by the receptive field limitations of convolution. Low-frequency structural information and high-frequency detail information are processed separately, making cross-modal complementary fusion more stable. The improved space complexity is... The time complexity is The intermodal similarity weight matrix A can be calculated using equation (9).

[0062] (9) in, This represents the scaling factor, a learnable frequency domain weighting parameter used to adaptively adjust the response intensity of each channel in the frequency domain. Indicates Fourier transform, This represents the conjugate transpose after the Fourier transform. The conjugate transpose operation ensures that the similarity between the Q-vector and the K-vector is calculated while considering phase information, thus accurately capturing their correlation in the frequency domain. This is achieved through a top-k sampling mechanism. Sparsification yields the sparsified similarity weight matrix. And then Performing an inverse Fourier transform yields the temporal similarity weight matrix. .

[0063] (10) in, This represents the inverse Fourier transform. To reduce scale shift and differences in attention response values ​​across channels, suppress extreme response values ​​generated by the frequency domain transform, and improve stability, [the following is implemented / implemented]. Standardization is performed within the channel, and the calculation method is shown in Equation (11).

[0064] (11) in, Represents the sparse similarity weight matrix. The mean of the spatial dimension. For spatial variance, To prevent the denominator from being zero.

[0065] The sparse sampling operation employs the top-k mechanism, sparsifying the frequency domain similarity weight matrix. It selectively focuses on information with high spectral energy, enhancing salient regions, suppressing noise and meaningless information, reducing ineffective spectral computation, and selecting the most contributing positions from a frequency domain perspective. This improves fusion performance while reducing memory usage. The sparse sampling mechanism structure is as follows: Figure 5 As shown.

[0066] The importance of a batch is evaluated using the modulus of a complex number, with a significance score. The calculation method is shown in equation (12).

[0067] (12) in, This indicates that the average value of each channel is taken. This indicates taking the modulus of a complex number. The significance scores are sorted, and the number of sampling blocks is obtained based on the set sampling ratio. Select those with higher significance scores Key frequency point tokens participate in subsequent fusion.

[0068] The sparsification selection process is shown in equation (13).

[0069] (13) in, This represents the set of indices representing the selected spatial locations. Indicates the period before obtaining the significance score The index of each token, based on the index exist Select from the options and keep only. The frequency domain characteristics corresponding to the set. The removed frequency domain locations have smaller amplitudes and weaker cross-modal responses, and are mostly noise redundant frequencies; conversely, the retained locations have larger amplitudes and are frequencies corresponding to key structures such as edges or targets.

[0070] Since the inverse Fourier transform requires a complete and regular two-dimensional spectrum as input, but the spectral scale is reduced after sparse sampling, it does not meet the input conditions of the inverse Fourier transform. Therefore, it is necessary to restore the N highly significant tokens to a regular two-dimensional frequency plane structure. Specifically, the sparsified spectrum is split into imaginary and real parts, and bilinear interpolation is used to upsample them to a complete spectrum with the same resolution as the original frequency domain. This process can smoothly amplify the sampling points in the frequency domain, reconstructing the complete spectrum while preserving the amplitude and scale information as much as possible, and finally obtaining the sparsified similarity weight matrix. .

[0071] The classification and regression branches consist of a classification branch and a regression branch. The classification branch outputs a classification response map, determining whether each location in the search region is a target region. The regression branch outputs a regression response map, predicting the distance of the target bounding box from the target center. The classification and regression branches calculate the similarity between the fused template features and the fused search features through deep cross-correlation. Following the SiamBAN implementation, the classification and regression heads aim to find the location features most similar to the target template features within the search feature region and predict the target bounding box. Deep cross-correlation is performed on the template features and search features to aggregate them. Then, a classification convolution is used to perform foreground-background classification on each point in the relevant layer, outputting two channels. A regression convolution is used to predict the offset positions from the four sides of the target box, outputting four channels.

[0072] As described above, the RGB-T target tracking network in this embodiment consists of a two-stream twin backbone network, a multi-scale edge enhancement module, a frequency domain sparse fusion module, and a classification and regression branch connected sequentially. Wherein: A dual-stream twin backbone network is used to extract multi-level features from visible light and infrared images; A multi-scale edge enhancement module is embedded between the second and third layers of the backbone network to enhance edge features; The frequency domain sparse fusion module is connected after the backbone network and is used to perform frequency domain sparse cross-attention fusion on bimodal features; The classification and regression branch is connected after the frequency domain sparse fusion module and is used to output target location and scale information.

[0073] This embodiment employs a two-stage training strategy to train the RGB-T target tracking network.

[0074] Phase 1: Pre-train the dual-stream twin backbone network on the large-scale visible light target tracking dataset GOT-10k to obtain the visible light basic model.

[0075] Phase Two: Load the aforementioned visible light base model and perform multimodal training on the entire network using RGB-T datasets (such as GTOT and RGBT234). During training, training samples are cropped and scaled to generate a 127×127 template image and a 225×225 search image as network input. Each training round contains 250,000 video samples, for a total of 30 epochs. In the first 10 epochs, the dual-stream twin backbone network is frozen, and only the frequency domain sparse fusion module and the classification / regression branch are trained to prevent sudden changes in the network that could lead to instability during training. Starting from the 11th epoch, the multi-scale edge enhancement module and the last two layers of the dual-stream twin backbone network are unfrozen and jointly trained with the frequency domain sparse fusion module and the classification / regression branch.

[0076] The base learning rate is 0.005. The first 5 epochs use a step-based warm-up method, with the learning rate gradually increasing from 0.001 to 0.005 to stabilize the model in the early stages of training. Subsequent epochs employ a log decay strategy, gradually reducing the learning rate to 1e-5. The batch size is 28.

[0077] During the tracking test, the window influence coefficient was set to 0.48, the penalty term coefficient was 0.15, and the learning rate was dynamically adjusted to 0.38 to ensure stable updates of the target state in the search frame.

[0078] This embodiment was completed in a Python 3.7 environment, using the PyTorch deep learning framework. PyCharm 2023.2.2 and MATLAB R2020b were installed on the Windows system. PyCharm was primarily used for model simulation, parameter tuning, and code debugging, while MATLAB was used for visualization of tracking results and performance evaluation. Furthermore, to ensure efficient model training, both training and testing were performed on a Linux system, with the server equipped with Python 3.8, PyTorch, CUDA 10.2, and relevant third-party libraries.

[0079] This embodiment conducted ablation and comparison experiments on the GTOT and RGBT234 datasets. Eight mainstream tracking algorithms on the GTOT dataset were selected and compared with the SiamFS algorithm. The eight mainstream tracking algorithms include: MDNet+RGBT, DAT, ECO, RT-MDNet, ADNet, CFnet+RGBT, SiamDW+RGBT, and SiameseFC. Among them, SiamDW+RGBT and SiameseFC are both Siamese network algorithms, and their evaluation curves are shown below. Figure 6 As shown.

[0080] PR curve as Figure 6 As shown in (a), the accuracy of the tracking algorithms all show a trend of first rapidly increasing and then gradually stabilizing as the positioning error threshold increases. When the positioning error threshold is less than 5 pixels, the accuracy difference between different algorithms is small. When the positioning error threshold is between 5 and 10 pixels, ADNet, CFnet+RGBT, SiamDW+RGBT, and SiameseFC show a significant difference from the other five algorithms. When the positioning error threshold is greater than 10 pixels, the accuracy of various algorithms tends to level off, with little change in value. Compared with other algorithms, SiamFS shows strong stability when the positioning error threshold is between 5 and 10 pixels, and then slightly lower than MDNet+RGBT.

[0081] SR curve as Figure 6As shown in (b) of the diagram, the success rates of each algorithm show a slow initial decline followed by a rapid decrease as the overlap ratio threshold increases. Higher overlap ratio thresholds place stricter requirements on target localization accuracy. When the overlap ratio threshold is 0.6, the success rates of CFnet+RGBT, SiamDW+RGBT, and SiameseFC are significantly lower than the other algorithms. Within the overlap ratio threshold range of 0.2 to 0.7, SiamFS has a significantly higher success rate than other algorithms, with a relatively gentle downward trend. The success rates of the other algorithms decrease rapidly within the high threshold range, reflecting their inadequacy in accurate target boundary localization.

[0082] To more comprehensively and meticulously analyze the applicability and robustness of SiamFS in complex real-world scenarios, comparative experiments were further introduced on the GTOT dataset under different challenge attributes. For each challenge attribute, the PR and SR of SiamFS compared with other mainstream algorithms were statistically analyzed. Among them, DuSiamRT, SiamFT, and SiamDW+RGBT are all Siamese network series algorithms, and the evaluation results are shown in Table 1.

[0083] Table 1. Comparison of PR / SR between SiamFS and mainstream algorithms under different attribute conditions on GTOT (%)

[0084] Looking at the GTOT dataset as a whole, compared to the Siamese network algorithms SiameseFC, SiamDW+RGBT, DuSiamRT, and SiamFT, the SiamFS algorithm improves PR by 16.5%, 14.0%, 5.4%, 6.2%, and , respectively, and SR by 12.5%, 10.0%, 3.7%, and 4.2%, respectively. Compared to the classic MDNet+RGBT algorithm, it improves accuracy by 2.0% and success rate by 2.8%. It also shows some performance improvement compared to other trackers.

[0085] Compared to other mainstream algorithms, the SiamFS algorithm performs relatively well under various challenging attributes. It achieves the best results in low-light and hot-crossing attributes, reaching accuracy and success rate of 80.9% / 65.2% and 84.4% / 67.6%, respectively. It also achieves good results in fast motion and target deformation attributes, achieving the best accuracy in target deformation attributes, suitable for fine-grained operations or recognition with low tolerance for positional errors. It achieves the best success rate in fast motion attributes, suitable for situations where stability is paramount and target loss is not an option. While it doesn't reach the best performance in target occlusion, scale changes, and small target attributes, it remains at a high level and doesn't exhibit extremely poor performance in any particular attribute. Compared to the other three Siamese network algorithms, it achieves better performance and relatively better results in other attributes, demonstrating greater stability.

[0086] Tracking tests were conducted on the RGBT234 dataset, which presents more complex scenarios and includes long-term tracking sequences, thus increasing the tracking difficulty. Six mainstream tracking algorithms were selected and compared with SiamFS on the RGBT234 dataset. These six mainstream tracking algorithms include: MDNet+RGBT, RT-MDNet, ECO, SOWP+RGBT, SiamDW+RGBT, and CFnet+RGBT. Performance comparison results are as follows: Figure 7 As shown.

[0087] PR curve as Figure 7 As shown in (a), the SiamDW+RGBT and CFnet+RGBT algorithms do not fully utilize the complementarity of the two modal images in the multimodal fusion process, and are significantly inferior to other algorithms. The curves of the SiamFS and RT-MDNet algorithms are similar. The SR curves are as follows. Figure 7 As shown in (b), when the overlap threshold is less than 0.5, SiamFS outperforms ECO, indicating that the method has good tracking stability under low to medium overlap constraints. However, when the overlap threshold is greater than 0.5, the success rate of SiamFS is slightly lower than that of ECO, indicating that the proportion of frames with a large intersection-union ratio (IU) between the predicted and ground truth bounding boxes is relatively small. The current network has room for improvement in high-precision bounding box regression, and further optimization of the model structure or fusion strategy is needed to increase the proportion of successful frames when the IU threshold is large.

[0088] Meanwhile, the performance of this dataset under different challenge attributes was analyzed. The results of SiamFS and some mainstream datasets on the RGBT234 dataset under different challenge attribute conditions are shown in Table 2.

[0089] Table 2 Comparison of PR / SR between SiamFS and mainstream algorithms under different attribute conditions on RGBT234 (%)

[0090] SiamFS achieved an accuracy of 71.6% and a success rate of 50.8%. Compared with the visible light SiamBAN algorithm, it improved accuracy by 3.5% and success rate by 1.9%. Compared with the Siam network series algorithms DuSiamRT and SiamFT, it improved accuracy by 0.6% and 2.8%, and success rate by 1.2% and 2.2%, respectively. Compared with the classic multi-domain algorithm MDNet, it improved accuracy by 0.6% and success rate by 1.8%. Although the success rate decreased by 0.6% compared with ECO, the accuracy improved by 1.4%, which to some extent made up for the shortcomings.

[0091] Based on the above network structure and training method, such as Figure 8As shown, the RGB-T target tracking method based on frequency domain sparse fusion in this embodiment specifically includes the following steps: Step 1: Construct a dual-stream twin backbone network and extract features from the input visible light template image, infrared template image, visible light search image, and infrared search image to obtain visible light template features, infrared template features, visible light search features, and infrared search features.

[0092] Step 2: The extracted visible light template features and visible light search features are convolved to generate query vectors, and the infrared template features and infrared search features are convolved to generate key vectors and value vectors, respectively. Cross-modal feature fusion is then performed using a frequency domain sparse fusion module. Specifically, for the template branch, the visible light template features and infrared template features are input into the frequency domain sparse fusion module to obtain fused template features; for the search branch, the visible light search features and infrared search features are input into the frequency domain sparse fusion module to obtain fused search features. The frequency domain sparse fusion module performs the following operations: Perform Fourier transforms on the query vector and key vector respectively to map them to the frequency domain; The similarity weight matrix between the query vector and the key vector is calculated in the frequency domain, and the similarity weight matrix is ​​sparsified by a sparse sampling mechanism. The sparsified similarity weight matrix is ​​mapped back to the spatial domain through inverse Fourier transform to obtain the sparse similarity weight matrix. The value vector is weighted using the sparse similarity weight matrix and added to the original query vector to obtain the fusion feature corresponding to that branch.

[0093] The sparse sampling mechanism specifically includes: calculating the significance score of each frequency point in the frequency domain, wherein the significance score is the channel average of the modulus of the complex number; selecting the k frequency points with the highest significance scores according to a preset sampling ratio; retaining the selected frequency points and discarding the remaining frequency points to obtain the sparsified frequency domain features.

[0094] After sparse sampling, the frequency domain features of the sparsified frequency domain are subjected to spectrum recovery to adapt to the input dimension of the inverse Fourier transform. Specifically, the sparsified frequency domain features are split into real and imaginary parts, and then upsampled to the complete spectrum with the same resolution as the original frequency domain using bilinear interpolation method to obtain a sparsed similarity weight matrix of the complete size.

[0095] When calculating the similarity weight matrix between the query vector and the key vector in the frequency domain, the query vector and the key vector are respectively subjected to Fourier transform, and the conjugate transpose of the query vector and the key vector are multiplied element-wise in the frequency domain, and then multiplied by the learnable frequency domain weight parameters to obtain the frequency domain similarity weight matrix.

[0096] The similarity weight matrix obtained by inverse Fourier transform is standardized within each channel: the mean and variance of the similarity weight matrix in the spatial dimension are calculated, and the features of each channel are standardized using the mean and variance to obtain the standardized sparse similarity weight matrix.

[0097] Step 3: Input the fused template features and fused search features into the classification and regression branch, calculate the similarity between the fused template features and fused search features through cross-correlation operation, and output the target location and scale information.

[0098] The classification and regression branches include a classification branch and a regression branch: the classification branch outputs a classification response map to determine whether each location in the search area is a target region; the regression branch outputs a regression response map to predict the distance of the target bounding box from the target center. The classification and regression branches calculate the similarity between the fused template features and the fused search features through a deep cross-correlation operation.

[0099] This embodiment also provides an RGB-T target tracking system based on frequency domain sparse fusion, including: The dual-stream twin backbone network module is used to extract features from the input visible light template image, infrared template image, visible light search image, and infrared search image, respectively. A frequency domain sparse fusion module is used to perform cross-modal feature fusion by using visible light features as query vectors and infrared features as key and value vectors. The frequency domain sparse fusion module includes: a Fourier transform unit for converting the query vector and key vector to the frequency domain; a sparse sampling unit for sparsifying the similarity weight matrix in the frequency domain, retaining the frequency points with the highest saliency scores; an inverse Fourier transform unit for mapping the sparsified frequency domain features back to the spatial domain; and a feature weighting unit for weighting the value vector using the sparse similarity weight matrix and adding it to the visible light features. The classification and regression module is used to perform cross-correlation calculations on the fused template features and fused search features, and output the target location and scale information.

[0100] The specific functions and operation methods of each module of the system are completely consistent with the corresponding steps in the above method embodiments, and will not be repeated here.

[0101] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method based on frequency domain sparse fusion of RGB-T, characterized in that, Includes the following steps: Step 1: Construct a dual-stream twin backbone network and extract features from the input visible light template image, infrared template image, visible light search image, and infrared search image to obtain visible light template features, infrared template features, visible light search features, and infrared search features. Step 2: The extracted visible light template features and visible light search features are convolved to generate query vectors, and the infrared template features and infrared search features are convolved to generate key vectors and value vectors, respectively. Cross-modal feature fusion is then performed through the frequency domain sparse fusion module. Specifically, for the template branch, the visible light template features and infrared template features are input into the frequency domain sparse fusion module to obtain fused template features; for the search branch, the visible light search features and infrared search features are input into the frequency domain sparse fusion module to obtain fused search features. Step 3: Input the fused template features and fused search features into the classification and regression branch, calculate the similarity between the fused template features and fused search features through cross-correlation operation, and output the target location and scale information.

2. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 1, characterized in that, The frequency domain sparse fusion module performs the following operations: Perform Fourier transforms on the query vector and key vector respectively to map them to the frequency domain; The similarity weight matrix between the query vector and the key vector is calculated in the frequency domain, and the similarity weight matrix is ​​sparsified by a sparse sampling mechanism. The sparsified similarity weight matrix is ​​mapped back to the spatial domain through inverse Fourier transform to obtain the sparse similarity weight matrix. The value vector is weighted using the sparse similarity weight matrix and added to the visible light features to form a residual connection, thus obtaining the fusion feature corresponding to this branch.

3. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 2, characterized in that, The sparse sampling mechanism specifically includes: Calculate the significance score for each frequency point in the frequency domain, wherein the significance score is the channel average of the modulus of the complex number; Based on the preset sampling ratio, select the top k frequency points with the highest significance scores; The selected frequency points are retained, while the remaining frequency points are discarded, resulting in the sparsified frequency domain features.

4. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 3, characterized in that, After sparse sampling, the frequency domain features are spectrally recovered to adapt to the input dimension of the inverse Fourier transform. Specifically, this includes: The sparsified frequency domain features are split into real and imaginary parts, and then upsampled to the complete spectrum with the same resolution as the original frequency domain using bilinear interpolation to obtain a sparsified similarity weight matrix of full size.

5. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 2, characterized in that, When calculating the similarity weight matrix between the query vector and the key vector in the frequency domain: After performing Fourier transforms on the query vector and key vector respectively, the conjugate transposes of the query vector and key vector are multiplied element-wise in the frequency domain, and then multiplied by a learnable frequency domain weight parameter to obtain the frequency domain similarity weight matrix.

6. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 5, characterized in that, Intra-channel standardization is performed on the similarity weight matrix obtained by inverse Fourier transform: Calculate the mean and variance of the similarity weight matrix in the spatial dimension, and use the mean and variance to standardize the features of each channel to obtain the standardized sparse similarity weight matrix.

7. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 1, characterized in that, The dual-stream twin backbone network adopts a structure with shallow-layer parameter sharing and deep-layer parameter independence: Shallow networks share parameters between the visible light mode and the infrared mode, while deep networks learn feature representations for their respective modes independently. The template branch and search branch in the dual-stream twin backbone network have the same structure and share weights.

8. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 7, characterized in that, A multi-scale edge enhancement module is embedded between the second and third layers of the dual-stream twin backbone network to strengthen edge features; the multi-scale edge enhancement module performs the following operations: The input features are reconstructed to generate basic edge responses; A progressive average pooling method is used to extract edge response features at different scales. The difference between the pooled features and the original features before pooling is calculated to highlight the edge position. The base edge response and the multi-scale edge-enhanced features are added together through residual connections to output the edge-enhanced features.

9. The RGB-T target tracking method based on frequency domain sparse fusion according to claim 1, characterized in that, The classification and regression branch includes a classification branch and a regression branch: The classification branch is used to output a classification response map, determining whether each location in the search area is the target area; The regression branch is used to output a regression response map, predicting the distance between the target bounding box and the target center; The classification and regression branches calculate the similarity between the fusion template features and the fusion search features through deep cross-correlation operations.

10. An RGB-T target tracking system based on frequency domain sparse fusion, characterized in that, include: The dual-stream twin backbone network module is used to extract features from the input visible light template image, infrared template image, visible light search image, and infrared search image, respectively. The frequency domain sparse fusion module is used to generate query vectors from visible light features through convolution, and generate key vectors and value vectors from infrared features through convolution, and perform cross-modal feature fusion through frequency domain sparse fusion. The frequency domain sparse fusion module includes: a Fourier transform unit for converting the query vector and key vector to the frequency domain; a sparse sampling unit for sparsifying the similarity weight matrix in the frequency domain and retaining the frequency points with the highest saliency scores; an inverse Fourier transform unit for mapping the sparsified frequency domain features back to the spatial domain; and a feature weighting unit for weighting the value vector using the sparse similarity weight matrix and adding it to the visible light features. The classification and regression module is used to perform cross-correlation calculations on the fused template features and fused search features, and output the target location and scale information.

Images