Uniflow RGB-D target tracking method based on LoRA fine tuning and frequency mixed attention

Abstract

The invention discloses a single-flow RGB-D target tracking method based on LoRA fine tuning and frequency mixed attention, and belongs to the technical field of computer vision, and the method comprises the steps: firstly, segmenting an input RGB image and a depth image into image blocks with the same size, and injecting a low-rank adaptation prompt into Patch Embedding through an LoRA mechanism, so as to achieve the feature alignment of two modes; designing a high-low frequency feature extraction module to extract high-frequency edge texture and low-frequency global structure information in parallel; a frequency mixing attention module is introduced into a single-flow Transform, intra-modal self-attention and inter-modal cross attention are executed in sequence, and high and low frequency features are subjected to weighted fusion to consider details and semantics; and completing classification and regression prediction through the prediction head. Compared with an existing double-flow structure, by means of the single-flow architecture design, RGB and depth information can be fused more efficiently, meanwhile, low calculation complexity is kept, and therefore balance between precision and efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a single-stream RGB-D target tracking method based on LoRA (Low-Rank Adaptation) fine-tuning and frequency mixed attention. Background Art

[0002] Existing RGB-D object tracking methods can be divided into two categories according to the backbone network architecture: CNN-based structure and Transformer-based structure.

[0003] CNN-based RGB-D trackers typically utilize complex convolution operations to accurately extract features within a pre-set receptive field, with a particular emphasis on effectively capturing high-frequency information. Specifically, CNN's convolutional layers utilize local connections, allowing their smaller convolution kernels to focus on local regions and exhibit high sensitivity to local variations in the data (such as high-frequency information in an image, including low-level textures and edge details). Furthermore, pooling layers and downsampling operations effectively preserve low-frequency information.

[0004] Among them, Transformer-based RGB-D trackers utilize a unique attention mechanism to build global information associations. Their progressively abstract multi-layered architecture filters out high-frequency noise, thereby prioritizing overall structural features (such as low-frequency information like target color and overall outline). However, this design can lead to insufficient attention to high-frequency features. In complex tracking scenarios, high-frequency features are crucial for distinguishing different instances and are essential for building robust and efficient trackers. In the RGB-D domain, depth information typically contains rich object contours and edge features, which are crucial for enhancing modality complementarity. Most current Transformer-based RGB-D object tracking methods exhibit significant limitations in capturing high-frequency details such as texture and edges, which restricts the tracker's ability to accurately match target details within the search area. Furthermore, Transformer-based RGB-D trackers generally employ a two-stream Transformer architecture, requiring independent parameter branches for the RGB and depth modalities. Compared to single-RGB object tracking methods, the introduction of the additional depth modality doubles the number of model parameters.

[0005] Based on this, the present invention designs a single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention to solve the above problems. Summary of the Invention

[0006] In response to the above-mentioned shortcomings of the existing technology, the present invention provides a single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency-mixed attention. First, the input RGB and depth images are divided into image blocks of the same size, and low-rank adaptation hints are injected into the Patch Embedding through the LoRA mechanism to achieve feature alignment of the two modalities; a high- and low-frequency feature extraction module is designed to extract high-frequency edge texture and low-frequency global structure information in parallel; a frequency-mixed attention module is introduced into the single-stream Transformer, which successively performs intra-modal self-attention and inter-modal cross-attention, and weightedly fuses high- and low-frequency features to take into account both details and semantics; classification and regression prediction are completed through the prediction head.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency mixed attention includes the following steps:

[0009] Step 1: Divide the input RGB template image, depth template image, RGB search image, and depth search image into image blocks and splice them along the channel to form tokens;

[0010] Step 2: Introduce the LoRA hint mechanism in the image patch embedding layer, add a low-rank update term on the basis of the original weight matrix to adjust the weight of the patch embedding, add position encoding and connect the template token and the search area token;

[0011] Step 3: The concatenated tokens are fed into L hybrid Transformer blocks for feature extraction and target information merging. Each hybrid Transformer block consists of two parts: a high- and low-frequency feature extraction module and a frequency mixing attention module.

[0012] Step 4: Use a series of fully convolutional networks to convert the token sequence after the frequency mixed attention operation into a two-dimensional spatial feature map, and output the target classification score map, offset and normalized bounding box.

[0013] Furthermore, the feature map input into the high and low frequency feature extraction module is represented as , represents the token length of the feature map, Represents the feature dimension of the token, which is decomposed into high-frequency feature maps along the channel dimension and low-frequency feature maps , The number of channels representing high-frequency features, The number of channels representing low-frequency features, where ,Will and Assigned to high-frequency mixer and low-frequency mixer for feature extraction respectively; the outputs of low-frequency mixer and high-frequency mixer are connected along the channel dimension; then, high-frequency features and low-frequency features exchange information between template tokens and search area tokens based on depth convolution combined with cross-channel linear layers;

[0014] The frequency mixture attention module performs frequency mixture attention operations.

[0015] Furthermore, step 1 includes the following steps: First, the depth search image and depth template image Convert the image to the same format as the RGB image through color mapping; then, convert the RGB template image to , depth template image , RGB search image and deep search images Divide them into image blocks respectively; Then, splice the RGB template image block and the depth template image block into template tokens ,in, Indicates the length of the template token, Represents the dimension of the token, and concatenates the RGB search area image block and the depth search image block into the search area token ,in, Indicates the length of the search area token.

[0016] Furthermore, the specific steps of step one are as follows:

[0017] 1.1. Express the RGB template image as , the depth template image is represented as , the RGB search image is represented as , the depth search image is represented as ;in, Indicates the height of the template image, Indicates the width of the template image, Indicates the height of the search image, Indicates the width of the search image;

[0018] 1.2. First, the depth search image and depth template image , converted to the same form as the number of RGB image channels through color mapping;

[0019] 1.3. Then, the RGB template image , depth template image , RGB search image and deep search images The resolution is divided into of and Non-overlapping image blocks; where , ;These image patches represent the token inputs of the template and search image respectively;

[0020] RGB template image The divided image blocks form a sequence of RGB template patches; the depth template image The divided image blocks form a sequence of depth template patches; RGB search image The divided image blocks form a sequence of RGB search area patches; the depth search image The divided image blocks form a sequence of deep search patches;

[0021] 1.4. Then, the RGB template patch sequence and the depth template patch sequence are concatenated into template tokens ,in, Indicates the length of the template token, Represents the dimension of the token, and concatenates the RGB search area patch sequence and the depth search patch sequence into a search area token ,in, Indicates the length of the search area token.

[0022] Furthermore, step 2 includes the following steps:

[0023] 2.1. The original weight matrix is , represents the input dimension, represents the output dimension; the low-rank update term is ,in and is a trainable low-rank matrix, is a low-rank dimension, and the weights of the patch embeddings are adjusted as follows: ,in, represents the updated weight matrix;

[0024] 2.2. Use two-dimensional bilinear interpolation to interpolate the pre-trained position embedding and encode the template token position and search area token position encoding Add to template tokens respectively and search zone tokens In the patch, generate template token embedding and search area token embedding , the formula is as follows:

[0025]

[0026]

[0027] in, , Indicates the template token length, , Indicates the search area token length, , , is the dimension of the token; Indicates the Embedding of template patches, Indicates the embedding of search area patches;

[0028] 2.3. Embedding Template Tokens Embedded with search area token Connect, get .

[0029] Furthermore, in step 3, at the high frequency stage, the input Along the channel is divided into and ; Embedded with max pooling and linear layers, is fed into a linear layer and a depthwise convolutional layer:

[0030]

[0031]

[0032] in, and represents the output of the high frequency mixer, represents the fully connected layer, represents the maximum pooling layer, represents a depth-wise convolutional layer;

[0033] In the low frequency stage, the output of the low frequency mixer ;in, represents upsampling, represents the multi-head self-attention mechanism, represents average pooling.

[0034] In step 3, the outputs of the low-frequency mixer and the high-frequency mixer are connected along the channel dimension. The formula is as follows:

[0035]

[0036] in, Indicates splicing.

[0037] Furthermore, in step 3, high-frequency features and low-frequency features exchange information between the template token and the search area token based on a combination of deep convolution and cross-channel linear layers. The formula is as follows:

[0038]

[0039]

[0040] in, is the final output of frequency fusion, It is a high and low frequency feature extraction module. is layer normalization; It represents the fusion result of high and low frequency features after the full connection layer and deep convolution processing. Represents the concatenated output of high and low frequency features.

[0041] In step 3, the formula of the frequency mixing attention module is as follows:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] in, represents the frequency mixed attention operation, represents the attention operation of the template, represents the attention operation of the search area; Represents a template, represents the matrix dimension, represents the activation function; Indicates the search area; Indicates concatenating the value matrices of the template and the search area.

[0048] Furthermore, in step three, The calculation of the attention weight is expanded to:

[0049]

[0050]

[0051]

[0052]

[0053] in, represents the relationship weight between the template and the search area, Represents the relationship weight within the search area; Represents the relationship weight within the template; Represents relationship modeling by aggregating features between images. Indicates that features are extracted by aggregating features within the image. Denotes the fusion template feature extraction via self-attention operation.

[0054] Furthermore, in step 4, the classification score graph , offset and normalized bounding box are obtained by the prediction head, Indicates that the image is converted into The patch size is , H represents the height of the feature map, and W represents the width of the feature map.

[0055] Furthermore, during the training of the model, classification loss and regression loss are used simultaneously, including weighted classification loss, IoU loss and L1 loss. The overall loss function is expressed as:

[0056]

[0057] in, represents the generalized IoU loss, represents the L1 norm loss, represents the classification loss, 、 Represents the weight parameter.

[0058] Compared with the existing technology, the present invention has the following beneficial effects: 1. The present invention introduces low-rank adaptation technology into the RGB-D tracking task for the first time, realizes cross-modal feature fusion through lightweight prompt fine-tuning, and achieves an inference speed of 68.2FPS, which effectively balances the computational burden from additional modality learning and improves the computing speed.

[0059] 2. The high- and low-frequency feature extraction module designed in this invention combines the characteristics of convolution and maximum pooling to capture high-frequency information and strengthen the high-frequency features of the two modalities. At the same time, it uses average pooling and downsampling to process the low-frequency information of the two modalities, thereby enhancing the model's perception of overall information and detailed information.

[0060] 3. The frequency hybrid attention module designed in the present invention includes self-attention to maintain intra-modal features and cross-attention to establish inter-modal and intra-modal relationships, enabling the model to adaptively enhance cross-modal frequency features.

[0061] 4. Experimental results show that our method achieves advanced tracking performance on the DepthTrack and VOT-RGBD2022 datasets, with significantly better inference efficiency than the two-stream model. Not only does it excel in standard testing scenarios, it also exhibits significant advantages in challenging tracking scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0063] Figure 1 This is the overall architecture diagram of the single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention of the present invention.

[0064] Figure 2 This is a structural diagram of the high and low frequency feature extraction module in the present invention.

[0065] Figure 3 This is the structural diagram of the frequency mixing attention module in the present invention.

[0066] Figure 4 This is a visualization of the RGB image tracking results using the method of the present invention and other RGB-D trackers under conditions of drastic lighting changes in the DepthTrack dataset.

[0067] Figure 5 This is a visualization of the depth image tracking results using the method of the present invention and other RGB-D trackers under conditions of drastic lighting changes in the DepthTrack dataset.

[0068] Figure 6 This is a visualization of the RGB image tracking results using the method of the present invention and other RGB-D trackers in the DepthTrack dataset under conditions of rapid target movement and scale changes.

[0069] Figure 7 This is a visualization of the depth image tracking results using the method of the present invention and other RGB-D trackers in the DepthTrack dataset under conditions of rapid target movement and scale changes. DETAILED DESCRIPTION

[0070] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0071] Example 1: Please refer to the accompanying drawings of the specification Figure 1 , a single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency mixed attention, includes the following steps:

[0072] Step 1: Divide the input RGB template image, depth template image, RGB search image, and depth search image into image blocks and splice them along the channel to form tokens;

[0073] The specific steps are as follows:

[0074] 1.1. Express the RGB template image as , the depth template image is represented as , the RGB search image is represented as , the depth search image is represented as .in, Indicates the height of the template image, Indicates the width of the template image, Indicates the height of the search image, Indicates the width of the search image.

[0075] 1.2. First, the depth search image and depth template image , converted to a form with the same number of channels as the RGB image (3 channels) through color mapping.

[0076] 1.3. Then, the RGB template image , depth template image , RGB search image and deep search images The resolution is divided into of and Non-overlapping image blocks. , These image patches represent the token inputs of the template and search image, respectively.

[0077] RGB template image The divided image blocks form a sequence of RGB template patches; the depth template image The divided image blocks form a sequence of depth template patches; RGB search image The divided image blocks form a sequence of RGB search area patches; the depth search image The divided image blocks form a sequence of depth search patches.

[0078] 1.4. Then, the RGB template patch sequence and the depth template patch sequence are concatenated into template tokens ,in, Indicates the length of the template token, Represents the dimension of the token, and concatenates the RGB search area patch sequence and the depth search patch sequence into a search area token ,in, Indicates the length of the search area token.

[0079] Step 2: Introduce the LoRA hint mechanism in the Patch Embedding layer. Add a low-rank update term to the original weight matrix to adjust the weight of the patch embedding, add position encoding, and connect the template token and the search area token.

[0080] Specifically, the method includes the following steps:

[0081] 2.1. The original weight matrix is , represents the input dimension, represents the output dimension; the low-rank update term is ,in and is a trainable low-rank matrix, is a low-rank dimension (usually ), the weights of the patch embeddings are adjusted as follows: ,in, Represents the updated weight matrix, the low-rank matrix rank r is 8;

[0082] 2.2. Use two-dimensional bilinear interpolation to interpolate the pre-trained position embedding and encode the template token position and search area token position encoding Add to template tokens respectively and search zone tokens In the patch, generate template token embedding and search area token embedding , the formula is as follows:

[0083]

[0084]

[0085] in, , Indicates the template token length, , Indicates the search area token length, , , is the dimension of the token; Indicates the Embedding of template patches, Indicates the embedding of search area patches;

[0086] 2.3. Embedding Template Tokens Embedded with search area token Connect, get ;

[0087] Step 3: The concatenated tokens are fed into L hybrid transformer (Hybrid Transformer) blocks for feature extraction and target information merging. Each hybrid transformer block consists of a high-low frequency feature extraction module (HLFM) and a frequency mixed attention module (FMAM), which fully exploit the complementarity between high-frequency details and low-frequency structure. The HLFM combines the advantages of convolution and max pooling to effectively capture high-frequency information in the Transformer architecture and enhance the high-frequency feature representation of the dual modality, thereby improving the model's ability to perceive detailed information. Simultaneously, low-frequency information is processed through average pooling and downsampling operations, achieving efficient global feature extraction. The frequency mixed attention module maintains intra-modal feature consistency through a self-attention mechanism and establishes inter-modal and intra-modal associations using a cross-attention mechanism, enabling the model to adaptively enhance the frequency characteristics of each modality.

[0088] In step 3, the feature map input to the high and low frequency feature extraction module is represented as , represents the token length of the feature map, Represents the feature dimension of the token, which is decomposed into high-frequency feature maps along the channel dimension and low-frequency feature maps , The number of channels representing high-frequency features, The number of channels representing low-frequency features, where ,Will and Assigned to high frequency mixer and low frequency mixer respectively for feature extraction.

[0089] At high frequency, the input Along the channel (channel is divided into channel 1 and channel 2) and Embedded with max pooling and linear layers, is fed into a linear layer and a depthwise convolutional layer:

[0090]

[0091]

[0092] in, and represents the output of the high frequency mixer, represents the fully connected layer, represents the maximum pooling layer, represents a depthwise convolutional layer.

[0093] In the low frequency stage, the output of the low frequency mixer ;in, represents upsampling, represents the multi-head self-attention mechanism, represents average pooling;

[0094] Finally, the outputs of the low- and high-frequency mixers are connected along the channel dimension as follows:

[0095]

[0096] in, Indicates splicing;

[0097] Subsequently, these high- and low-frequency features integrate modality information by exchanging information between template tokens and search area tokens and combining position-based modeling (depthwise convolution) with cross-channel linear layers, as shown below:

[0098]

[0099]

[0100] in, is the final output of frequency fusion, It is a high and low frequency feature extraction module. is layer normalization; It represents the fusion result of high and low frequency features after the full connection layer and deep convolution processing. Represents the concatenated output of high- and low-frequency features;

[0101] In step 3, the frequency mixing attention module performs frequency mixing attention operation;

[0102] Given a target template and a concatenated token sequence of the search region (referring to the concatenated template token embedding Embedded with search area token ,Right now ); The high- and low-frequency feature extraction module extracts high-frequency and low-frequency features and fuses them. Then, these fused features The result is fed into the frequency mixture attention module for further processing. A depthwise convolution is applied to each feature map (i.e., query, key, and value), and the key and value matrices are downsampled to reduce the amount of computation. The processed feature maps are then flattened and linearly projected to generate the query matrix q, key matrix k, and value matrix v required for the frequency mixture attention operation.

[0103] The frequency mixture attention module performs the following calculations:

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] in, represents the frequency mixed attention operation, represents the attention operation of the template, represents the attention operation of the search area; Represents a template, represents the matrix dimension, represents the activation function; Indicates the search area; Indicates concatenating the value matrices of the template and the search area;

[0110] in, The calculation of the attention weight can be expanded as follows:

[0111]

[0112]

[0113]

[0114]

[0115] in, represents the relationship weight between the template and the search area, Represents the relationship weight within the search area; Represents the relationship weight within the template; Represents relationship modeling by aggregating features between images. Indicates that features are extracted by aggregating features within the image. Denotes the fusion template feature extraction via self-attention operation.

[0116] Step 4: Use a series of fully convolutional networks (FCNs) to transform the token sequence (frequency mixed attention operation) ) is converted into a two-dimensional spatial feature map and outputs a target classification score map (indicating the target location), offset, and normalized bounding box.

[0117] Specifically, the classification score map , offset and normalized bounding box are obtained by the prediction head, Indicates that the image is converted into The patch size is , H represents the height of the feature map, and W represents the width of the feature map.

[0118] During the training process of the model, classification loss and regression loss are used simultaneously, including weighted classification loss, IoU loss and L1 loss. The overall loss function is expressed as:

[0119]

[0120] in, represents the generalized IoU loss, represents the L1 norm loss, represents the classification loss (weighted focal loss), 、 Represents the weight parameter, value =5, =2.

[0121] This paper introduces low-rank adaptation technology into the RGB-D tracking task for the first time, realizes cross-modal feature fusion through lightweight prompt fine-tuning, and achieves an inference speed of 68.2FPS, effectively balancing the computational burden from additional modality learning and improving the computing speed.

[0122] The high- and low-frequency feature extraction module designed in this invention combines the characteristics of convolution and maximum pooling to capture high-frequency information and strengthen the high-frequency features of the two modalities. At the same time, it uses average pooling and downsampling to process the low-frequency information of the two modalities, enhancing the model's perception of overall information and detailed information.

[0123] The frequency hybrid attention module designed in the present invention includes self-attention to maintain intra-modal features and cross-attention to establish inter-modal and intra-modal relationships, enabling the model to adaptively enhance cross-modal frequency features.

[0124] Experimental Example 1: To better illustrate the effectiveness of the tracking method of the present invention, a comprehensive comparison is conducted between the present invention and the current advanced RGB-D tracker on the DepthTrack and VOT-RGBD2022 datasets. The results are shown in Tables 1 and 2.

[0125] Table 1 Comparison results between the proposed method and the current state-of-the-art RGB-D tracker on the DepthTrack dataset

[0126]

[0127] In the table, DAL - A Deep Depth-aware Long-term Tracker;

[0128] DeT — DepthTrack: Unveiling the Power of RGBD Tracking

[0129] SPT — RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking

[0130] ViPT—Visual Prompt Multi-Modal Tracking;

[0131] VADT — Visual Adapt for RGBD Tracking

[0132] SDSTrack — Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking

[0133] UBPT——Unidirectional and Bidirectional Prompts for RGBD Tracking.

[0134] In the table, CA3DMS, MixFormer, ATCAIS, TSDM, ProTrack, Un-Track, and OneTracker are also trackers that have been disclosed in the prior art.

[0135] As can be seen, our method achieves comparable performance to current state-of-the-art methods, achieving excellent precision, recall, and F-score scores of 0.618, 0.610, and 0.614, respectively. Furthermore, our method can run at a speed of up to 68 FPS.

[0136] Table 2 Comparison results between the proposed method and the current state-of-the-art RGB-D tracker on the VOT-RGBD2022 dataset

[0137]

[0138] In the table, DiMP, OSTrack, and SBT_RGBD are also trackers that have been disclosed in the prior art.

[0139] It can be seen that the method of the present invention achieved excellent performance of 0.722, 0.821 and 0.872 in the three key indicators of EAO, accuracy and robustness, respectively. This result fully demonstrates the generalization ability of the present invention in different dataset scenarios. Compared with other RGB-D object trackers, the present invention maintains a leading position in all performance indicators. This advantage is mainly due to the single-stream architecture design of the present invention, which enables it to more efficiently fuse RGB and depth information while maintaining low computational complexity, thereby achieving a balance between accuracy and efficiency.

[0140] Experimental results show that our method achieves advanced tracking performance on the DepthTrack and VOT-RGBD2022 datasets, with significantly better inference efficiency than the two-stream model. It not only excels in standard testing scenarios but also has significant advantages in challenging tracking scenarios.

[0141] Experimental Example 2:

[0142] Figure 4 and Figure 5 The performance of the proposed method under conditions of drastic lighting fluctuations was tested. The experiment involved a challenging scenario involving a lighted balloon moving in a complex background lighting environment. The results demonstrated that the proposed method effectively handles the interference between target and background lighting, and its adaptive feature fusion mechanism significantly outperforms comparable tracking algorithms.

[0143] Figure 6 and Figure 7The proposed method was evaluated for its performance under conditions of rapid target movement and scale changes. The experimental sequences involved targets moving rapidly in different directions, accompanied by significant scale changes and motion blur. Through its multi-scale feature extraction and motion prediction modules, the proposed method was able to accurately predict target positions, demonstrating excellent robustness.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. Single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention, characterized by: The following steps are involved: Step 1: Divide the input RGB template image, depth template image, RGB search image, and depth search image into image blocks and splice them along the channel to form tokens; Step 2: Introduce the LoRA hint mechanism in the image patch embedding layer, add a low-rank update term on the basis of the original weight matrix to adjust the weight of the patch embedding, add position encoding and connect the template token and the search area token; Step 3: The concatenated tokens are fed into L hybrid Transformer blocks for feature extraction and target information merging. Each hybrid Transformer block consists of two parts: a high- and low-frequency feature extraction module and a frequency mixing attention module. Step 4: Use a series of fully convolutional networks to convert the token sequence after the frequency mixed attention operation into a two-dimensional spatial feature map, and output the target classification score map, offset and normalized bounding box.

2. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 1 is characterized in that The input feature map in the high and low frequency feature extraction module is represented as , represents the token length of the feature map, Represents the feature dimension of the token, which is decomposed into high-frequency feature maps along the channel dimension and low-frequency feature maps , The number of channels representing high-frequency features, The number of channels representing low-frequency features, where ,Will and Assigned to high frequency mixer and low frequency mixer respectively for feature extraction; Connect the outputs of the low-frequency mixer and the high-frequency mixer along the channel dimension; Subsequently, high-frequency features and low-frequency features exchange information between template tokens and search area tokens based on a combination of depthwise convolution and cross-channel linear layers; The frequency mixture attention module performs frequency mixture attention operations.

3. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 2 is characterized in that Step 1 includes the following steps: First, the depth search image and depth template image Convert the image to the same format as the RGB image through color mapping; then, convert the RGB template image to , depth template image , RGB search image and deep search images Divide them into image blocks respectively; Then, splice the RGB template image block and the depth template image block into template tokens ,in, Indicates the length of the template token, Represents the dimension of the token, and concatenates the RGB search area image block and the depth search image block into the search area token ,in, Indicates the length of the search area token.

4. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 3 is characterized in that Step 1 The specific steps are as follows: 1.

1. Express the RGB template image as , the depth template image is represented as , the RGB search image is represented as , the depth search image is represented as ;in, Indicates the height of the template image, Indicates the width of the template image, Indicates the height of the search image, Indicates the width of the search image; 1.

2. First, the depth search image and depth template image , converted to the same form as the number of RGB image channels through color mapping; 1.

3. Then, the RGB template image , depth template image , RGB search image and deep search images The resolution is divided into of and Non-overlapping image blocks; where , ;These image patches represent the token inputs of the template and search image respectively; RGB template image The divided image blocks form a sequence of RGB template patches; the depth template image The divided image blocks form a sequence of depth template patches; RGB search image The divided image blocks form a sequence of RGB search area patches; the depth search image The divided image blocks form a sequence of deep search patches; 1.

4. Then, the RGB template patch sequence and the depth template patch sequence are concatenated into template tokens ,in, Indicates the length of the template token, Represents the dimension of the token, and concatenates the RGB search area patch sequence and the depth search patch sequence into a search area token ,in, Indicates the length of the search area token.

5. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 4 is characterized in that Step 2 includes the following steps: 2.

1. The original weight matrix is , represents the input dimension, represents the output dimension; the low-rank update term is ,in and is a trainable low-rank matrix, is a low-rank dimension, and the weights of the patch embeddings are adjusted as follows: ,in, represents the updated weight matrix; 2.

2. Use two-dimensional bilinear interpolation to interpolate the pre-trained position embedding and encode the template token position and search area token position encoding Add to template tokens respectively and search zone tokens In the patch, generate template token embedding and search area token embedding , the formula is as follows: ； ； in, , Indicates the template token length, , Indicates the search area token length, , , is the dimension of the token; Indicates the Embedding of template patches, Indicates the j embedding of search area patches; 2.

3. Embedding Template Tokens Embedded with search area token Connect, get .

6. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 5 is characterized in that In step 3, at the high frequency stage, the input Along the channel is divided into and ; Embedded with max pooling and linear layers, is fed into a linear layer and a depthwise convolutional layer: ； ； in, and represents the output of the high frequency mixer, represents the fully connected layer, represents the maximum pooling layer, represents a depth-wise convolutional layer; In the low frequency stage, the output of the low frequency mixer ;in, represents upsampling, represents the multi-head self-attention mechanism, represents average pooling; In step 3, the outputs of the low-frequency mixer and the high-frequency mixer are connected along the channel dimension. The formula is as follows: ； in, Indicates splicing.

7. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 6 is characterized in that In step 3, high-frequency features and low-frequency features exchange information between the template token and the search area token based on a combination of deep convolution and cross-channel linear layers. The formula is as follows: ； ； in, is the final output of frequency fusion, It is a high and low frequency feature extraction module. is layer normalization; It represents the fusion result of high and low frequency features after the full connection layer and deep convolution processing. Represents the concatenated output of high- and low-frequency features; In step 3, the formula of the frequency mixing attention module is as follows: ； ； ； ； ； in, represents the frequency mixed attention operation, represents the attention operation of the template, represents the attention operation of the search area; Represents a template, represents the matrix dimension, represents the activation function; Indicates the search area; Indicates concatenating the value matrices of the template and the search area.

8. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 7 is characterized in that In step three, The calculation of the attention weight is expanded to: ； ； ； ； in, represents the relationship weight between the template and the search area, Represents the relationship weight within the search area; Represents the relationship weight within the template; Represents relationship modeling by aggregating features between images. Indicates that features are extracted by aggregating features within the image. Denotes the fusion template feature extraction via self-attention operation.

9. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 8 is characterized in that In step 4, the classification score graph , offset and normalized bounding box are obtained by the prediction head, Indicates that the image is converted into The patch size is , H represents the height of the feature map, and W represents the width of the feature map.

10. The single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention according to claim 9 is characterized in that During the training of the model, classification loss and regression loss are used simultaneously, including weighted classification loss, IoU loss and L1 loss. The overall loss function is expressed as: ； in, represents the generalized IoU loss, represents the L1 norm loss, represents the classification loss, 、 Represents the weight parameter.

Images