A Single-Stream RGB-D Target Tracking Method Based on LoRA Fine-Tuning and Frequency Hybrid Attention

Abstract

The application discloses a single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency mixed attention, and belongs to the technical field of computer vision. Firstly, input RGB and depth images are respectively divided into image blocks of the same size, and low-rank adaptive prompts are injected in Patch Embedding through a LoRA mechanism to realize feature alignment of two modalities. A high-low frequency feature extraction module is designed to extract high-frequency edge texture and low-frequency global structure information in parallel. A frequency mixed attention module is introduced in the single-stream Transformer to perform intra-modal self-attention and inter-modal cross-attention in sequence, and high-low frequency features are weighted and fused to balance details and semantics. Classification and regression prediction are completed through a prediction head. Compared with the existing double-stream structure, the single-stream architecture design of the application can more efficiently fuse RGB and depth information while maintaining a relatively low computational complexity, thereby achieving a balance between precision and efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a single-stream RGB-D target tracking method based on LoRA (Low-Rank Adaptation) fine-tuning and frequency-mixed attention. Background Technology

[0002] Existing RGB-D target tracking methods can be divided into two categories based on the backbone network architecture: CNN-based structures and Transformer-based structures.

[0003] Among them, CNN-based RGB-D trackers typically utilize complex convolutional operations to accurately extract features within a predefined receptive field, with a particular focus on effectively capturing high-frequency information. Specifically, the convolutional layers of CNNs employ local connectivity, and their smaller convolutional kernels can focus on local regions, exhibiting high sensitivity to local changes in data (such as high-frequency information in images, including low-level textures and edge details). Simultaneously, pooling layers and downsampling operations effectively preserve low-frequency information.

[0004] Among them, Transformer-based RGB-D trackers construct global information associations through a unique attention mechanism. Their progressively abstract, multi-layered structure filters high-frequency noise, thus focusing more on overall structural features (such as target color and overall contour, which are low-frequency information). However, this design may lead to insufficient attention to high-frequency features. In complex tracking scenarios, high-frequency features are crucial for distinguishing different instances and are indispensable elements for building robust and efficient trackers. Especially in the RGB-D domain, depth information typically contains rich target contours and edge features, which are significant for enhancing modal complementarity. Most current Transformer-based RGB-D target tracking methods have significant limitations in capturing high-frequency details such as texture and edges, which restricts the tracker's ability to accurately match target details within the search region. Furthermore, Transformer-based RGB-D trackers generally employ a two-stream Transformer structure, requiring separate parameter branches for RGB and depth modalities. Compared to single-RGB target tracking methods, the introduction of the additional depth modality leads to a doubling of model parameters.

[0005] Based on this, the present invention designs a single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency hybrid attention to solve the above problems. Summary of the Invention

[0006] To address the aforementioned shortcomings of existing technologies, this invention provides a single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency-mixed attention. First, the input RGB and depth images are segmented into image patches of equal size. Low-rank adaptation cues are injected into the Patch Embedding using the LoRA mechanism to achieve feature alignment between the two modalities. A high- and low-frequency feature extraction module is designed to extract high-frequency edge textures and low-frequency global structural information in parallel. A frequency-mixed attention module is introduced into the single-stream Transformer, sequentially performing intra-modal self-attention and inter-modal cross-attention, and weighted fusing high- and low-frequency features to balance detail and semantics. Classification and regression prediction are then performed using a prediction head.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency-hybrid attention includes the following steps:

[0009] Step 1: Divide the input RGB template image, depth template image, RGB search image, and depth search image into image blocks, and stitch them together along the channels to form tokens;

[0010] Step 2: Introduce the LoRA hint mechanism in the image patch embedding layer. Add a low-rank update term to the original weight matrix to adjust the weights of the patch embedding, add position encoding and connect the template token and the search region token.

[0011] Step 3: Input the connected token into L hybrid Transformer blocks for feature extraction and target information merging. Each hybrid Transformer block contains two parts: a high-frequency and low-frequency feature extraction module and a frequency mixing attention module.

[0012] Step 4: Use a series of fully convolutional networks to convert the token sequence after frequency mixing attention operation into a two-dimensional spatial feature map, and output the target classification score map, offset and normalized bounding box.

[0013] Furthermore, the feature map input to the high- and low-frequency feature extraction module is represented as follows: , Indicates the token length of the feature map. The token's feature dimension is represented by its decomposition into high-frequency feature maps along the channel dimension. and low-frequency feature maps , The number of channels representing high-frequency characteristics. The number of channels representing low-frequency characteristics, where ,Will and Feature extraction is performed on the high-frequency mixer and the low-frequency mixer respectively; the outputs of the low-frequency mixer and the high-frequency mixer are concatenated along the channel dimension; subsequently, the high-frequency features and the low-frequency features exchange information between the template token and the search region token, based on a combination of depthwise convolution and cross-channel linear layers;

[0014] The frequency mixing attention module performs frequency mixing attention operations.

[0015] Furthermore, step one includes the following steps: First, the depth search image... and depth template image The image is converted to a form with the same number of channels as the RGB image through color mapping; then, the RGB template image is... Depth template image RGB search image and depth search images The image is divided into image blocks; then, the RGB template image blocks and the depth template image blocks are concatenated to form a template token. ,in, Indicates the length of the template token. This represents the dimension of the token, and it concatenates the RGB search region image patch and the depth search image patch to form the search region token. ,in, Indicates the length of the search area token.

[0016] Furthermore, the specific steps of step one are as follows:

[0017] 1.1 Represent the RGB template image as The depth template image is represented as RGB search image representation The depth search image is represented as ;in, Indicates the height of the template image. Indicates the width of the template image. Indicates the height of the search image. Indicates the width of the search image;

[0018] 1.2 First, the depth search image and depth template image It is converted to a form with the same number of RGB image channels through color mapping;

[0019] 1.3. Then, use the RGB template image. Depth template image RGB search image and depth search images Divided into resolutions of of and Non-overlapping image patches; where, , These image blocks represent the token inputs for the template and the search image, respectively.

[0020] RGB template image The divided image blocks form an RGB template patch sequence; depth template image The divided image patches form a depth template patch sequence; RGB search image The divided image patches form a sequence of RGB search regions; depth search image The divided image patches form a depth search patch sequence;

[0021] 1.4 Then, concatenate the RGB stencil patch sequence and the depth stencil patch sequence into a stencil token. ,in, Indicates the length of the template token. This represents the dimension of the token, and concatenates the RGB search region patch sequence and the depth search patch sequence to form the search region token. ,in, Indicates the length of the search area token.

[0022] Furthermore, step two includes the following steps:

[0023] 2.1 The original weight matrix is , Indicates the input dimension. Indicates the output dimension; the low-rank update term is ,in and It is a trainable low-rank matrix. Since it is a low-rank dimension, the method for adjusting the weights of patch embeddings is as follows: ,in, This represents the updated weight matrix;

[0024] 2.2. Two-dimensional bilinear interpolation is used to interpolate the pre-trained position embeddings to encode the template token positions. and search area token location encoding Add to template tokens respectively and search area token In the patch, template tokens are generated and embedded. and search area token embedding The formula is as follows:

[0025]

[0026]

[0027] in, , Indicates the length of the template token. , Indicates the length of the search area token. , , Let be the dimension of the token; Indicates the first Embedding of a template patch Indicates the first Embedding of a search area patch;

[0028] 2.3 Embed the template token Embedded with search area token Connect, get .

[0029] Furthermore, in step three, during the high-frequency phase, the input... Divided along the channel and ; Embedded max pooling and linear layers, It is fed into linear layers and deep convolutional layers:

[0030]

[0031]

[0032] in, and This indicates the output of the high-frequency mixer. Indicates a fully connected layer. Indicates the max pooling layer. Indicates a deep convolutional layer;

[0033] In the low-frequency phase, the output of the low-frequency mixer ;in, Indicates upsampling, This indicates a multi-head self-attention mechanism. This indicates average pooling.

[0034] In step three, the outputs of the low-frequency mixer and the high-frequency mixer are connected along the channel dimensions, as shown in the following formula:

[0035]

[0036] in, Indicates splicing.

[0037] Furthermore, in step three, high-frequency and low-frequency features exchange information between the template token and the search region token, based on a combination of depthwise convolution and cross-channel linear layers, as shown in the following formula:

[0038]

[0039]

[0040] in, The final output of frequency fusion, It is a high- and low-frequency feature extraction module. It is layer normalization; This represents the result of fusing high- and low-frequency features after processing with fully connected layers and depthwise convolutions. This represents the spliced ​​output of high and low frequency characteristics.

[0041] In step three, the formula for the frequency mixing attention module is as follows:

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] in, This indicates a frequency-mixed attention operation. This indicates the attention operation of the template. This indicates attention operations within the search area; Represents a template. Represents the matrix dimension. Indicates the activation function; Indicates the search area; This indicates that the value matrix of the template and the search region will be concatenated.

[0048] Furthermore, in step three, The calculation of attention weights is extended as follows:

[0049]

[0050]

[0051]

[0052]

[0053] in, This indicates the weight of the relationship between the template and the search area. Indicates the relation weights within the search region; Indicates the relational weights within the template; This approach represents modeling relationships by aggregating features between images. This indicates that features are extracted by aggregating features within the image. This indicates the extraction of fusion template features through self-attention operations.

[0054] Furthermore, in step four, the classification score chart... Offset and normalized bounding box All are obtained from the prediction head. This indicates that the image is converted during the embedding process. The patch size, where H represents the height of the feature map and W represents the width of the feature map.

[0055] Furthermore, during model training, both classification and regression losses were used, including weighted classification loss, IoU loss, and L1 loss. The overall loss function is expressed as:

[0056]

[0057] in, This represents the generalized IoU loss. Indicates L1 norm loss, Represents classification loss, , This represents the weighting parameter.

[0058] Compared with the prior art, the beneficial effects of this invention are as follows: 1. This invention introduces low-rank adaptation technology into the RGB-D tracking task for the first time, and achieves cross-modal feature fusion through lightweight prompts and fine-tuning, with an inference speed of 68.2 FPS, effectively balancing the computational burden from additional modal learning and improving the computing speed.

[0059] 2. The high- and low-frequency feature extraction module designed in this invention combines the characteristics of convolution and max pooling to capture high-frequency information and enhance the high-frequency features of the two modalities. At the same time, it uses average pooling and downsampling to process the low-frequency information of the two modalities, thereby enhancing the model's perception of overall information and detailed information.

[0060] 3. The frequency hybrid attention module designed in this invention includes self-attention to maintain intramodal features and cross-attention to establish intermodal and intramodal relationships, enabling the model to adaptively enhance cross-modal frequency features.

[0061] 4. Experimental results show that the method of this invention achieves state-of-the-art tracking performance on the DepthTrack and VOT-RGBD2022 datasets, with inference efficiency significantly superior to the two-stream model. It not only performs excellently in standard test scenarios but also demonstrates significant advantages in challenging tracking scenarios. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a diagram illustrating the overall architecture of the single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency-mixed attention according to the present invention.

[0064] Figure 2 This is a structural diagram of the high- and low-frequency feature extraction module in this invention.

[0065] Figure 3 This is a structural diagram of the frequency mixing attention module in this invention.

[0066] Figure 4 This is a visualization of the RGB image tracking results using the method of this invention and other RGB-D trackers under drastic lighting conditions in the DepthTrack dataset.

[0067] Figure 5 This is a visualization of the Depth image tracking results under drastic lighting conditions in the DepthTrack dataset, using the method of this invention and other RGB-D trackers.

[0068] Figure 6 This is a visualization of the RGB image tracking results using the method of this invention and other RGB-D trackers in the DepthTrack dataset under conditions of rapid target movement and scale changes.

[0069] Figure 7 This is a visualization of the Depth image tracking results using the method of this invention and other RGB-D trackers in the DepthTrack dataset under conditions of rapid target movement and scale changes. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0071] Example 1: Please refer to the accompanying drawings in the instruction manual. Figure 1 A single-stream RGB-D target tracking method based on LoRA fine-tuning and frequency-mixed attention includes the following steps:

[0072] Step 1: Divide the input RGB template image, depth template image, RGB search image, and depth search image into image blocks, and stitch them together along the channels to form tokens;

[0073] The specific steps are as follows:

[0074] 1.1 Represent the RGB template image as The depth template image is represented as RGB search image representation The depth search image is represented as .in, Indicates the height of the template image. Indicates the width of the template image. Indicates the height of the search image. This indicates the width of the search image.

[0075] 1.2 First, the depth search image and depth template image It is converted to a form with the same number of channels as the RGB image (3 channels) through color mapping.

[0076] 1.3. Then, use the RGB template image. Depth template image RGB search image and depth search images Divided into resolutions of of and Non-overlapping image patches. Among them, , These image blocks represent the token inputs for the template and the search image, respectively.

[0077] RGB template image The divided image blocks form an RGB template patch sequence; depth template image The divided image patches form a depth template patch sequence; RGB search image The divided image patches form a sequence of RGB search regions; depth search image The divided image blocks form a depth search patch sequence.

[0078] 1.4 Then, concatenate the RGB stencil patch sequence and the depth stencil patch sequence into a stencil token. ,in, Indicates the length of the template token. This represents the dimension of the token, and concatenates the RGB search region patch sequence and the depth search patch sequence to form the search region token. ,in, Indicates the length of the search area token.

[0079] Step 2: Introduce the LoRA hint mechanism in the Patch Embedding layer (image patch embedding layer), add a low-rank update term to the original weight matrix to adjust the weight of patch embedding, add position encoding and connect the template token and the search region token;

[0080] Specifically, it includes the following steps:

[0081] 2.1 The original weight matrix is , Indicates the input dimension. Indicates the output dimension; the low-rank update term is ,in and It is a trainable low-rank matrix. It is a low-rank dimension (usually) The method for adjusting the weight of patch embedding is as follows: ,in, This represents the updated weight matrix, where the rank r of the low-rank matrix is ​​8.

[0082] 2.2. Two-dimensional bilinear interpolation is used to interpolate the pre-trained position embeddings to encode the template token positions. and search area token location encoding Add to template tokens respectively and search area token In the patch, template tokens are generated and embedded. and search area token embedding The formula is as follows:

[0083]

[0084]

[0085] in, , Indicates the length of the template token. , Indicates the length of the search area token. , , Let be the dimension of the token; Indicates the first Embedding of a template patch Indicates the first Embedding of a search area patch;

[0086] 2.3 Embed the template token Embedded with search area token Connect, get ;

[0087] Step 3: The concatenated tokens are input into L hybrid Transformer (Hyformer) blocks for feature extraction and target information merging. Each hybrid Transformer block contains two parts: a High-Low Frequency Module (HLFM) and a Frequency Mixed Attention Module (FMAM), which fully leverages the complementarity between high-frequency details and low-frequency structures. The HLFM combines the advantages of convolution and max pooling to effectively capture high-frequency information in the Transformer architecture, strengthening the high-frequency feature representation of the dual modality and thus improving the model's ability to perceive detailed information. Simultaneously, it processes low-frequency information through average pooling and downsampling operations, achieving efficient extraction of global features. The Frequency Mixed Attention Module maintains the consistency of features within a modality through self-attention and establishes inter-modal and intra-modal associations using cross-attention, enabling the model to adaptively enhance the frequency features of each modality.

[0088] In step three, the feature map input to the high- and low-frequency feature extraction module is represented as follows: , Indicates the token length of the feature map. The token's feature dimension is represented by its decomposition into high-frequency feature maps along the channel dimension. and low-frequency feature maps , The number of channels representing high-frequency characteristics. The number of channels representing low-frequency characteristics, where ,Will and Feature extraction is performed separately by the high-frequency mixer and the low-frequency mixer.

[0089] In the high-frequency phase, the input Divided along the channel (the channel is divided into channel 1 and channel 2) and Embedded max pooling and linear layers, It is fed into linear layers and deep convolutional layers:

[0090]

[0091]

[0092] in, and This indicates the output of the high-frequency mixer. Indicates a fully connected layer. Indicates the max pooling layer. This indicates a deep convolutional layer.

[0093] In the low-frequency phase, the output of the low-frequency mixer ;in, Indicates upsampling, This indicates a multi-head self-attention mechanism. Indicates average pooling;

[0094] Finally, the outputs of the low-frequency and high-frequency mixers are connected along the channel dimensions, as shown in the following formula:

[0095]

[0096] in, Indicates splicing;

[0097] Subsequently, these high- and low-frequency features integrate modal information by exchanging information between template tokens and search region tokens, and by combining location-based modeling (depth convolution) with cross-channel linear layers, as shown below:

[0098]

[0099]

[0100] in, The final output of frequency fusion, It is a high- and low-frequency feature extraction module. It is layer normalization; This represents the result of fusing high- and low-frequency features after processing with fully connected layers and depthwise convolutions. The spliced ​​output represents the high and low frequency characteristics;

[0101] In step three, the frequency mixing attention module performs frequency mixing attention operations;

[0102] Given a target template and a concatenated token sequence for the search region (referring to the concatenated template token embeddings) Embedded with search area token ,Right now The high- and low-frequency feature extraction module extracts high-frequency and low-frequency features and fuses them. Then, these fused features... The data is fed into the frequency-mixed attention module for further processing. A depthwise convolution is applied to each feature map (i.e., query, key, and value), and the key and value matrices are downsampled to reduce computation. Then, the processed feature maps are flattened and linearly projected to generate the query matrix q, key matrix k, and value matrix v required for the frequency-mixed attention operation.

[0103] The frequency mixing attention module performs the following calculations:

[0104]

[0105]

[0106]

[0107]

[0108]

[0109] in, This indicates a frequency-mixed attention operation. This indicates the attention operation of the template. This indicates attention operations within the search area; Represents a template. Represents the matrix dimension. Indicates the activation function; Indicates the search area; This indicates that the value matrices of the template and the search region will be concatenated;

[0110] in, The calculation of attention weights can be expanded as follows:

[0111]

[0112]

[0113]

[0114]

[0115] in, This indicates the weight of the relationship between the template and the search area. Indicates the relation weights within the search region; Indicates the relational weights within the template; This approach represents modeling relationships by aggregating features between images. This indicates that features are extracted by aggregating features within the image. This indicates the extraction of fusion template features through self-attention operations.

[0116] Step 4: Use a series of fully convolutional networks (FCNs) to mix the token sequence (frequency after attention operation) The target is converted into a two-dimensional spatial feature map and outputs a target classification score map (indicating the target location), offset, and normalized bounding box.

[0117] Specifically, the classification score chart Offset and normalized bounding box All are obtained from the prediction head. This indicates that the image is converted during the embedding process. The patch size, where H represents the height of the feature map and W represents the width of the feature map.

[0118] During model training, both classification and regression losses were used, including weighted classification loss, IoU loss, and L1 loss. The overall loss function is expressed as:

[0119]

[0120] in, This represents the generalized IoU loss. Indicates L1 norm loss, This represents the classification loss (weighted focus loss). , This represents the weight parameter, and its value is... =5, =2.

[0121] This invention introduces low-rank adaptation technology into RGB-D tracking tasks for the first time, achieving cross-modal feature fusion through lightweight cue fine-tuning, with an inference speed of 68.2 FPS. This effectively balances the computational burden from additional modality learning and improves the computing speed.

[0122] The high- and low-frequency feature extraction module designed in this invention combines the characteristics of convolution and max pooling to capture high-frequency information and enhance the high-frequency features of the two modalities. At the same time, it uses average pooling and downsampling to process the low-frequency information of the two modalities, thereby enhancing the model's perception of overall information and detailed information.

[0123] The frequency hybrid attention module designed in this invention includes self-attention to maintain intramodal features and cross-attention to establish intermodal and intramodal relationships, enabling the model to adaptively enhance cross-modal frequency features.

[0124] Experimental Example 1: To better illustrate the effectiveness of the tracking method of the present invention, a comprehensive comparison was made between the present invention and the current state-of-the-art RGB-D trackers on the DepthTrack and VOT-RGBD2022 datasets. The results are shown in Tables 1-2.

[0125] Table 1. Comparison results of the present invention with current state-of-the-art RGB-D trackers on the DepthTrack dataset.

[0126]

[0127] In the table, DAL stands for A Deep Depth-aware Long-term Tracker.

[0128] DeT—DepthTrack: Unveiling the Power of RGBD Tracking;

[0129] SPT—RGBD1K: A Large-scale Dataset and Benchmark for RGB-D Object Tracking.

[0130] ViPT – Visual Prompt Multi-Modal Tracking;

[0131] VADT – Visual Adapt for RGBD Tracking;

[0132] SDSTrack—Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking;

[0133] UBPT – Unidirectional and Bidirectional Prompts for RGBD Tracking.

[0134] In the table, CA3DMS, MixFormer, ATCAIS, TSDM, ProTrack, Un-Track, and OneTracker are all existing trackers that have been publicly disclosed.

[0135] As can be seen, the method of this invention achieves a performance level comparable to current advanced methods, with excellent results of 0.618, 0.610, and 0.614 in precision, recall, and F-score, respectively. Furthermore, the method of this invention can achieve a speed of 68 FPS.

[0136] Table 2. Comparison results of the present invention with current advanced RGB-D trackers on the VOT-RGBD2022 dataset.

[0137]

[0138] In the table, DiMP, OSTrack, and SBT_RGBD are all publicly available trackers.

[0139] As can be seen, the method of this invention achieves excellent performance of 0.722, 0.821, and 0.872 in the three key metrics of EAO, accuracy, and robustness, respectively. This result fully demonstrates the generalization ability of this invention in different dataset scenarios. Compared with other RGB-D object trackers, this invention maintains a leading position in all performance metrics. This advantage is mainly due to the single-stream architecture design of this invention, which enables it to more efficiently fuse RGB and depth information while maintaining low computational complexity, thus achieving a balance between accuracy and efficiency.

[0140] Experimental results show that the method of this invention achieves state-of-the-art tracking performance on the DepthTrack and VOT-RGBD2022 datasets, with inference efficiency significantly superior to the two-stream model. It not only performs well in standard test scenarios but also demonstrates significant advantages in challenging tracking scenarios.

[0141] Experimental Example 2:

[0142] Figure 4 and Figure 5 The performance of this invention under conditions of drastic lighting changes was tested. The experiment simulated a challenging scenario where a balloon with lights moved through a complex background lighting environment. The results show that this invention effectively handles the mutual interference between the target and background lighting, and its designed adaptive feature fusion mechanism significantly outperforms the comparison algorithms in tracking performance.

[0143] Figure 6 and Figure 7The performance of this invention under conditions of rapid target movement and scale changes was evaluated. In the experimental sequence, the target moved rapidly in different directions, accompanied by significant scale changes and motion blur. Through its multi-scale feature extraction and motion prediction modules, this invention was still able to accurately predict the target position, demonstrating excellent robustness.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention, characterized in that, Comprising the following steps: Step one, the input RGB template image, depth template image, RGB search image, depth search image are divided into image blocks, and the tokens are formed by channel splicing; Step two, introduce LoRA prompt mechanism in image block embedding layer, add a low rank update item on the basis of original weight matrix to adjust the weight of patch embedding, add position coding and connect template token and search area token; Step three, input the connected token into L mixed Transformer block for feature extraction and target information merging, each mixed Transformer block contains high and low frequency feature extraction module and frequency mixed attention module; High frequency feature and low frequency feature exchange information between template token and search area token, based on depth convolution and cross channel linear layer combination, formula as follows: ； ； wherein, is the final output of frequency fusion, is a high-low frequency feature extraction module, is layer normalization; represents the high-low frequency feature fusion result after full connection layer and deep convolution processing, represents the spliced output of high-low frequency features; The formula of the frequency mixed attention module is as follows: ； ； ； ； ； wherein, denotes a frequency-mixed attention operation, denotes an attention operation for a template, denotes an attention operation for a search region; denotes a template, denotes a matrix dimension, denotes an activation function; denotes a search region; denotes concatenating value matrices of the template and the search region; Step four, use a series of fully convolutional network to convert the token sequence after frequency mixed attention operation into two-dimensional spatial feature map, and output target classification score map, offset and normalized bounding box.

2. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 1, characterized in that, The input feature map in the high-low frequency feature extraction module is represented as , Token length representing the feature map, Token feature dimension, which is decomposed into high-frequency feature map and low-frequency feature map , Number of channels representing high-frequency features, Number of channels representing low-frequency features, wherein , respectively and are assigned to the high-frequency mixer and the low-frequency mixer for feature extraction; The outputs of the low frequency mixer and the high frequency mixer are connected along the channel dimension; Then, high frequency feature and low frequency feature exchange information between template token and search area token, based on depth convolution and cross channel linear layer combination; The frequency mixed attention module performs frequency mixed attention operation.

3. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 2, wherein, Step one includes the following steps: first, the depth search image and the depth template image are converted into the same form as the RGB image channel number by color mapping; then, the RGB template image , the depth template image , the RGB search image and the depth search image are respectively divided into image blocks; then, the RGB template image block and the depth template image block are spliced into a template token , wherein, denotes the length of the template token, denotes the dimension of the token, and the RGB search area image block and the depth search image block are spliced into a search area token , wherein, denotes the length of the search area token.

4. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 3, characterized in that, The specific steps of step one are as follows: 1.

1. Represent the RGB template image as , the depth template image as , the RGB search image as , and the depth search image as ; where represents the height of the template image, represents the width of the template image, represents the height of the search image, represents the width of the search image. 1.

2. First, the depth search image and the depth template image are converted into the same form as the RGB image channel number by color mapping. 1.

3. Then, the RGB template image , the depth template image , the RGB search image , and the depth search image are divided into non-overlapping image patches of resolution , respectively , and ; where , ; these image patches represent the token inputs of the template and search images, respectively; RGB template image The divided image blocks form an RGB template patch sequence; depth template image The divided image blocks form a depth template patch sequence; RGB search image The divided image blocks form an RGB search region patch sequence; depth search image The divided image blocks form a depth search patch sequence; 1.

4. Then, the RGB template patch sequence and the depth template patch sequence are stitched into a template token wherein, denotes the length of the template token, denotes the dimension of the token, while the RGB search region patch sequence and the depth search patch sequence are stitched into a search region token wherein, denotes the length of the search region token.

5. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 4, wherein, Step two includes the following steps: 2.1, the original weight matrix is , represents the input dimension, represents the output dimension; the low-rank update term is , wherein and are trainable low-rank matrices, is a low-rank dimension, and the weight method for adjusting the patch embedding is: , wherein represents the updated weight matrix; 2.

2. Two-dimensional bilinear interpolation is used to interpolate the pre-trained position embeddings to encode the template token positions. and search area token location encoding Add to template tokens respectively and search area token In the patch, template tokens are generated and embedded. and search area token embedding The formula is as follows: ； ； wherein, , denotes the template token length, , denotes the search region token length, , , is the dimensionality of the token; denotes the embedding of the th template patch, denotes the embedding of the th search region patch; 2.3, Embedding template tokens With search area token embedding Connection, resulting in .

6. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 5, wherein, In step three, in the high frequency phase, the input is split along the channel and ; The max-pooling and linear layer are embedded, into the linear layer and the deep convolutional layer: ； ； wherein and denotes the output of the high-frequency mixer, denotes the fully connected layer, denotes the max-pooling layer, denotes the deep convolutional layer; In the low frequency stage, the output of the low frequency mixer ; wherein, denotes up-sampling, denotes multi-head self-attention mechanism, denotes average pooling; In step three, the outputs of the low frequency mixer and the high frequency mixer are connected along the channel dimension, and the formula is as follows: ； wherein represents splicing.

7. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 6, wherein, In step three, The calculation of the attention weight of the i-th feature point is extended as ； ； ； ； wherein, denotes the relationship weight between the template and the search region, denotes the relationship weight within the search region; denotes the relationship weight within the template; represents relationship modeling by aggregating inter-image features, denotes feature extraction by aggregating intra-image features, denotes fusion template feature extraction by self-attention operation.

8. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention according to claim 7, characterized in that, In step four, the classification score chart... Offset and normalized bounding box All are obtained from the prediction head. This indicates that the image is converted during the embedding process. The patch size, where H represents the height of the feature map and W represents the width of the feature map.

9. The single-stream RGB-D object tracking method based on LoRA fine-tuning and frequency-mixed attention of claim 8, wherein, In the training process of the model, classification loss and regression loss are used at the same time, including weighted classification loss, IoU loss and L1 loss, and the total loss function is represented as: ； wherein, denotes a generalized IoU loss, denotes an L1 norm loss, denotes a classification loss, , denotes a weight parameter.

Images