Multi-modal visual target tracking method based on dynamic adaptation

Abstract

The invention discloses a multi-modal visual target tracking method based on dynamic adaptation. The method comprises the following steps: constructing a dynamically adaptive multi-modal feature extraction and fusion network; constructing a dynamic bridging fusion module; constructing a parameter efficient adaptation mechanism based on low-rank adaptation; constructing a complete multi-modal visual target tracking model; training the model on computing equipment such as a server, and optimizing network parameters by reducing an overall loss value of a network loss function until the network converges; tracking a specified single target in a to-be-tracked video sequence by using the trained visual target tracking model; and performing performance evaluation on the trained model. According to the method, adaptive integration and unified target tracking of different visual modal information are realized through dynamic modal fusion and an efficient parameter adaptation mechanism.

Description

Technical Field

[0001] The present invention relates to a multimodal visual target tracking method based on dynamic adaptation, which belongs to the field of computer vision and image processing technology, and belongs to related fields such as multimodal perception, target detection and tracking, and is suitable for visual target tracking applications in various complex environments. Background Art

[0002] With the rapid development of computer vision technology, visual object tracking, as a critical foundational task, has been widely applied in fields such as intelligent surveillance, autonomous driving, and behavioral analysis. Traditional object tracking methods, most of which rely on single-modality visual information (such as RGB images), are prone to tracking loss or mistracking in complex environments, especially with rapid motion, changing lighting, object occlusion, and background interference. This makes it difficult to ensure system robustness and adaptability.

[0003] To overcome the limitations of a single modality, multimodal visual object tracking technology has gradually emerged. This technology integrates multiple visual perception modalities (such as RGB, depth map, thermal infrared, and event camera), leveraging the complementarity between these modalities to improve the performance and stability of object tracking. For example, the depth map modality can provide information about the geometric structure of the scene, which helps to resolve occlusion issues; the thermal infrared modality is more stable in low-light or nighttime conditions; and the event camera modality has the advantages of high temporal resolution and low latency, providing richer dynamic information in high-speed motion scenes.

[0004] Existing multimodal visual object tracking methods primarily employ static modal fusion strategies, such as feature concatenation and fixed weighting. While these methods have improved tracking robustness to some extent, they still suffer from the following major issues and shortcomings. First, they lack dynamic modal adaptability. Existing methods typically assign fixed fusion weights to each modality, failing to dynamically adjust the contributions of different modalities based on environmental changes or scenario requirements. When certain modalities fail or introduce noise in specific scenarios, static fusion can introduce invalid or even interfering information, impacting overall tracking performance. For example, in low-light scenarios, infrared modality is more effective than RGB modality, but static weighting strategies lack the flexibility to prioritize the most valuable modality, resulting in performance degradation. Second, model structure redundancy and high adaptation costs are associated with high model complexity. Due to the significant differences in feature distributions between modalities, many multimodal object tracking methods design independent branches or dedicated networks for each modality. While this architecture can extract modality-specific information, it is highly redundant, significantly increasing the number of parameters and computational overhead. In addition, traditional methods generally require fine-tuning of all parameters of the pre-trained model, resulting in high training and deployment costs and large storage overhead, which limits the popularization and application of multimodal tracking technology in actual scenarios, especially in environments with limited resources or frequent task switching. Summary of the Invention

[0005] Purpose of the Invention: The above-mentioned existing problems arise mainly due to the following reasons: On the one hand, the static fusion strategy lacks the ability to dynamically perceive environmental changes and modal validity, making it impossible to achieve targeted information screening and adjustment; on the other hand, independent modeling and full parameter fine-tuning lead to insufficient flexibility and efficiency of multimodal target tracking models, making it difficult to achieve a unified and efficient multimodal tracking solution. For this reason, how to achieve dynamic modal fusion and improve parameter adaptation efficiency has become a technical challenge that urgently needs to be solved in the field of multimodal visual target tracking.

[0006] The technical problem to be solved by the present invention is that in practical applications, the existing multimodal visual target tracking method lacks the ability to dynamically and adaptively fuse information from different modalities, and has problems such as model structure redundancy and low parameter adaptation efficiency. As a result, the target tracking system lacks robustness and flexibility in complex and changing environments, has high training and deployment costs, and is difficult to strike a balance between performance and efficiency.

[0007] The present invention provides a unified visual target tracking method that can realize dynamic fusion of multimodal information and efficient parameter adaptation, so as to improve the adaptability and practicality of the tracking system.

[0008] Technical solution: A multimodal visual target tracking method based on dynamic adaptation. This method achieves adaptive integration of different visual modal information and unified target tracking through dynamic modal fusion and efficient parameter adaptation mechanism. It can effectively improve the robustness and accuracy of the target tracking system in complex environments, while significantly reducing the training and deployment costs of the model.

[0009] The method supports a variety of common multimodal combinations, including RGB images and thermal infrared images (RGB-T), RGB images and depth images (RGB-D), and RGB images and event camera images (RGB-E). First, the image input of the above multimodal combinations is received. The image input of each modality is processed separately through a feature embedding layer, which divides the input image into non-overlapping image blocks of fixed size and projects them into high-dimensional feature vectors (tokens) through linear mapping. At the same time, a learnable position encoding is superimposed to preserve spatial information and achieve a unified feature representation.

[0010] In response to the problem of large differences in representation between different modal features, the present invention proposes a dynamic bridging fusion module. This module includes three sub-modules: feature compression, dynamic weight generation, and channel attention enhancement. First, the feature maps of each modality are compressed by 1×1 convolution to reduce the dimensionality and reduce the computational complexity while retaining the key information of each modality. Then, global average pooling and multi-layer perceptron (MLP) are used to obtain the global description vector of each modality, and adaptive fusion weights are obtained through Softmax normalization to achieve dynamic adjustment of the contribution of different modal features. Finally, a channel attention mechanism (such as Squeeze-and-Excitation) is used to perform channel weighted enhancement on the fused features, highlighting the key information channels and further improving the distinguishing ability of the fused features. The dynamic bridging fusion module can adaptively optimize the fusion ratio of different modalities according to changes in the input scene, thereby improving tracking robustness and adaptability.

[0011] The fused multimodal features serve as a unified representation input and are fed into a feature interaction network based on the Transformer. This network, composed of a multi-layer self-attention structure and feedforward layers, can model long-range dependencies between different modalities and time series, further improving global representation capabilities.

[0012] In terms of efficient parameter adaptation, the present invention introduces a parameter adaptation mechanism (Low-Rank Adaptation, LoRA) based on low-rank decomposition in the self-attention layer and feedforward network of the Transformer encoder. This mechanism only performs low-rank adjustment on a subset of key parameters, and the backbone network parameters remain frozen, thereby significantly reducing the amount of parameters and training overhead. For different multimodal tasks, the LoRA parameter configuration is optimized by grid search to achieve efficient adaptation and flexible switching of various multimodal scenarios. Specifically, during reasoning and task switching, only the LoRA adaptation parameters of the corresponding task need to be loaded, without reloading or fine-tuning all model parameters, which greatly improves the real-time performance and flexibility of the system.

[0013] The present invention realizes multimodal visual target tracking of multimodal combined images without the need to design a separate network branch for each modality. Through a unified dynamic fusion and efficient parameter adaptation mechanism, it achieves high flexibility and efficiency in multimodal target tracking, significantly reduces model redundancy and resource consumption, improves the feasibility and scalability of practical applications, and has good engineering application prospects.

[0014] A computer device, characterized in that: the computer device includes a memory, a processor and a computer program stored in the memory and capable of running on the processor, and when the processor executes the above-mentioned computer program, the steps of the multimodal visual target tracking method based on dynamic adaptation are implemented as described above.

[0015] A computer-readable storage medium is characterized in that: the computer-readable storage medium stores a computer program for executing the multimodal visual target tracking method based on dynamic adaptation as described above.

[0016] Beneficial effects: The multimodal visual target tracking method based on dynamic adaptation provided by the present invention can effectively overcome the shortcomings of existing multimodal tracking technologies in information fusion and parameter adaptation. Compared with the existing technology, the present invention realizes the adaptive integration of multimodal features by introducing a dynamic bridging fusion module, so that the system can automatically adjust the fusion ratio of each modal feature according to the actual scene changes, significantly improving the tracking robustness and image target recognition accuracy in complex environments. At the same time, an efficient parameter adaptation mechanism based on low-rank decomposition is adopted, and the adaptation of different modal tasks can be completed by updating only a very small number of key parameters, which greatly reduces the amount of model parameters and computing resource consumption, and reduces training and deployment costs. This method does not require a separate network branch to be designed for each modal image, and supports efficient switching between multiple modal tasks by loading different adaptation parameters. It is easy to operate and meets the real-time and flexibility requirements in practical applications. In addition, the fusion and adaptation strategy proposed in the present invention has strong versatility and scalability, and can adapt to images of multiple modal combinations and different application scenarios, providing a high-performance, low-cost solution for multimodal visual target tracking in the fields of intelligent security, autonomous driving, industrial detection, etc. Overall, the present invention not only outperforms existing technologies in tracking accuracy, efficiency, and system resource utilization, but also simplifies the process and operation procedures, and improves the feasibility and practical value of large-scale engineering deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a main flow chart of the multimodal visual target tracking method in a specific embodiment;

[0018] Figure 2 Schematic diagram of the overall system architecture of the method of the present invention;

[0019] Figure 3 It is a structural diagram of the dynamic bridge fusion module in a specific embodiment;

[0020] Figure 4 A comparison chart of tracking accuracy and success rate curves of a specific embodiment on the LasHeR dataset;

[0021] Figure 5 The following is a curve diagram of tracking performance evaluation of a specific embodiment on the VisEvent dataset. DETAILED DESCRIPTION

[0022] The present invention is further illustrated below with reference to specific examples. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0023] A multimodal visual target tracking method based on dynamic adaptation is proposed. The method uses Transformer as the backbone network, innovatively introduces a dynamic bridging fusion module and a parameter efficient adaptation mechanism based on low-rank adaptation (LoRA) to achieve efficient fusion of multimodal information and flexible switching of multiple tasks. Figure 1 The specific implementation steps are as follows:

[0024] S1. Construct a dynamic adaptive multimodal feature extraction and fusion network;

[0025] S2, building a dynamic bridging fusion module;

[0026] S3. Build an efficient parameter adaptation mechanism based on low-rank adaptation (LoRA);

[0027] S4, integrating dynamic bridging fusion modules and efficient parameter adaptation mechanisms to build a complete multimodal visual target tracking model;

[0028] S5. Training the model on a server or other computing device to optimize network parameters by reducing the overall loss value of the network loss function until the network converges;

[0029] S6. Tracking a single target specified in the video sequence to be tracked using the trained visual target tracking model;

[0030] S7. Perform performance evaluation on the trained model.

[0031] Preferably, the step S1 specifically includes the following steps:

[0032] S1.1. Multimodal Input Preparation. Collect or receive image data from two specified modalities (RGB and thermal infrared / depth / event camera). Each frame of data is divided into a template frame (the target's initial appearance frame) and a search frame (the frame of the area to be tracked) based on the target tracking task requirements.

[0033] S1.2. Image preprocessing. Perform basic preprocessing operations such as normalization, resizing, and denoising on each modality's images to ensure consistency and comparability of input data from different modalities. Preferably, all input images are resized to a fixed size (e.g., 256×256 pixels).

[0034] S1.3. Divide the image into patches. Divide the template frame and search frame of each modality into a number of fixed-size, non-overlapping patches (image blocks), for example, each patch is 16×16 pixels. Assuming a frame size of H×W pixels, each frame is divided into N = (H×W) / (16×16) patches.

[0035] S1.4. High-dimensional embedding of patch features. For each patch, a separate linear mapping layer (fully connected layer) is used to project its pixel data into a high-dimensional feature space to obtain a patch token representation. For example, if the feature dimension after embedding is C = 768, each patch is mapped to a 768-dimensional vector.

[0036] S1.5. Positional encoding is added. To preserve the spatial structure of the patch sequence, a learnable positional encoding is added to each patch token. This positional encoding can be implemented using either a sine-cosine function or a trainable parameter matrix.

[0037] S1.6. Modality-specific embedding layer parameter strategy. Different training strategies are used for the embedding layer parameters for different modalities: the patch embedding layer parameters for the RGB modality are frozen to maintain consistency with the pre-trained model, ensuring the stability of the main modality's features. The embedding layer parameters for other modalities (such as T, D, and E) are allowed to be updated during training to enhance cross-modal feature extraction capabilities and adaptability.

[0038] S1.7. Multimodal feature sequence output. Through the above steps, the template frame and search frame of each modality are mapped into a high-dimensional feature sequence containing N patch tokens, resulting in a multimodal feature sequence. This serves as the input to the subsequent dynamic bridging fusion module, providing the foundation for multimodal information fusion and unified modeling.

[0039] Preferably, step 2 constructs a dynamic bridging fusion module (see Figure 2 、 Figure 3 ). Input the multimodal feature sequence into the dynamic bridging fusion module to achieve adaptive multimodal feature fusion. The specific steps are as follows:

[0040] S2.1. Multimodal feature input. Receive the two modal high-dimensional patch token sequences output from step S1. Let the features of the two modalities be and Where C is the number of channels, H and W are the spatial sizes of the feature map.

[0041] S2.2, Feature dimensionality reduction compression. The feature maps F0 and F1 of each modality are subjected to channel dimensionality reduction using 1×1 convolution operations to obtain the reduced-dimensional features F0′ and F1′, namely

[0042] F′0=W0*F0,F′1=W1*F1

[0043] Where W0 and W1 are trainable 1×1 convolution kernels. This step can effectively reduce the subsequent computational complexity and improve fusion efficiency.

[0044] S2.3. Feature concatenation and joint representation. The reduced multimodal features F0′ and F1′ are concatenated in the channel dimension to form a joint feature representation. The concatenated joint features provide the basis for subsequent weight allocation and fusion.

[0045] S2.4, Global descriptor extraction and dynamic fusion weight generation. Perform global average pooling on F0′ and F1′ respectively to obtain their respective global descriptors g0 and g1:

[0046]

[0047] Each descriptor g i Input a two-layer fully connected network (the first layer has ReLU activation, and the second layer can use Sigmoid or no activation), and then normalize it with Softmax to obtain the modal fusion weights w0 and w1:

[0048] w i =Softmax(σ(W2(ReLU(W1g i ))))

[0049] Where W1 and W2 are trainable parameters, and σ is the Sigmoid activation function.

[0050] S2.5, Dynamic weighted fusion. Using the modal fusion weights obtained above, perform weighted fusion on the dimensionality reduction features F0′ and F1′ to obtain the dynamic fusion feature F fused :

[0051] F fused =w0F′0+w1F′1

[0052] S2.6, channel attention enhancement. Further improve the discriminative ability of fusion features, fused Perform channel attention processing. In order to further improve the discriminative ability of fusion features, the method introduces the channel attention mechanism. Specifically, for the fusion features Perform global average pooling on each channel c (where c = 1, 2, ..., C) to obtain channel statistics:

[0053]

[0054] Subsequently, the channel descriptor s cInput two 1×1 convolutional layers (with ReLU and Sigmoid activation) to reduce the dimension and restore the number of channels respectively, and generate channel attention scores:

[0055] a c =σ(W2(ReLU(W1s c )))

[0056] Finally, the fusion features are weighted channel by channel attention weight to obtain the output features:

[0057] F output =a c ·F fused

[0058] S2.7, output fusion features. The fusion features F after channel attention enhancement are output As the output of the dynamic bridging fusion module, it is used by subsequent modules such as the Transformer encoder to perform deep feature interaction and target tracking tasks.

[0059] Preferably, step S3 specifically includes the following steps:

[0060] S3.1. Locate the adaptation layers in the Transformer backbone network. Select the Multi-Head Self-Attention (MHSA) and Feed Forward Network (FFN) layers in the Transformer backbone network as the target layers for parameter adaptation. Insert low-rank decomposition adaptation branches at the locations of the linear transformations of these layers (such as the Q, K, V weight matrices and the feedforward weight matrix).

[0061] S3.2, Design and integration of low-rank decomposition adaptation branches. The weight matrix to be adapted in the target layer for each parameter adaptation Introduce a set of low-rank decomposition parameter matrices Where r<<d. During forward propagation, a low-rank compensation term is added to the output of the original weight W:

[0062] ΔW=BA

[0063] Implement the following mapping:

[0064] h=Wx+αBAx

[0065] Where x is the input feature vector of the layer, with dimension d, which is consistent with the input dimension of the weight matrix W, and α is the scaling factor (which can be set to 1 or determined by parameter adjustment).

[0066] S3.3. Parameter Freezing and Adaptive Branch Training Strategy. During training, the original weights W in the Transformer backbone network are frozen, and only the low-rank branch parameters B and A are trained. This significantly reduces the number of training parameters, improves adaptation efficiency, and prevents catastrophic forgetting in the backbone model.

[0067] S3.4. Task-specific adaptation parameter configuration. For different multimodal tasks (such as RGB-T, RGB-D, RGB-E, etc.), a separate set of LoRA branch parameters (B and A matrices and α scaling factor) is configured for each task. By performing a grid search on the validation set, the optimal low-rank r value and scaling factor α for each task are determined, achieving efficient adaptation for different modal combinations.

[0068] S3.5, Reasoning and Task Switching Mechanism. During the reasoning or actual application phase, based on the required multimodal task, only the LoRA parameter branch corresponding to the task is loaded, without the need to load or fine-tune all the backbone network parameters, thereby greatly reducing storage and computing consumption and enabling fast and flexible switching between multiple tasks.

[0069] S3.6. Scalability of the LoRA adaptation mechanism. This adaptation mechanism can be flexibly extended to all linear transformation positions in the Transformer structure, including the Q, K, and V matrices in the attention mechanism and the weight matrix of the feedforward network. It can also add a separate LoRA branch for newly introduced modal tasks, which is highly scalable.

[0070] S3.7. Output adapted features. The low-rank adapted Transformer network outputs adapted deep features, providing parameter-efficient, dynamically switchable feature support for subsequent target regression and tracking tasks.

[0071] Preferably, step S4 specifically includes the following steps:

[0072] S4.1. Integrate the patch embedding layer, dynamic bridging fusion module and LoRA parameter adaptation mechanism into the same backbone network. Specifically, the input images of each modality are first patch-divided and linearly embedded to obtain a high-dimensional patch token sequence, which is then input into the dynamic bridging fusion module to achieve dimensionality reduction, splicing, weighted fusion and channel attention enhancement of each modality feature; the fused features are then sent to a multi-layer Transformer backbone network integrated with the LoRA low-rank adaptation branch to achieve parameter-efficient deep feature interaction and adaptive adjustment.

[0073] S4.2. The backbone network adopts a multi-layer Transformer structure, and a target regression head is set at the network output end. Specifically: the backbone network is composed of multiple Transformer encoder layers in series, and a LoRA branch is inserted at the linear transformation position of each layer (such as the self-attention module and the feedforward network); a target regression head is set at the end of the network, including a fully connected layer or a lightweight convolutional structure, which maps the fused features output by the backbone network to the bounding box coordinates or center position of the target, thereby realizing the precise positioning of the tracked target and the output of the final tracking result.

[0074] Preferably, the step S5 specifically includes the following steps:

[0075] S5.1. On a server or high-performance computing device, select a multimodal public dataset (such as LasHeR, VisEvent, etc.) as training data, pair and enhance the multimodal template frames and search frames, and construct training samples.

[0076] S5.2. Use optimizers such as AdamW to set a joint loss function, including target position regression loss (such as L1 loss or IoU loss) and target classification loss (such as cross entropy loss), which are jointly used to optimize network parameters.

[0077] S5.3. During training, keep the backbone Transformer and its original parameters frozen, and only train and adjust the LoRA low-rank branch parameters, the channel attention parameters in the dynamic bridge fusion module, and the embedding layer parameters of the non-RGB modality, so as to reduce the amount of parameter updates and improve adaptation efficiency.

[0078] S5.4. Continue end-to-end training until the loss function on the validation set converges and the evaluation indicators meet the preset requirements, and finally obtain a well-trained multimodal target tracking model with strong generalization ability.

[0079] Preferably, step S6 specifically includes the following steps:

[0080] S6.1. During the inference or actual application phase, input a template frame of the target to be tracked and continuous video image frames, and simultaneously preprocess the images of each modality.

[0081] S6.2. Perform multimodal feature extraction, fusion, and deep feature interaction on each frame through the patch embedding layer, dynamic bridge fusion module, and Transformer backbone network with integrated LoRA adaptation mechanism, and output fused features.

[0082] S6.3. Input the fused features output by the backbone network into the target regression head, automatically calculate and output the position of the target in the current frame (such as the bounding box coordinates), and realize real-time, multimodal target tracking in continuous image frames.

[0083] S6.4. If you need to switch to different multimodal target tracking tasks, you only need to load the corresponding LoRA parameter branch and adaptation parameters without retraining the backbone Transformer main network, achieving efficient multi-task adaptation and expansion.

[0084] The step S7 performance evaluation and effect verification. Figure 4 and Figure 5 As shown in Figure 3, the target tracking performance evaluation results of the method of the present invention on the LasHeR dataset and the VisEvent dataset show that the method is significantly better than existing similar multimodal tracking methods in terms of both accuracy curve and success rate indicators.

[0085] Obviously, those skilled in the art should understand that the various steps of the multimodal visual target tracking method based on dynamic adaptation of the above-mentioned embodiment of the present invention can be implemented using a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, they can be implemented using program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in an order different from that shown here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A multimodal visual target tracking method based on dynamic adaptation, characterized in that: The proposed method achieves adaptive integration of different visual modal information and unified target tracking through dynamic modality fusion and efficient parameter adaptation mechanism. Specifically, it includes the following contents: Acquire a multimodal image. Each modality is processed separately through a feature embedding layer, which divides the input image into fixed-size non-overlapping image patches and projects them into high-dimensional feature vectors through linear mapping, while superimposing a learnable positional encoding. The dynamic bridging fusion module is used to solve the representation differences between different modal features. The dynamic bridging fusion module can adaptively optimize the fusion ratio of different modalities according to the changes in the input scene; the dynamic bridging fusion module includes three sub-modules: feature compression, dynamic weight generation and channel attention enhancement; first , the feature compression module performs dimensionality reduction and compression on the feature maps of each modality through 1×1 convolution; then, the dynamic weight generation module uses global average pooling and multi-layer perceptron to obtain the global description vector of each modality, and obtains the adaptive fusion weight through softmax normalization; finally, the channel attention enhancement module uses the channel attention mechanism to perform channel weighted enhancement on the fused features; The fused multimodal features are input as a unified representation and sent to a feature interaction network with a Transformer encoder as the backbone; the network includes a multi-layer self-attention structure and a feedforward network layer, which can model long-distance dependencies between different modalities and between time series, and obtain a multimodal visual target tracking model. The model is trained and the trained visual target tracking model is used to track a specified single target in the video sequence to be tracked, thereby realizing multimodal visual target tracking of multimodal combined images.

2. The multimodal visual target tracking method based on dynamic adaptation according to claim 1, characterized in that: The multimodal combined images include RGB images and thermal infrared images RGB-T, RGB images and depth images RGB-D, and RGB images and event camera images RGB-E.

3. The multimodal visual target tracking method based on dynamic adaptation according to claim 1, characterized in that: A parameter adaptation mechanism based on low-rank decomposition is introduced in the self-attention layer and feedforward network of the Transformer encoder. This mechanism only performs low-rank adjustments on a subset of identified key parameters, and the backbone network parameters remain frozen. For different multimodal tasks, the LoRA parameter configuration is optimized through grid search. Specifically, during inference and task switching, only the LoRA adaptation parameters of the corresponding task need to be loaded, without the need to reload or fine-tune all model parameters.

4. The multimodal visual target tracking method based on dynamic adaptation according to claim 1, characterized in that: For multimodal images, each frame is divided into a template frame and a search frame according to the target tracking task requirements. Normalization, resizing, and denoising preprocessing are performed on each modality's images. The template frame and search frame of each modality are divided into several fixed-size non-overlapping patches. For each patch, an independent linear mapping layer is used to project its pixel data into a high-dimensional feature space to obtain a patch token representation. A learnable positional encoding is superimposed on each patch token. Differentiated training strategies are used for the embedding layer parameters for different modalities: the patch embedding layer parameters for the RGB modality are frozen to maintain consistency with the pre-trained model. The embedding layer parameters of other modalities are allowed to be updated during training; the template frame and search frame of each modality are mapped into a high-dimensional feature sequence containing N patch tokens as the input of the dynamic bridging fusion module.

5. The multimodal visual target tracking method based on dynamic adaptation according to claim 1, characterized in that: The dynamic bridging fusion module receives the high-dimensional patch token sequence of each modality, and assumes that the features of the two modalities are and Where C is the number of channels, H and W are the spatial sizes of the feature map; The 1×1 convolution operation is used to perform channel dimensionality reduction on the feature maps F0 and F1 of each modality to obtain the reduced-dimensional features F0′ and F1′, that is, F′0=W0*F0,F′1=W1*F1 Where W0 and W1 are trainable 1×1 convolution kernels; The multimodal features F0′ and F1′ after dimensionality reduction are concatenated in the channel dimension to form a joint feature representation; Perform global average pooling on F0′ and F1′ respectively to obtain their respective global descriptors g0 and g1: Each descriptor g i Input two layers of fully connected network and then normalize them by Softmax to obtain modal fusion weights w0 and w1: w i =Softmax(σ(W2(ReLU(W1g i )))) Where W1 and W2 are trainable parameters, and σ is the Sigmoid activation function; Perform weighted fusion on the dimensionality reduction features F0′ and F1′ to obtain the dynamic fusion feature F fused : F fused =w0F′0+w1F′1 F fused Perform channel attention processing to fusion features Perform global average pooling to obtain channel statistics: Subsequently, the channel descriptor s c Input two 1×1 convolutional layers to reduce the dimension and restore the number of channels respectively, and generate channel attention scores: from c =σ(W2(ReLU(W1s c ))) Finally, the fusion features are weighted channel by channel attention weight to obtain the output features: F output =a c ·F fused The fusion feature F after channel attention enhancement output As the output of the dynamic bridging fusion module.

6. The multimodal visual target tracking method based on dynamic adaptation according to claim 1, characterized in that: Locate the adaptation layer in the Transformer backbone network, select the self-attention layer and feedforward network layer in the Transformer backbone network as the target layer for parameter adaptation, and insert the adaptation branch at the linear transformation position; for each weight matrix to be adapted Introduce a set of low-rank decomposition parameter matrices Where r<<d; during forward propagation, a low-rank compensation term is added to the output of the original weight W: ΔW=BA Implement the following mapping: h=Wx+αBAx Where x is the input feature and α is the scaling factor; During the training process, the original weight W parameters in the Transformer backbone network are kept frozen, and only the low-rank branch parameters B and A are trained. For different multimodal tasks, an independent set of LoRA branch parameters is configured for each task, and the optimal low-rank r value and scaling factor α for each task is determined by grid search on the validation set. During the inference or actual application phase, according to the required multimodal task, only the LoRA parameter branch corresponding to the task is loaded; The LoRA adaptation mechanism can be extended to all linear transformation positions in the Transformer structure, including the Q, K, and V matrices in the attention mechanism and the weight matrix of the feedforward network. It can also add a separate LoRA branch for newly introduced modal tasks. The low-rank adapted Transformer network outputs the adapted deep features.

7. The multimodal visual target tracking method based on dynamic adaptation according to claim 1, characterized in that: The patch embedding layer, dynamic bridging fusion module, and LoRA parameter adaptation mechanism are integrated into the same backbone network. Specifically, the input images of each modality are first patch-partitioned and linearly embedded to obtain a high-dimensional patch token sequence, which is then input into the dynamic bridging fusion module to achieve dimensionality reduction, splicing, weighted fusion, and channel attention enhancement of each modality feature. The fused features are then fed into a multi-layer Transformer backbone network integrated with the LoRA low-rank adaptation branch to achieve parameter-efficient deep feature interaction and adaptive adjustment. The backbone network adopts a multi-layer Transformer structure, and a target regression head is set at the network output end. Specifically, the backbone network is composed of multiple Transformer encoder layers in series, and a LoRA branch is inserted at the linear transformation position of each layer; a target regression head is set at the end of the network, including a fully connected layer or a lightweight convolutional structure, which maps the fused features output by the backbone network to the bounding box coordinates or center position of the target, thereby achieving accurate positioning of the tracked target and output of the final tracking results.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the multimodal visual target tracking method based on dynamic adaptation as described in any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the multimodal visual target tracking method based on dynamic adaptation as described in any one of claims 1 to 7.

Images