RGBT tracking system based on pixel-level fusion and use method

Abstract

The present application provides a kind of RGBT tracking network based on pixel-level fusion and use method, including: pixel-level fusion adapter: first, each modality is divided by a low-level feature extraction layer, then fed into separate Vim block to encode specific features.Next, apply token and channel connection to merge two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information.Finally, the fused features are decoded into images using convolution layers with efficient local detail modeling capabilities.The present application proposes a two-stage task-oriented progressive learning framework.In the first stage, multi-expert adaptive distillation (MAD).Aims to inherit superior fusion capabilities from multiple image fusion models with different structures.In the second stage, decoupled representation fine-tuning strategy (DRF), improves fusion accuracy by explicitly separating task-related and task-irrelevant information through repulsion loss, and ensures the completeness of information decoupling through reconstruction loss, thereby improving fusion robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and image processing, in particular to an RGBT tracking system based on pixel-level fusion and a use method. BACKGROUND

[0002] As a basic task in the field of computer vision, visual tracking aims to locate the target object in subsequent frames according to the initial state of the target. Due to the complementary advantages of visible light (RGB) and thermal infrared (TIR) modalities, which can enhance the tracking robustness in complex scenes, visible light thermal infrared (RGBT) tracking has attracted great attention from researchers in recent years, and a large number of works have shown impressive tracking performance. Current research mainly explores various fusion strategies, which can be mainly divided into three categories, including pixel-level fusion, feature-level fusion and decision-level fusion.

[0003] The first category of pixel-level fusion, which directly performs straightforward fusion at the pixel level. Yang et al., Prompting for multi-modal tracking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3492-3500, 2022. explore prompt learning in multi-modal pixel-level interaction. While Tang et al., Exploring fusion strategies for accurate rgbt visual object tracking. Information Fusion, page 101881, 2023. introduce an existing image fusion network for RGBT tracking.

[0004] The second type of feature-level fusion. Due to the powerful representation ability of deep neural networks, feature-level fusion schemes are widely used in RGBT tracking and have achieved remarkable results. Liu et al., Quality-aware rgbt tracking via supervised reliability learning and weighted residual guidance. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3129-3137, 2023. A feature-weighted fusion architecture based on the importance or quality of each modality in the last layer of features is adopted to achieve robust fusion. Chen et al., Simplifying cross-modal interaction via modality-shared features for rgbt tracking. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1573-1582, 2024. After the 4th module, a cross-attention-based channel-shared information aggregation module is integrated into the ViT backbone to aggregate channel-shared information.

[0005] The third type of decision-level fusion. By processing the tracking results of visible light and thermal infrared respectively, and then fusing the decisions of the two, the final comprehensive tracking result is obtained. This method is relatively simple, and usually relies on the optimization of independent tracking algorithms for each modality. For example: Tang et al., Exploring fusion strategies for accurate rgbt visual object tracking. Information Fusion, page 101881, 2023. Zhang et al., Multi-modal fusion for end-to-end rgb-t tracking. In Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019.

[0006] Due to the limited feature representation ability of shallow networks, the current pixel-level fusion method is difficult to mine and utilize the inter-modal complementary information related to the task in complex tracking scenes, resulting in a significant performance gap compared with the feature-level fusion method. SUMMARY

[0007] The technical problem to be solved by the present application is how to use a lightweight model to mine and utilize inter-modal complementary information related to tasks in a complex tracking scene.

[0008] The present application solves the above technical problems by the following technical means:

[0009] The RGBT tracking network based on pixel-level fusion, characterized in that it comprises:

[0010] The pixel-level fusion adapter comprises three Vim blocks and two convolutional layers; the RGB and TIR two modalities are divided by a low-level feature extraction layer, then fed into the first Vim block to encode specific features; then the token and channel connection is applied to merge the two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information; finally, the fused features are decoded into images to obtain a fused image

[0011] The task-oriented progressive learning framework comprises a task-oriented multi-expert distillation framework and a decoupled representation fine-tuning framework; the multi-expert distillation framework is used for parameter initialization of the pixel-level fusion adapter;

[0012] The decoupled representation fine-tuning framework comprises two auxiliary branches and a tracking branch; during training, the two auxiliary branches and the tracking branch operate in parallel, and the two auxiliary branches are used to independently encode task-independent information from RGB and TIR to obtain RGB modality features TIR modality features The tracking branch extracts features from the fused image I f to obtain fused features and performs task loss; then combined to perform exclusion loss and fusion reconstruction loss to decouple the task relevance and task independence of the two modalities; output target classification and positioning; during testing, the trained tracking branch is used for target tracking.

[0013] The present application designs a lightweight pixel-level fusion adapter (PFA), first, each modality (RGB and TIR) is divided by a low-level feature extraction layer, then fed into a separate Vim block to encode specific features. Next, the token and channel connection is applied to merge the two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information. Finally, the fused features are decoded into images using convolutional layers with efficient local detail modeling capabilities.

[0014] The application proposes a two-stage task-oriented progressive learning framework (TPL). In the first stage, multi-expert adaptive distillation (MAD) is used to inherit superior fusion capabilities from multiple image fusion models with different structures. MAD can adaptively adjust the importance weight of different experts to ensure that the best expert is selected for distillation in different scenarios. In the second stage, a decoupled representation fine-tuning strategy (DRF) is used to explicitly guide the integration of PFA and tracker and extract task-related information. DRF improves fusion accuracy by explicitly separating task-related and task-irrelevant information through a repulsion loss, and improves fusion robustness by ensuring the completeness of information decoupling through a reconstruction loss.

[0015] As an optimization scheme of the above scheme, it further includes a nearest neighbor dynamic template updating module for comparing the predicted target classification score with a preset threshold to determine whether the model needs to update the target template.

[0016] As an optimization scheme of the above scheme, the multi-expert distillation framework library includes a CNN-based image fusion model, a diffusion-based image fusion model and a Mamba-based image fusion model; given an RGB modality image and a TIR modality image, I PFA , I c , I d , I m respectively represent the fusion images of PFA, the CNN-based image fusion model, the diffusion-based image fusion model and the Mamba-based image fusion model; therefore, the process of multi-expert adaptive distillation is as follows:

[0017]

[0018] wherein represents the IOU predictor of the single-stream tracker, represents the weight of the i-th expert; the pixel-level fusion adapter parameters are optimized by minimizing the distillation loss .

[0019] As an optimization scheme of the above scheme, the two auxiliary branches include a feature extraction backbone and a decoder; the tracking branch includes a feature extraction backbone with the same structure as the auxiliary branch, and a tracking head.

[0020] As an optimization scheme of the above scheme, the repulsion loss is defined as:

[0021]

[0022] wherein (x) + represents max(0,x), and alpha is a margin parameter for controlling the similarity threshold.

[0023] As an optimization scheme of the above scheme, the fusion reconstruction operation is to combine the task-related and task-irrelevant features and put them into the decoders of the two auxiliary branches to reconstruct the original modality image, and the process is represented as:

[0024]

[0025] wherein D rgb and D tir represent the RGB and TIR mode encoders respectively, representing the mean square error function.

[0026] The application also provides a pixel-level fusion method using the above-mentioned RGBT tracking network based on pixel-level fusion, comprising the following steps:

[0027] The step of pixel-level fusion feature extraction: the RGB and TIR two modalities are input into the pixel-level fusion adapter, and are divided by a low-level feature extraction layer respectively, and then are fed into the first Vim block to encode specific features; then the token and channel connection are applied to combine the two modalities along different feature dimensions, and two additional Vim blocks are further used to encode the fused information; finally, the fused features are decoded into images;

[0028] Task-oriented progressive learning: including the steps of pixel-level fusion adapter parameter initialization and decoupled representation fine-tuning;

[0029] The step of adjusting the parameters of the pixel-level fusion adapter is to initialize the parameters of the pixel-level fusion adapter using a task-oriented multi-expert distillation framework;

[0030] The step of decoupled representation fine-tuning is that during training, the two auxiliary branches and the tracking branch operate in parallel, and the two auxiliary branches are used to independently encode the task-irrelevant information from the RGB and TIR to obtain the RGB modality feature

[0031] TIR modality feature The tracking branch extracts features from the fused image I f to obtain the fused feature and performs task loss; then combines to perform exclusion loss and fusion reconstruction loss to realize the decoupling of the task-relatedness and task-irrelevance of the two modality features; outputs target classification and positioning; during testing, the trained tracking branch is used for target tracking.

[0032] As an optimization scheme of the above scheme, the expert library includes a CNN-based image fusion model, a diffusion-based image fusion model and a Mamba-based image fusion model; given the RGB modality image and the TIR modality image, IPFA , I c , I d , I m respectively represent the fusion image of PFA, CNN-based image fusion model, diffusion-based image fusion model and Mamba-based image fusion model; therefore, the process of multi-expert adaptive distillation is as follows:

[0033]

[0034] wherein represents the IOU predictor of the single-stream tracker, represents the weight of the i-th expert; by minimizing the distillation loss to optimize the pixel-level fusion adapter parameters.

[0035] As an optimization scheme of the above scheme, the two auxiliary branches include a feature extraction backbone and a decoder; the tracking branch includes a feature extraction backbone with the same structure as the auxiliary branch, and a tracking head.

[0036] As an optimization scheme of the above scheme, the definition of the repulsion loss is:

[0037]

[0038] wherein (x) + represents max(0,x), and alpha is a margin parameter for controlling the similarity threshold;

[0039] The fusion reconstruction operation is to combine task-related and task-irrelevant features and put them into the decoder of the two auxiliary branches to reconstruct the original modality image, and this process is represented as:

[0040]

[0041] wherein D rgb and D tir respectively represent the RGB and TIR mode encoders, represents the mean square error function.

[0042] The advantages of the present application are:

[0043] The present application designs a lightweight pixel-level fusion adapter (PFA) which adopts a Mamba architecture with global modeling capability and linear computational complexity to realize effective pixel-level fusion between two modal images. First, each modality (RGB and TIR) is divided by a low-level feature extraction layer, and then fed into separate Vim blocks to encode specific features. In order to enhance the cross-channel representation, a parameter-free channel exchange mechanism is introduced to solve the limitation of Vim model channel dimension. Next, token and channel connections are applied to merge the two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information. Finally, the fused features are decoded into a fused image using a convolution layer with high-efficiency local detail modeling capability.

[0044] The present application proposes a two-stage task-oriented progressive learning framework (TPL). In the first stage, multi-expert adaptive distillation (MAD). It aims to inherit superior fusion capabilities from multiple image fusion models with different structures. MAD can adaptively adjust the importance weight of different experts to ensure the selection of the best expert for distillation in different scenarios. In the second stage, decoupled representation fine-tuning strategy (DRF). It explicitly guides the integration of PFA and tracker to extract task-related information. DRF improves fusion accuracy by explicitly separating task-related and task-irrelevant information through a repulsion loss, and ensures the completeness of information decoupling through a reconstruction loss, thereby improving fusion robustness.

[0045] In order to solve the problem that the appearance difference between the initial template and the search frame in the pixel-level fusion scheme changes significantly over time, which in turn limits the performance of the pixel-level fusion tracker, the present application proposes a method combining an additional dynamic template, a nearest neighbor dynamic template update strategy (NDTU), which aims to bridge the appearance difference. In the tracking inference stage, the dynamic template is updated appropriately by identifying reliable frames. And in order to improve efficiency, only the latest reliable frames in history are saved, thereby ensuring timely updating of the template during tracking, while avoiding accumulation of excessive redundant data. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The framework of the RGBT tracking network based on pixel-level fusion in the embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0048] To reveal the potential of pixel-level fusion in RGBT tracking, this embodiment proposes a novel task-driven RGBT tracking network based on pixel-level fusion, such as... Figure 1 As shown, this network possesses an efficient and effective progressive learning framework called TPF. To balance the efficiency and capability of pixel-level fusion, this embodiment employs Mamba with linear computational complexity to construct a lightweight pixel-level fusion adapter (PFA), containing only 14.3KB of parameters to ensure image fusion efficiency. Subsequently, a novel task-driven progressive learning (TPL) framework is proposed, comprising two stages of multi-expert adaptive extraction and decoupled representation fine-tuning to improve the fusion capability of PFA. In the first stage, the multi-expert adaptive extraction strategy guides PFA to inherit the advantages of different state-of-the-art image fusion models in different tracking scenarios, laying a solid foundation for fusion. The second stage proposes a decoupled representation fine-tuning strategy, decoupling RGB and TIR features into task-related and task-independent components, realizing a pixel-level fusion model focused on task-related features. Specifically, it utilizes two auxiliary branches to independently encode task-independent information from RGB and TIR, while the tracking branch combines modality extraction to extract task-related features. This process is guided by three key losses: a task loss to ensure relevance, a rejection loss to separate task-independent information, and a reconstruction loss to maintain input integrity. Therefore, with explicit guidance, PFA and the tracker can integrate task-relevant information while filtering out task-irrelevant information. It is worth noting that the repulsion loss and reconstruction loss are only effective during training. Furthermore, as time progresses, changes in the target appearance between the initial template and the search frame introduce additional challenges to pixel-level fusion. To overcome this, this embodiment designs a simple yet effective Nearest Neighbor Dynamic Template Update (NDTU) strategy, selecting the most reliable tracking result closest to the current search region as the dynamic template.

[0049] The network in this embodiment consists of three parts: a pixel-level fusion adapter (PFA), a task-oriented progressive learning framework (TPL), and a nearest neighbor dynamic template update strategy (NDTU).

[0050] 1. To achieve pixel-level fusion, this embodiment does not directly connect or sum the image matrices. Inspired by research on visible light and infrared image fusion, this embodiment designs a lightweight pixel-level fusion adapter (PFA). Unlike existing computationally intensive image fusion methods, this embodiment employs the Mamba architecture with global modeling capabilities and linear computational complexity to achieve efficient pixel-level fusion between two modalities, thus better adapting to tracking tasks.

[0051] 2. To balance the efficiency and capacity of pixel-level fusion and improve the fusion capacity of PFA, the embodiment proposes a two-stage task-oriented progressive learning framework (TPL). Specifically: the first stage, multi-expert adaptive distillation. PFA can provide efficient image fusion due to its low parameter setting, but also limits the fusion performance. Although previous methods have tried to adopt distillation strategies to improve fusion capacity, it is difficult to provide sufficient guidance in various tracking scenarios relying solely on the knowledge of a single expert model. To overcome this limitation, the embodiment proposes a method called multi-expert adaptive distillation (MAD), which aims to inherit superior fusion capacity from multiple image fusion models with different structures. MAD can adaptively adjust the importance weight of different experts to ensure the selection of the best expert for distillation in different scenarios. The second stage, decoupled representation fine-tuning (DRF). Since all expert models are still human vision oriented, further fine-tuning of PFA is necessary to effectively adapt to the tracking task. For this purpose, the embodiment proposes a new decoupled representation fine-tuning strategy (DRF) to explicitly guide PFA and tracker to integrate and extract task-related information. DRF improves fusion accuracy by explicitly separating task-related and task-irrelevant information through a repulsion loss, and improves the robustness of fusion by ensuring the use of complete modal information for supervision through a reconstruction loss.

[0052] 3. Since in the pixel-level fusion scheme, the appearance difference between the initial template and the search frame changes significantly over time, which also limits the performance of the pixel-level fusion tracker. For this purpose, the embodiment introduces a simple and effective nearest neighbor dynamic template update strategy (NDTU) to bridge the appearance difference by combining additional dynamic templates.

[0053] The three parts of the network of the embodiment are described in detail as follows:

[0054] 1. Pixel-level fusion adapter (PFA)

[0055] The network details of PFA are shown in Figure 1 . PFA mainly consists of three Vim blocks with channel sizes of 8, 8 and 16, and two convolutional layers. First, each modality (RGB and TIR) is divided by a low-level feature extraction layer, then fed into separate Vim blocks to encode specific features. To enhance cross-channel representation, the embodiment introduces a parameter-free channel exchange mechanism to address the limitation of Vim model channel dimension. Next, token and channel connections are applied to merge the two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information. Finally, the embodiment uses convolutional layers with efficient local detail modeling capability to decode the fused features into images.

[0056] 2. Task-oriented progressive learning framework (TPL)

[0057] The first stage, multi-expert adaptive distillation (MAD). Specifically, three different image fusion models are selected in this embodiment, including a CNN-based model (Zhao et al., Cddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusion. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 5906-5916, 2023), a diffusion-based model (Yi et al., Diff-if: Multi-modality image fusion via diffusion model with fusion knowledge prior. Information Fusion, 110:102450, 2024.2, 3, 4, 7, 1830), and a Mamba-based model (Li et al., Mambadfuse: A mamba-based dual-phase model for multi-modality image fusion. arXiv preprint arXiv:2404.08406, 2024.2, 4, 7, 1740), as three experts of PFA. Given the RGB modality image and the TIR modality image, this embodiment uses I PFA c d m denote the fusion images of PFA, the CNN-based image fusion model, the diffusion-based image fusion model, and the Mamba-based image fusion model, respectively. Therefore, the process of MAD is as follows:

[0058]

[0059] where denotes the IOU (intersection over union) predictor of the single-stream tracker, denotes the weight of the i-th expert. By minimizing the distillation loss to optimize the PFA parameters, the existing multi-expert fusion capability is inherited, and a low computational complexity is guaranteed.

[0060] ​​​The second stage is a decoupled representation fine-tuning strategy (DRF). Specifically, the decoupled representation fine-tuning framework includes two auxiliary branches and one tracking branch. The embodiment introduces two auxiliary branches, each of which includes a feature extraction backbone identical in structure to the tracking branch and a decoder specific to a certain modality. The tracking branch includes a tracking head in addition to the feature extraction backbone. These auxiliary branches operate in parallel with the tracking branch, and the two auxiliary branches are used to independently encode task-independent information from the RGB and TIR modalities to obtain RGB modality features TIR modality features The tracking branch performs feature extraction on the fused image I f to obtain fused features and performs a task loss; then combines to perform a repulsion loss and a fusion reconstruction loss to achieve decoupling of task-relatedness and task-independence of the two modalities contains task-related information of the two modalities, contains task-independent information of each modality); outputs target classification and localization; and during testing, the trained tracking branch is used for target tracking. To separate the task-related and task-independent features in each channel, the embodiment applies a task loss to optimize and introduces a repulsion loss to push and away from each other

[0061] The definition of the repulsion loss is:

[0062]

[0063] where (x) + represents max(0, x), and alpha is a margin parameter that controls the similarity threshold. The repulsion loss prevents unnecessary overlap between these features during fine-tuning, allowing the model to focus on task-related information in each modality and minimize interference.

[0064] To ensure the integrity of feature decoupling, the embodiment combines the task-related and task-independent features and places them into the corresponding modality-specific decoder to reconstruct the original modality image. This process can be represented as:

[0065]

[0066] where D rgb and D tir represent the RGB and TIR mode encoders, respectively, and denotes the reconstruction loss function. The reconstruction loss ensures that each channel preserves task-related and task-unrelated information during fine-tuning, enables dynamic adjustment of these features, and minimizes the loss of key task details during fusion.

[0067] Finally, the final loss of the fine-tuning stage is defined as the combination of the reconstruction loss, the repulsion loss, and the specific task loss in this embodiment, as follows:

[0068]

[0069] wherein denotes the tracking task loss. The coefficient λ is used to balance the contribution of each loss component, and is set to 1 for simplicity in the study of this embodiment.

[0070] 3. Nearest Neighbor Dynamic Template Update Strategy (NDTU)

[0071] Since in the pixel-level fusion scheme, the appearance difference between the initial template and the search frame will change significantly over time, which also limits the performance of the pixel-level fusion tracker. To this end, a simple and effective nearest neighbor dynamic template update strategy (NDTU) can be introduced to bridge the appearance difference by combining additional dynamic templates. In order to update the dynamic template appropriately during the tracking inference stage, this embodiment first identifies reliable frames, defined as frames whose predicted target classification scores P exceed a reliable threshold p. For all reliable frames in history, this embodiment only saves the latest reliable frame S new to improve efficiency. Every N frames, this embodiment introduces S new as the dynamic template of the current frame. In the training stage, this embodiment introduces dynamic updated templates sampled from intermediate frames as additional inputs. Benefited from these templates are the fusion results from frames with different appearance differences at different time instants, so that the model can better capture the appearance changes of the target by modeling the global relationship between the spatial and temporal dimensions.

[0072] The performance of the algorithm of the present application is evaluated by comparing the performance of the present embodiment with the most advanced RGBT tracking methods on GTOT, RGBT210, RGBT234 and LasHeR and dataset to verify the effectiveness of the proposed method (Table 1). The evaluation indicators most commonly used for the dataset are selected for quantitative performance evaluation in each dataset. The PR, NPR and SR are used as evaluation indicators for the LasHeR dataset, and the PR and SR are used as evaluation indicators for the rest of the datasets. Among them, PR refers to the percentage of total frames in which the distance between the center point of the tracking result and the center of the real bounding box is less than a specified threshold. SR measures the ratio of frames in which the overlap area of the tracking result box and the real bounding box is higher than the threshold, and then calculates the SR metric by changing the threshold and calculating the area under the curve (AUC). In order to eliminate the influence of image resolution and target size on the PR metric, the PR is normalized to obtain the normalized precision rate NPR when evaluating the tracking performance on the LasHeR dataset. From the results, it can be seen that the method proposed in the present embodiment can be applied to RGBT target tracking dataset, and the results have obvious improvement compared with other methods.

[0073] Table 1 Method result comparison chart

[0074]

[0075]

[0076] Table 1 Method result comparison chart (continued)

[0077]

[0078]

[0079] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An RGBT tracking system based on pixel-level fusion, characterized in that, Comprise: Pixel-level fusion adapter: comprising three Vim blocks and two convolutional layers; Two modalities of RGB and TIR are divided by a low-level feature extraction layer respectively, and then fed into the first Vim block to encode specific features; The token and channel connection are then applied to merge the two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information; finally, a decoder is used to decode the fused features into an image, resulting in a fused image I f ; Task-oriented progressive learning framework: including a task-oriented multi-expert distillation framework library and a decoupled representation fine-tuning module framework; the expert distillation framework library is used for parameter adjustment initialization of the pixel-level fusion adapter; The decoupled representation fine-tuning module framework comprises two auxiliary branches and a tracking branch; During training, two auxiliary branches operate in parallel with the tracking branch, and the two auxiliary branches are used to independently encode task-irrelevant information from RGB and TIR to obtain RGB modal features TIR modal features The tracking branch performs feature extraction on the fused image I f to obtain fused features and performs a task loss; then combines to perform a repulsion loss and a fusion reconstruction loss to achieve decoupling of task correlation and task irrelevance of the two modal features; outputs target classification and positioning; during testing, the trained tracking branch is used for target tracking; The multi-expert distillation framework includes a CNN-based image fusion model, a diffusion-based image fusion model, and a Mamba-based image fusion model; given an RGB modality image and a TIR modality image, I PFA , I c , I d , I m respectively represent the fusion images of PFA, the CNN-based image fusion model, the diffusion-based image fusion model, and the Mamba-based image fusion model; therefore, the process of multi-expert adaptive distillation is as follows: where T denotes the IOU predictor of the single-stream tracker, W i denotes the weight of the i-th expert; the pixel-level fusion adapter parameters are optimized by minimizing the distillation loss .

2. The pixel-level fusion based RGB-T tracking system of claim 1, wherein, It also includes a nearest neighbor dynamic template update module: for comparing the predicted target classification score with the preset threshold, and then judging whether the model needs to be updated.

3. The pixel-level fusion based RGB-T tracking system of claim 1 or 2, wherein, Two auxiliary branches include a feature extraction backbone and a decoder; the tracking branch includes a feature extraction backbone with the same structure as the auxiliary branch, and a tracking head.

4. The pixel-level fusion based RGB-T tracking system of claim 1 or 2, wherein, The definition of the repulsion loss is: where (x) + denotes max(0, x), and a is a margin parameter that controls the similarity threshold.

5. The pixel-level fusion based RGB-T tracking system of claim 1 or 2, wherein, The fusion reconstruction operation is to combine task-related and task-irrelevant features and put them into the decoder of the two auxiliary branches to reconstruct the original modality image, which is represented as: where D rgb and D tir denote the RGB and TIR mode encoders, respectively, denotes the mean square error function.

6. A pixel-level fusion method using the pixel-level fusion-based RGBT tracking system according to any one of claims 1 to 5, characterized in that, Comprise the following steps: The step of pixel-level fusion feature extraction: RGB and TIR two modalities as the input of the pixel-level fusion adapter, are divided by a low-level feature extraction layer respectively, and then fed into the first Vim block to encode specific features; The token and channel connection are then applied to merge the two modalities along different feature dimensions, and two additional Vim blocks further encode the fused information; finally, a decoder is used to decode the fused features into an image, resulting in the fused image I f ; Task-oriented progressive learning: including the steps of pixel-level fusion adapter parameter initialization and decoupled representation fine-tuning; The step of adjusting the parameters of the pixel-level fusion adapter is to initialize the parameters of the pixel-level fusion adapter using a task-oriented multi-expert distillation framework; The step of decoupling the representation fine-tuning is: during training, the two auxiliary branches operate in parallel with the tracking branch, and the two auxiliary branches are used to independently encode the task-independent information from the RGB and TIR to obtain the RGB modal feature The TIR modal feature The tracking branch fuses the image I f Feature extraction is performed to obtain the fusion feature And perform task loss; then combine To perform exclusion loss and fusion reconstruction loss to achieve the decoupling of the task correlation and task independence of the two modal features; output target classification and positioning; during testing, the trained tracking branch is used for target tracking; The multi-expert distillation framework includes a CNN-based image fusion model, a diffusion-based image fusion model, and a Mamba-based image fusion model; given an RGB modality image and a TIR modality image, I PFA , I c , I d , I m respectively represent the fusion images of PFA, the CNN-based image fusion model, the diffusion-based image fusion model, and the Mamba-based image fusion model; therefore, the process of multi-expert adaptive distillation is as follows: wherein denotes the IOU predictor of the single-stream tracker, denotes the weight of the i-th expert; the pixel-level fusion adapter parameters are optimized by minimizing the distillation loss ​ 7. The pixel-level fusion method of claim 6, wherein, Two auxiliary branches include a feature extraction backbone and a decoder; the tracking branch includes a feature extraction backbone with the same structure as the auxiliary branch, and a tracking head.

8. The pixel-level fusion method of claim 7, wherein, The definition of the repulsion loss is: where (x) + denotes max(0, x), and a is a margin parameter that controls the similarity threshold. The fusion reconstruction operation is to combine task-related and task-irrelevant features and put them into the decoder of the two auxiliary branches to reconstruct the original modality image, which is represented as: where D rgb and D tir denote the RGB and TIR mode encoders, respectively, denotes the mean square error function.

Images