RGBT tracking enhancement fusion method and system based on difference information

Abstract

The invention provides an RGBT tracking enhancement fusion method and system based on difference information. The method comprises the following steps: processing input data; and multi-modal ViT feature extraction, in which an ADEF module is integrated in a specific layer (such as a fourth layer, a seventh layer and a tenth layer) of ViT and is used for enhancing modal specific feature representation. The ADEF module calculates and subtracts the cross-modal fusion attention matrix; and feature difference enhancement fusion is carried out, and after the last layer of the ViT, an FDEF module is applied. The FDEF module retains and amplifies the unique features of each mode by subtracting the cross-modal similarity matrix, so that the quality of multi-modal fusion is further improved; and target state prediction. According to the method, the technical problems of impure feature representation, incomplete fusion strategy and limited adaptive capacity are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology target tracking, and in particular to an RGBT tracking enhancement fusion method and system based on difference information. Background Art

[0002] As one of the core tasks in the field of computer vision, video object tracking has not only aroused extensive research interest in academia in recent years, but also triggered a large number of application explorations in the industry. For details, please refer to the following literature: F. Leon and M. Gavrilescu, "A review of tracking and trajectory prediction methods for autonomous driving," Mathematics, vol. 9, no. 6, p. 660, 2021. / S. Angadi and S. Nandyal, "A review on object detection and tracking in video surveillance," International Journal of Advanced Research in Engineering and Technology, vol. 11, no. 9, 2020. The complexity and challenge of this task are mainly reflected in several aspects: the non-specificity of the tracked target, the unpredictable changes of the target in the video sequence, and various interference factors. Due to the non-specificity of the target, the tracker cannot improve its performance through pre-training or modeling as in some specific tasks. In the actual tracking process, the target may disappear, change its appearance, be disturbed by the background, or move quickly, which greatly increases the difficulty of target tracking.

[0003] Traditionally, object tracking mainly relies on RGB images and is achieved by extracting features such as color, texture, and spatial relationships through carefully designed algorithms. For details, please refer to the following literature: D.S. Bolme, J.R. Beveridge, B.A. D. Raper, and Y.M. Lui, "Visual object tracking using adaptive correlation filters," in 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2010, pp. 2544–2550 / J.F. Henriques, R. Caseiro, P. Martins, and J. Batista, "High-speed tracking ing with kernelized correlation filters," IEEE transactions on pattern analysis and machine intelligence, vol. 37, no. 3, pp. 583–596, 2014. However, the performance of these methods is often greatly reduced in complex scenes such as illumination changes, occlusion, and object deformation.With the rapid development of deep learning technology, especially the widespread application of convolutional neural networks (CNN) and Transformer models, see the following literature for details: L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, "Fully-convolutional siamese networks for object tracking," in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14. Springer, 2016, pp. 850–865 / D. Guo, J. Wang, Y. Cui, Z. Wang, and S. Chen, "Siamcar: Siamese fully convolutional classification and regression for visual tracking," in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 6269–6277, the accuracy and robustness of target tracking have been significantly improved. Researchers have begun to use these deep learning models to extract deep features of targets, which are more expressive and generalizable than traditional hand-designed features, thereby significantly improving tracking performance. In addition, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) have also been introduced into target tracking to process temporal information in video sequences, further promoting the development of target tracking technology.

[0004] With the continuous development of target tracking technology, its application areas are also expanding. In the field of security, target tracking technology is widely used in scenarios such as video surveillance and face recognition; in intelligent transportation systems, it helps to achieve vehicle detection and tracking, and improves the level of intelligent traffic management; in the field of human-computer interaction, target tracking technology enables computers to collect video signals of humans in front of computers without contact, and understand human actions and intentions. In addition, target tracking is also widely used in medical imaging, agricultural automation, smart industry and other fields.

[0005] Although deep learning has brought significant progress to target tracking, single-modality target tracking methods (such as using only RGB images) still face many limitations in complex and changing scenarios. To overcome these limitations, researchers have begun to explore multimodal fusion techniques, especially the fusion of RGB (visible light) and TIR (thermal infrared) modalities. RGB images provide rich texture and color information, which is essential for target recognition and tracking. However, in scenes with insufficient lighting or drastic lighting changes, the performance of RGB images may be severely affected. In contrast, thermal infrared images are not affected by lighting conditions and can provide stable target information at night or in harsh lighting conditions. Therefore, the fusion of RGB and thermal infrared images is expected to achieve all-weather target tracking.

[0006] However, most existing RGBT tracking methods directly fuse the information of RGB and thermal infrared modalities, while ignoring the difference information between different modalities. This simple fusion method may lead to the loss of key features and the introduction of background noise, thus affecting the tracking performance. For details, see the following literature: B.Li, J.Yan, W.Wu, Z.Zhu, and X.Hu, "Highperformance visual tracking with siamese region proposal network," in Proceedings of the IEEE conference on computer vision and patternrecognition, 2018, pp.8971–8980 / B.Li, W.Wu, Q.Wang, F.Zhang, J.Xing, and J.Yan, "Siamrpn++: Evolution of siamese visual tracking with very deep networks," in Proceedings of the IEEE / CVF conference on computer vision and patternrecognition, 2019, pp.4282–4291 / Z.Zhang, H.Peng, J.Fu, B.Li, and W.Hu, "Ocean: Object-aware anchor-free tracking," in Computer Vision–ECCV 2020:16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16. Springer, 2020, pp. 771–787. To solve this problem, some studies have tried to improve the multimodal fusion effect by introducing additional modules or strategies.For example, a dual-branch network is constructed to process RGB and TIR modalities respectively to reduce the interference between modalities; or a specific attention mechanism is designed to enhance the interaction between modalities, as shown in the following references: T.Hui, Z.Xun, F.Peng, J.Huang, X.Wei, X.Wei, J.Dai, J.Han, and S.Liu, “Bridging search region interactionwith template for rgb-t tracking,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp.13630–13639 / B.Cao, J.Guo, P.Zhu, and Q.Hu, “Bi-directional adapter for multi modal tracking,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol.38, no.2, 2024, pp.927–935, so as to extract more useful features.

[0007] Nevertheless, these methods still have shortcomings in processing the difference information between RGB and TIR modalities. They ignore the important differences between RGB and thermal infrared modalities during feature extraction and cross-modal information fusion, which may suppress some useful features and thus limit tracking performance. On the one hand, simple shared weights or cross-attention mechanisms often cannot effectively distinguish the unique features between different modalities, resulting in the introduction of background noise during the fusion process, reducing the purity of feature representation and tracking accuracy. On the other hand, most existing methods focus on the fusion of shared information between modalities, while ignoring the importance of difference information between modalities. This practice of ignoring difference information limits the adaptability of tracking models in complex scenarios, because different modalities often provide complementary information. Only by making full use of this information can more robust tracking be achieved.

[0008] In order to solve the problem of impure feature representation, existing methods often lack effective means to accurately distinguish and retain the unique features of each modality when processing RGB and TIR bimodal data. This results in a large amount of background noise being mixed in during the feature representation process, which seriously interferes with the tracking accuracy.

[0009] In response to the problem of imperfect fusion strategy, the existing attention mechanism mainly focuses on the shared information between modalities when extracting and fusing RGB and TIR features, but ignores the rich value contained in the difference information between modalities. This fusion method not only limits the further improvement of the fusion effect, but also may lead to the loss of key information.

[0010] In order to address the problem of limited adaptability, existing methods often exhibit poor adaptability and robustness when faced with complex scenes such as illumination changes, occlusions, and scale changes.

[0011] The existing invention patent application document with publication number CN118736363A is "A Transformer-based RGB-T visual multimodal feature extraction method". The existing method includes: taking an RGB-TIR image pair as input, extracting the features of the RGB image and the features of the TIR image respectively, obtaining an RGB feature vector and a TIR feature vector, and processing them to make them have the same shape; taking the processed feature vector as input, fusing the template area features and the search area features, and obtaining the initial visual cue vectors of the RGB image and the TIR image respectively; using a feature weight allocation module to interactively learn and fuse the RGB features and the TIR features, and obtaining weighted features with interactive fusion information.

[0012] The above-mentioned prior art proposes a Transformer-based RGB-T visual multimodal feature extraction method, which focuses on fusing RGB and TIR features through self-attention mechanism and interactive learning. This prior art fuses RGB features and TIR features through interactive learning of feature weight allocation module, and the enhancement effect of feature representation and multimodal fusion effect need to be improved. The above-mentioned prior art has deficiencies in the unified representation of synergy and specificity between modalities, and fails to fully utilize the difference information between modalities when fusing features.

[0013] The existing invention patent application document "Feature-enhanced multimodal target tracking method and device based on attention query" with publication number CN117333876A, the existing method includes: obtaining a multi-scale image feature vector set and a text feature vector, performing a channel aggregation-based attention query on the image feature vector set, calculating the correlation weights of each feature channel and the text feature vector, and weighting the multi-scale feature tensor to generate an image feature representation; according to the modal self-enhancement strategy, the image feature representation and the text feature vector are respectively enhanced in internal characteristics, and the image feature representation and the text feature vector are complemented and strengthened by the cross-modal mutual enhancement strategy; using a deep decoding network, the precise boundary and position information of the target in the image are parsed to generate a multimodal target tracking result. The existing scheme proposes a feature-enhanced multimodal target tracking method based on attention query, focusing on extracting features using pre-trained deep image encoders and text encoders, and enhancing features through an attention query mechanism. The existing scheme proposes a modal self-enhancement strategy and a cross-modal mutual enhancement strategy. In terms of the unified representation of inter-modal synergy and specificity, the prior art is insufficient in terms of the unified representation of inter-modal synergy and specificity.

[0014] The existing invention patent application document "Multimodal Adaptive Fusion Method and System Based on Second-Order Attention" with publication number CN117746200A, the existing method includes: introducing second-order cross-attention on the attention block to realize multimodal attention mutual enhancement, while taking different inputs for different areas of attention, and taking horizontally adjacent areas into consideration; when screening background tokens, all tokens are sorted using the attention weights of the two modal branches, and background tokens are eliminated according to a fixed ratio.

[0015] The aforementioned prior art is based on a multimodal adaptive fusion method of second-order attention, focusing on introducing second-order cross attention on the attention block to achieve multimodal attention mutual enhancement. The aforementioned prior art proposes a unified candidate elimination module, which sorts and screens all features by integrating multimodal attention weights. The aforementioned prior art has a large amount of computation in the feature fusion algorithm, resulting in a low inference speed.

[0016] In summary, the existing technologies have technical problems such as impure feature representation, imperfect fusion strategy and limited adaptability. Summary of the invention

[0017] The technical problem to be solved by the present invention is: how to solve the technical problems of impure feature representation, imperfect fusion strategy and limited adaptability in the prior art.

[0018] The present invention adopts the following technical solution to solve the above technical problem: an RGBT tracking enhancement fusion method based on difference information includes:

[0019] S1, receiving and segmenting RGB image and thermal infrared TIR image pairs, obtaining and flattening image blocks to obtain image sequences; using embedding layers and position encoding, converting the image sequences into feature vectors;

[0020] S2. Perform multimodal ViT feature extraction operations, pass the feature vector through ViT, perform feature extraction and relationship modeling, and capture the internal correlation information of the modality based on the self-attention mechanism; integrate the ADEF module in ViT, calculate and subtract the cross-modal fusion attention matrix, and obtain the RGB modality feature representation enhancement vector and the TIR modality feature representation enhancement vector;

[0021] S3, perform feature difference enhancement fusion operation, integrate and apply the FDEF module in the last layer of ViT to obtain and subtract the cross-modal similarity matrix, retain and amplify the modal unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and fuse them through element-level addition to obtain enhanced multimodal features;

[0022] S4. Predict the target state based on the enhanced multimodal features, use the preset tracking head to predict the target position and state, input the multimodal image to be tracked into DEFNet, and output the tracking result.

[0023] This paper proposes a novel difference-based enhanced fusion network of DEFNet, a novel difference-based enhanced fusion network. By introducing the cross-modal difference enhancement module ADEF and the feature difference enhancement module FDEF, the cross-modal difference information is used to enhance the feature representation and multimodal fusion. Redundant background information will be filtered out, and more meaningful target areas will be enhanced, thereby achieving more robust and accurate RGBT tracking in complex and changing visual scenes.

[0024] This paper attempts to use difference information to enhance feature representation and multimodal fusion for the first time in the field of RGBT tracking. By introducing two modules, ADEF and FDEF, DEFNet not only successfully solves the problems of impure feature representation, imperfect fusion strategy and limited adaptability in existing methods, but also significantly improves the robustness and accuracy of RGBT tracking.

[0025] In a more specific technical solution, S2 includes:

[0026] S21, calculate the self-attention matrix for the RGB mode and TIR mode in the feature vector respectively;

[0027] S22, for RGB modality and TIR modality, calculate the cross-modal fusion attention matrix;

[0028] S23. Subtract the cross-modal fusion attention matrix from the self-attention matrix to obtain the difference attention matrix:

[0029] Attn′ r =Attn r -Attn fused

[0030] Attn′ t =Attn t -Attn fused

[0031] S24, processing to obtain difference features according to the difference attention matrix;

[0032] S25, add the difference feature to the original feature and pass it through the residual connection to complete the feature fusion operation in the ADEF module:

[0033]

[0034] The present invention adopts a differential attention mechanism to enhance modality-specific features by reducing the impact of shared information between modalities. The present invention adopts a differential feature fusion strategy to improve the quality of multimodal fusion by bidirectionally fusing differential feature information.

[0035] In a more specific technical solution, in S21, it is assumed that and They represent the feature representation of RGB mode and TIR mode after the l-th layer Transformer respectively, where: is the template feature, To search for regional features;

[0036] Using the following logic, we can obtain the query matrix, key matrix, and value matrix through linear transformation:

[0037]

[0038] Where W Q ,W K ,W V is the linear transformation matrix.

[0039] Using the following logic, the self-attention matrix is ​​calculated:

[0040]

[0041] Where, d k is the dimension of the key.

[0042] In a more specific technical solution, in S22, the following logic is used to calculate the cross-modal fusion attention matrix between the RGB modality and the TIR modality:

[0043]

[0044] In the formula, Represents element-wise multiplication.

[0045] In a more specific technical solution, in S24, the following logic is used to perform weighted summation on the value matrix using the difference attention matrix to obtain the difference feature:

[0046] F r ′=Attn′ r V r

[0047] F t ′=Attn′ t V t .

[0048] In more specific technical solutions, S3 includes:

[0049] S31, performing cross-modal attention calculation, calculating cross-modal attention weights for the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and generating cross-modal fusion features;

[0050] S32. Calculate the difference feature, subtract the cross-modal fusion feature from the original feature, and obtain the unique difference feature:

[0051]

[0052] S33, performing a cross-modal attention recalculation operation on the unique difference feature to obtain an enhanced difference feature;

[0053] S34, perform multimodal feature fusion, add the enhanced difference feature to the query matrix of the TIR modality, and fuse them to obtain the enhanced multimodal feature:

[0054] F fused =MLP(F″ rt +Q t )

[0055] Where MLP stands for multi-layer perceptron.

[0056] DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) ​​modal features by introducing the attention difference base fusion module (ADEF) and feature difference base fusion module (FDEF). Compared with methods that only rely on simple cross-attention mechanisms, DEFNet can more accurately extract the unique information of each modality, reduce the interference of background noise, improve the quality of feature representation, and enhance feature representation capabilities.

[0057] In a more specific technical solution, in S31, the following logic is used to obtain the cross-modal attention weight:

[0058]

[0059] In the formula, Q t ,K r ,V r They are the query matrix of TIR modality, the key matrix and the value matrix of RGB modality respectively.

[0060] In a more specific technical solution, in S33, the following logic is used to perform a cross-modal attention recalculation operation:

[0061]

[0062] Where ΔK is optional and is used to fine-tune the key matrix. Here, it is simplified to directly use K r .

[0063] In a more specific technical solution, in S4, the enhanced multimodal features are input into the tracking head, and the target position and state are predicted through classification and regression operations; training is performed on no less than 2 benchmark data sets, and the network parameters are optimized by minimizing the tracking error; the multimodal image to be tracked is input into DEFNet to obtain the tracking result.

[0064] Existing multimodal fusion methods often focus on the fusion of shared information between modalities, while ignoring the retention of modality-specific information. The present invention optimizes multimodal fusion. DEFNet uses differential information modeling to retain the complementary advantages between modalities and highlight the unique characteristics of each modality, thereby achieving higher-quality multimodal fusion. This feature has been verified in the experimental results on multiple benchmark datasets, and the fusion effect of DEFNet is significantly better than other algorithms.

[0065] In a more specific technical solution, an RGBT tracking enhancement fusion system based on difference information includes:

[0066] An input data processing module is used to receive and segment the RGB image and thermal infrared TIR image pairs, obtain and flatten image blocks, and obtain image sequences; and use an embedding layer and position encoding to convert the image sequence into a feature vector;

[0067] The multimodal ViT feature extraction module is used to perform multimodal ViT feature extraction operations, pass the feature vector through ViT, perform feature extraction and relationship modeling, and capture the internal correlation information of the modality based on the self-attention mechanism; the ADEF module is integrated in ViT to calculate and subtract the cross-modal fusion attention matrix to obtain the RGB modality feature representation enhancement vector and the TIR modality feature representation enhancement vector. The multimodal ViT feature extraction module is connected to the input data processing module;

[0068] The feature difference enhancement fusion module is used to perform the feature difference enhancement fusion operation. The FDEF module is integrated and applied in the last layer of ViT to obtain and subtract the cross-modal similarity matrix, retain and amplify the modal unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and fuse them through element-level addition to obtain enhanced multimodal features. The feature difference enhancement fusion module is connected to the multimodal ViT feature extraction module;

[0069] The target state prediction module is used to predict the target state based on the enhanced multimodal features, use the preset tracking head to predict the target position and state, input the multimodal image to be tracked into DEFNet, and output the tracking result. The target state prediction module is connected to the feature difference enhancement fusion module.

[0070] Compared with the prior art, the present invention has the following advantages:

[0071] This paper proposes a novel difference-based enhanced fusion network of DEFNet, a novel difference-based enhanced fusion network. By introducing the cross-modal difference enhancement module ADEF and the feature difference enhancement module FDEF, the cross-modal difference information is used to enhance the feature representation and multimodal fusion. Redundant background information will be filtered out, and more meaningful target areas will be enhanced, thereby achieving more robust and accurate RGBT tracking in complex and changing visual scenes.

[0072] The present invention focuses on enhancing feature representation through differential information, emphasizing the use of ADEF modules and FDEF modules to enhance feature representation and multimodal fusion. The present invention calculates and subtracts the cross-modal fusion attention matrix through the ADEF module to reduce background noise and enhance target-related features. The FDEF module then subtracts the cross-modal similarity matrix to retain and amplify the unique features of each modality, further improving the quality of multimodal fusion. The present invention enhances the unified representation of synergy and specificity between modalities through the use of differential information, thereby improving the accuracy and robustness of tracking and solving the problem of insufficient feature fusion.

[0073] The present invention focuses on enhancing feature representation through difference information, especially between RGB and TIR modalities. The present invention specifically implements the self-enhancement and mutual enhancement of features through two modules, ADEF and FDEF. The present invention enhances the unified representation of synergy and specificity between modalities through the use of difference information, thereby improving the accuracy and robustness of tracking.

[0074] The present invention focuses on enhancing feature representation and multimodal fusion through ADEF and FDEF modules. The present invention realizes feature enhancement and fusion through ADEF and FDEF modules. The present invention reduces the interference of background noise and improves the quality of feature representation through ADEF and FDEF modules, thereby improving the inference speed.

[0075] This paper attempts to use difference information to enhance feature representation and multimodal fusion for the first time in the field of RGBT tracking. By introducing two modules, ADEF and FDEF, DEFNet not only successfully solves the problems of impure feature representation, imperfect fusion strategy and limited adaptability in existing methods, but also significantly improves the robustness and accuracy of RGBT tracking.

[0076] The present invention adopts a differential attention mechanism to enhance modality-specific features by reducing the impact of shared information between modalities. The present invention adopts a differential feature fusion strategy to improve the quality of multimodal fusion by bidirectionally fusing differential feature information.

[0077] DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) ​​modal features by introducing the attention difference base fusion module (ADEF) and feature difference base fusion module (FDEF). Compared with methods that only rely on simple cross-attention mechanisms, DEFNet can more accurately extract the unique information of each modality, reduce the interference of background noise, improve the quality of feature representation, and enhance feature representation capabilities.

[0078] Existing multimodal fusion methods often focus on the fusion of shared information between modalities, while ignoring the retention of modality-specific information. The present invention optimizes multimodal fusion. DEFNet uses differential information modeling to retain the complementary advantages between modalities and highlight the unique characteristics of each modality, thereby achieving higher-quality multimodal fusion. This feature has been verified in the experimental results on multiple benchmark datasets, and the fusion effect of DEFNet is significantly better than other algorithms.

[0079] The present invention solves the technical problems existing in the prior art of impure feature representation, imperfect fusion strategy and limited adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1This is a schematic diagram of the overall network framework of an RGBT tracking enhancement fusion method based on difference information according to Example 1 of the present invention;

[0081] Figure 2 This is a schematic diagram of basic steps of an RGBT tracking enhancement fusion method based on difference information according to Example 1 of the present invention;

[0082] Figure 3 This is a data flow processing schematic diagram of an RGBT tracking enhancement fusion method based on difference information according to Example 1 of the present invention;

[0083] Figure 4 Schematic diagram of the ADEF module structure of Example 1 of the present invention;

[0084] Figure 5 This is a schematic diagram of the FDEF module structure of Example 1 of the present invention;

[0085] Figure 6 Schematic diagram of attention visualization of the ADEF module according to Embodiment 1 of the present invention;

[0086] Figure 7 This is an experimental comparison diagram of an RGBT tracking enhancement fusion method based on difference information in Example 2 of the present invention and other algorithms. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] Example 1

[0089] like Figure 1 and Figure 2 As shown, the present invention provides an RGBT tracking enhancement fusion method based on difference information, including the following basic steps:

[0090] S1. Input data processing;

[0091] In the image input process of this embodiment, an input RGB image and a thermal infrared (TIR) ​​image pair are received, including a template image and a search area image.

[0092] In the image preprocessing process of this embodiment, the RGB and TIR images are segmented into small blocks, and these small blocks are flattened into sequences. Subsequently, the embedding layer and position encoding are applied to convert these sequences into feature vectors.

[0093] S2, multimodal ViT feature extraction;

[0094] In the feature extraction process of this embodiment, the processed feature vector is passed through the Vision Transformer (ViT) with shared parameters for feature extraction and relationship modeling. ViT captures the correlation within each modality through the self-attention mechanism.

[0095] In this embodiment, ADEF module integration is performed, specifically, ADEF modules are integrated in specific layers of ViT (such as layer 4, layer 7, and layer 10) to enhance modality-specific feature representation. The ADEF module reduces background noise and enhances target-related features by calculating and subtracting the cross-modal fusion attention matrix.

[0096] like Figure 6 As shown in FIG. 1 , in this embodiment, the ADEF (Attention Discrepancy-based Enhancement Fusion) module aims to enhance the unique feature representation of each modality by suppressing the shared information between modalities. Figure 6 ,a, represents the RGB and TIR search areas. b, represents the original self-attention diagram. c, represents the difference-based attention diagram.

[0097] In the ADEF module of this embodiment, the self-attention matrix is ​​calculated: for the RGB mode and the TIR mode, the self-attention matrix is ​​calculated respectively. and Respectively represent the feature representation of RGB and TIR modalities after the lth layer Transformer, where is the template feature, To search for regional features.

[0098] In this embodiment, the query (Query), key (Key) and value (Value) matrices are obtained through linear transformation:

[0099]

[0100] Among them, W Q ,W K ,W V is the linear transformation matrix.

[0101] In this embodiment, the self-attention matrix is ​​calculated as:

[0102]

[0103]

[0104] Among them, dk is the dimension of the key.

[0105] In the ADEF module of this embodiment, a cross-modal fusion attention matrix is ​​calculated to calculate the cross-modal fusion attention matrix between RGB and TIR modalities:

[0106]

[0107] in, Represents element-wise multiplication.

[0108] In the ADEF module of this embodiment, the difference attention matrix is ​​calculated. Specifically, the difference attention matrix is ​​obtained by subtracting the cross-modal fusion attention matrix from the self-attention matrix:

[0109] Attn′ r =Attn r -Attn fused

[0110] Attn′ t =Attn t -Attn fused

[0111] In the ADEF module of this embodiment, the difference feature calculation is performed. Specifically, the difference attention matrix is ​​used to perform weighted summation on the value matrix to obtain the difference feature:

[0112] F r ′=Attn′ r V r

[0113] F t ′=Attn′ t V t

[0114] In the ADEF module of this embodiment, feature fusion is performed. Specifically, the difference feature is added to the original feature and transmitted through the residual connection:

[0115]

[0116] S3, feature difference enhancement fusion;

[0117] In the FDEF module application process of this embodiment, the FDEF module is applied after the last layer of ViT. The FDEF module retains and amplifies the unique features of each modality by subtracting the cross-modal similarity matrix, further improving the quality of multimodal fusion.

[0118] In this embodiment, feature fusion is performed. Specifically, the features of the RGB and TIR modalities are fused by element-level addition to obtain enhanced multi-modal features.

[0119] The FDEF (Feature Discrepancy-based Enhancement Fusion) module calculates the feature similarity matrix through a cross-modal attention mechanism and uses the difference information to amplify the unique features of each modality to achieve high-quality multimodal fusion.

[0120] In the FDEF module of this embodiment, cross-modal attention calculation is performed. Specifically, taking the RGB modality as an example (the TIR modality operation is the same), the cross-modal attention weight of the TIR modality to the RGB modality is calculated, and a fusion feature is generated:

[0121]

[0122] Among them, Q t ,K r ,V r They are the query matrix of TIR modality, the key matrix and the value matrix of RGB modality respectively.

[0123] In the FDEF module of this embodiment, the difference feature is calculated. Specifically, the difference feature is obtained by subtracting the cross-modal fusion feature from the original feature. This step is intended to retain the unique information of the RGB modality:

[0124]

[0125] In the FDEF module of this embodiment, cross-modal attention calculation is performed again. Specifically, in order to make fuller use of the difference information, cross-modal attention calculation is performed again on the difference feature to obtain enhanced difference features:

[0126]

[0127] Among them, ΔK is optional and is used to fine-tune the bond matrix. Here, it is simplified to directly use K r .

[0128] In the FDEF module of this embodiment, multimodal feature fusion is performed. Specifically, the enhanced difference feature is added to the query matrix of the TIR modality (or concatenated and passed through an MLP) to achieve multimodal feature fusion:

[0129] F fused =MLP(F″ rt +Q t )

[0130] Among them, MLP is a multi-layer perceptron, which is used to further process the fused features.

[0131] S4, target state prediction;

[0132] In this embodiment, tracking head processing is performed. Specifically, the fused features are input into the tracking head, and the position and state of the target, such as a bounding box, are predicted through classification and regression operations.

[0133] Output results, specifically, training is performed on multiple benchmark data sets to optimize network parameters by minimizing tracking errors. The multimodal image to be tracked is input into DEFNet, and the tracking results are output. In this embodiment, the tracking results include but are not limited to: the bounding box and confidence of the target.

[0134] Example 2

[0135] In the RGBT tracking enhancement fusion method based on difference information provided by the present invention, DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) ​​modal features by introducing the attention difference base fusion module (ADEF) and the feature difference base fusion module (FDEF). Compared with the method that only relies on a simple cross-attention mechanism, DEFNet can more accurately extract the unique information of each modality, reduce the interference of background noise, improve the quality of feature representation, and enhance the feature representation capability.

[0136] Existing multimodal fusion methods often focus on the fusion of shared information between modalities, while ignoring the preservation of modality-specific information. DEFNet retains the complementary advantages between modalities and highlights the unique characteristics of each modality through differential information modeling, thereby achieving higher-quality multimodal fusion. This feature has been verified in experimental results on multiple benchmark datasets. The fusion effect of DEFNet is significantly better than other algorithms, optimizing multimodal fusion:

[0137] In order to evaluate the performance of the tracker, this embodiment conducts experiments on four RGBT tracking benchmark datasets, including GTOT (see reference: C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, and L. Lin, "Learning collaborative sparse representation for grayscale-thermal tracking," IEEE Transactions on Image Processing, vol. 25, no. 12, pp. 5743–5756, 2016), RGBT210 (see reference: C. Li, N. Zhao, Y. Lu, C. Zhu, and J. Tang, "Weighted sparse representation regularized graph learning for rgb-t object tracking," Proceediings of the 25th ACM international conference on Multimedia, 2017.. JOURNAL OF LATEX CLASSFILES, VOL. 14, NO. 8, AUGUST 2021 11), RGBT234 (see reference: C.Li, X.Liang, Y.Lu, N.Zhao, and J.Tang, "RGB-t object tracking: Benchmark and baseline," PatternRecognition, vol.96, p.106977, 2019) and LasHeR (see reference: C.Li, W.Xue, Y.Jia, Z.Qu, B.Luo, J.Tang, and D.Sun, "Lasher: A large-scale high-diversity benchmark for RGBT tracking," IEEE Trans.Image Process., vol.31, pp.392–404, 2022). Experimental results The experimental results show that DEFNet performs well in key indicators such as PR and SR, significantly outperforming existing advanced algorithms. In particular, when dealing with complex scenes (such as occlusion, fast motion, low light, etc.), DEFNet shows stronger robustness and accuracy. The tracking performance of the model is significantly improved, as shown in Table 1 below:

[0138]

[0139] As shown in Table 1 above, on the GTOT, RGBT210, RGBT234 and LasHeR datasets, the PR, NPR and SR scores of the DEFNet of the present invention are compared with those of other trackers. The best result, the second and the third results are marked in red, blue and green fonts respectively.

[0140] like Figure 7As shown, in this embodiment, compared with other algorithms, DEFNet used in the present invention shows higher efficiency in processing large-scale, high-diversity data sets. For example, the experimental results on the LasHeR data set show that DEFNet not only leads in tracking performance, but also has a significant advantage in processing speed. This shows that DEFNet can complete the target tracking task more efficiently when processing complex data sets. DEFNet and a variety of advanced algorithms (such as CAT

[17] , ADRNet

[18] , TBSI

[19] , see the literature: C.Li, L.Liu, A.Lu, Q.Ji, and J.Tang, "Challenge-aware rgbttracking," in European conference on computer vision. Springer, 2020, pp.222237 / P.Zhang, D.Wang, H.Lu, and X.Yang, "Learning adaptive attribute driven representation for real-time rgb-t tracking," International Journal of Computer Vision, vol.129, pp.2714–2729, 2021 / T.Hui, Z.Xun, F.Peng, J.Huang, X.Wei, X.Wei, J.Dai, J.Han, and S.Liu, "Bridging search region interaction with template for rgb-t tracking,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13630–13639) conducted comprehensive comparative experiments on multiple benchmark datasets. The experimental results show that DEFNet performs well in all indicators, especially in dealing with challenging scenes. This further proves the advancement and practicality of DEFNet in RGBT tracking tasks. DEFNet also has good scalability and flexibility, and can easily adapt to different tracking tasks and scene requirements. In future research, the application potential of DEFNet in more complex scenarios can be further explored. At the same time, DEFNet can also be combined with other advanced deep learning algorithms to further improve the performance and accuracy of target tracking.It is believed that with the continuous advancement of technology and the continuous optimization of algorithms, DEFNet will play a more important role in the field of RGBT tracking and contribute new strength to the development of computer vision technology. This invention introduces an enhanced fusion method based on difference information, designs two modules, and enhances modality-specific information from the perspective of difference information by suppressing modality-shared information that is not related to the target. It effectively solves the problems of impure feature representation, imperfect fusion strategy, and limited adaptability in existing solutions, and improves the robustness and accuracy of RGBT tracking.

[0141] In summary, the present invention proposes a novel difference-based enhanced fusion network of DEFNet, a novel difference-based enhanced fusion network. By introducing the cross-modal difference enhancement module ADEF and the feature difference enhancement module FDEF, the cross-modal difference information is used to enhance the feature representation and multimodal fusion. Redundant background information will be filtered out, and more meaningful target areas will be enhanced, thereby achieving more robust and accurate RGBT tracking in complex and changing visual scenes.

[0142] This paper attempts to use difference information to enhance feature representation and multimodal fusion for the first time in the field of RGBT tracking. By introducing two modules, ADEF and FDEF, DEFNet not only successfully solves the problems of impure feature representation, imperfect fusion strategy and limited adaptability in existing methods, but also significantly improves the robustness and accuracy of RGBT tracking.

[0143] The present invention adopts a differential attention mechanism to enhance modality-specific features by reducing the impact of shared information between modalities. The present invention adopts a differential feature fusion strategy to improve the quality of multimodal fusion by bidirectionally fusing differential feature information.

[0144] DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) ​​modal features by introducing the attention difference base fusion module (ADEF) and feature difference base fusion module (FDEF). Compared with methods that only rely on simple cross-attention mechanisms, DEFNet can more accurately extract the unique information of each modality, reduce the interference of background noise, improve the quality of feature representation, and enhance feature representation capabilities.

[0145] Existing multimodal fusion methods often focus on the fusion of shared information between modalities, while ignoring the retention of modality-specific information. The present invention optimizes multimodal fusion. DEFNet uses differential information modeling to retain the complementary advantages between modalities and highlight the unique characteristics of each modality, thereby achieving higher-quality multimodal fusion. This feature has been verified in the experimental results on multiple benchmark datasets, and the fusion effect of DEFNet is significantly better than other algorithms.

[0146] The present invention solves the technical problems existing in the prior art of impure feature representation, imperfect fusion strategy and limited adaptability.

[0147] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A RGBT tracking enhancement fusion method based on difference information, characterized in that: The method comprises: S1, receiving and segmenting RGB image and thermal infrared TIR image pair, obtaining and flattening image blocks, and obtaining image sequence; using embedding layer and position encoding, converting the image sequence into feature vector; S2. Perform a multimodal ViT feature extraction operation, pass the feature vector through ViT, perform feature extraction and relationship modeling, so as to capture the internal correlation information of the modality based on the self-attention mechanism; integrate the ADEF module in the ViT, calculate and subtract the cross-modal fusion attention matrix, and obtain the RGB modality feature representation enhancement vector and the TIR modality feature representation enhancement vector; S3, performing feature difference enhancement fusion operation, integrating and applying the FDEF module in the last layer of the ViT to obtain and subtract the cross-modal similarity matrix, retaining and amplifying the modal unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and fusing them through element-level addition to obtain enhanced multimodal features; S4. Predict the target state according to the enhanced multimodal features, use the preset tracking head to predict the target position and state, input the multimodal image to be tracked into DEFNet, and output the tracking result.

2. According to the RGBT tracking enhancement fusion method based on difference information according to claim 1, it is characterized in that: The S2 includes: S21, calculating the self-attention matrix for the RGB mode and the TIR mode in the feature vector respectively; S22. Calculate a cross-modal fusion attention matrix for the RGB modality and the TIR modality; S23. Subtract the cross-modal fusion attention matrix from the self-attention matrix to obtain a difference attention matrix: Attn′ r =Attn r -Attn fused Attn′ t =Attn t -Attn fused S24, processing to obtain difference features according to the difference attention matrix; S25, adding the difference feature to the original feature and transferring them through the residual connection to complete the feature fusion operation in the ADEF module:

3. The RGBT tracking enhancement fusion method based on difference information according to claim 2 is characterized in that: In S21, it is assumed that and They represent the feature representation of the RGB mode and the TIR mode after the l-th layer Transformer, respectively, where: is the template feature, To search for regional features; Using the following logic, we can obtain the query matrix, key matrix, and value matrix through linear transformation: Where W Q ,W K ,W V is the linear transformation matrix. The self-attention matrix is ​​calculated using the following logic: Where, d k is the dimension of the key.

4. The RGBT tracking enhancement fusion method based on difference information according to claim 2 is characterized in that: In S22, the cross-modal fusion attention matrix between the RGB modality and the TIR modality is calculated using the following logic: Where ° represents element-by-element multiplication.

5. The RGBT tracking enhancement fusion method based on difference information according to claim 2 is characterized in that: In S24, the following logic is used to perform weighted summation on the value matrix using the difference attention matrix to obtain the difference feature: F r ′=Attn′ r V r F t ′=Attn′ t V t 。 6. The RGBT tracking enhancement fusion method based on difference information according to claim 1, characterized in that: The S3 includes: S31, performing cross-modal attention calculation, calculating cross-modal attention weights for the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and generating cross-modal fusion features; S32. Calculate the difference feature, and subtract the cross-modal fusion feature from the original feature to obtain a unique difference feature: S33, performing a cross-modal attention recalculation operation on the unique difference feature to obtain an enhanced difference feature; S34, performing multimodal feature fusion, adding the enhanced difference feature to the query matrix of the TIR modality, and fusing to obtain the enhanced multimodal feature: F fused =MLP(F″ rt +Q t ) Where MLP stands for multi-layer perceptron.

7. The RGBT tracking enhancement fusion method based on difference information according to claim 6 is characterized in that: In S31, the cross-modal attention weight is obtained using the following logic: In the formula, Q t ,K r ,V r They are the query matrix of TIR modality, the key matrix and the value matrix of RGB modality respectively.

8. The RGBT tracking enhancement fusion method based on difference information according to claim 6, characterized in that: In S33, the cross-modal attention recalculation operation is performed using the following logic: Where ΔK is optional and is used to fine-tune the key matrix. Here, it is simplified to directly use K r .

9. The RGBT tracking enhancement fusion method based on difference information according to claim 1, characterized in that: In S4, the enhanced multimodal features are input into the tracking head, and the target position and state are predicted through classification and regression operations; training is performed on no less than 2 benchmark data sets, and network parameters are optimized by minimizing the tracking error; the multimodal image to be tracked is input into DEFNet to obtain the tracking result.

10. An RGBT tracking enhancement fusion system based on difference information, characterized in that: The system comprises: An input data processing module is used to receive and segment the RGB image and the thermal infrared TIR image pair, obtain and flatten the image blocks, and obtain an image sequence; and convert the image sequence into a feature vector using an embedding layer and position encoding; A multimodal ViT feature extraction module is used to perform a multimodal ViT feature extraction operation, pass the feature vector through ViT, perform feature extraction and relationship modeling, so as to capture the internal correlation information of the modality based on the self-attention mechanism; integrate the ADEF module in the ViT, calculate and subtract the cross-modal fusion attention matrix, obtain the RGB modality feature representation enhancement vector and the TIR modality feature representation enhancement vector, and the multimodal ViT feature extraction module is connected to the input data processing module; A feature difference enhancement fusion module is used to perform a feature difference enhancement fusion operation, integrate and apply the FDEF module in the last layer of the ViT to obtain and subtract the cross-modal similarity matrix, retain and amplify the modal unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and fuse them through element-level addition to obtain enhanced multimodal features. The feature difference enhancement fusion module is connected to the multimodal ViT feature extraction module; The target state prediction module is used to predict the target state according to the enhanced multimodal features, use the preset tracking head to predict the target position and state, input the multimodal image to be tracked into DEFNet, and output the tracking result. The target state prediction module is connected to the feature difference enhancement fusion module.

Images