An RGBT tracking enhancement fusion method and system based on difference information

Abstract

The application provides an RGBT tracking enhancement fusion method and system based on difference information, and the method comprises the following steps: input data processing; multi-modal ViT feature extraction, wherein an ADEF module is integrated in a specific layer (such as the 4th layer, the 7th layer or the 10th layer) of the ViT, and is used for enhancing the specific feature representation of the mode. The ADEF module calculates and subtracts a cross-modal fusion attention matrix; feature difference enhancement fusion, wherein an FDEF module is applied after the last layer of the ViT. The FDEF module retains and amplifies the unique features of each mode by subtracting a cross-modal similarity matrix, and further improves the quality of multi-modal fusion; target state prediction. The application solves the technical problems of impure feature representation, imperfect fusion strategy and limited adaptability.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision target tracking, and in particular to an RGBT tracking enhancement fusion method and system based on difference information. BACKGROUND

[0002] Video target tracking, as one of the core tasks in the field of computer vision, has not only aroused extensive research interest in the academic field in recent years, but also triggered a large number of application explorations in the industrial field. For details, see the following references: F. Leon and M. Gavrilescu, “A review of tracking and trajectory prediction methods for autonomous driving,” Mathematics, vol. 9, no. 6, p. 660, 2021. / S. Angadi and S. Nandyal, “A review on object detection and tracking in video surveillance,” International Journal of Advanced Research in Engineering and Technology, vol. 11, no. 9, 2020. The complexity and challenge of this task mainly lie in the following aspects: the non-specificity of the tracked target, the unpredictable changes of the target in the video sequence, and various interference factors. Due to the non-specificity of the target, the tracker cannot improve the performance by pre-training or modeling as in some specific tasks. However, in the actual tracking process, the target may disappear, change in appearance, be disturbed by the background, or move quickly, all of which greatly increase the difficulty of target tracking.

[0003] Traditionally, target tracking mainly relies on RGB images, and features such as color, texture, spatial relationship, etc. are extracted by carefully designed algorithms to achieve. See the following references for details: D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui, “Visual object tracking using adaptive correlation filters,” in 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 2010, pp. 2544-2550 / J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” IEEE transactions on pattern analysis and machine intelligence, vol. 37, no. 3, pp. 583-596, 2014. However, in complex scenes such as illumination changes, occlusions, target deformations, etc., the performance of these methods is often greatly reduced.With the rapid development of deep learning technology, especially the wide application of convolutional neural networks (CNN) and Transformer models, see the following references: L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II 14. Springer, 2016, pp. 850-865 / D. Guo, J. Wang, Y. Cui, Z. Wang, and S. Chen, “Siamcar: Siamese fully convolutional classification and regression for visual tracking,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2020, pp. 6269-6277, the accuracy and robustness of target tracking have been significantly improved. Researchers began to use these deep learning models to extract the deep features of the target, which are more expressive and generalizable than traditional hand-designed features, thus significantly improving the tracking performance. In addition, recurrent neural networks (RNN), long short-term memory networks (LSTM), etc. are also introduced into target tracking to process the temporal information in video sequences, further promoting the development of target tracking technology.

[0004] With the continuous development of target tracking technology, its application fields are also expanding. In the security field, target tracking technology is widely used in video surveillance, face recognition, etc. In the intelligent transportation system, it helps to realize vehicle detection and tracking, and improves the intelligent level of traffic management. In the field of human-computer interaction, target tracking technology enables computers to collect video signals of humans in front of computers without contact, and understand human actions and intentions. In addition, target tracking has a wide range of applications in medical imaging, agricultural automation, intelligent industry, etc.

[0005] Despite the significant progress brought by deep learning, single modality (e.g., using only RGB images) based object tracking methods still face many limitations in complex and dynamic scenarios. To overcome these limitations, researchers have begun to explore multi-modal fusion techniques, particularly the fusion of RGB (visible light) and TIR (thermal infrared) modalities. RGB images provide rich texture and color information, which is crucial for object identification and tracking. However, in low-light or highly variable lighting conditions, the performance of RGB images can be severely affected. In contrast, thermal infrared images are not affected by lighting conditions and can provide stable target information at night or in poor lighting conditions. Therefore, the fusion of RGB and thermal infrared images is expected to enable all-weather object tracking.

[0006] However, most of the existing RGBT tracking methods directly fuse the information of RGB and thermal modalities, ignoring the diversity information between different modalities. This simple fusion method may lead to the loss of key features and the introduction of background noise, thus affecting the tracking performance, see the following references: B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High performance visual tracking with siamese region proposal network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8971-8980 / B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan, “Siamrpn++: Evolution of siamese visual tracking with very deep networks,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019, pp. 4282-4291 / Z. Zhang, H. Peng, J. Fu, B. Li, and W. Hu, “Ocean: Object-aware anchor-free tracking,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16. Springer, 2020, pp. 771-787. To solve this problem, some researches try to improve the multi-modal fusion effect by introducing additional modules or strategies.For example, a dual-branch network is constructed to process RGB and TIR modalities separately to reduce inter-modal interference; or a specific attention mechanism is designed to enhance inter-modal interaction, see the following references: T. Hui, Z. Xun, F. Peng, J. Huang, X. Wei, X. Wei, J. Dai, J. Han, and S. Liu, “Bridging search region interaction with template for rgb-t tracking,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13630-13639 / B. Cao, J. Guo, P. Zhu, and Q. Hu, “Bi-directional adapter for multi modal tracking,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 2, 2024, pp. 927-935, to extract more useful features.

[0007] However, these methods still have deficiencies in handling the difference information between RGB and TIR modalities, and ignore the important differences between RGB and thermal infrared modalities in feature extraction and cross-modal information fusion, which can suppress some useful features and limit tracking performance. On the one hand, simple shared weights or cross-attention mechanisms often fail to effectively distinguish unique features between different modalities, leading to the introduction of background noise in the fusion process, reducing the purity of feature representation and the accuracy of tracking. On the other hand, existing methods mostly focus on the fusion of shared information between modalities, ignoring the importance of difference information between modalities, which limits the adaptability of tracking models in complex scenes, because different modalities often provide complementary information. Only by fully utilizing these information can more robust tracking be achieved.

[0008] To address the problem of impure feature representation, existing methods often lack effective means to accurately distinguish and retain unique features of each modality when processing RGB and TIR dual-modal data. This leads to a large amount of background noise being mixed in during the feature representation process, seriously interfering with the accuracy of tracking.

[0009] To address the problem of imperfect fusion strategy, existing attention mechanisms mainly focus on shared information between modalities when extracting and fusing RGB and TIR features, ignoring the rich value of difference information between modalities. This fusion method not only limits the further improvement of fusion effect, but also may lead to the loss of key information.

[0010] To address the problem of limited adaptability, existing methods often show poor adaptability and robustness when facing complex scenes such as illumination changes, occlusions, and scale changes.

[0011] The existing patent application document with publication number CN118736363A, "RGB-T visual multi-modal feature extraction method based on Transformer", includes: taking RGB-TIR image pairs as input, extracting features of RGB images and TIR images respectively, obtaining RGB feature vectors and TIR feature vectors, and processing them to make their shapes the same; taking the processed feature vectors as input, fusing template region features and search region features to obtain initial visual hint vectors of RGB images and TIR images respectively; using a feature weight distribution module to interactively learn and fuse RGB features and TIR features to obtain weighted features with interactive fusion information.

[0012] The foregoing prior art proposes an RGB-T visual multi-modal feature extraction method based on Transformer, focusing on fusing RGB and TIR features through self-attention mechanism and interactive learning. This existing technology fuses RGB features and TIR features through a feature weight distribution module, and the enhancement effect of feature representation and the multi-modal fusion effect need to be improved. The foregoing prior art has deficiencies in the unified representation of inter-modal collaboration and specificity, and fails to fully utilize the difference information between modalities when fusing features.

[0013] The prior art patent application document with the publication number CN117333876A discloses a feature enhancement multi-modal target tracking method and device based on attention query. The method includes: obtaining a multi-scale image feature vector set and a text feature vector, performing channel aggregation based attention query on the image feature vector set, calculating the correlation weight of each feature channel and the text feature vector, weighting the multi-scale feature tensor, and generating an image feature representation; according to a modal self-enhancement strategy, the image feature representation and the text feature vector are respectively enhanced in internal characteristics, and a cross-modal mutual enhancement strategy is used to mutually supplement and strengthen the image feature representation and the text feature vector; and a deep decoding network is used to analyze the accurate boundary and position information of the target in the image, and generate a multi-modal target tracking result. The prior art discloses a feature enhancement multi-modal target tracking method based on attention query, which focuses on extracting features by using a pre-trained deep image encoder and a text encoder, and enhancing the features by using an attention query mechanism. The prior art discloses a modal self-enhancement strategy and a cross-modal mutual enhancement strategy. However, the prior art has defects in the unified representation of inter-modal collaboration and specificity.

[0014] The prior art patent application document with the publication number CN117746200A discloses a multi-modal adaptive fusion method and system based on second-order attention. The method includes: introducing second-order cross-attention on the attention block to realize multi-modal attention mutual enhancement, and taking different inputs for different regions of attention, and also considering the horizontally adjacent regions; when screening background tokens, the attention weights of the two modal branches are used to sort all tokens, and the background tokens are eliminated according to a fixed proportion.

[0015] The prior art multi-modal adaptive fusion method based on second-order attention focuses on introducing second-order cross-attention on the attention block to realize multi-modal attention mutual enhancement. The prior art discloses a unified candidate elimination module, which sorts and eliminates all features by comprehensively considering multi-modal attention weights. The prior art has a large calculation amount in the feature fusion algorithm, which leads to a low inference speed.

[0016] In summary, the prior art has technical problems of impure feature representation, imperfect fusion strategy and limited adaptability. SUMMARY

[0017] The technical problem to be solved by the present application is how to solve the technical problems of impure feature representation, imperfect fusion strategy and limited adaptability in the prior art.

[0018] The present application solves the above technical problems by adopting the following technical solution: an RGBT tracking enhancement fusion method based on difference information includes:

[0019] S1, receive and split the RGB image and the thermal infrared TIR image pair, get and flatten the image blocks, get the image sequence; use the embedding layer and the position coding to convert the image sequence into a feature vector;

[0020] S2, perform a multi-modal ViT feature extraction operation, pass the feature vector through the ViT to perform feature extraction and relationship modeling, capture the intra-modal correlation information based on the self-attention mechanism; integrate the ADEF module in the last layer of the ViT, calculate and subtract the cross-modal fusion attention matrix, obtain the RGB modality feature representation enhancement vector and the TIR modality feature representation enhancement vector;

[0021] S3, perform a feature difference enhancement fusion operation, integrate and apply the FDEF module in the last layer of the ViT, to obtain and subtract the cross-modal similarity matrix, retain and amplify the modal unique features in the RGB modality feature representation enhancement vector and the TIR modality feature representation enhancement vector, and perform fusion through element-level addition to obtain enhanced multi-modal features;

[0022] S4, predict the target state according to the enhanced multi-modal features, predict the target position and state by using the preset tracking head, input the multi-modal image to be tracked into the DEFNet, and output the tracking result.

[0023] The application proposes a new difference-based enhanced fusion network DEFNet, which is a new difference-based enhanced fusion network. By introducing the cross-modal difference enhancement module ADEF and the feature difference enhancement module FDEF, the difference information between cross-modal is used to enhance the feature representation and multi-modal fusion. The redundant background information is filtered out, and the more meaningful target area is enhanced, so that more robust and accurate RGBT tracking can be realized in complex visual scenes.

[0024] The application first attempts to use difference information to enhance feature representation and multi-modal fusion in the field of RGBT tracking. By introducing the ADEF and FDEF modules, the DEFNet not only successfully solves the problems of impure feature representation, imperfect fusion strategy and limited adaptability in existing methods, but also significantly improves the robustness and accuracy of RGBT tracking.

[0025] In a more specific technical solution, S2 includes:

[0026] S21, calculate the self-attention matrix for the RGB modality and the TIR modality in the feature vector respectively;

[0027] S22, calculate the cross-modal fusion attention matrix for the RGB modality and the TIR modality;

[0028] S23, subtract the cross-modal fusion attention matrix from the self-attention matrix to obtain a difference attention matrix:

[0029] Attn' r = Attn r - Attn fused

[0030] Attn' t = Attn t - Attn fused

[0031] S24, according to the difference attention matrix, the difference feature is obtained by processing;

[0032] S25, add the difference feature to the original feature, and pass through the residual connection to complete the feature fusion operation in the ADEF module:

[0033]

[0034] The application adopts a difference attention mechanism, reduces the influence of shared information between modes, and enhances mode-specific features. The application adopts a difference feature fusion strategy, and improves the multi-modal fusion quality by bidirectional fusion of difference feature information.

[0035] In a more specific technical solution S21, let and respectively represent the feature representation of the RGB mode and the TIR mode after the lth layer of the Transformer, wherein, is the template feature, is the search area feature;

[0036] Using the following logic, the query matrix, the key matrix and the value matrix are obtained by linear transformation:

[0037]

[0038] In the formula, W Q ,W K ,W V is a linear transformation matrix.

[0039] Using the following logic, the self-attention matrix is calculated:

[0040]

[0041] In the formula, d k is the dimension of the key.

[0042] In a more specific technical solution, S22, using the following logic, the cross-modal fusion attention matrix between the RGB mode and the TIR mode is calculated:

[0043]

[0044] In the formula, represents element-wise multiplication.

[0045] In a more specific technical solution, in S24, the difference attention matrix is used to weight and sum the value matrix to obtain the difference feature using the following logic:

[0046] F r ′=Attn′ r V r

[0047] F t ′=Attn′ t V t .

[0048] In a more specific technical solution, S3 includes:

[0049] S31, cross-modal attention calculation is performed to enhance the RGB modality feature representation vector and the TIR modality feature representation vector, respectively calculate the cross-modal attention weight, and generate a cross-modal fusion feature;

[0050] S32, difference feature calculation is performed, and the cross-modal fusion feature is subtracted from the original feature to obtain a unique difference feature:

[0051]

[0052] S33, cross-modal attention recalculation is performed on the unique difference feature to obtain an enhanced difference feature;

[0053] S34, multi-modal feature fusion is performed, and the enhanced difference feature is added to the TIR modality query matrix to obtain an enhanced multi-modal feature:

[0054] F fused =MLP(F″ rt +Q t )

[0055] In the formula, MLP represents a multi-layer perceptron.

[0056] DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) modality features by introducing an attention difference base fusion module (ADEF) and a feature difference base fusion module (FDEF). Compared with methods that rely only on simple cross-attention mechanisms, DEFNet can more accurately extract unique information from each modality, reduce background noise interference, improve feature representation quality, and enhance feature representation ability.

[0057] In a more specific technical solution, in S31, the cross-modal attention weight is obtained by using the following logic:

[0058]

[0059] In the formula, Q t ,K r ,V r are the query matrix of the TIR modality, the key matrix and the value matrix of the RGB modality respectively.

[0060] In a more specific technical solution, in S33, the cross-modal attention recalculation operation is performed by using the following logic:

[0061]

[0062] In the formula, Delta K is optional and is used to fine-tune the key matrix, which is simplified here to directly use K r .

[0063] In a more specific technical solution, in S4, the enhanced multi-modal feature is input to the tracking head, and the target position and state are predicted through classification and regression operations; the network parameters are optimized by minimizing the tracking error on no less than two benchmark datasets; and the multi-modal image to be tracked is input to the DEFNet to obtain the tracking result.

[0064] The existing multi-modal fusion method often focuses on the fusion of shared information between modes, and ignores the preservation of mode-specific information. The present application optimizes multi-modal fusion, and the DEFNet models the difference information, which not only preserves the complementary advantages between modes, but also highlights the unique features of each mode, thereby realizing higher quality multi-modal fusion. This feature is verified in the experimental results on multiple benchmark datasets, and the fusion effect of the DEFNet is obviously better than that of other algorithms.

[0065] In a more specific technical solution, an RGBT tracking enhancement fusion system based on difference information includes:

[0066] The input data processing module is used to receive and segment the RGB image and the thermal infrared TIR image pair, obtain and flatten the image blocks, and obtain the image sequence; the image sequence is converted into a feature vector by using an embedding layer and position encoding;

[0067] The multi-modal ViT feature extraction module is used for multi-modal ViT feature extraction operation, and the feature vector is extracted through the ViT to perform feature extraction and relationship modeling, so as to capture the internal correlation information of the modal based on the self-attention mechanism; in the ViT integrated ADEF module, the cross-modal fusion attention matrix is calculated and subtracted, and the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector are obtained; the multi-modal ViT feature extraction module is connected with the input data processing module;

[0068] The feature difference enhancement fusion module is used for feature difference enhancement fusion operation, and the FDEF module is integrated and applied in the last layer of the ViT to obtain and subtract the cross-modal similarity matrix, so as to retain and amplify the modal unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and the enhanced multi-modal features are obtained through element-level addition, and the feature difference enhancement fusion module is connected with the multi-modal ViT feature extraction module;

[0069] The target state prediction module is used for target state prediction according to the enhanced multi-modal features, and the preset tracking head is used to predict the target position and state, the multi-modal image to be tracked is input into the DEFNet, and the tracking result is output, and the target state prediction module is connected with the feature difference enhancement fusion module.

[0070] Compared with the prior art, the present application has the following advantages:

[0071] The present application proposes a new difference-based enhancement fusion network DEFNet, which is a new difference-based enhancement fusion network. By introducing the cross-modal difference enhancement module ADEF and the feature difference enhancement module FDEF, the difference information between the cross-modal is used to enhance the feature representation and the multi-modal fusion. The redundant background information is filtered out, and the more meaningful target area is enhanced, so that more robust and accurate RGBT tracking is realized in the complex and variable visual scene.

[0072] The present application focuses on enhancing the feature representation through difference information, and emphasizes the ADEF module and the FDEF module to enhance the feature representation and the multi-modal fusion. The present application calculates and subtracts the cross-modal fusion attention matrix through the ADEF module, so as to reduce the background noise and enhance the target related features, and then the FDEF module subtracts the cross-modal similarity matrix to retain and amplify the unique features of each modal, so as to further improve the quality of multi-modal fusion. Through the use of difference information, the present application enhances the unity representation of the collaboration and specificity between the modes, so as to improve the accuracy and robustness of tracking, and solves the problem of insufficient feature fusion.

[0073] The present application focuses on enhancing feature representation through differential information, especially between RGB and TIR modalities. The present application specifically implements self-enhancement and mutual enhancement of features through ADEF and FDEF two modules. The present application enhances the unity of inter-modality synergy and specificity through the use of differential information, thereby improving the accuracy and robustness of tracking.

[0074] The present application focuses on enhancing feature representation and multi-modal fusion through ADEF and FDEF modules. The present application implements feature enhancement and fusion through ADEF and FDEF modules. The present application reduces the interference of background noise through ADEF and FDEF modules, improves the quality of feature representation, and thereby improves the inference speed.

[0075] The present application first attempts to use differential information to enhance feature representation and multi-modal fusion in the field of RGBT tracking. By introducing ADEF and FDEF two modules, DEFNet not only successfully solves the problems of impure feature representation, imperfect fusion strategy and limited adaptability in existing methods, but also significantly improves the robustness and accuracy of RGBT tracking.

[0076] The present application adopts a differential attention mechanism to enhance modality-specific features by reducing the influence of shared information between modalities. The present application adopts a differential feature fusion strategy to improve multi-modal fusion quality by bidirectional fusion of differential feature information.

[0077] DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) modality features by introducing an attention difference base fusion module (ADEF) and a feature difference base fusion module (FDEF). Compared with methods that rely only on simple cross-attention mechanisms, DEFNet can more accurately extract unique information from each modality, reduce the interference of background noise, improve the quality of feature representation, and enhance feature representation capability.

[0078] Existing multi-modal fusion methods often focus on the fusion of shared information between modalities, while ignoring the preservation of modality-specific information. The present application optimizes multi-modal fusion. DEFNet models differential information, preserves the complementary advantages between modalities, and highlights the unique features of each modality, thereby achieving higher quality multi-modal fusion. This feature is verified in the experimental results on multiple benchmark datasets, and the fusion effect of DEFNet is significantly better than other algorithms.

[0079] The present application solves the technical problems of impure feature representation, imperfect fusion strategy and limited adaptability in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1This is a schematic diagram of the overall network framework of an RGBT tracking enhancement fusion method based on differential information according to Embodiment 1 of the present invention;

[0081] Figure 2 This is a schematic diagram of the basic steps of an RGBT tracking enhancement fusion method based on differential information according to Embodiment 1 of the present invention;

[0082] Figure 3 This is a schematic diagram of data stream processing for an RGBT tracking enhancement fusion method based on differential information according to Embodiment 1 of the present invention;

[0083] Figure 4 This is a schematic diagram of the ADEF module structure in Embodiment 1 of the present invention;

[0084] Figure 5 This is a schematic diagram of the FDEF module structure in Embodiment 1 of the present invention;

[0085] Figure 6 This is a schematic diagram of the attention visualization of the ADEF module in Embodiment 1 of the present invention;

[0086] Figure 7 This is a comparison diagram of an RGBT tracking enhancement fusion method based on differential information in Embodiment 2 of the present invention with other algorithms. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] Example 1

[0089] like Figure 1 and Figure 2 As shown, the present invention provides an RGBT tracking enhancement fusion method based on differential information, which includes the following basic steps:

[0090] S1, Input data processing;

[0091] In the image input process of this embodiment, the input RGB image and thermal infrared (TIR) ​​image pair are received, including the template image and the search area image.

[0092] In the image preprocessing process of this embodiment, the RGB and TIR images are segmented into small blocks, and these blocks are flattened into sequences. Subsequently, embedding layers and positional encoding are applied to convert these sequences into feature vectors.

[0093] S2, multi-modal ViT feature extraction;

[0094] In the feature extraction process of the present embodiment, the processed feature vector is subjected to feature extraction and relationship modeling by VisionTransformer (ViT) with shared parameters. ViT captures the correlation within each modality through self-attention mechanism.

[0095] In the present embodiment, ADEF module integration is performed, specifically, ADEF modules are integrated in specific layers (such as the 4th layer, the 7th layer, and the 10th layer) of ViT for enhancing modality-specific feature representation. ADEF modules reduce background noise and enhance target-related features by calculating and subtracting cross-modal fusion attention matrices.

[0096] As shown in Figure 6 , in the present embodiment, the ADEF (Attention Discrepancy-based Enhancement Fusion) module aims to enhance the unique feature representation of each modality by suppressing the shared information between modalities. Referring to Figure 6 , a represents the RGB and TIR search region. b represents the original self-attention diagram. c represents the difference-based attention diagram.

[0097] In the ADEF module of the present embodiment, self-attention matrix calculation is performed: for the RGB modality and the TIR modality, self-attention matrices are calculated respectively. Let and represent the feature representation of the RGB and TIR modalities after the lth layer of Transformer respectively, where is the template feature, is the search region feature.

[0098] In the present embodiment, the Query, Key, and Value matrices are obtained through linear transformation:

[0099]

[0100] where W Q , W K , and W V are linear transformation matrices.

[0101] In the present embodiment, the self-attention matrix calculation is:

[0102]

[0103]

[0104] where dk is the dimension of the key.

[0105] In the ADEF module of the embodiment, cross-modal fusion attention matrix calculation is performed to calculate the cross-modal fusion attention matrix between the RGB and TIR modalities:

[0106]

[0107] wherein, represents element-wise multiplication.

[0108] In the ADEF module of the embodiment, difference attention matrix calculation is performed, specifically, the difference attention matrix is obtained by subtracting the cross-modal fusion attention matrix from the self-attention matrix:

[0109] Attn′ r =Attn r -Attn fused

[0110] Attn′ t =Attn t -Attn fused

[0111] In the ADEF module of the embodiment, difference feature calculation is performed, specifically, the difference feature is obtained by weighted sum of the value matrix using the difference attention matrix:

[0112] F r ′=Attn′ r V r

[0113] F t ′=Attn′ t V t

[0114] In the ADEF module of the embodiment, feature fusion is performed, specifically, the difference feature is added to the original feature and transmitted through the residual connection:

[0115]

[0116] S3, feature difference enhancement fusion;

[0117] In the application process of the FDEF module of the embodiment, the FDEF module is applied after the last layer of ViT. The FDEF module retains and amplifies the unique features of each modality by subtracting the cross-modal similarity matrix, further improving the quality of multi-modal fusion.

[0118] In this embodiment, feature fusion is performed, specifically, the features of the RGB and TIR modalities are fused through element-level addition to obtain enhanced multi-modal features.

[0119] The FDEF (Feature Discrepancy-based Enhancement Fusion) module calculates a feature similarity matrix through a cross-modal attention mechanism, and uses difference information to amplify unique features of each modality, thereby achieving high-quality multi-modal fusion.

[0120] In the FDEF module of this embodiment, cross-modal attention calculation is performed, specifically, taking the RGB modality as an example (the TIR modality operates the same), the cross-modal attention weight of the TIR modality to the RGB modality is calculated, and a fused feature is generated:

[0121]

[0122] wherein Q t ,K r ,V r are the query matrix of the TIR modality, the key matrix of the RGB modality, and the value matrix, respectively.

[0123] In the FDEF module of this embodiment, difference feature calculation is performed, specifically, the difference feature is obtained by subtracting the cross-modal fused feature from the original feature, which aims to retain unique information of the RGB modality:

[0124]

[0125] In the FDEF module of this embodiment, cross-modal attention calculation is performed again, specifically, in order to make more full use of the difference information, the difference feature is calculated again through cross-modal attention, to obtain an enhanced difference feature:

[0126]

[0127] wherein ΔK is optional and is used to fine-tune the key matrix, and here it is simplified to directly use K r .

[0128] In the FDEF module of this embodiment, multi-modal feature fusion is performed, specifically, the enhanced difference feature is added to (or spliced and then passed through an MLP) the query matrix of the TIR modality, to realize fusion of the multi-modal features:

[0129] F fused =MLP(F″ rt +Q t )

[0130] Wherein, the MLP is a multi-layer perceptron, used for further processing the fused features.

[0131] S4, target state prediction;

[0132] In this embodiment, the tracking head processing is performed, specifically, the fused features are input into the tracking head, and the position and state of the target are predicted through classification and regression operations, for example, a bounding box.

[0133] Output results, specifically, the network parameters are optimized by minimizing the tracking error through training on multiple benchmark datasets. The multi-modal image to be tracked is input into the DEFNet, and the tracking result is output. In this embodiment, the tracking result includes but is not limited to: the bounding box of the target, the confidence.

[0134] Embodiment 2

[0135] The DEFNet effectively enhances the model's extraction ability of RGB and thermal infrared (TIR) modal features by introducing an attention difference base fusion module (ADEF) and a feature difference base fusion module (FDEF). Compared with methods that only rely on simple cross-attention mechanisms, DEFNet can more accurately extract unique information of each modality, reduce the interference of background noise, improve the quality of feature representation, and enhance the feature representation ability.

[0136] Existing multi-modal fusion methods often focus on the fusion of shared information between modalities, while ignoring the preservation of unique information of modalities. DEFNet models the difference information, which not only preserves the complementary advantages between modalities, but also highlights the unique features of each modality, thereby achieving higher quality multi-modal fusion. This feature is verified in the experimental results on multiple benchmark datasets, and the fusion effect of DEFNet is obviously better than other algorithms, optimizing the multi-modal fusion:

[0137] To evaluate the performance of the tracker, the present embodiment conducts experiments on four RGBT tracking benchmark datasets, including GTOT (see document: C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, and L. Lin, “Learning collaborative sparse representation for grayscale-thermal tracking,” IEEE Transactions on Image Processing, vol. 25, no. 12, pp. 5743-5756, 2016), RGBT210 (see document: C. Li, N. Zhao, Y. Lu, C. Zhu, and J. Tang, “Weighted sparse representation regularized graph learning for rgb-t object tracking,” Proceedings of the 25th ACM international conference on Multimedia, 2017.. JOURNAL OF LATEX CLASSFILES, VOL. 14, NO. 8, AUGUST 2021 11), RGBT234 (see document: C. Li, X. Liang, Y. Lu, N. Zhao, and J. Tang, “Rgb-t object tracking: Benchmark and baseline,” Pattern Recognition, vol. 96, p. 106977, 2019), and LasHeR (see document: C. Li, W. Xue, Y. Jia, Z. Qu, B. Luo, J. Tang, and D. Sun, “Lasher: A large-scale high-diversity benchmark for RGBT tracking,” IEEE Trans. Image Process., vol. 31, pp. 392-404, 2022). The experimental results show that DEFNet performs excellently in key indicators such as PR and SR, significantly outperforming existing advanced algorithms. Especially in handling complex scenes such as occlusion, fast motion, low light, etc., DEFNet exhibits stronger robustness and accuracy. The tracking performance of the model is significantly improved, see Table 1 below for details:

[0138]

[0139] As shown in Table 1 above, the PR, NPR and SR scores of the DEFNet of the present application are compared with those of other trackers on GTOT, RGBT210, RGBT234 and LasHeR datasets. The best, second and third results are marked with red, blue and green fonts, respectively.

[0140] As Figure 7As shown, in this embodiment, compared with other algorithms, the DEFNet adopted by the present application shows higher efficiency in processing large-scale and high-variety data sets. For example: the experimental results on the LasHeR data set show that DEFNet not only leads in tracking performance, but also has obvious advantages in processing speed. This shows that DEFNet can more efficiently complete the target tracking task when processing complex data sets. DEFNet and various advanced algorithms (such as CAT

[17] , ADRNet

[18] , TBSI

[19] , see the literature: C. Li, L. Liu, A. Lu, Q. Ji, and J. Tang, “Challenge-aware rgbt tracking,” in European conference on computer vision. Springer, 2020, pp. 222237 / P. Zhang, D. Wang, H. Lu, and X. Yang, “Learning adaptive attribute driven representation for real-time rgb-t tracking,” International Journal of Computer Vision, vol. 129, pp. 2714-2729, 2021 / T. Hui, Z. Xun, F. Peng, J. Huang, X. Wei, X. Wei, J. Dai, J. Han, and S. Liu, “Bridging search region interaction with template for rgb-t tracking,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13630-13639) have carried out comprehensive comparative experiments on multiple benchmark data sets. The experimental results show that DEFNet performs well in various indicators, especially in processing challenging scenes with significant advantages. This further proves the advancement and practicality of DEFNet in the RGBT tracking task. DEFNet also has good scalability and flexibility, and can easily adapt to different tracking tasks and scene requirements. In future research, the application potential of DEFNet in more complex scenes can be further explored, and DEFNet can also be combined with other advanced deep learning algorithms to further improve the performance and accuracy of target tracking.It is believed that with the continuous progress of technology and the continuous optimization of algorithms, DEFNet will play a more important role in the field of RGBT tracking and contribute new strength to the development of computer vision technology. The present application introduces an enhanced fusion method based on difference information, designs two modules, and enhances the modal specific information by suppressing the modal shared information irrelevant to the target from the difference information angle, effectively solving the problems of impure feature representation, imperfect fusion strategy and limited adaptability in the existing scheme, and improving the robustness and accuracy of RGBT tracking.

[0141] In summary, the present application proposes a new difference-based enhanced fusion network DEFNet. By introducing the cross-modal difference enhancement module ADEF and the feature difference enhancement module FDEF, the difference information between cross-modal is used to enhance the feature representation and multi-modal fusion. Redundant background information will be filtered out, and more meaningful target regions will be enhanced, so that more robust and accurate RGBT tracking can be achieved in complex visual scenes.

[0142] The present application first attempts to use difference information to enhance feature representation and multi-modal fusion in the field of RGBT tracking. By introducing ADEF and FDEF two modules, DEFNet not only successfully solves the problems of impure feature representation, imperfect fusion strategy and limited adaptability in existing methods, but also significantly improves the robustness and accuracy of RGBT tracking.

[0143] The present application adopts a difference attention mechanism to enhance modal specific features by reducing the influence of shared information between modalities. The present application adopts a difference feature fusion strategy to improve the quality of multi-modal fusion by bidirectional fusion of difference feature information.

[0144] DEFNet effectively enhances the model's ability to extract RGB and thermal infrared (TIR) modal features by introducing an attention difference-based fusion module (ADEF) and a feature difference-based fusion module (FDEF). Compared with methods that rely only on simple cross-attention mechanisms, DEFNet can more accurately extract unique information from each modality, reducing background noise interference, improving feature representation quality, and enhancing feature representation capabilities.

[0145] Existing multi-modal fusion methods often focus on the fusion of shared information between modalities, while ignoring the preservation of modal specific information. The present application optimizes multi-modal fusion, and DEFNet models difference information, preserves the complementary advantages between modalities, and highlights the unique features of each modality, thereby achieving higher quality multi-modal fusion. This feature is verified in the experimental results on multiple benchmark datasets, and the fusion effect of DEFNet is significantly better than other algorithms.

[0146] The present application solves the technical problems of impure feature representation, imperfect fusion strategy and limited adaptability in the prior art.

[0147] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An RGBT tracking enhancement fusion method based on differential information, characterized in that, The method includes: S1. Receive and segment the RGB image and TIR image pair, obtain and flatten the image blocks to obtain the image sequence; use the embedding layer and position encoding to convert the image sequence into a feature vector; S2. Perform multimodal ViT feature extraction operation, extract features and model relationships through ViT to capture intramodal correlation information based on self-attention mechanism; integrate ADEF module in ViT, calculate and subtract cross-modal fusion attention matrix to obtain RGB modal feature representation enhancement vector and TIR modal feature representation enhancement vector; S2 includes: S21. Calculate the self-attention matrix Attn for the RGB and TIR modes in the feature vectors, respectively. r Attn t ; S22. For RGB and TIR modes, calculate the cross-modal fusion attention matrix Attn. fused ; S23. Subtract the cross-modal fusion attention matrix from the self-attention matrix to obtain the differential attention matrix Attn'. r Attn' t : Attn' r =Attn r -Attn fused Attn' t =Attn t -Attn fused S24. Based on the difference attention matrix, process to obtain the difference features; S25. Add the differential features to the original features and pass them through the residual connection to complete the feature fusion operation in the ADEF module: S3. Perform feature difference enhancement fusion operation. Integrate and apply the FDEF module in the last layer of ViT to obtain and subtract the cross-modal similarity matrix, retain and amplify the modal unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and fuse them through element-level addition to obtain enhanced multimodal features. S3 includes: S31. Perform cross-modal attention calculation, calculate the cross-modal attention weights for the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector respectively, and generate cross-modal fusion features; S32. Perform differential feature calculation: subtract the cross-modal fusion features from the original features to obtain the unique differential feature F'. rt : S33. Perform cross-modal attention recomputation on unique differential features to obtain enhanced differential features; S34. Perform multimodal feature fusion by adding the enhanced differential features to the query matrix of the TIR modality and fusing them to obtain the enhanced multimodal features: F fused =MLP(F″ rt +Q t ) In the formula, MLP represents a multilayer perceptron; S4. Predict the target state based on the enhanced multimodal features. Use the preset tracking head to predict the target position and state. Input the multimodal image to be tracked into DEFNet and output the tracking results.

2. The RGBT tracking enhancement fusion method based on differential information according to claim 1, characterized in that, In S21, let and The RGB mode and the TIR mode after the l-th Transformer are respectively represented, where, For template features, Features of the search region; The query matrix, key matrix, and value matrix are obtained through linear transformation: In the formula, W Q W K W V It is a linear transformation matrix; Calculate the self-attention matrix: In the formula, d k It is the dimension of the key.

3. The RGBT tracking enhancement fusion method based on differential information according to claim 1, characterized in that, In step S22, the cross-modal fusion attention matrix between the RGB modality and the TIR modality is calculated: In the formula, ° represents element-wise multiplication.

4. The RGBT tracking enhancement fusion method based on differential information according to claim 1, characterized in that, In step S24, the value matrix is ​​weighted and summed using the difference attention matrix to obtain the difference features: F r '=To n 'rV r F t '=Attn' t V t 。 5. The RGBT tracking enhancement fusion method based on differential information according to claim 1, characterized in that, In step S31, the cross-modal attention weights are calculated: In the formula, Q t ,K r V r These are the query matrix for the TIR mode, and the key and value matrices for the RGB mode, respectively.

6. The RGBT tracking enhancement fusion method based on differential information according to claim 1, characterized in that, In step S33, the cross-modal attention recalculation operation is performed: In the formula, ΔK is optional and used to fine-tune the bond matrix; here, it is simplified to use K directly. r .

7. The RGBT tracking enhancement fusion method based on differential information according to claim 1, characterized in that, In step S4, the enhanced multimodal features are input into the tracking head, and the target position and state are predicted through classification and regression operations; the network is trained on at least two benchmark datasets, and the network parameters are optimized by minimizing the tracking error; the multimodal image to be tracked is input into DEFNet to obtain the tracking result.

8. An RGBT tracking enhancement fusion system based on difference information, used to execute the RGBT tracking enhancement fusion method based on difference information as described in any one of claims 1 to 7, characterized in that, The system includes: The input data processing module receives and segments RGB image and TIR image pairs, obtains and flattens image blocks to obtain an image sequence; and uses an embedding layer and positional encoding to convert the image sequence into a feature vector. A multimodal ViT feature extraction module is used to perform multimodal ViT feature extraction operations. It extracts features and models relationships using ViT to capture intramodal correlation information based on a self-attention mechanism. In the ViT integrated ADEF module, a cross-modal fusion attention matrix is ​​calculated and subtracted to obtain RGB modal feature representation enhancement vectors and TIR modal feature representation enhancement vectors. The multimodal ViT feature extraction module is connected to the input data processing module. The feature difference enhancement and fusion module is used to perform feature difference enhancement and fusion operations. It integrates and applies the FDEF module in the last layer of the ViT to obtain and subtract the cross-modal similarity matrix, retain and amplify the modal-unique features in the RGB modal feature representation enhancement vector and the TIR modal feature representation enhancement vector, and fuses them through element-level addition to obtain enhanced multimodal features. The feature difference enhancement and fusion module is connected to the multimodal ViT feature extraction module. The target state prediction module is used to predict the target state based on the enhanced multimodal features, predict the target position and state using a preset tracking head, input the multimodal image to be tracked into DEFNet, and output the tracking result. The target state prediction module is connected to the feature difference enhancement fusion module.

Images