RGBT tracking method and system based on progressive fusion transformer and dynamic guidance learning

Abstract

The application provides an RGBT tracking method and system based on progressive fusion Transformer and dynamic guidance learning, which comprises a training process and a test process. The training process comprises data sampling and preprocessing, feature extraction, cross-branch feature fusion, template frame and search frame feature information interaction, search frame feature input into a classifier and calculation of classification loss, and training optimization by using a collaborative learning algorithm. The test process comprises data sampling and preprocessing, feature extraction, cross-branch feature fusion, template frame and current frame feature information interaction, current frame feature input into a classifier and a regressor, and prediction of the bounding box of the target in the current frame. The application solves the technical problems of loss of heterogeneous information of different modalities, neglect of the relationship between fused features and specific modal features in learning discriminative multi-modal representation, and lack of global representation capability.

Description

An RGBT tracking method and system based on progressive fusion of Transformer and dynamic guided learning. Technical Field

[0001] This invention relates to the field of computer vision, and more specifically to an RGBT tracking method and system based on progressive fusion of Transformer and dynamic guided learning. Background Technology

[0002] Visual tracking refers to the detection, extraction, recognition, and tracking of moving targets in an image sequence to obtain motion parameters such as position, velocity, acceleration, and trajectory. This allows for further processing and analysis, leading to an understanding of the moving target's behavior and enabling more advanced detection tasks. Visible light images, also known as RGB images, are visible to the naked eye and possess high texture detail and spatial resolution. Thermal infrared images, obtained based on the heat radiated by an object, are insensitive to lighting conditions. The combined use of features from both visible light and thermal infrared images enables visual trackers to perform accurate and robust tracking in challenging scenarios, such as changing lighting and inclement weather. Therefore, RGBT tracking has attracted increasing attention in the field of visual tracking.

[0003] Regarding multimodal fusion methods, the existing invention patent application document "Multimodal fusion for end-to-end RGB-T tracking" (publication number CN111476823A) and "Real-time RGBT tracking method and device based on multi-challenge interactive learning" include the following steps: S1, training a network model; the network model includes a first backbone network module, a second backbone network module, a third backbone network module, an adaptive region of interest alignment operation layer, and a fully connected layer, all connected in sequence; the first backbone network module includes a first two-stream CNN network module and a first challenge perceptron branch module; S2, tracking based on the trained network model to determine the target position. From the specific implementation details and methods of the aforementioned schemes, it can be seen that the aforementioned prior art uses simple addition and concatenation operations to fuse multimodal features, but this method ignores the information interaction between the two modalities. The paper "Duality-gated mutual condition network for RGB-T tracking" uses an attention mechanism to learn the contribution of each modality to the adaptive fusion of dual modalities. The paper "Mfgnet: Dynamic modality-aware filter generation for RGB-T tracking" designs a dynamic modality-aware filter generation network based on dynamic convolution, enhancing information transfer between visible light and thermal infrared data by adaptively adjusting the convolution kernel. Furthermore, the existing invention patent application "RGBT target tracking method based on cross-modal sharing and specific representation" (publication number CN113077491A) registers and annotates visible light video and corresponding thermal infrared video, dividing them into training and testing sets as required. It uses a VGG-M network to extract modality-specific features of different modalities, utilizes a shared feature extraction module to extract modality-shared features between two modalities, and employs a multi-scale feature fusion strategy in the shared feature extraction module to enhance features and improve robustness.In this existing technology, an RGBT target tracking model is constructed. The deep network framework used in this RGBT target tracking model is PyTorch, which includes an attention mechanism module, a shared feature extraction module, and a specific feature extraction module. The specific steps are as follows: S3.1, inputting pre-processed visible light images and thermal infrared images into the network simultaneously; S3.2, extracting specific features of different modalities, i.e., loading two VGG-M networks with the same structure but different parameters to extract features from the visible light images and thermal infrared images in the training set respectively; S3.3, using the attention mechanism module to improve the discriminativeness of the specific features of the two modalities obtained in step S3.2; S3.4, using a multi-scale feature fusion method in the shared feature extraction module to improve the robustness of the features and outputting the shared features between the two modalities; S3.5, fusing the specific features and shared features of the two modalities respectively, and then concatenating the fused features of the two modalities. However, the aforementioned existing methods either ignore modality-specific information modeling or over-model the information modeling between shared modalities. The paper "Attribute-based progressive fusion network for RGBT tracking" attempts to model global contextual information using Transformer, which can enhance the information of specific modalities in fusion features, but cannot effectively integrate complementary information between two modalities.

[0004] Regarding the utilization of the relationship between fused and specific modalities, the method "Object tracking in RGB-T videos using modal-aware attention network and competitive learning" uses a multi-branch classification network to mine the relationships between modalities. However, it treats all branches equally in joint optimization, which may lead to some branches not optimizing well. The method "RGB-T tracking by trident fusion network" uses a multi-branch classification network to integrate the learning of heterogeneous modal information. However, since convolutional schemes can only learn local features well, this method may achieve suboptimal results. These methods, however, cannot effectively model the relationship between fused and separate branches for high-quality discriminative representation learning.

[0005] In summary, existing technologies suffer from technical problems such as loss of heterogeneous information in modalities, neglect of the relationship between fused features and specific modal features when learning to discriminate multimodal representations, and lack of global representation capabilities. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to solve the technical problems of existing technologies, such as the loss of heterogeneous information in modality, the neglect of the relationship between fused features and specific modal features when learning to discriminate multimodal representations, and the lack of global representation capabilities.

[0007] This invention solves the above-mentioned technical problems by employing the following technical solution: an RGBT tracking method and system based on progressive fusion of Transformer and dynamic guided learning, comprising: a training process and a testing process.

[0008] The training process includes:

[0009] S1. From the preset sequence, acquire and preprocess the visible light and thermal infrared image pairs of the template frame and the visible light and thermal infrared image pairs of the search frame, and adjust the size of the preset template frame area and the preset search frame area to obtain the visible light and thermal infrared template frame and the visible light and thermal infrared search frame.

[0010] S2. Extract features from the visible light and thermal infrared template frames and the visible light and thermal infrared search frames to obtain template frame features and search frame features. Then, use the fusion branch network of the Transformer module to perform cross-branch feature fusion to obtain template frame fusion branch output and search frame fusion branch output. Then, interactively process the feature information of the template frame and the search frame to obtain multi-branch features of the search frame.

[0011] S3. The multi-branch features of the searched frame are fed into the classifiers of their respective branches to calculate the classification loss.

[0012] S4. Based on the classification loss, determine the applicable specific modality branch, and then use the collaborative learning algorithm to train and optimize the fusion branch to obtain the applicable network model parameters;

[0013] The testing process includes:

[0014] S1' Sample and preprocess the test data, initialize the network model using the applicable network model parameters, and process it to obtain the test template frame and the current test frame;

[0015] S2' By executing steps S2 to S4, feature extraction, cross-branch feature fusion, and information interaction operations are performed on the test template frame and the current test frame to obtain the fused modal branch features of the current frame;

[0016] S3': The current frame's fused modal branch features are fed into the classifier and regressor to predict the bounding box of the target in the current frame.

[0017] This invention proposes an RGBT visual tracking method based on Absorption Transformer. Unlike previous RGBT tracking methods, this method uses the Absorption Transformer module to effectively absorb information between fused features and specific modal features, thereby preserving information from both shared and specific modalities as much as possible, avoiding over-modeling, and improving modality integration. Furthermore, a collaborative learning algorithm is proposed to optimize model training by integrating information from specific modalities into the fused modality, resulting in more comprehensive optimization of each branch.

[0018] In a more specific technical solution, step S1 includes:

[0019] S11. Randomly sample and align the preset sequence to obtain the visible light and thermal infrared image pairs of the template frame and the visible light and thermal infrared image pairs of the search frame.

[0020] S12. Based on the preset initial bounding box, perform cropping and adjustment operations on the template frame region and the search frame region to obtain the visible light and thermal infrared template frame and the visible light and thermal infrared search frame.

[0021] In a more specific technical solution, step S2 includes:

[0022] S21. Input the visible light and thermal infrared template frames and the visible light and thermal infrared search frames into the preset backbone network ResNet50 to extract the template frame features and search frame features.

[0023] S22. Add the different modal features of the template frame features and the search frame features and feed them into the fusion branch network. Use multi-head attention and trans-encoder to obtain enhanced fusion features.

[0024] S23. Based on the enhanced fusion features, perform cross-attention calculation using pre-set logic to obtain the cross-attention output of the template frame and the cross-attention output of the search frame;

[0025] S24. Add the template frame cross-attention output and the search frame cross-attention output together and send them to the layer normalization module (Norm) to obtain the template frame fusion branch output and the search frame fusion branch output.

[0026] S25. Combine the outputs of each branch of the template frame and the outputs of each branch of the search frame into feature pairs, and interactively process the feature information of the template frame and the search frame to obtain the enhanced multi-branch features of the search frame.

[0027] In a more specific technical solution, in step S21, the pre-configured backbone network includes ResNet50.

[0028] In a more specific technical solution, step S22 includes:

[0029] S221. Add the features of different modalities to obtain the fused modal feature to be enhanced, F. x ;

[0030] S222, F, the feature to be enhanced in the fusion modality. x The data is fed into a fusion branch network, and a multi-head attention mechanism is used to enhance the fusion features, resulting in multi-head attention-enhanced fusion features.

[0031] S223. The different modal features are fed into the cross encoder for enhancement.

[0032] This invention utilizes a feature fusion enhancement mechanism. By performing cross-attention calculation on the fused modal features of the enhanced template frame and search frame and the enhanced single modal features, it avoids the use of existing technologies that directly apply cross-attention to the fusion of two modalities when there are significant differences between visible light and thermal infrared modalities. This preserves the heterogeneous information of different modalities, improves the multimodal fusion effect, and enhances the global representation capability.

[0033] In a more specific technical solution, in step S23, the enhanced fusion feature is used as the query input Q, and the single-modal features of the enhanced template frame and the search frame are used as the key K and value V, respectively. Cross-attention calculation is performed using the following logic to obtain the cross-attention outputs of different modalities of the template frame and the search frame:

[0034]

[0035]

[0036] This invention obtains query Q from search frame features and key K and value V from template frame features, and performs cross-attention calculation to obtain enhanced search frame features. Since the template frame contains rich target state information, information interaction between template frame features and search frame features is realized.

[0037] In a more specific technical solution, in step S25, the feature pairs are respectively fed into the absorbing Transformer encoder and the absorbing Transformer decoder to process and obtain the multi-branch features of the search frame.

[0038] This invention addresses the shortcomings of existing RGBT visual tracking tasks by introducing an Absorbed Transformer (ATM) component and a collaborative learning algorithm, thereby further improving the effectiveness and usability of the RGBT visual tracking algorithm.

[0039] In a more specific technical solution, step S4 includes:

[0040] S41. Compare the classification loss obtained by the fusion branch with the classification loss obtained by the single-modal branch to obtain the comparison results;

[0041] S42. Based on the comparison results, the total loss of the fusion branch is obtained.

[0042] The relationship between fused features and specific modality features is crucial, but it is often overlooked when learning discriminative multimodal representations.

[0043] In a more specific technical solution, step S41 includes:

[0044] S411. If the classification loss obtained by the single-modal branch is less than the classification loss obtained by the fusion branch, then the square of the difference between the classification score of the fusion branch and the classification score of the single-modal branch shall be used as part of the total loss of the fusion branch.

[0045] S412. Otherwise, the total loss of the fusion branch is composed of the classification loss and the bounding box prediction loss of the fusion branch.

[0046] This invention utilizes a collaborative learning algorithm to minimize the classification score gap between the fusion branch and the higher-performing specific modality branch, thereby optimizing the learning of the fusion branch and improving the quality of discriminative representation learning.

[0047] In more specific technical solutions, the RGBT tracking system based on progressive fusion of Transformer and dynamic guided learning includes:

[0048] The training module and the testing module, among which,

[0049] The training module includes:

[0050] The acquisition and preprocessing module is used to acquire and preprocess the visible light and thermal infrared image pairs of the template frame and the visible light and thermal infrared image pairs of the search frame from the preset sequence, and adjust the size of the preset template frame area and the preset search frame area to obtain the visible light and thermal infrared template frame and the visible light and thermal infrared search frame.

[0051] The Absorption Transformer module is used to extract features from visible light and thermal infrared template frames and visible light and thermal infrared search frames to obtain template frame features and search frame features. Based on these features, the fusion branch network of the Absorption Transformer module is used to perform cross-branch feature fusion to obtain template frame fusion branch output and search frame fusion branch output. Based on this, the feature information of the template frame and search frame is interactively processed to obtain multi-branch features of the search frame. The Absorption Transformer module is connected to the acquisition preprocessing module.

[0052] The classification loss calculation module is used to feed the multi-branch features of the search frame into the classifiers of their respective branches to calculate the classification loss. The classification loss calculation module is connected to the absorption Transformer module.

[0053] The model parameter optimization module is used to determine the applicable specific modality branch based on the classification loss, and then use the collaborative learning algorithm to train and optimize the fusion branch to obtain the applicable network model parameters. The model parameter optimization module is connected to the classification loss calculation module.

[0054] The test module includes:

[0055] The test data acquisition and preprocessing module is used to sample and preprocess test data, initialize the network model using applicable network model parameters, and process the data to obtain the test template frame and the current test frame.

[0056] The test data fusion module is used to perform feature extraction, cross-branch feature fusion, and information interaction operations on the test template frame and the current test frame to obtain the fused modal branch features of the current frame. The test data fusion module is connected to the test data acquisition and preprocessing module.

[0057] The current frame target bounding box prediction module is used to feed the current frame fused modality branch features into the classifier and regressor to predict the bounding box of the target in the current frame. The current frame target bounding box prediction module is connected to the test data fusion module.

[0058] Compared with existing technologies, this invention has the following advantages: It proposes an RGBT tracking method and system based on progressive fusion Transformer and dynamic guided learning. Unlike previous RGBT tracking methods, this method uses an absorption Transformer module to effectively absorb information between fused features and specific modal features, thereby preserving information from shared and specific modalities as much as possible, avoiding over-modeling, and improving modality integration. Furthermore, it proposes a collaborative learning algorithm to optimize model training, integrating information from specific modalities into the fused modality, resulting in more comprehensive optimization of each branch.

[0059] This invention utilizes a feature fusion enhancement mechanism. By performing cross-attention calculation on the fused modal features of the enhanced template frame and search frame with the enhanced single modal features, it avoids the use of existing technologies that directly apply cross-attention to the fusion of two modalities when there are significant differences between visible light and thermal infrared modalities. This preserves the heterogeneous information of different modalities, improves the multimodal fusion effect, and enhances the global representation capability.

[0060] This invention obtains query Q from search frame features and key K and value V from template frame features, and performs cross-attention calculation to obtain enhanced search frame features. Since the template frame contains rich target state information, information interaction between template frame features and search frame features is realized.

[0061] This invention addresses the shortcomings of existing RGBT visual tracking tasks by introducing an Absorbed Transformer (ATM) component and a collaborative learning algorithm, thereby further improving the effectiveness and usability of the RGBT visual tracking algorithm.

[0062] This invention utilizes a collaborative learning algorithm to minimize the classification score gap between the fusion branch and the higher-performing specific modality branch, thereby optimizing the learning of the fusion branch and improving the quality of discriminative representation learning.

[0063] This invention solves the technical problems of existing technologies, such as the loss of heterogeneous information in modalities, the neglect of the relationship between fused features and specific modal features when learning to discriminate multimodal representations, and the lack of global representation capabilities. Attached Figure Description

[0064] Figure 1 is a schematic diagram of the basic steps of the training process of the RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning in Embodiment 1 of the present invention.

[0065] Figure 2 is a schematic diagram of the network model structure of Embodiment 1 of the present invention;

[0066] Figure 3 is a schematic diagram of the basic steps of the test process of the RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning in Embodiment 1 of the present invention.

[0067] Figure 4 is a comparison diagram of the ATM component and collaborative learning algorithm proposed in Embodiment 2 of the present invention with the benchmark method;

[0068] Figure 5 is a visualization of the tracking results of Embodiment 2 of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Example 1

[0071] As shown in Figure 1, the RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning provided by the present invention includes a training process and a testing process.

[0072] In this embodiment, the training process includes the following basic steps:

[0073] S1. Data sampling and preprocessing;

[0074] In this embodiment, the sequence is randomly sampled to obtain visible light and thermal infrared image pairs aligned with the template frame and visible light and thermal infrared image pairs aligned with the search frame. The template frame region and the search frame region are cropped and resized according to the initialized bounding box to obtain {(Z v Z i ),(X v ,X i Z represents the template frame, X represents the search frame, v represents the visible light mode, and i represents the thermal infrared mode;

[0075] S2, Feature Extraction;

[0076] In this embodiment, the template frame and the search frame are respectively input into the backbone network ResNet50 to extract features, thus obtaining the features.

[0077] S3, Cross-branch feature fusion;

[0078] As shown in Figure 2, in this embodiment, taking the search frame features as an example, the two modal features are added together to obtain the fused modal feature F. x The data is then fed into a fusion branch network, and a multi-head attention mechanism is used to enhance the fusion modal features. Simultaneously, the two unimodal features are fed into the cross-encoder for enhancement. Then, the enhanced fused feature is used as the query input Q, and the two enhanced unimodal features are used as keys K and values ​​V, respectively, for cross-attention calculation, i.e.:

[0079]

[0080]

[0081] The outputs of the two cross-attention points are then summed and fed into the layer normalization and feedforward network (FFN) to obtain the output of the fusion branch. Perform the same operation on the template frame features to obtain the output of the template frame fusion branch.

[0082] In this embodiment, the Transformer encoder and Transformer decoder work as follows:

[0083] The query Q is obtained from the search frame features, and the key K and value V are obtained from the template frame features. Cross-attention calculation is performed to obtain enhanced search frame features. Since the template frame contains rich target state information, this method can realize information interaction between template frame features and search frame features.

[0084] S4, interaction of feature information between template frame and search frame;

[0085] In this embodiment, the outputs of the template frame, search frame, and three other branches are as follows:

[0086]

[0087] Three pairs of features are formed by the same branch:

[0088]

[0089] The features of the three branches of the search frame are obtained by feeding them into the Transformer encoder and Transformer decoder respectively.

[0090] S5. The search frame features are fed into the classifier and the classification loss is calculated.

[0091] In this embodiment, the features of the three branches of the search frame are fed into the classifier of each branch to obtain their respective score maps and the classification loss is calculated.

[0092] S6. Use collaborative learning algorithms for training optimization.

[0093] In this embodiment, the collaborative learning algorithm adaptively utilizes high-performing branches to guide the learning of the remaining branches. The working principle of the collaborative learning algorithm in this embodiment is as follows:

[0094] Taking visible light branch-guided learning as an example, the classification loss obtained from the fusion branch is compared with that obtained from the visible light branch. If the visible light branch's loss is smaller, the square of the difference between the fusion branch's classification score and the visible light branch's classification score is used as part of the total loss of the fusion branch; otherwise, the total loss of the fusion branch only includes the fusion branch's classification loss and bounding box prediction loss. The same applies to the thermal infrared branch. This learning method minimizes the classification score gap between the fusion branch and the better-performing specific modality branch, thereby optimizing the learning of the fusion branch.

[0095] As shown in Figure 2, in this embodiment, the testing process includes the following basic steps:

[0096] S1', Data sampling and preprocessing;

[0097] In this embodiment, the network model for testing is initialized using the trained and optimized model parameters. The first frame image pair of the two modalities of the given sequence is sampled and cropped as the template frame. Each frame in the subsequent frames of the sequence is used as the search frame, and the first frame is used to form a template frame search frame pair.

[0098] S2', Feature extraction;

[0099] S3', Cross-branch feature fusion;

[0100] S4', template frame and search frame feature information interaction;

[0101] In this embodiment, by performing feature extraction, cross-branch feature fusion, and interaction of feature information between template frame and search frame during the training process, the features of the fused modal branch of the search frame, i.e. the current frame, are obtained.

[0102] S5', The current frame features are fed into the classifier and regressor;

[0103] S6': Predict the bounding box of the target in the current frame.

[0104] In this embodiment, the features of the current frame fusion modality branch are fed into the classifier and regressor, and the target position is predicted to obtain the predicted target bounding box coordinates.

[0105] Example 2

[0106] As shown in Figure 4, in this embodiment, the left side (Our) represents the simultaneous use of the ATM component and the collaborative learning algorithm, the middle side (Our-ATM) represents the use of only the ATM component, and the right side (ToMP50-RGBT*) represents the baseline method that uses neither the ATM component nor the collaborative learning algorithm. The comparisons in Figure 4 show that the Absorb Transformer module (ATM) and the collaborative learning algorithm used in this invention both demonstrate significant performance improvements compared to the baseline method.

[0107] Comparison table of experimental results of this invention with other methods

[0108]

[0109]

[0110] As shown in the table above, the present invention achieved the best tracking results on all four test sets. The mean accuracy (MPR) and mean success rate (MSR) were tested on the RGBT210, RGBT234 and VTUAV test sets. The mean accuracy of the present invention was improved by 2.7%, 3.1% and 3.2% respectively compared with the state-of-the-art methods on the three test sets. On the LasHeR test set, the accuracy (PR) was improved by 18.8% and the success rate (SR) was improved by 15.1% compared with the state-of-the-art methods.

[0111] As shown in Figure 5, in this embodiment, the black box represents the tracking result of the present invention, and the gray box represents the tracking result diagram of the benchmark method of the present invention.

[0112] In summary, this invention proposes an RGBT tracking method based on progressive fusion Transformer and dynamic guided learning. Unlike previous RGBT tracking methods, this method uses an absorption Transformer module to effectively absorb information between fused features and specific modality features, thereby preserving information from shared and specific modalities as much as possible, avoiding over-modeling, and improving modality integration. Furthermore, a collaborative learning algorithm is proposed to optimize model training by integrating information from specific modalities into the fused modality, resulting in more comprehensive optimization of each branch.

[0113] This invention utilizes a feature fusion enhancement mechanism. By performing cross-attention calculation on the fused modal features of the enhanced template frame and search frame with the enhanced single modal features, it avoids the use of existing technologies that directly apply cross-attention to the fusion of two modalities when there are significant differences between visible light and thermal infrared modalities. This preserves the heterogeneous information of different modalities, improves the multimodal fusion effect, and enhances the global representation capability.

[0114] This invention obtains query Q from search frame features and key K and value V from template frame features, and performs cross-attention calculation to obtain enhanced search frame features. Since the template frame contains rich target state information, information interaction between template frame features and search frame features is realized.

[0115] This invention addresses the shortcomings of existing RGBT visual tracking tasks by introducing an Absorbed Transformer (ATM) component and a collaborative learning algorithm, thereby further improving the effectiveness and usability of the RGBT visual tracking algorithm.

[0116] This invention utilizes a collaborative learning algorithm to minimize the classification score gap between the fusion branch and the higher-performing specific modality branch, thereby optimizing the learning of the fusion branch and improving the quality of discriminative representation learning.

[0117] This invention solves the technical problems of existing technologies, such as the loss of heterogeneous information in modalities, the neglect of the relationship between fused features and specific modal features when learning to discriminate multimodal representations, and the lack of global representation capabilities.

[0118] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning, characterized in that, The method includes a training process and a testing process. The training process includes: S1, acquiring and preprocessing visible light and thermal infrared image pairs of template frames and search frames from a preset sequence, adjusting the size of the preset template frame region and the preset search frame region to obtain visible light and thermal infrared template frames and visible light and thermal infrared search frames; S2, extracting features from the visible light and thermal infrared template frames and the visible light and thermal infrared search frames to obtain template frame features and search frame features, and using the fusion branch network of the Absorbing Transformer module to perform cross-branch feature fusion to obtain template frame fusion branch output and search frame fusion branch output, and interactively processing the feature information of the template frame and the search frame to obtain multi-branch features of the search frame; wherein, the branch outputs of the template frame and the branch outputs of the search frame are combined into feature pairs to interactively process the feature information of the template frame and the search frame to obtain enhanced multi-branch features of the search frame; the feature pairs are respectively fed into the Absorbing Transformer encoder and the Absorbing Transformer encoder. The Ansformer decoder processes and obtains the multi-branch features of the search frame; S3, the multi-branch features of the search frame are fed into the classifiers of their respective branches to calculate the classification loss; S4, the applicable specific modality branch is determined based on the classification loss, and the fusion branch is trained and optimized using a collaborative learning algorithm to obtain the applicable network model parameters; S4 includes: S41, comparing the classification loss obtained from the fusion branch with the classification loss obtained from the single modality branch to obtain a comparison result; S42, processing the comparison result to obtain the total loss of the fusion branch; the testing process includes: S1', sampling and preprocessing the test data, initializing the network model using the applicable network model parameters to obtain the test template frame and the test current frame; S2', by executing steps S2 to S4, performing feature extraction, cross-branch feature fusion, and information interaction operations on the test template frame and the test current frame to obtain the fused modality branch features of the current frame; S3', feeding the fused modality branch features of the current frame into the classifier and regressor to predict the bounding box of the target in the current frame.

2. The RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning as described in claim 1, characterized in that, Step S1 includes: S11, performing random sampling and alignment processing on the preset sequence to obtain the visible light and thermal infrared image pairs of the template frame and the visible light and thermal infrared image pairs of the search frame; S12, performing cropping and adjustment operations on the template frame region and the search frame region according to the preset initial bounding box to obtain the visible light and thermal infrared template frame and the visible light and thermal infrared search frame.

3. The RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning as described in claim 1, characterized in that, Step S2 includes: S21, inputting the visible light and thermal infrared template frames and the visible light and thermal infrared search frames into a preset backbone network ResNet50 to extract the template frame features and the search frame features; S22, adding the different modal features of the template frame features and the search frame features and sending them into the fusion branch network, and using a multi-head attention mechanism and cross-encoder to obtain enhanced fusion features; S23, performing cross-attention calculation using preset logic based on the enhanced fusion features to obtain the template frame cross-attention output and the search frame cross-attention output; S24, adding the template frame cross-attention output and the search frame cross-attention output and sending them to the layer normalization module to obtain the template frame fusion branch output and the search frame fusion branch output.

4. The RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning as described in claim 3, characterized in that, In step S21, the pre-configured backbone network includes ResNet50.

5. The RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning as described in claim 3, characterized in that, Step S22 includes: S221, adding the different modal features to obtain the fused modal feature to be enhanced. S222, The features to be enhanced in the fusion mode. The data is fed into the fusion branch network, and a multi-head attention mechanism is used to enhance the fusion features, resulting in multi-head attention-enhanced fusion features. S223. The different modal features are fed into the cross encoder for enhancement.

6. The RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning according to claim 3, characterized in that, In step S23, the enhanced fusion feature is used as the query input Q, and the enhanced single-modal features of the template frame and the search frame are used as the key K and value V, respectively. Cross-attention calculation is performed using the following logic to obtain the cross-attention outputs of different modalities of the template frame and the search frame: In the formula, To merge the output of the branch, This is the output of the template frame fusion branch, where x represents the search frame. Represents visible light modes, This indicates the thermal infrared mode.

7. The RGBT tracking method based on progressive fusion of Transformer and dynamic guided learning as described in claim 1, characterized in that, S41 includes: S411, if the classification loss obtained by the single-modal branch is less than the classification loss obtained by the fusion branch, then the square of the difference between the classification score of the fusion branch and the classification score of the single-modal branch is used as part of the total loss of the fusion branch; S412, otherwise the total loss of the fusion branch is composed of the classification loss of the fusion branch and the bounding box prediction loss.

8. An RGBT tracking system based on progressive fusion of Transformer and dynamic guided learning, characterized in that, The system includes a training module and a testing module. The training module includes: an acquisition and preprocessing module, used to acquire and preprocess visible light and thermal infrared image pairs of template frames and search frames from a preset sequence, adjusting the sizes of the preset template frame region and the preset search frame region to obtain visible light and thermal infrared template frames and visible light and thermal infrared search frames; and an absorption Transformer module, used to extract features from the visible light and thermal infrared template frames and the visible light and thermal infrared search frames to obtain template frame features and search frame features, thereby utilizing the fusion branch network of the absorption Transformer module for cross-branch processing. Feature fusion is performed to obtain the template frame fusion branch output and the search frame fusion branch output. These are then used to interactively process the feature information of the template frame and the search frame to obtain multi-branch features for the search frame. The absorption Transformer module is connected to the acquisition preprocessing module. Specifically, the branch outputs of the template frame and the search frame are combined into feature pairs for interactive processing of the feature information of the template frame and the search frame to obtain enhanced multi-branch features for the search frame. These feature pairs are then fed into the absorption Transformer encoder and the absorption Transformer decoder to process and obtain the multi-branch features for the search frame. A classification loss calculation module is also included. The module is used to input the multi-branch features of the search frame into the classifiers of their respective branches to calculate the classification loss. The classification loss calculation module is connected to the absorption Transformer module. The model parameter optimization module is used to determine the applicable specific modality branch based on the classification loss, and then use a collaborative learning algorithm to train and optimize the fusion branch to obtain applicable network model parameters. This model parameter optimization module is also connected to the classification loss calculation module. The module compares the classification loss obtained from the fusion branch with the classification loss obtained from the single modality branch to obtain a comparison result. The total loss of the fusion branch is obtained based on the comparison result. The testing module includes: test data acquisition and preprocessing. The test data acquisition and preprocessing module is used to sample and preprocess test data, initialize the network model using the applicable network model parameters, and process the data to obtain the test template frame and the current test frame. The test data fusion module is used to perform feature extraction, cross-branch feature fusion, and information interaction operations on the test template frame and the current test frame to obtain the current frame fused modal branch features. The test data fusion module is connected to the test data acquisition and preprocessing module. The current frame target bounding box prediction module is used to feed the current frame fused modal branch features into the classifier and regressor to predict the bounding box of the target in the current frame. The current frame target bounding box prediction module is connected to the test data fusion module.

Images