Real-time rgbt target tracking method based on multi-modal interaction and multi-stage optimization

Abstract

This invention proposes a real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization. The tracking model includes a feature extraction module, a multimodal interaction module, a target classifier, and a result optimization module. The tracking model is trained using a publicly available RGBT dataset, including both offline and online training phases. This invention constructs a multimodal interaction module to learn robust feature representations, improves the attention calculation method between cross-modal features, and achieves complementary enhancement between the two modalities. By introducing a gating function, the influence of redundant noise is effectively removed. The multi-stage optimization module combines optical flow algorithms and optimization models to achieve accurate relocalization of the tracked target, effectively mitigating the effects of camera shake, local occlusion, and other factors, thus improving the robustness and real-time performance of the tracking model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision target tracking technology, specifically relating to a real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization. Background Technology

[0002] Visible light sensors are widely used in various fields, but due to the weak light at night and the limited visibility in rainy or foggy weather, visible light cameras cannot work properly. In order to meet the requirements of all-weather operation, current research will combine multimodal optoelectronic sensors, namely visible light and thermal infrared devices, to achieve complementary information on heat sources and environmental details, build robust features, and improve model performance.

[0003] RGBT tracking aims to design a powerful all-weather tracker by integrating complementary features from the visible and thermal spectra, with broad applications in security patrols, assisted driving, and crowd temperature measurement. RGBT target tracking effectively integrates bimodal complementary cues while suppressing noise, predicting the target's bounding box in subsequent frames based on the initial frame's label. Many methods leverage modal complementarity to improve algorithm performance in various domains; however, in environments with constantly changing lighting, the visible light mode is unstable, and the thermal infrared mode experiences thermal crossover when the target and background temperatures are close, leading to decreased resolution. Without dynamic feature selection, this can actually increase noise, weaken the discriminative feature ratio, and cause target loss. Therefore, further exploration of the intrinsic relationships between multimodal modes is needed to achieve feature complementarity and redundant noise removal.

[0004] Existing methods focus on extracting multi-layer features to generate a reliable feature representation, thereby improving tracking accuracy. Another approach investigates modal fusion weights to achieve feature selection. However, the performance of these existing tracking methods degrades in challenging scenarios such as varying illumination and dynamic interference. Furthermore, common challenges in visual tracking require greater attention. Observational changes in camera motion and scaling can lead to target loss, in which case many trackers cannot predict accurate results. Therefore, there is still room for improvement in multimodal interaction design and the refinement of tracking results. Summary of the Invention

[0005] The problem this invention aims to solve is to address the shortcomings of existing tracking technologies by proposing a real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization, thereby improving the real-time performance and robustness of the tracker model.

[0006] The technical solution of this invention is as follows: a real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization. Based on infrared and visible light modal interactions, a target tracking model is constructed. First, offline training is performed using existing RGBT target tracking data. During online tracking, the target tracking model parameters are fine-tuned online based on the target determined in the first frame. Then, real-time target tracking is performed on the video sequence. The target tracking model includes a feature extraction module, a multimodal interaction module, a target classifier, and a result optimization module, implemented as follows:

[0007] Step 1): Use the spatiotemporally registered thermal infrared and visible light images as input to the target tracking model;

[0008] Step 2): Construct a shared-weighted dual-stream feature extraction module, which uses the first three layers of lightweight VGG-M convolution to extract infrared and visible light depth features from the input thermal infrared and visible light images.

[0009] Step 3): Construct a multimodal interaction fusion module to calculate the feature residual map generated by multi-head cross-modal attention for the extracted infrared and visible light depth features, thereby improving modal quality in a bidirectional enhancement manner.

[0010] The discriminative feature transformation from infrared to visible light modes for single cross-modal attention is represented as follows: The formula is as follows:

[0011]

[0012] In the above formula, Let R represent a pair of visible light and infrared depth features obtained from step 2), where R represents visible light and T represents infrared. In formula (1), Q, K, and V represent the query, key, and value, respectively, all of which undergo 1×1 convolution operations to change their dimensions. k To represent the scaling factor, first, we establish Q. R and K T The cross-modal correlations between them are then analyzed, and an attention matrix is ​​generated using a softmax function. Finally, the cross-modal features are derived from the attention matrix and V. T Weighted generation, in order to take into account the attention distribution in different spaces, extends single cross-modal attention to multi-head structures, as shown in the following formula:

[0013] MultiHead(Q R ,K T V T = Concat(H1,…,H) n W O (2)

[0014]

[0015] In the above formula, W represents the weight matrix of Q, K, and V under the i-th attention head. O This represents the weight matrix for concatenating multi-head attention, where n represents the number of attention heads, i = 1, ..., n, and Concat represents the concatenation operation.

[0016] Obtain the feature residual map F of multi-head cross-modal attention T-R =MultiHead(Q R ,K T V T Based on the bidirectional enhancement characteristics, F can be calculated by reversing the above formula. R-T =MultiHead(Q T ,K R V R ), F R-T The discriminative features representing the shift from visible light mode to infrared mode in multi-head cross-modal attention are represented by two complementary feature residual maps F. R-T and F T-R ;

[0017] Step 4): Use a gating function to remove redundant feature information from the result obtained in step 3), and then use a cascading method to aggregate the enhanced multimodal features;

[0018] Step 5): Use ROIAlign to sample the target and background on the fused feature map generated in step 4), and construct training samples with the target as positive examples and the background as negative examples to train the target classifier;

[0019] Step 6): Construct a target classifier consisting of three fully connected layers. Use softmax to calculate the binary classification loss, thereby updating the model parameters. The target classifier outputs the tracking prediction result.

[0020] Step 7): Construct the result optimization module, which consists of an optical flow algorithm and an optimization model. The optical flow is used to predict the large-scale movement of the target, calculate the offset between two adjacent frames, and correct the tracking results. The optimization model performs a second regression on the prediction results to obtain a more accurate positioning.

[0021] Furthermore, the present invention includes the following steps:

[0022] Step S1: Construct the dataset by dividing the data from the publicly available RGBT target tracking dataset for model training and testing;

[0023] Step S2: Construct the target tracking model;

[0024] Step S3: Offline model training. Load the pre-trained model VGG-M and repeat steps 1)-6). Use the AdamW algorithm to train the feature extraction module, multimodal interaction module, and target classifier of the target tracking model until the loss converges. Obtain the trained model parameters. The offset prediction of the result optimization module adopts the Lucas-Kanade optical flow algorithm, and the optimization model is fine-tuned using RGBT data.

[0025] Step S4: Online training and tracking, as detailed below:

[0026] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, fine-tune the parameters of the feature extraction module, multimodal interaction module and target classifier online in order to learn the target information, and then select the region with the highest confidence score to obtain the preliminary tracking results;

[0027] Step S4.2: Determine whether to optimize the tracking result based on the confidence score, and use the tracking result of this frame as the input for the next frame;

[0028] Step S4.3: Repeat steps S4.1-S4.2 to calculate the target tracking result for each frame step by step, thereby achieving target tracking of the entire RGBT sequence.

[0029] The beneficial effects of this invention are: it provides a highly efficient and accurate RGBT target tracking technology. Compared with existing technologies, it has the following advantages.

[0030] (1) This invention proposes a multimodal feature interaction module that combines infrared modal information to make up for the defects of visible light modality. It uses multi-head cross-modal attention to calculate attention matrices in multiple spatial dimensions, guides modal enhancement, reduces interference from low-quality modality, effectively realizes multimodal information interaction and fusion, and removes redundant feature channels, making the overall features concise and robust, thereby improving tracking performance and generalization ability.

[0031] (2) This invention proposes a multi-stage optimization strategy, which uses optical flow to predict the target offset, uses an optimization model to perform secondary regression on the tracking results, and then selects a stage-by-stage optimization method based on the confidence score. This effectively saves computational costs while improving tracking accuracy and real-time performance. Attached Figure Description

[0032] Figure 1 This is the overall flowchart of the present invention.

[0033] Figure 2 This is a schematic diagram of the cross-modal attention module of the present invention.

[0034] Figure 3 This is a schematic diagram of the gating network of the present invention.

[0035] Figure 4 The figures show the overall tracking performance curves of this invention on the GTOT dataset. (a) is the precision plot, where the horizontal axis represents the localization error threshold and the vertical axis represents the maximum accuracy. (b) is the success plot, where the horizontal axis represents the overlap threshold and the vertical axis represents the maximum success. The accuracy score and success score are shown in the figures. The method of this invention is MFG. MANet, MDNet+RGBT, and L1-PF are other existing similar tracking methods.

[0036] Figure 5 The figures show the overall tracking performance curves of this invention on the RGBT234 dataset. (a) is the precision plot, where the horizontal axis represents the localization error threshold and the vertical axis represents the maximum accuracy. (b) is the success plot, where the horizontal axis represents the overlap threshold and the vertical axis represents the maximum success rate. The accuracy and success scores are shown in the figures. The method used in this invention is MFG; MANet, RT, L1-PF, and JSR are other existing similar tracking methods.

[0037] Figure 6 This is a qualitative result diagram of a video sequence tested in this invention. Detailed Implementation

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings:

[0039] like Figure 1 As shown in the overall flowchart, this invention proposes a real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization, including the following steps:

[0040] Step S1: Construct a dataset by dividing the data from the publicly available RGBT234 and GTOT target tracking datasets for model training and testing.

[0041] Step S2: Construct the target tracking model, including a feature extraction module, a multimodal interaction module, a target classifier, and a result optimization module, as detailed below:

[0042] Step S2.1: Using existing RGBT target tracking data, the spatiotemporally registered visible light and thermal infrared images are used as inputs to the target tracking model, and the images are input into the model in pairs according to the video frame time sequence.

[0043] Step S2.2: Construct a dual-stream feature extraction module with shared weights, utilizing the first three convolutional layers of lightweight VGG-M, in which dilated convolutions are added to increase the receptive field. The feature channels output by the third convolutional layer are 512, extracting preliminary depth features of infrared and visible light.

[0044] Step S2.3: Construct a multimodal interaction fusion module, such as... Figure 2 As shown. For the features extracted in step S2.2, multi-head cross-modal attention is calculated to generate feature residual maps, improving low-quality modalities in a bidirectional enhancement manner. Multi-head cross-modal attention explores potential RGBT modal correlations, perceiving global information, thereby guiding one modality to acquire discriminative features from another modality. The discriminative feature transformation from infrared to visible light modes by single cross-modal attention can be represented as... The formula is as follows:

[0045]

[0046] In the above formula, This represents a pair of RGBT depth features obtained from step S2.2, namely visible light and infrared depth features; in formula (1), Q, K and V represent query, key and value respectively, the superscript R represents visible light and T represents infrared, all of which are transformed by 1×1 convolution, d k This represents the scaling factor. First, we establish Q... R and K T The cross-modal correlations between them are then analyzed; then, an attention matrix is ​​generated using a softmax function; finally, the cross-modal features are derived from the attention matrix and V. T Weighted generation. To accommodate attention distributions across different spaces, this invention uses a single cross-modal attention mechanism (Attention(Q)). R ,K T V T Extended to multi-head structure MultiHead(Q) R ,K T V T The formula is as follows:

[0047] MultiHead(Q R ,K T V T = Concat(H1,…,H) n W O (2)

[0048]

[0049] In the above formula, W represents the weight matrix of Q, K, and V for the i-th attention head. O The weight matrix represents the concatenation of multi-head attention, where n represents the number of attention heads, i = 1, ..., n, and Concat represents the concatenation operation; the feature residual map F of multi-head cross-modal attention is obtained. T-R =MultiHead(Q R ,KT V T Based on the bidirectional enhancement characteristics, F can be calculated by reversing the above formula. R-T =MultiHead(Q T ,K R V R ), F R-T The discriminative features representing the shift from visible light mode to infrared mode in multi-head cross-modal attention are represented by two complementary feature residual maps F. R-T and F T-R .

[0050] Step S2.4: Construct the gating function as follows Figure 3 As shown. Redundant features are removed, and combined with the attention-enhanced features from step S2.3, i.e., the complementary residual maps of the two features, multimodal feature interaction and aggregation are achieved. The detailed method is as follows:

[0051]

[0052] W R =softmax(Concat(S R ,S T (5)

[0053]

[0054]

[0055] Formulas (4)-(6) remove redundant information from the original features and generate the feature map from the previous generation. and Formula (7) will and respectively and F T-R and F R-T The features are summed to obtain the final feature map. Formula (4) calculates the channel attention vectors for the two modalities respectively, using S... M In this representation, M is R, indicating attention to the visible light feature channel; M is T, indicating attention to the infrared feature channel; X... M This represents the original depth features, specifically the visible light and infrared depth features X obtained from step 2). R ,X T GAP indicates Global Average Pooling operation. Let S represent convolution, σ represent the activation function; and let S represent the attention vector. M The process is cascaded, and then the attention scores are weighted using the softmax function, with weights W. R By S R and S T Cascaded to obtain, W T =(1-W) R), using W R Suppressing the generation of redundant channel features and Then, the enhanced feature F generated in step 3) is compared with... T-R and F R-T Add them together to obtain the final robust feature representation. and As shown in formula (7).

[0056] This invention proposes a gate function to adaptively control the information flow of two modalities and establish a long-term channel dependency relationship through cross-modal channel attention. First, the channel attention of the two modalities is calculated separately using formula (4), and then the attention vectors are concatenated. Finally, the attention scores are weighted using the softmax function, and redundant channel features are suppressed using this weight. Then, the enhanced features obtained in step S2.3 are fused to generate the final robust feature representation, as shown in formula (7). The enhanced multimodal features are then aggregated using a concatenation method.

[0057] Step S2.5: Use ROIAlign to sample the target and background on the fused feature map generated in step S2.3, and use the target as a positive example and the background as a negative example to build training samples for training the target classifier. Since sampling directly from the feature map avoids pixel-level sampling, the computation is accelerated.

[0058] Step S2.6: Construct the target classifier, which consists of three fully connected layers. The loss function consists of binary classification loss and instance embedding loss, thereby guiding the update of model parameters.

[0059] Step S2.7: Construct the result optimization module, which consists of an optical flow algorithm and an optimization model. Optical flow is used to predict the offset of the target's large-scale movement. To overcome the target tracking drift problem caused by camera shake or occlusion, considering the stability of the infrared mode, the optical flow algorithm is used on the infrared mode to construct feature points and calculate the offset between two adjacent frames, thereby correcting the tracking results. The optimization model performs a quadratic regression on the prediction results to obtain more accurate positioning. The optimization model is a plug-and-play structure, consisting of a pixel-level correlation layer and a spatially aware nonlocal layer. The model parameters are obtained through fine-tuning training using publicly available infrared data.

[0060] Step S3: Offline model training. Load the pre-trained VGG-M model on ImageNet, follow the multi-domain learning strategy, and repeat steps S2.1-S2.6. Train the model using the AdamW algorithm until the loss decreases and converges. Set the number of iterations to 200, and set the learning rate of the last fully connected layer to 1e. -4 The remaining layers are set to 1e -3We obtain the parameters of the trained model; the optical flow algorithm is based on traditional hand-designed features and adopts the Lucas-Kanade optical flow algorithm, which is simple and efficient, so it is not used in training; the optimized model is fine-tuned on the pre-trained model using publicly available infrared data. Fine-tuning is a common transfer learning method and will not be described in detail.

[0061] Step S4: Online training and tracking phase, as detailed below:

[0062] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target. Based on the label of the first frame, generate 500 positive samples and 5000 negative samples. Use the AdamW algorithm to fine-tune the feature extraction module, multimodal interaction module, and fully connected layer parameters online, with 50 iterations, so that the tracker can learn the target information. Then, select the top 5 regions with the highest confidence scores, average them, and obtain preliminary tracking results. Maintain a sample queue for updating target features.

[0063] Step S4.2: Determine whether to optimize the tracking results based on the confidence score. To save computational costs, an optimization strategy is designed based on the confidence score. When the confidence score is less than 0, the optical flow algorithm is used to re-search for the target. If the offset is greater than T, the target position is corrected. When the confidence score is greater than U, the optimized model is used to obtain more accurate tracking results. For other confidence score ranges, the multi-stage optimization strategy is not used. According to experimental tests, setting parameter U to 30 and parameter T to 10 can achieve relatively good results. Furthermore, the tracking result of this frame is used as the input for the next frame.

[0064] Step S4.3: Repeat steps S4.1-S4.2 to calculate the target tracking result for each frame step by step, thereby achieving target tracking of the entire RGBT sequence.

[0065] To verify the effectiveness of the tracking model, comparative experiments were conducted on two widely used RGBT datasets. The experimental results are shown in the metrics below. Figure 4 and Figure 5 As shown. Compared with the most advanced multimodal tracking methods currently available, the MFG method of this invention significantly improves the accuracy and success rate metrics on the GTOT and RGBT234 datasets, compared with the real-time multi-domain tracking network RT-MDNet and the advanced MANet method.

[0066] Finally, representative challenging scenarios are selected as examples for qualitative analysis of tracking performance. Four frames are shown in the examples. Figure 6 As shown, the top row displays the tracking results for the visible light mode, and the bottom row displays the tracking results for the infrared mode. It can be seen that the method of this invention achieves satisfactory results even in complex environments, including those with camera motion and extreme lighting conditions. The tracked target is marked with a dashed box. Figure 6 The points are highlighted with black arrows. For example, targets are affected by illumination, scale changes, and partial occlusion, which makes it difficult for most trackers to achieve satisfactory tracking results. However, thanks to robust multimodal feature interaction fusion and tracking result optimization, the method of this invention achieves stable all-weather tracking and effectively handles the effects of illumination and partial occlusion. In addition, the method of this invention runs at approximately 30 frames per second, meeting real-time requirements.

Claims

1. A real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization, characterized in that... Based on the interaction of infrared and visible light modes, a target tracking model is constructed. First, offline training is performed using existing RGBT target tracking data. During online tracking, the tracking target model parameters are fine-tuned online based on the target determined in the first frame. Then, real-time target tracking is performed on the video sequence. The target tracking model includes a feature extraction module, a multimodal interaction module, a target classifier, and a result optimization module, implemented as follows: Step 1): Use the spatiotemporally registered thermal infrared and visible light images as input to the target tracking model; Step 2): Construct a shared-weighted dual-stream feature extraction module, which uses the first three layers of lightweight VGG-M convolution to extract infrared and visible light depth features from the input thermal infrared and visible light images. Step 3): Construct a multimodal interaction fusion module to calculate the feature residual map generated by multi-head cross-modal attention for the extracted infrared and visible light depth features, thereby improving modal quality in a bidirectional enhancement manner. The discriminative feature transformation from infrared to visible light modes for single cross-modal attention is represented as follows: The formula is as follows: In the above formula, Let R represent a pair of visible light and infrared depth features obtained from step 2), where R represents visible light and T represents infrared. In formula (1), Q, K, and V represent the query, key, and value, respectively, all of which undergo 1×1 convolution operations to change their dimensions. k To represent the scaling factor, first, we establish Q. R and K T The cross-modal correlations between them are then analyzed, and an attention matrix is ​​generated using a softmax function. Finally, the cross-modal features are derived from the attention matrix and V. T Weighted generation, in order to take into account the attention distribution in different spaces, extends single cross-modal attention to multi-head structures, as shown in the following formula: MultiHead(Q R ,K T ,V T )=Concat(H1,…,H n )W O (2) In the above formula, W represents the weight matrix of Q, K, and V under the i-th attention head. O This represents the weight matrix for concatenating multi-head attention, where n represents the number of attention heads, i = 1, ..., n, and Concat represents the concatenation operation. Obtain the feature residual map F of multi-head cross-modal attention T-R =MultiHead(Q R ,K T V T Based on the bidirectional enhancement characteristics, F can be calculated by reversing the above formula. R-T =MultiHead(Q T ,K R V R ), F R-T The discriminative features representing the shift from visible light mode to infrared mode in multi-head cross-modal attention are represented by two complementary feature residual maps F. R-T and F T-R ; Step 4): Use a gating function to remove redundant feature information from the result obtained in step 3), and then use a cascading method to aggregate the enhanced multimodal features; Step 5): Use ROIAlign to sample the target and background on the fused feature map generated in step 4), and construct training samples with the target as positive examples and the background as negative examples to train the target classifier; Step 6): Construct a target classifier consisting of three fully connected layers. Use softmax to calculate the binary classification loss, thereby updating the model parameters. The target classifier outputs the tracking prediction result. Step 7): Construct the result optimization module, which consists of an optical flow algorithm and an optimization model. The optical flow is used to predict the large-scale movement of the target, calculate the offset between two adjacent frames, and correct the tracking results. The optimization model performs a second regression on the prediction results to obtain a more accurate positioning.

2. The real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization according to claim 1, characterized in that... Includes the following steps: Step S1: Construct the dataset by dividing the data from the publicly available RGBT target tracking dataset for model training and testing; Step S2: Construct the target tracking model; Step S3: Offline model training. Load the pre-trained model VGG-M and repeat steps 1)-6). Use the AdamW algorithm to train the feature extraction module, multimodal interaction module, and target classifier of the target tracking model until the loss converges. Obtain the trained model parameters. The offset prediction of the result optimization module adopts the Lucas-Kanade optical flow algorithm, and the optimization model is fine-tuned using RGBT data. Step S4: Online training and tracking, as detailed below: Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, fine-tune the parameters of the feature extraction module, multimodal interaction module and target classifier online in order to learn the target information, and then select the region with the highest confidence score to obtain the preliminary tracking results; Step S4.2: Determine whether to optimize the tracking result based on the confidence score, and use the tracking result of this frame as the input for the next frame; Step S4.3: Repeat steps S4.1-S4.2 to calculate the target tracking result for each frame step by step, thereby achieving target tracking of the entire RGBT sequence.

3. The real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization according to claim 2, characterized in that, In step S4.2, a multi-stage optimization strategy is designed based on the confidence score. When the confidence score is less than 0, the optical flow algorithm is used to search for the target again. If the offset is greater than T, the target position is corrected. When the confidence score is greater than U, the optimization model is used to obtain more accurate tracking results. In other confidence intervals, the optimization strategy is not enabled.

4. The real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization according to claim 1, characterized in that, Step 4) Use a gating function to remove redundant feature information from the result obtained in Step 3), and then use a concatenation method to aggregate the enhanced multimodal features, as follows: W R =softmax(Concat(S R ,S T )) (5) Formula (4) calculates the channel attention vectors for the two modalities respectively, using S M In this representation, M is R, indicating attention to the visible light feature channel; M is T, indicating attention to the infrared feature channel; X... M This represents the original depth features of the corresponding modality, and GAP represents the global average pooling operation. Let S represent convolution, σ represent the activation function; and let S represent the attention vector. M Cascaded, then the attention scores are weighted using the softmax function, and the weights W are used. R Suppressing the generation of redundant channel features and Then, the enhanced feature F generated in step 3) is compared with... T-R and F R-T Add them together to obtain the final robust feature representation. and As shown in formula (7).

5. The real-time RGBT target tracking method based on multimodal interaction and multi-stage optimization according to claim 1, characterized in that, In step 7), an optical flow algorithm is used on the infrared mode to construct feature points and calculate the offset between two adjacent frames, thereby correcting the tracking results and optimizing the model into a plug-and-play structure consisting of a pixel-level correlation layer and a spatially aware nonlocal layer. The model parameters are obtained through fine-tuning training using publicly available infrared data.

Images