RGB-t object tracking method and system

Abstract

This application discloses an RGB-T target tracking method and system, relating to the technical fields of image processing and pattern recognition. The method includes: extracting features from visible light images and thermal infrared images based on a dual-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features, and thermal infrared search features; fusing the above features to obtain multimodal template features and multimodal search features; performing a mutual attention operation based on the multimodal template features and the multimodal search features, and calculating the cross-correlation parameters of the multimodal template features and the multimodal search features; inputting the cross-correlation features into a corner prediction network and a mask prediction network, predicting the corner coordinates of the target bounding box through the corner prediction network, generating a binary segmentation mask of the target through the mask prediction network, and combining the corner coordinates and the binary segmentation mask to output a refined localization result of the target.

Description

Technical Field

[0001] This application relates to the technical fields of image processing and pattern recognition, and in particular to an RGB-T target tracking method and system. Background Technology

[0002] Video target tracking technology refers to the continuous prediction of the target's position and state in subsequent frames after determining the initial position of the target in the first frame of a given video sequence. It is widely used in fields such as autonomous driving, video surveillance, and security.

[0003] Existing target tracking methods are mostly based on visible light images, achieving good results under favorable lighting conditions. However, in insufficient light, overexposure under strong light, at night, or in complex weather conditions, the quality of visible light imaging deteriorates, reducing the contrast between the target and the background and leading to decreased tracking accuracy and stability. To overcome the limitations of single visible light imaging, RGB-T target tracking technology, which integrates visible light and thermal infrared images, is gaining increasing attention. Thermal infrared imaging is independent of external lighting and insensitive to changes in illumination, maintaining relatively stable imaging performance even in environments with smoke, haze, etc., effectively supplementing the insufficient information in visible light images.

[0004] Currently, most RGB-T target tracking methods use rectangular bounding boxes to describe the target location. However, when the target shape changes, scale changes, or there is partial occlusion, the rectangular bounding box often contains a lot of background area. The introduction of background information can easily interfere with the target feature expression, and as the error accumulates, it further reduces the accuracy and robustness of the tracking results.

[0005] Therefore, there is an urgent need for a solution that can further refine the target setting to address the above problems. Summary of the Invention

[0006] This application proposes an RGB-T target tracking method and system to address the shortcomings of the prior art.

[0007] According to a first aspect of the embodiments of this application, an RGB-T target tracking method is provided, comprising: Visible light images and thermal infrared images are acquired, and features are extracted from the visible light images and the thermal infrared images respectively based on a two-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features, and thermal infrared search features; Based on modal importance perception, the visible light template features and the thermal infrared template features are fused to obtain multimodal template features, and the visible light search features and the thermal infrared search features are fused to obtain multimodal search features; Based on the multimodal template features and the multimodal search features, a mutual attention operation is performed to implicitly update the template information, and the cross-correlation parameters of the multimodal template features and the multimodal search features are calculated using variable convolution enhancement processing to obtain cross-correlation features; The cross-correlation features are input into the corner prediction network and the mask prediction network. The corner prediction network predicts the corner coordinates of the target bounding box, and the mask prediction network generates a binary segmentation mask of the target. The fine-grained localization result of the target is output by combining the corner coordinates and the binary segmentation mask.

[0008] In some implementations, the step of fusing the visible light template features and the thermal infrared template features based on modal importance perception to obtain multimodal template features, and fusing the visible light search features and the thermal infrared search features to obtain multimodal search features, includes: Based on the modal importance perception, importance parameters of the visible light modal features and the thermal infrared modal features are calculated respectively, wherein the visible light modal features include visible light template features and visible light search features, and the thermal infrared modal features include thermal infrared template features and thermal infrared search features; The importance parameters of the visible light mode are compared with those of the thermal infrared mode to determine the relative importance of the visible light mode features and the thermal infrared mode features, and to output the corresponding comparison results. Based on the comparison results, guidance information is generated from the first target modal feature with higher importance, the second target modal feature with lower importance is enhanced, and the enhanced second target modal feature is fused with the first target modal feature with higher importance. Wherein, when the first target modal feature is the visible light modal feature, the second target modal feature is the thermal infrared modal feature; when the first target modal feature is the thermal infrared modal feature, the second target modal feature is the visible light modal feature.

[0009] In some implementations, generating guidance information from the first target modal features of higher importance includes: Based on the first target modality features with higher importance in the comparison results, spatial dimension guidance information is generated according to the spatial attention mechanism.

[0010] In some implementations, fusing the enhanced second target modal feature with the first target modal feature of higher importance includes: The visible light modal features and the thermal infrared modal features are weighted using the importance parameters of the visible light modal features and the thermal infrared modal features respectively, and then cascaded in the channel dimension to obtain preliminary fused features. The preliminary fused features are then enhanced using a channel attention mechanism.

[0011] In some implementations, the step of performing a mutual attention operation based on the multimodal template features and the multimodal search features to implicitly update template information, and using variable convolution enhancement processing to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features to obtain cross-correlation features, includes: In the channel dimension, first attention information is generated based on the multimodal template features to enhance the multimodal search features, and second attention information is generated based on the enhanced multimodal search features to update the multimodal template features; Deformable convolution is applied to the enhanced multimodal search features and the updated multimodal template features respectively, and the similarity between the multimodal search features and the multimodal template features in the spatial dimension is calculated to obtain the initial features; The initial features are weighted using a channel attention mechanism to obtain the cross-correlation features.

[0012] In some implementations, the generation of first attention information based on the multimodal template features at the channel dimension to enhance the multimodal search features is calculated using the following mathematical expression:

[0013]

[0014] in, Represents multimodal template features; The transpose matrix representing the features of the multimodal template; Represents the first attention information in the channel dimension; ⊙ represents the element-wise multiplication operation between channels; Indicates by row (or by column) Normalization function; Represents multimodal search features; ⊕ represents element-wise addition between channels; This represents the enhanced multimodal search features; The number of channels representing the feature; Indicates the spatial dimension of the feature.

[0015] In some implementations, predicting the corner coordinates of the target bounding box using the corner prediction network includes: The cross-correlation features are input into two parallel prediction branches. Each prediction branch sequentially passes through multiple convolutional blocks, a convolutional layer, and a soft maximum index operation to output the coordinates of the top-left and bottom-right corners of the target bounding box, respectively.

[0016] In some implementations, generating a binary segmentation mask for the target via the mask prediction network includes: Based on modal importance perception, visible light search features and thermal infrared search features at multiple different levels are fused to obtain fused search features at multiple levels. An encoder-decoder structure is used to aggregate and upsample the fusion search features of the multiple levels layer by layer to generate the binary segmentation mask.

[0017] In some embodiments, the method further includes: The corner prediction network and the mask prediction network are trained using a joint loss function, which includes bounding box prediction loss, binary segmentation mask loss, and target edge loss.

[0018] According to a second aspect of this application, an RGB-T target tracking system is provided, comprising: The feature extraction module is used to acquire visible light images and thermal infrared images, and to extract features from the visible light images and thermal infrared images respectively based on a two-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features, and thermal infrared search features; The feature fusion module is used to fuse the visible light template features and the thermal infrared template features respectively to obtain multimodal template features based on modal importance perception, and to fuse the visible light search features and the thermal infrared search features to obtain multimodal search features; The cross-correlation feature determination module is used to perform a cross-attention operation based on the multimodal template features and the multimodal search features, to implicitly update the template information, and to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features using variable convolution enhancement processing, in order to obtain cross-correlation features; The result output module is used to input the cross-correlation features into the corner prediction network and the mask prediction network, predict the corner coordinates of the target bounding box through the corner prediction network, generate a binary segmentation mask of the target through the mask prediction network, and output the fine-grained localization result of the target by combining the corner coordinates and the binary segmentation mask.

[0019] The beneficial effects of the RGB-T target tracking method and system in this application include at least the following: This application's embodiments, by simultaneously processing visible light and thermal infrared images, can comprehensively utilize the advantages of both modalities. In good lighting conditions, it relies on rich visible light texture information, while in low light, nighttime, or adverse weather conditions, it relies on the stability of thermal infrared imaging, thus significantly improving the adaptability and robustness of the tracking system in different environments. Based on a modal importance-aware fusion strategy, it can dynamically evaluate and compare the importance of features from two modalities, guiding important information from the more important modality to the less important modality, achieving adaptive fusion, and generating more discriminative and reliable multimodal fusion features, effectively suppressing interference from unreliable modal information. Through mutual injection... The intention mechanism enables bidirectional interaction between the template and the search region information, implicitly updating the template information to adapt to changes in the target's appearance. Combined with deformable convolution enhancement processing, it improves the model's ability to model and match complex deformations of the target, resulting in more accurate and robust cross-correlation features. By jointly predicting corner coordinates and the target mask, it can simultaneously utilize the regression accuracy of the bounding box and the mask's ability to finely characterize the target shape. The combination of the two achieves more accurate target localization and reduces the problem of traditional rectangular boxes containing too much background area, thereby improving the accuracy and robustness of tracking in challenging scenarios such as target occlusion and scale changes. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an RGB-T target tracking method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an RGB-T target tracking system according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the various embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this disclosure to facilitate a better understanding of the disclosure. However, the technical solutions claimed in this disclosure can be implemented even without these technical details and with various variations and modifications based on the following embodiments. The division of the various embodiments below is for ease of description and should not constitute any limitation on the specific implementation of this disclosure. The various embodiments can be combined with and referenced by each other without contradiction.

[0022] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present application, but merely to illustrate selected embodiments of the present application. Other embodiments obtained by those skilled in the art based on the embodiments of the present application without inventive effort are all within the scope of protection of the embodiments of the present application.

[0023] This application discloses an RGB-T target tracking method and system. The RGB-T target tracking method is implemented based on the RGB-T target tracking system. The purpose is to overcome the shortcomings of the prior art. First, the initial or coarse bounding box of the target is obtained by using a basic tracker, and the target candidate region is determined accordingly. Then, within the target candidate region, a specially laid-out tracking refinement model is introduced to further accurately predict the target position, thereby reducing the influence of the background region on the tracking result.

[0024] See attached document Figure 1 The diagram shows a flowchart of an RGB-T target tracking method according to an embodiment of this application. The method is based on a novel RGB-T target tracking refinement model (hereinafter referred to as the "model") built in this embodiment and includes the following steps 110-140.

[0025] Step 110: Acquire visible light image and thermal infrared image, and extract features from the visible light (RGB) image and the thermal infrared (TIR) ​​image respectively based on the dual-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features and thermal infrared search features.

[0026] The acquired visible light image and thermal infrared image can be understood as the visible light image and thermal infrared image obtained from the input model. Furthermore, by using a two-stream twin network, feature extraction is performed on the input visible light RGB image and thermal infrared TIR image respectively to obtain the corresponding RGB template features, RGB search features, TIR template features and TIR search features.

[0027] For example, step 110 includes: inputting the visible light image and thermal infrared image from the template region image and the search region image into the corresponding twin feature extraction network for processing, thereby obtaining single-modal template features at different levels. , , , ) and single-modal search features ( , , , (The subscript is missing from the original text.) ir represents the modality type to which different features belong.

[0028] Step 120: Based on modal importance perception, the visible light template feature and the thermal infrared template feature are fused to obtain multimodal template features, and the visible light search feature and the thermal infrared search feature are fused to obtain multimodal search features.

[0029] In this embodiment, modal importance perception is implemented based on a pre-built guided fusion module (IGFM) used for adaptive fusion processing of visible light modal features and thermal infrared modal features. This guided fusion module comprises three parts: a modal importance calculation submodule, a modal importance comparison submodule, and a guided fusion submodule. Specifically, the modal importance calculation submodule calculates the importance of RGB single-modal features and TIR single-modal features; the modal importance comparison submodule compares the relative importance of RGB and TIR modes to further select an appropriate guided fusion strategy; and the guided fusion submodule first uses high-importance modes to guide low-importance modes to generate more discriminative features, and then performs multimodal feature fusion. In other words, the modal importance calculation part is used to process the input RGB single-modal features and thermal infrared single-modal features respectively to obtain the corresponding modal importance parameters; the modal importance comparison part is used to compare the importance parameters of the two modalities to determine the relative importance of the two modal features in the fusion process; the guided fusion strategy uses the modal features with higher importance to guide the processing of the modal features with lower importance in order to enhance the effective information in the low-importance modal features, and then the two modal features are fused to obtain multimodal fused features.

[0030] In some implementations, the method of fusing the visible light template feature and the thermal infrared template feature to obtain multimodal template features based on modal importance perception, and fusing the visible light search feature and the thermal infrared search feature to obtain multimodal search features, includes: calculating the importance parameters of the visible light modal feature and the thermal infrared modal feature based on modal importance perception, wherein the visible light modal feature includes the visible light template feature and the visible light search feature, and the thermal infrared modal feature includes the thermal infrared template feature and the thermal infrared search feature; comparing the importance parameters of the visible light modal feature and the thermal infrared modal feature to determine the relative importance of the visible light modal feature and the thermal infrared modal feature and outputting the corresponding comparison result; generating guidance information through the first target modal feature with higher importance based on the comparison result, enhancing the second target modal feature with lower importance, and fusing the enhanced second target modal feature with the first target modal feature with higher importance.

[0031] Wherein, when the first target modal feature is the visible light modal feature, the second target modal feature is the thermal infrared modal feature; when the first target modal feature is the thermal infrared modal feature, the second target modal feature is the visible light modal feature.

[0032] In some implementations, the generation of guidance information through the first target modality features with higher importance includes: generating spatial dimension guidance information based on the first target modality features with higher importance in the comparison results, according to a spatial attention mechanism.

[0033] In some implementations, fusing the enhanced second target modal feature with the first target modal feature of higher importance includes: weighting the visible light modal feature and the thermal infrared modal feature using the importance parameters of the visible light modal feature and the thermal infrared modal feature respectively, cascading them in the channel dimension to obtain a preliminary fused feature, and enhancing the preliminary fused feature through a channel attention mechanism.

[0034] For example, the multimodal fusion module processes RGB features from the fourth level of the template branch. With TIR characteristics The RGB features from the first to fourth levels extracted by the feature extraction network from the search region branch are fused together. and TIR characteristics The process involves fusion. Mathematically, the calculation is as follows: ; ; in, and These are the fused multimodal template and search features, respectively. and These are the fourth-level single-modal RGB template and TIR template features, respectively; and These are the unimodal RGB and TIR search features of the i-th level, respectively, i=1,2,3,4; Indicates the proposed piece.

[0035] In IGFM, the fourth-level unimodal RGB template feature is used. (recorded as) ) and TIR template features (recorded as) Taking the fusion process as an example, it includes the following three parts.

[0036] The first part is the modal importance calculation submodule. This module utilizes a simple architecture to determine the relative importance of RGB and thermal infrared data. The features of the two modes are then processed... After convolutional and batch normalization (BN) layers, global average pooling (GAP) is used to capture global information, resulting in two vectors. Finally, these two vectors are concatenated along the channel dimension and convolution is performed to obtain the numerical representation of modality importance. and The calculation process is as follows: ; ; ; in, This indicates that the kernel size is Convolutional layers; Indicates a cascading operation; (*) indicates the Sigmoid activation function.

[0037] The second part is the modal importance calculation submodule. This submodule calculates the importance values ​​of the obtained RGB and thermal infrared modes. and By comparing these strategies, a more suitable fusion guidance strategy can be selected. If the RGB modal features are more important than the thermal infrared modal features, then strategy 1 is selected, using the RGB modal features to generate guidance information and using this information to enhance the thermal infrared features. Conversely, if If so, choose strategy 2, which uses thermal infrared modal features to generate guidance information and uses this information to enhance RGB modal features.

[0038] The third part is based on the guided fusion submodule. The IGFM module implements a guided fusion strategy, which transfers discriminative features from high-importance modalities to low-importance modalities, and then fuses the high-importance single-modal features with the enhanced low-importance single-modal features. Specifically, first, the modal features with higher importance values ​​are used to generate guiding information through spatial attention (SAM), and this information is transferred to the modal features with lower importance values. Then, the two single-modal features are weighted by their modal importance values, and finally concatenated along the channel dimension to obtain the final fusion result. Subsequently, a channel attention mechanism is used to emphasize channels with discriminative information in the fused features, and residual connections and The convolutional layer operation yields the final multimodal fusion features. .when When selecting Guided Fusion Strategy 1, use RGB modal features to guide the generation of more discriminative thermal infrared modal features. To obtain multimodal fusion features ;when At that time, select Guided Fusion Strategy 2, which uses thermal infrared modal features to guide the generation of more discriminative RGB modal features. To obtain multimodal fusion features From a mathematical perspective, the calculation process is as follows: ; ; ; ; ; ; in, This represents element-wise addition; BM(*) represents element-wise multiplication; BM(*) represents broadcast multiplication.

[0039] The Spatial Attention Module (SAM) is represented by the following calculation process: ; The Channel Attention Module (CAM) is represented by the following calculation process: ; This represents the multimodal fusion features after the IGFM module; MLP represents the multilayer perceptron; GMP represents the max pooling operation.

[0040] Step 130: Based on the multimodal template features and the multimodal search features, perform a mutual attention operation to implicitly update the template information, and use variable convolution enhancement processing to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features to obtain cross-correlation features.

[0041] In this embodiment, step 130 is implemented based on a pre-set cross-correlation module (AttCorr) configured with an attention mechanism. By performing template-search region matching on the multimodal template features and the multimodal search features input to the cross-correlation module, cross-correlation features are obtained, thereby improving the ability of the model in this application to adapt to changes in the appearance of the target.

[0042] In some implementations, the method involves performing a mutual attention operation based on the multimodal template features and the multimodal search features to implicitly update the template information, and using variable convolution enhancement processing to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features to obtain cross-correlation features. This includes: generating first attention information based on the multimodal template features in the channel dimension to enhance the multimodal search features, and generating second attention information based on the enhanced multimodal search features to update the multimodal template features; applying deformable convolution to the enhanced multimodal search features and the updated multimodal template features respectively, and calculating the similarity between the multimodal search features and the multimodal template features in the spatial dimension to obtain initial features; and applying a channel attention mechanism to weight the initial features to obtain the cross-correlation features.

[0043] In some implementations, at the channel dimension, first attention information is generated based on the multimodal template features to enhance the multimodal search features, and is calculated using the following mathematical expression:

[0044]

[0045] in, Represents multimodal template features; The transpose matrix representing the features of the multimodal template; Represents the first attention information in the channel dimension; ⊙ represents the element-wise multiplication operation between channels; Indicates by row (or by column) Normalization function; Represents multimodal search features; ⊕ represents element-wise addition between channels; This represents the enhanced multimodal search features; The number of channels representing the feature; Indicates the spatial dimension of the feature.

[0046] For example, the AttCorr module in this application implements implicit template information updates through the interaction of template and search region features. Specifically, the AttCorr module first interacts with template information and search region information through a channel mutual attention strategy to achieve implicit updates of template information; secondly, it introduces deformable convolution operations to enhance the model's ability to handle complex deformations of the target; subsequently, it uses a pairwise relationship module to calculate the similarity between the template and search features in the spatial dimension to obtain cross-correlation features; finally, it enhances the cross-correlation features by applying global average pooling in the channel dimension. The deployed cross-correlation module uses fourth-level multimodal template features. With multimodal search features The input is used to obtain cross-correlation features. Mathematically, the calculation process is as follows: ; in, The features are the fused multimodal template features; The fused multimodal search features; This indicates the proposed AttCorr module.

[0047] In the AttCorr module, multimodal template features at the fourth level are used. (recorded as) ) and multimodal search features (recorded as) The input consists of the following four parts.

[0048] The first part is based on the mutual attention submodule. First, mutual attention information from the template branch is embedded into the search features to enhance them; second, mutual attention information from the search branch is embedded into the template features to update them. The search feature enhancement process includes: first, obtaining attention information from the template branch, and then... It will first be reshaped into Where C represents the number of feature channels, W and H represent their width and height respectively, and N=HW; and Multiplying it by its transpose matrix and passing it through the Softmax function yields the channel attention information. Then, attention information is embedded into the search features. In this process, search features learn the target representation in the template through attention information, thereby improving the discriminative power of the search features. Mathematically, the calculation process is as follows: ; ; in, yes Obtained through reshaping; Represents matrix multiplication; This represents the enhanced search features. The template feature update process includes: first, obtaining attention information from the enhanced search features through an operation similar to the channel mutual attention calculation in the template branch. Subsequently, attention information is embedded into the template features, thereby updating the target information in the template features. Mathematically, the calculation process is as follows: ; ; in, Indicates the updated template features; features Will be reshaped into ,feature Will be reshaped into , and This represents the output characteristics of the mutual attention module.

[0049] The second part focuses on the deformable attention submodule. After the template features and search region features are interacted and updated through the mutual attention module, deformable attention is further introduced to enhance the tracking algorithm's ability to handle complex target deformations, such as viewpoint changes, pose changes, and occlusion. The features obtained from the mutual attention module... and Application 3 3. Deformable convolution (Dconv) is used to further enhance the search features and template features. Mathematically, its calculation process is as follows: ; ; in, and This represents the enhanced search features and template features.

[0050] The third part focuses on the pairwise relation submodule. To obtain more detailed spatial information about the target, this paper chooses to use the pairwise relation module in the cross-correlation module to calculate the location of the tracked target in the search region. The pairwise relation module (PRM) calculates the affinity between each pixel in the template features and all pixels in the search features, which in turn determines the location of the target in the search region. After 1 1. After convolution, it is transformed into C The characteristics of HW size will After 1 1. Convolution followed by reshaping into HW Features of size C are then multiplied by matrix multiplication of the deformed template features and the search region features, and then passed through 1 1. Convolution operation yields cross-correlation features Accordingly, the process can be represented as: ; ; in, This indicates matrix resizing.

[0051] Part Four: Channel Attention Submodule. A channel attention module is used after the PRM module to reduce noise in the cross-correlation features. First, the cross-correlation features are processed along the spatial dimension. Global information is captured through global average pooling, and then passed through two 1-channel arrays with activation functions. One convolutional layer is used to obtain channel attention features. Then use right The mathematical description of the weighted operation is as follows: ; ; ; in, This represents the cross-correlation feature obtained after passing through the AttCorr module.

[0052] Step 140: Input the cross-correlation feature into the corner prediction network and the mask prediction network. The corner prediction network predicts the corner coordinates of the target bounding box, and the mask prediction network generates a binary segmentation mask of the target. The corner coordinates and the binary segmentation mask are combined to output the refined localization result of the target.

[0053] In some implementations, predicting the corner coordinates of the target bounding box through the corner prediction network includes: inputting the cross-correlation feature into two parallel prediction branches, each prediction branch sequentially passing through multiple convolutional blocks, a convolutional layer, and a soft maximum index operation to output the coordinates of the upper left and lower right corners of the target bounding box, respectively.

[0054] In some implementations, the generation of the binary segmentation mask of the target by the mask prediction network includes: fusing multiple visible light search features and thermal infrared search features at different levels based on modal importance perception to obtain multiple levels of fused search features; and using an encoder-decoder structure to aggregate and upsample the multiple levels of fused search features layer by layer to generate the binary segmentation mask.

[0055] In some implementations, the method further includes training the corner prediction network and the mask prediction network using a joint loss function, which includes bounding box prediction loss, binary segmentation mask loss, and target edge loss.

[0056] For example, this application embodiment sets up two prediction networks: a corner prediction network and a mask prediction network. The corner prediction network takes the cross-correlation features output by the AttCorr module as input to the target bounding box corner prediction network, feeding them into two parallel and symmetrical corner prediction branches to predict the coordinates of the top-left and bottom-right corners of the target bounding box. Each corner prediction branch consists of four stacked 3x3... 3 Convolution - Batch Normalization - ReLU Block, One 1 It consists of one convolutional layer and a soft-argmax operation. In each stacked convolutional block, the number of feature channels decreases progressively, followed by a 1-channel... A single convolutional layer generates a corner prediction heatmap with one channel, and finally, the coordinates of the corresponding corner points are calculated using a soft-argmax operation. A mask prediction network is used to generate a binary segmentation mask for the target. In the Target Tracking Refinement Model (RFN), the visible light and thermal infrared features of each layer's backbone features are first fused using the IGFM module. Then, a U-Net structure is used to aggregate the multimodal fusion features of the three layers in the search branch layer by layer to obtain the segmentation result of the target region. During the training phase, a multi-stage mask segmentation loss is introduced to constrain the mask prediction network to improve the stability and accuracy of the target mask prediction results. The loss function is used to jointly train the corner prediction network and the mask prediction network during training. The bounding box loss is used during training. ) and binary segmentation mask loss ( The prediction results are constrained. Furthermore, target edge loss supervision is introduced into the mask branch. ) as an auxiliary constraint. Among them, the bounding box loss ( The loss function is calculated using the mean squared error loss function, and the binary segmentation mask loss is ( ) and edge loss supervision ( All losses are calculated using the pixel-level binary cross-entropy loss function, the mathematical expression of which is as follows: ; ; ; ; in, Let be the true value of the i-th sample. Let N be the predicted value of the i-th sample, and N represent the number of training samples. Let be the true value of the i-th sample. Let be the predicted value for the i-th sample; Let be the true value of the i-th sample. Let be the predicted value for the i-th sample. For joint losses; and Select 1000.

[0057] This application's embodiments, by simultaneously processing visible light and thermal infrared images, can comprehensively utilize the advantages of both modalities. In good lighting conditions, it relies on rich visible light texture information, while in low light, nighttime, or adverse weather conditions, it relies on the stability of thermal infrared imaging, thus significantly improving the adaptability and robustness of the tracking system in different environments. Based on a modal importance-aware fusion strategy, it can dynamically evaluate and compare the importance of features from two modalities, guiding important information from the more important modality to the less important modality, achieving adaptive fusion, and generating more discriminative and reliable multimodal fusion features, effectively suppressing interference from unreliable modal information. Through mutual injection... The intention mechanism enables bidirectional interaction between the template and the search region information, implicitly updating the template information to adapt to changes in the target's appearance. Combined with deformable convolution enhancement processing, it improves the model's ability to model and match complex deformations of the target, resulting in more accurate and robust cross-correlation features. By jointly predicting corner coordinates and the target mask, it can simultaneously utilize the regression accuracy of the bounding box and the mask's ability to finely characterize the target shape. The combination of the two achieves more accurate target localization and reduces the problem of traditional rectangular boxes containing too much background area, thereby improving the accuracy and robustness of tracking in challenging scenarios such as target occlusion and scale changes.

[0058] To verify the effectiveness of the proposed novel tracking refinement model for RGB-T target tracking, embodiments of this application can be evaluated on public datasets (RGBT234, GTOT, and LasHeR) to assess performance metrics.

[0059] The following simulation experiments further illustrate the embodiments of this application. The simulation conditions include: all simulation experiments were conducted on the Ubuntu 20.04 operating system, using an Nvidia RTX A4000 GPU, and implemented using the PyTorch deep learning framework. The simulation content and result analysis include: comparing the embodiments of this application with several state-of-the-art RGB-T VOT models, including SiamMLAA, DMCNet, MFNet, CMD, JTPMA, APFNet, M5L, SiamCDA, JMMAC, CMPP, CAT, DAFNet, and mfDiMP. mfDiMP is the baseline of the embodiments of this application, and the baseline tracker was retrained, denoted as mfDiMP-r. Experiments were conducted on publicly available RGB-T VOT datasets RGBT234, GTOT, and LasHeR, and the results were objectively evaluated using recognized evaluation metrics. The simulation results are shown in Table 1. Table 1: Quantitative Results of Different Models

[0060] Table 2: Experimental Results of Model Structure Research

[0061] Table 3: Quantitative Results of Different Variants of the IGFM Module

[0062] Table 4: Quantitative Results of Different Variants of the AttCorr Module

[0063] Table 5: Experimental Results of Ablation Study Using Predictive Networks

[0064] In this context, PR stands for Precision, which is the percentage of frames in the video where the Euclidean distance between the center of the predicted bounding box and the center of the ground truth label is less than a manually set threshold (5 pixels in GTOT, 20 pixels in RGBT234, and 20 pixels in LasHeR). SR stands for Success Rate, which is the percentage of frames in the video where the intersection-over-union (IoU) ratio between the predicted bounding box and the ground truth label is greater than a specified threshold; it is typically calculated using the area under the curve (AUC). NPR stands for Normalized Precision, which is the percentage of frames where the predicted bounding box exceeds the ground truth label's accuracy. After normalizing the size of the bounding box, the precise value is calculated; Params represents the parameter size; FPS represents the number of frames transmitted per second; Base represents mfDiMP+RFN-v0, where RFN-v0 means removing the mutual attention module (CrAM) and deformable attention module (DAM) in the AttCorr module, and replacing other fusion strategies with multimodal fusion strategies, i.e., using element-wise addition to fuse single-modal features. In addition, only the corner prediction network is used; MF-v1 and MF-v2 represent the base of Ours. The above replaces IGFM with other variants of the multimodal fusion module; in modal fusion strategy MF-v1, the modal importance calculation submodule in the IGFM module is deleted, and the part of the guided fusion strategy that transmits cross-modal discriminative information in spatial dimension is replaced with: single-modal features are only self-enhanced through the spatial attention module; in modal fusion strategy MF-v2, only the modal importance calculation and comparison submodules in the IGFM module are deleted; Corr-v1, Corr-v2, and Corr-v3 represent replacing AttCorr with Ours. These are variations of the cross-correlation module; Corr-v1, Corr-v2, and Corr-v3 represent removing ChAM, DAM, and CrAM from the Attcorr module, respectively; Corner-Head removes the mask prediction head network from the RFN, retaining only the corner prediction head network; Mask-Head removes the corner prediction head network from the RFN, retaining only the mask prediction head network; Both-Head is the setting in the RFN, using two prediction head networks; higher PR, NPR, and SR all result in better performance.

[0065] As shown in Table 1, the embodiment of this application achieves optimal performance across all metrics on the three public datasets, demonstrating a significant performance improvement. Table 2 shows that by progressively adding a Mask-Head network, replacing the IGFM module, and adding a mutual attention module (CrAM) and a deformable attention module (DAM) to the Base model, the deployed modules all improve model performance. Table 3 compares MF-v1 with IGFM, indicating that directly fusing features from two modalities without information interaction between different modalities can easily lead to insufficient fusion of the two single-modal features. A comparison between MF-v2 and IGFM shows that the IGFM setting can reduce interference from unreliable modal information and improve the discriminative power of multimodal features. A comparison of the IGFM strategy with the aforementioned multimodal fusion strategy variants demonstrates the effectiveness of the IGFM strategy. Table 4 shows that removing the Channel Attention (ChAM) or CrAM sub-modules significantly reduces model performance, indicating that ChAM reduces redundant information in features, thus enhancing the discriminative power of cross-correlation features. Removing the CrAM sub-module, using only the first frame's target region as a template, fails to adapt well to changes in target appearance. Additionally, DAM enhances the model's ability to handle complex target deformations. Table 5 shows that the Mask-Head lacks a corner prediction head network, resulting in a significant drop in tracking performance. This is likely due to unreliable pseudo-mask label data, leading to decreased accuracy in bounding box prediction. Furthermore, Corner-Head shows a marked performance decrease compared to Both-Head, demonstrating that introducing target mask prediction significantly improves tracker performance.

[0066] This application's embodiments improve the detail of target region description by introducing a target binary segmentation task and combining it with bounding box prediction results; by designing a guided fusion module based on modal importance awareness (IGFM), more discriminative multimodal features are obtained; and by designing an attention-based cross-correlation module (AttCorr) to improve the model's adaptability to changes in target appearance, this application's embodiments greatly improve the tracking performance of the basic tracker in several challenging scenarios such as occlusion and cluttered backgrounds.

[0067] See attached document Figure 2 The diagram shows the structure of an RGB-T target tracking system according to an embodiment of this application, including: a feature extraction module 210, a feature fusion module 220, a cross-correlation feature determination module 230, and a result output module 240.

[0068] For example, the feature extraction module 210 is used to acquire a visible light image and a thermal infrared image, and to extract features from the visible light image and the thermal infrared image respectively based on a two-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features and thermal infrared search features.

[0069] For example, the feature fusion module 220 is used to fuse the visible light template feature and the thermal infrared template feature respectively to obtain multimodal template features based on modal importance perception, and to fuse the visible light search feature and the thermal infrared search feature to obtain multimodal search features.

[0070] For example, the cross-correlation feature determination module 230 is used to perform a cross-attention operation based on the multimodal template features and the multimodal search features to implicitly update the template information, and to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features using variable convolution enhancement processing to obtain cross-correlation features.

[0071] For example, the result output module 240 is used to input the cross-correlation feature into the corner prediction network and the mask prediction network, predict the corner coordinates of the target bounding box through the corner prediction network, generate a binary segmentation mask of the target through the mask prediction network, and output the fine-grained localization result of the target by combining the corner coordinates and the binary segmentation mask.

[0072] The RGB-T target tracking system of this application significantly improves tracking robustness and adaptability in complex environments such as changing lighting, nighttime, and severe weather by extracting and adaptively fusing visible light and thermal infrared dual-stream features, effectively overcoming the limitations of a single mode. Employing mutual attention and deformable convolution mechanisms, it achieves dynamic updating of template information and enhances the modeling ability for changes in target appearance and deformation, thereby improving the accuracy of tracking and matching as well as the robustness to target motion and attitude changes. By jointly outputting the corner coordinates of the target bounding box and a fine-grained binary segmentation mask, it can more accurately locate the target subject, greatly reducing the problem of excessive background interference in traditional rectangular boxes. Ultimately, it achieves a significant improvement in tracking accuracy in challenging scenarios such as occlusion and scale changes.

[0073] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of this application, and this application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this application, and these modifications and improvements are also considered to represent the scope of protection of this application.

Claims

1. An RGB-T target tracking method, characterized in that, include: Visible light images and thermal infrared images are acquired, and features are extracted from the visible light images and the thermal infrared images respectively based on a two-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features, and thermal infrared search features; Based on modal importance perception, the visible light template features and the thermal infrared template features are fused to obtain multimodal template features, and the visible light search features and the thermal infrared search features are fused to obtain multimodal search features; Based on the multimodal template features and the multimodal search features, a mutual attention operation is performed to implicitly update the template information, and the cross-correlation parameters of the multimodal template features and the multimodal search features are calculated using variable convolution enhancement processing to obtain cross-correlation features; The cross-correlation features are input into the corner prediction network and the mask prediction network. The corner prediction network predicts the corner coordinates of the target bounding box, and the mask prediction network generates a binary segmentation mask of the target. The fine-grained localization result of the target is output by combining the corner coordinates and the binary segmentation mask.

2. The method according to claim 1, characterized in that, The method based on modal importance perception involves fusing the visible light template features and the thermal infrared template features to obtain multimodal template features, and fusing the visible light search features and the thermal infrared search features to obtain multimodal search features, including: Based on the modal importance perception, importance parameters of the visible light modal features and the thermal infrared modal features are calculated respectively, wherein the visible light modal features include visible light template features and visible light search features, and the thermal infrared modal features include thermal infrared template features and thermal infrared search features; The importance parameters of the visible light mode are compared with those of the thermal infrared mode to determine the relative importance of the visible light mode features and the thermal infrared mode features, and to output the corresponding comparison results. Based on the comparison results, guidance information is generated from the first target modal feature with higher importance, the second target modal feature with lower importance is enhanced, and the enhanced second target modal feature is fused with the first target modal feature with higher importance. Wherein, when the first target modal feature is the visible light modal feature, the second target modal feature is the thermal infrared modal feature; when the first target modal feature is the thermal infrared modal feature, the second target modal feature is the visible light modal feature.

3. The method according to claim 2, characterized in that, The generation of guidance information through the first target modality features, which are of higher importance, includes: Based on the first target modality features with higher importance in the comparison results, spatial dimension guidance information is generated according to the spatial attention mechanism.

4. The method according to claim 2, characterized in that, The step of fusing the enhanced second target modal features with the first target modal features of higher importance includes: The visible light modal features and the thermal infrared modal features are weighted using the importance parameters of the visible light modal features and the thermal infrared modal features respectively, and then cascaded in the channel dimension to obtain preliminary fused features. The preliminary fused features are then enhanced using a channel attention mechanism.

5. The method according to claim 1, characterized in that, The process involves performing a mutual attention operation based on the multimodal template features and the multimodal search features to implicitly update template information, and using variable convolution enhancement processing to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features to obtain cross-correlation features, including: In the channel dimension, first attention information is generated based on the multimodal template features to enhance the multimodal search features, and second attention information is generated based on the enhanced multimodal search features to update the multimodal template features; Deformable convolution is applied to the enhanced multimodal search features and the updated multimodal template features respectively, and the similarity between the multimodal search features and the multimodal template features in the spatial dimension is calculated to obtain the initial features; The initial features are weighted using a channel attention mechanism to obtain the cross-correlation features.

6. The method according to claim 5, characterized in that, In the channel dimension, first attention information is generated based on the multimodal template features to enhance the multimodal search features, and is calculated using the following mathematical expression: in, Represents multimodal template features; The transpose matrix representing the features of the multimodal template; Represents the first attention information in the channel dimension; ⊙ represents the element-wise multiplication operation between channels; Indicates by row (or by column) Normalization function; Represents multimodal search features; ⊕ represents element-wise addition between channels; This represents the enhanced multimodal search features; The number of channels representing the feature; Indicates the spatial dimension of the feature.

7. The method according to claim 1, characterized in that, The prediction of the corner coordinates of the target bounding box through the corner prediction network includes: The cross-correlation features are input into two parallel prediction branches. Each prediction branch sequentially passes through multiple convolutional blocks, a convolutional layer, and a soft maximum index operation to output the coordinates of the top-left and bottom-right corners of the target bounding box, respectively.

8. The method according to claim 1, characterized in that, The generation of the binary segmentation mask for the target through the mask prediction network includes: Based on modal importance perception, visible light search features and thermal infrared search features at multiple different levels are fused to obtain fused search features at multiple levels. An encoder-decoder structure is used to aggregate and upsample the fusion search features of the multiple levels layer by layer to generate the binary segmentation mask.

9. The method according to claim 1, characterized in that, The method further includes: The corner prediction network and the mask prediction network are trained using a joint loss function, which includes bounding box prediction loss, binary segmentation mask loss, and target edge loss.

10. An RGB-T target tracking system, characterized in that, include: The feature extraction module is used to acquire visible light images and thermal infrared images, and to extract features from the visible light images and thermal infrared images respectively based on a two-stream twin network to obtain visible light template features, visible light search features, thermal infrared template features, and thermal infrared search features; The feature fusion module is used to fuse the visible light template features and the thermal infrared template features respectively to obtain multimodal template features based on modal importance perception, and to fuse the visible light search features and the thermal infrared search features to obtain multimodal search features; The cross-correlation feature determination module is used to perform a cross-attention operation based on the multimodal template features and the multimodal search features, to implicitly update the template information, and to calculate the cross-correlation parameters of the multimodal template features and the multimodal search features using variable convolution enhancement processing, in order to obtain cross-correlation features; The result output module is used to input the cross-correlation features into the corner prediction network and the mask prediction network, predict the corner coordinates of the target bounding box through the corner prediction network, generate a binary segmentation mask of the target through the mask prediction network, and output the fine-grained localization result of the target by combining the corner coordinates and the binary segmentation mask.

Images