An rgbt target tracking method based on modal perception feature learning

Abstract

The application discloses an RGBT target tracking method based on modal perception feature learning, constructs a target tracking model for an RGBT target tracking task, and adopts a transfer learning strategy for training.The target tracking model comprises a feature extraction module, a modal perception module, a Transform fusion module, a three-branch prediction head and a template updating module.The modal perception module combines an aggregation and distribution mechanism of channel features and a similarity perception mechanism of spatial features, fully excavates multi-modal discriminative features, and adopts a mixed attention to effectively capture global dependency relationships to enhance instance representation.The Transform fusion module is used for fusing features of a template and a search area.The three-branch prediction head combines a mutual constraint loss function, improves target positioning accuracy, and the template updating module is used for optimizing a target template, alleviates influences of challenge factors such as scale change, background interference and partial occlusion, and further improves stability of target tracking.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision target tracking, and particularly relates to an RGBT target tracking method based on modal perception feature learning. BACKGROUND

[0002] Visible light sensors are widely used, but they are subject to many limitations in weak light environments and harsh weather, leading to a decline in image quality and visible distance, and cannot meet the all-weather application requirements. Therefore, researchers use the strategy of multi-modal photoelectric sensor fusion to combine visible light sensors with thermal infrared sensors to obtain more comprehensive information. Thermal infrared sensors can detect the infrared radiation emitted by objects to obtain their thermal energy, thereby providing information that visible light sensors cannot capture. Multi-modal information fusion will achieve complementarity of the two types of information, build a more robust feature representation, and improve the performance of deep models to cope with more complex application scenarios.

[0003] The main goal of the RGBT target tracking task is to use the complementary features of visible light and thermal light to build a robust all-weather tracker and apply it to many fields such as military operations, field search and rescue, security inspection, assisted driving, and crowd density estimation. In the RGBT target tracking task, according to the label given by a person in the initial frame, the target position information in the subsequent frames is continuously predicted. Currently, various tracking methods have achieved good performance by utilizing the complementary characteristics of visible light and thermal light, but further research is needed to extract discriminative features from heterogeneous modalities that are beneficial to instance representation, accurately estimate target bounding boxes, and solve tracking challenges such as scale changes and occlusions.

[0004] There are currently two main research approaches: one is to explore the extraction of multi-layer features to build reliable feature representations to improve tracking accuracy; the other is to study the distribution of modal fusion weights to achieve adaptive selection of features. However, in the face of challenging scenarios such as target occlusion, appearance change, and scale change, the performance of existing tracking methods may decline. SUMMARY

[0005] The problem to be solved by the present application is that existing RGBT target tracking schemes have declined in performance in complex scenarios such as target occlusion, appearance change, and scale change, and cannot meet the use requirements. The present application improves the stability and performance of the RGBT target tracker by introducing a modal perception feature learning method.

[0006] The technical scheme of the present application is: an RGBT target tracking method based on modal perception feature learning, a target tracking model is constructed for the RGBT target tracking task, comprising the following steps:

[0007] Step S1: Construct a target tracking dataset, collect RGBT tracking video sequences for model training and testing;

[0008] Step S2: Construct a target tracking model, including a feature extraction module, a modal perception module, a Transformer fusion module, a three-branch prediction head, and a template update module, as follows:

[0009] Step S2.1: Obtain the images of the target template and the search region from the pre-processed registered visible light and thermal infrared images, respectively, as inputs to the target tracking model;

[0010] Step S2.2: The feature extraction module is a dual-channel feature extraction module that extracts target template and search region features for visible light and thermal infrared images, a total of 4 groups of features;

[0011] Step S2.3: The modal perception module first uses a channel feature aggregation and distribution mechanism to learn multi-modal robust features for visible light and thermal infrared using an attention mechanism to enhance the features of the target template and the search region. The enhanced features of the target template from different modalities are superimposed to obtain template fusion features. Then, spatial feature similarity perception is performed on the channel enhanced features generated by the attention mechanism to filter and enhance the features of the search region from different modalities. The superimposed search region fusion features are obtained;

[0012] Step S2.4: Perform feature flattening operation on the template fusion features and the search region fusion features, and convert the features into vectors using 1x1 convolution;

[0013] Step S2.5: Calculate the correlation between the target template features and the search region features from the flattened feature vectors using the Transformer fusion module to obtain a fusion vector;

[0014] Step S2.6: The fusion vector is input into the three-branch prediction head to obtain the prediction results. The three-branch prediction head consists of three branches: classification, positioning, and regression. The three branches are correlated through constraint loss, and the target tracking results are output based on confidence score ranking;

[0015] Step S2.7: Based on the prediction results of step S2.6, the template update module classifies the target template state according to the confidence score. The target template is updated or not updated according to the update strategy to achieve adaptive update of the target template and maintain the reliability of the target template;

[0016] Step S3: Offline training, use AdamW optimizer to train the model until the loss converges, and obtain the trained model parameters. The template update module does not need to be trained;

[0017] Step S4: online tracking:

[0018] Step S4.1: obtaining the label of the first frame of the video sequence as the initial tracking target, outputting the prediction value through the target tracking model, and then selecting the region with the highest confidence score to obtain the preliminary tracking result;

[0019] Step S4.2: judging the current target template state according to the confidence score of the result, and selecting whether to update the template or reset the template according to the update strategy and the current state;

[0020] Step S4.3: repeating steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, adaptively updating the template to maintain the reliability of the target template, and completing the target tracking task of the overall RGBT sequence.

[0021] Further, in step S2.3, the channel feature enhancement is realized by using the channel aggregation and distribution mechanism, which is as follows:

[0022] d g =F g (GAP(f R +f T )) (1)

[0023]

[0024] In the above formula, denotes a pair of RGBT target template features obtained from step S2.2, GAP in formula (1) denotes a global average pooling operation, F g denotes a fully connected layer, d g denotes a global pooling vector feature, in the channel feature aggregation stage, the visible light feature f R is added to the thermal infrared feature f T , then the global average pooling operation is performed, and finally the channel attention weight is output through the fully connected layer; in the channel feature distribution stage, F i in formula (2) denotes a double-branch fully connected layer distribution feature, R denotes the visible light modality, T denotes the thermal infrared modality, and σ denotes a Sigmoid function, and finally the weight is multiplied with the original feature f i z to generate the enhanced feature denotes a pair of RGBT search region features obtained from step S2.2, and is obtained from

[0025] Further, in S2.3, the spatial feature similarity perception adopts a spatial similarity calculation mechanism to generate the residual of the perceived instance through similarity learning, which is as follows:​

[0026]

[0027]

[0028]

[0029] In the above formula, S i represents a similarity feature map, f conv represents a convolution operation, and up represents a bilinear interpolation up-sampling, represents an enhanced feature, * represents a convolution operation, σ represents a Sigmoid function, and formula (4) represents a final template fusion feature is obtained by adding the thermal infrared and visible light enhanced features, and formula (5) represents a final search region fusion feature are multiplied to generate a residual map, and then added with corresponding thermal infrared and visible light enhanced features. i

[0030] As a further improvement of the present application, in step S2.6, the three branches are associated with each other by a constraint loss, which is specifically as follows:

[0031] L cls = -∑ j ((y j log(p j )IoU+(1-y j )log(1-p j ))) (6)

[0032]

[0033]

[0034] Loss = n1L cls + n2L reg + n3L loc (9)

[0035] In the above formula, L cls represents a classification loss, which is constructed by using an IoU weighted binary cross-entropy loss, y j represents a label of the jth sample, y j = 1 represents a positive sample, p j represents a probability of belonging to the foreground, and IoU represents an intersection of a predicted value and a true value; L reg represents a regression loss, which includes an L1 norm loss and an L CIoU loss, b j represents the jth bounding box, and p j ​denotes the classification confidence corresponding to the positive sample, and λ C both denote regularization parameters; L loc denotes the positioning loss, O j is the IoU score calculated for the regression branch, denotes the prediction value of the positioning branch; Loss denotes the overall loss obtained by weighted summation of the above three loss functions, wherein n1, n2 and n3 denote hyperparameters.

[0036] As a further improvement of the present application, the updating strategy in step S4.2 is: according to the confidence score output by the three-branch prediction head, the target tracking state is divided into stable state, temporary stable state and unstable state, the stable state is defined as the state that the confidence score of M consecutive frames is greater than 0.9, once the stable state is reached, the current target template is updated to replace the initial target template; if the confidence score is between 0.7 and 0.9, it is considered that the tracker is in a temporary stable state, and the target template remains unchanged in this period of time; if the confidence is less than 0.7 and accumulates to N times, the tracker is in an unstable state, and the current target template is reset to the initial target template of the first frame.

[0037] The present application improves in the aspects of multi-modal feature mining, fusion and template updating, thereby improving the RGBT tracking performance. The present application has the following advantages compared with the prior art.

[0038] (1) The modal perception module designed in the target tracking model constructed by the present application fully mines multi-modal discriminative features in combination with the aggregation and distribution mechanism of channel features and the similarity calculation mechanism of spatial features.

[0039] (2) The Transformer fusion module in the target tracking model constructed by the present application is used to fuse the features of the template and the search area, and effectively captures global dependency relationships by using hybrid attention to learn modal perception representation.

[0040] (3) The target tracking model constructed by the present application uses a mutual constraint loss function, and a three-branch prediction head is designed accordingly, thereby improving the positioning accuracy.

[0041] (4) The target tracking model of the present application proposes a state-aware template updating strategy to improve the tracking performance. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the overall flowchart of the present application.

[0043] Figure 2 is a schematic diagram of the channel aggregation and distribution mechanism of the present application.

[0044] Figure 3For the spatial feature similarity perception sketch of the application.

[0045] Figure 4 For the overall tracking performance curve of the application on the LasHeR dataset, (a) the precision plot, (b) the success plot, the precision score and the success rate score are shown in the figure. DETAILED DESCRIPTION

[0046] The technical solutions of the application will be further described in detail below in combination with the corresponding drawings:

[0047] As Figure 1 shown in the tracking overall flowchart, the application proposes an RGBT target tracking method based on modal perception feature learning, which constructs and trains a target tracking model for the RGBT target tracking task, including the following steps.

[0048] Step S1: Constructing a dataset, dividing the corresponding video sequences from the public medium-sized RGBT target tracking dataset for model training and testing, in this embodiment, the corresponding data from the public LasHeR target tracking dataset is divided for model training and testing.

[0049] Step S2: Constructing a target tracking model, including a feature extraction module, a modal perception module, a Transformer fusion module, a three-branch prediction head and a template update module, as follows:

[0050] Step S2.1: Obtain the images of the target template and the search area from the pre-processed and registered visible light and thermal infrared images, respectively, as the input of the target tracking model;

[0051] Step S2.2: Constructing a dual-channel feature extraction module, extracting features from the target template and search area images of the visible light and thermal infrared modalities, in this embodiment, ResNet50 is used as the backbone network to extract the template and search area deep features of the thermal infrared and visible light images;

[0052] Step S2.3: Constructing a modal perception module, for the features extracted in step S2.2, first use the channel feature aggregation and distribution mechanism to remove the interference of redundant information, and fully utilize the attention mechanism to realize multi-modal robust feature learning, as Figure 2 shown, the formula is as follows:

[0053] d g =F g (GAP(f R +f T ))(1)

[0054]

[0055] In the above formula, Let F represent a pair of RGBT target template features obtained from step S2.2. In formula (1), GAP represents the global average pooling operation. g Represents a fully connected layer, d g This represents the vector features of global pooling. During the channel feature aggregation stage, the visible light features f... R With thermal infrared characteristics f T The values ​​are added together; then, they are processed through global average pooling; finally, the channel attention weights are output through a fully connected layer. In the channel feature distribution stage, F in equation (2)... i This represents the feature distribution of a two-branch fully connected layer, where R represents the visible light mode, T represents the thermal infrared mode, and σ represents the sigmoid function. Finally, the weights are multiplied by the original feature f. i z Multiplication generates target template-enhanced features Similarly, This represents a pair of RGBT search region features obtained from step S2.2, based on the channel aggregation and distribution mechanism, by f i x Enhanced features of the search region The channel feature aggregation and distribution mechanism realizes feature enhancement of the target template and the search region. The template fusion feature is obtained by superimposing the enhanced features of the target template under different modalities.

[0056] Spatial feature similarity perception is performed during feature fusion. For channel enhancement features of different modalities generated via the attention mechanism, spatial feature similarity calculation is used. Residuals of perceived instances are generated through similarity learning to achieve feature selection, guiding the enhancement of specific high-quality modalities, and further strengthening more reliable modalities. The process is as follows: Figure 3 As shown, details are as follows:

[0057]

[0058]

[0059]

[0060] In the above formula, S i Represents the similarity feature map, f conv This represents a 3×3 convolution operation, and up represents bilinear interpolation upsampling. The formula represents the enhanced features of different modalities of the target template and the search region, where * represents the convolution operation and σ represents the Sigmoid function. Formula (4) represents the final template features. The enhanced features of the two modalities of visible light and thermal infrared are added. Formula (5) represents that the final search area fusion feature is obtained by adding the visible light and thermal infrared two modalities features and the corresponding similarity map S i The residual map is generated by multiplying the two, and then added to the corresponding thermal infrared and visible light enhanced features.

[0061] Step S2.4: Perform feature flattening operation on the template fusion feature and the search area fusion feature obtained in step S2.3, and convert the features into vectors by using 1x1 convolution, so as to facilitate subsequent fusion module processing.

[0062] Step S2.5: Build a Transformer fusion module, calculate the correlation between the target template and the search area feature for the flattened feature in step S2.4, and obtain a fusion vector. The module includes self-attention and cross-attention mechanisms, which are stacked four times to consider the global dependence of the features and fully fuse the template and search area features.

[0063] Step S2.6: In order to overcome the inconsistency between classification and regression prediction, the present application constructs a three-branch prediction head, and the fusion vector obtains a prediction result through the three-branch prediction head. The three-branch prediction head is composed of three branches of classification, positioning and regression, and is trained by a constraint loss function to realize the information correlation between the branches and suppress unreasonable tracking candidate regions. The three-branch prediction head is sorted according to the confidence score, and the target tracking result is output. The constraint loss function is as follows:

[0064] L cls = -∑ j ((y j log(p j )IoU+(1-y j )log(1-p j ))) (6)

[0065]

[0066]

[0067] Loss = n1L cls + n2L reg + n3L loc (9)

[0068] In the above formula, L cls represents the classification loss, which is improved in the present application, and a binary cross-entropy loss based on IoU weighting is constructed, y j represents the label of the jth sample, y j = 1 represents a positive sample, and p jdenotes the probability belonging to the foreground, IoU denotes the intersection of the predicted value and the true value; L reg denotes the regression loss, including L1 norm loss and L CIoU loss two parts; b j denotes the jth bounding box, p j denotes the classification confidence corresponding to the positive sample, λ1 and λ C both denote the regularization parameter; L loc denotes the positioning loss, O j is the IoU score calculated for the regression branch, denotes the predicted value of the positioning branch; Loss denotes the overall loss, which is the weighted sum of the above three loss functions, wherein n1, n2 and n3 denote hyperparameters.

[0069] Step S2.7: According to the prediction result of step S2.6, the template updating module classifies the target template state according to the confidence score, selects whether to update the target template according to the update strategy, realizes the adaptive update of the target template, and maintains the reliability of the target template. The update strategy of the template updating module is as follows: according to the confidence score output by the prediction head, the target tracking state is divided into stable state, temporary stable state and unstable state. The stable state is defined as a state in which the confidence score of M consecutive frames is greater than 0.9. Once the stable state is reached, the current template will replace the initial template. If the confidence score is between 0.7 and 0.9, it is considered that the tracker is in a temporary stable state, and the template remains unchanged in this time period. If the confidence is less than 0.7 and accumulates to N times, the tracker is in an unstable state, and the current template is reset to the initial target template of the first frame, and subsequent tracking is performed.

[0070] Step S3: Offline training, load the weight of the pre-trained backbone network model on the GOT10K dataset, repeat steps S2.1-S2.6, train the model using the AdamW optimizer until the loss converges, and the iteration number is set to 40 and the learning rate is set to 1e -4 , thereby obtaining the trained model parameters, in addition, the template updating module does not need to be trained.

[0071] Step S4: Online tracking, specifically as follows:

[0072] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, output the predicted value through the feature extraction module, the modal perception module, the Transformer fusion module and the three-branch prediction head, and then select the region with the highest confidence score to obtain the preliminary tracking result;

[0073] Step S4.2: Determine the target state according to the confidence score, and select whether to update the template or reset the template according to the update condition and the current state;

[0074] Step S4.3: repeat steps S4.1-S4.2, gradually calculate the target tracking result of each frame, maintain the update of the reliable target template, so as to realize the target tracking of the whole RGBT sequence.

[0075] In order to verify the effectiveness of the tracking model, comparative experiments are carried out on the latest large-scale RGBT dataset, and the experimental result indexes are as shown in Figure 4 Figure 4 (a) is the accuracy score curve, Figure 4 (b) is the success rate score curve. Compared with the latest multi-modal tracking method, the accuracy and success rate indexes of the present application on the LasHeR dataset are significantly improved, and the performance is better than that of the APFNet and the advanced DMCNet tracking method.

[0076] Finally, the present application also selects representative challenge scenes as visual examples for qualitative analysis of the tracking method performance. In the complex environment, including the challenge of background interference, partial occlusion and scale change, the present application method can obtain satisfactory results. In addition, the running rate of the present application method is about 55 frames per second, which basically meets the real-time requirement.​

Claims

1. A method for RGBT object tracking based on modal perception feature learning, characterized in that The target tracking model is constructed for an RGBT target tracking task, including the following steps. Step S1: Constructing a target tracking dataset, collecting RGBT tracking video sequences for model training and testing; Step S2: Constructing a target tracking model, including a feature extraction module, a modal perception module, a Transformer fusion module, a three-branch prediction head, and a template update module, specifically as follows: Step S2.1: Obtaining the images of the target template and the search region from the preprocessed registered visible light and thermal infrared images, respectively, as inputs of the target tracking model; Step S2.2: The feature extraction module is a dual-channel feature extraction module, which extracts target template and search region features of visible light and thermal infrared images, a total of 4 groups of features; Step S2.3: The modal perception module first uses a channel feature aggregation and distribution mechanism to learn multi-modal robust features of visible light and thermal infrared through attention mechanisms, and then performs spatial feature similarity perception on the enhanced features of the target template and the search region, and finally obtains the search region fusion features by stacking the enhanced features of the target template and the search region under different modalities; Step S2.4: Performing feature flattening operation on the template fusion features and the search region fusion features, and converting the features into vectors by using 1x1 convolution; Step S2.5: The Transformer fusion module calculates the correlation between the target template features and the search region features based on the flattened feature vectors to obtain a fusion vector; Step S2.6: The fusion vector is input into the three-branch prediction head to obtain a prediction result, and the three-branch prediction head is composed of three branches, namely classification, positioning and regression branches, which are connected through constraint loss, and the target tracking result is output according to the confidence score ranking; Step S2.7: According to the prediction result of step S2.6, the template update module classifies the target template state according to the confidence score, and selects whether to update the target template according to the update strategy to realize the adaptive update of the target template and maintain the reliability of the target template; Step S3: Offline training, using the AdamW optimizer to train the model until the loss converges, obtaining the trained model parameters, wherein the template update module does not need to be trained; Step S4: Online tracking: Step S4.1: Obtaining the label of the first frame of the video sequence as the initial tracking target, outputting the prediction value through the target tracking model, and then selecting the region with the highest confidence score to obtain the preliminary tracking result; Step S4.2: According to the confidence score of the result, the current target template state is determined, and whether to update the template or reset the template is selected according to the update strategy and the current state; Step S4.3: Repeating steps S4.1-S4.2, gradually calculating each frame of target tracking result, adaptively updating the template to maintain the reliability of the target template, and completing the overall RGBT sequence target tracking task. 2.The modality-aware feature learning based RGB-T object tracking method according to claim 1, characterized in that In step S2.3, the channel feature enhancement is implemented by using a channel aggregation and distribution mechanism, and the specific implementation is as follows: d g = F g (GAP(f R + f T ))(1) In the above formula, Let F represent a pair of RGBT target template features obtained from step S2.

2. In formula (1), GAP represents the global average pooling operation. g Indicates a fully connected layer, d g The vector features representing global pooling are used in the channel feature aggregation stage to combine visible light features f. R With thermal infrared characteristics f T The values ​​are added together, then subjected to global average pooling, and finally output as channel attention weights through a fully connected layer. In the channel feature distribution stage, F in formula (2)... i This represents the feature distribution of a two-branch fully connected layer, where R represents the visible light mode, T represents the thermal infrared mode, and σ represents the sigmoid function. Finally, the weights are multiplied by the original feature f. i z Multiplication generates enhanced features by This represents a pair of RGBT search region features obtained from step S2.2, and the same reasoning applies to f. i x get 3.The modality-aware feature learning based RGB-T object tracking method according to claim 1, characterized in that In step S2.3, the spatial feature similarity perception adopts a spatial similarity calculation mechanism, and the residual of the perceived instance is generated through similarity learning, and the specific implementation is as follows: In the above formula, S i denotes the similarity feature map, f conv denotes the convolution operation, up denotes the bilinear interpolation up-sampling, denotes the enhanced feature, * denotes the convolution operation, σ denotes the Sigmoid function, and formula (4) denotes the final template fusion feature obtained by adding the thermal infrared and visible light enhanced features, and formula (5) denotes the final search region fusion feature obtained by multiplying the visible light and thermal infrared two modal features with the corresponding similarity map S i respectively, and then added with the corresponding thermal infrared and visible light enhanced features. 4.The modality-aware feature learning based RGB-T object tracking method according to claim 1, characterized in that In step S2.6, the three-branch interrelation is realized by using a constraint loss, and the specific implementation is as follows: L cls = -∑ j ((y j log(p j ) + (1 - y j ) log(1 - p j )))(6) Loss = n1L cls + n2L reg + n3L loc (9) In the above formula, L cls represents the classification loss, which is constructed by using the IoU weighted binary cross-entropy loss, y j represents the label of the jth sample, y j = 1 represents a positive sample, p j represents the probability of belonging to the foreground, and IoU represents the intersection of the predicted value and the true value; L reg represents the regression loss, including L1 norm loss and L CIoU loss two parts, b j represents the jth bounding box, p j represents the classification confidence corresponding to the positive sample, λ1 and λ C both represent regularization parameters; L loc represents the positioning loss, O j is the IoU score calculated for the regression branch, represents the predicted value of the positioning branch; Loss represents the overall loss, which is obtained by weighted summation of the above three loss functions, wherein n1, n2 and n3 represent hyperparameters.

5. The modality-aware feature learning based RGBT target tracking method according to claim 1, characterized in that In step S4.2, the update strategy is as follows: according to the confidence score output by the three-branch prediction head, the target tracking state is divided into stable state, temporary stable state and unstable state, the stable state is defined as the state that the confidence score of M consecutive frames is greater than 0.9, once the stable state is reached, the current target template is updated to replace the initial target template; if the confidence score is between 0.7 and 0.9, it is considered that the tracker is in a temporary stable state, and the target template remains unchanged in this time period; If the confidence is less than 0.7 and the cumulative number reaches N times, the tracker is in an unstable state, and the current target template is reset to the initial target template of the first frame.

Images