RGBT target tracking method based on modal perception feature learning
Through modal-aware feature learning and adaptive template update methods, the performance degradation of RGBT target tracking in complex scenarios is solved, the stability and performance of the tracker is improved, and efficient multimodal feature fusion and positioning accuracy are achieved. .
Patent Information
- Application Number
- CN202310625057.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2043-05-30
Smart Images

Figure CN119068016A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision target tracking, and in particular relates to an RGBT target tracking method based on modal perception feature learning. Background Art
[0002] Visible light sensors are widely used, but they are subject to many limitations in low-light environments and bad weather, resulting in reduced image quality and visual distance, and cannot meet the needs of all-weather applications. To this end, researchers adopted a multimodal photoelectric sensor fusion strategy to combine visible light sensors with thermal infrared sensors to obtain more comprehensive information. Thermal infrared sensors can obtain thermal energy by detecting infrared radiation emitted by objects, thereby providing information that is difficult for visible light sensors to capture. Multimodal information fusion will achieve the complementarity of the two types of information, build a more robust feature representation, and improve the performance of deep models to cope with more complex application scenarios.
[0003] The main goal of the RGBT target tracking task is to use the complementary features of visible light and thermal spectra to build a robust all-weather tracker, which is applied to many fields such as military operations, field search and rescue, security inspection, assisted driving, and crowd density estimation. In the RGBT target tracking task, the target position information in subsequent frames is continuously predicted based on the artificially given labels in the initial frame. At present, a variety of tracking methods have achieved good performance by using the complementary characteristics of visible light and thermal spectra, but further research is still needed in terms of extracting discriminative features that are conducive to instance representation from heterogeneous modalities, accurately estimating the target bounding box, and solving tracking challenges such as scale changes and occlusions.
[0004] There are currently two mainstream research ideas: one is to explore the extraction of multi-layer features and build reliable feature expressions to improve tracking accuracy; the other is to study the allocation of modal fusion weights to achieve adaptive feature selection. However, in the face of challenging scenarios such as target occlusion, appearance changes, and scale changes, the performance of existing tracking methods may decline. Summary of the invention
[0005] The problem to be solved by the present invention is that the existing RGBT target tracking solution has degraded performance and cannot meet the usage requirements in complex scenes, such as target occlusion, appearance change and scale change. The present invention improves the stability and performance of the RGBT target tracker by introducing a method of modal perception feature learning.
[0006] The technical solution of the present invention is: a RGBT target tracking method based on modal perception feature learning, constructing a target tracking model for RGBT target tracking tasks, comprising the following steps:
[0007] Step S1: Construct a target tracking dataset and collect RGBT tracking video sequences for model training and testing;
[0008] Step S2: Construct a target tracking model, including a feature extraction module, a modality perception module, a Transformer fusion module, a three-branch prediction head, and a template update module, as follows:
[0009] Step S2.1: Obtain the target template and the image of the search area from the pre-processed and registered visible light and thermal infrared images respectively, as the input of the target tracking model;
[0010] Step S2.2: The feature extraction module is a dual-channel feature extraction module, which extracts target template and search area features of visible light and thermal infrared images respectively, with a total of 4 groups of features;
[0011] Step S2.3: The modal perception module first adopts the channel feature aggregation and distribution mechanism for the features extracted in step S2.2, and uses the attention mechanism to perform multimodal robust feature learning of visible light and thermal infrared respectively, to achieve feature enhancement of the target template and the search area, and obtains the template fusion feature by superimposing the enhanced features of the target template under different modalities, and then performs spatial feature similarity perception. For the channel enhanced features of different modalities generated by the attention mechanism, the spatial feature similarity calculation is used to perform feature screening, enhance the features of different modalities in the search area, and obtain the search area fusion feature after superposition;
[0012] Step S2.4: performing feature flattening operation on the template fusion feature and the search area fusion feature, and converting the features into vectors using 1×1 convolution;
[0013] Step S2.5: The Transformer fusion module calculates the correlation between the target template features and the search area features on the flattened feature vector to obtain a fusion vector;
[0014] Step S2.6: The fused vector is passed through a three-branch prediction head to obtain a prediction result. The three-branch prediction head consists of three branches, namely classification, localization and regression branches. The three branches are interrelated through constraint loss, sorted according to confidence scores, and the target tracking result is output;
[0015] Step S2.7: Based on the prediction result of step S2.6, the template updating module classifies the target template state according to the confidence score, and chooses whether to update the target template according to the updating strategy, so as to realize the adaptive updating of the target template and maintain the reliability of the target template;
[0016] Step S3: Offline training, using the AdamW optimizer to train the model until the loss converges, and obtain the trained model parameters, wherein the template update module does not need to be trained;
[0017] Step S4: Online tracking:
[0018] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, output the prediction value through the target tracking model, and then select the area with the highest confidence score to obtain the preliminary tracking result;
[0019] Step S4.2: Determine the current state of the target template based on the confidence score of the result, and choose whether to update or reset the template based on the update strategy and the current state;
[0020] Step S4.3: Repeat steps S4.1-S4.2, gradually calculate the target tracking result of each frame, adaptively update the template to maintain the reliability of the target template, and complete the target tracking task of the entire RGBT sequence.
[0021] Furthermore, in step S2.3, channel feature enhancement is achieved by using a channel aggregation and distribution mechanism, as follows:
[0022] d g =F g (GAP(f R +f T )) (1)
[0023]
[0024] In the above formula, represents a pair of RGBT target template features obtained from step S2.2. In formula (1), GAP represents the global average pooling operation, and F g represents the fully connected layer, d g Represents the global pooled vector feature. In the channel feature aggregation stage, the visible light feature f R With thermal infrared characteristics f T Add, then perform global average pooling, and finally output channel attention weights through the fully connected layer; in the channel feature distribution stage, F in formula (2) i represents a dual-branch fully connected layer distribution feature, R represents the visible light modality, T represents the thermal infrared modality, σ represents the Sigmoid function, and finally the weight is combined with the original feature f i z Multiply to generate enhanced features by represents a pair of RGBT search region features obtained from step S2.2, and the same reason get
[0025] Furthermore, in S2.3, spatial feature similarity perception uses a spatial similarity calculation mechanism to generate residuals of perception instances through similarity learning, as follows:
[0026]
[0027]
[0028]
[0029] In the above formula, S i Represents the similarity feature map, f conv represents the convolution operation, up represents the bilinear interpolation upsampling, represents enhanced features, * represents convolution operation, σ represents Sigmoid function, and formula (4) represents the final template fusion feature Obtained by adding thermal infrared and visible light enhancement features, formula (5) represents the final search area fusion feature The two modal features of visible light and thermal infrared and the corresponding similarity graph S i The residual map is generated by multiplying them respectively, and then added to the corresponding thermal infrared and visible light enhancement features.
[0030] As a further improvement of the present invention, in step S2.6, the three branches are correlated with each other through constraint loss, as follows:
[0031] L cls =-∑ j ((y j log(p j )IoU+(1-y j )log(1-p j ))) (6)
[0032]
[0033]
[0034] Loss = n1L cls +n2L reg +n3L loc (9)
[0035] In the above formula, L cls represents the classification loss, which is constructed based on the IoU-weighted binary cross entropy loss, y j Indicates the label of the jth sample, y j =1 indicates a positive sample, p j represents the probability of belonging to the foreground, and IoU represents the intersection of the predicted value and the true value; L reg represents the regression loss, including L1 norm loss and L CIoU Loss of two parts, b j represents the jth bounding box, p jRepresents the classification confidence corresponding to the positive sample, λ1 and λ C Both represent regularization parameters; L loc represents the positioning loss, O j The IoU score calculated for the regression branch, Represents the predicted value of the positioning branch; Loss represents the overall loss, which is the weighted sum of the above three loss functions, where n1, n2 and n3 represent hyperparameters.
[0036] As a further improvement of the present invention, the update strategy in step S4.2 is: according to the confidence score output by the three-branch prediction head, the target tracking state is divided into a steady state, a temporary steady state and an unstable state. The steady state is defined as a state in which the confidence score of M consecutive frames is greater than 0.9. Once the stable state is reached, the current target template is updated to replace the initial target template; if the confidence score is between 0.7 and 0.9, the tracker is considered to be in a temporary steady state, and the target template remains unchanged during this time period; if the confidence score is less than 0.7 and reaches N times cumulatively, the tracker is in an unstable state, and the current target template is reset to the initial target template of the first frame.
[0037] The present invention improves multimodal feature mining, fusion and template updating, thereby improving RGBT tracking performance. The present invention has the beneficial effect of providing a high-performance RGBT target tracking technology, which has the following advantages over the prior art.
[0038] (1) The modal perception module designed in the target tracking model constructed by the present invention combines the aggregation and distribution mechanism of channel features and the similarity calculation mechanism of spatial features to fully exploit multimodal discriminative features.
[0039] (2) The Transformer fusion module in the target tracking model constructed by the present invention is used to fuse the features of the template and the search area, and adopts hybrid attention to effectively capture the global dependency to learn the modal perception representation.
[0040] (3) The target tracking model constructed by the present invention adopts a mutually constrained loss function and a corresponding three-branch prediction head is designed to improve the positioning accuracy.
[0041] (4) The target tracking model of the present invention proposes a state-aware template update strategy to improve tracking performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is the overall flow chart of the present invention.
[0043] Figure 2 Schematic diagram of the channel aggregation and distribution mechanism of the present invention.
[0044] Figure 3This is a schematic diagram of spatial feature similarity perception of the present invention.
[0045] Figure 4 1 is the overall tracking performance curve of the present invention on the LasHeR dataset, wherein (a) the precision plot, (b) the success plot, the accuracy score and the success rate score are shown in the figure respectively. DETAILED DESCRIPTION
[0046] The technical solution of the present invention is further described in detail below in conjunction with the corresponding drawings:
[0047] like Figure 1 As shown in the overall tracking flowchart shown, the present invention proposes an RGBT target tracking method based on modal perception feature learning, which constructs and trains a target tracking model for RGBT target tracking tasks, including the following steps.
[0048] Step S1: construct a data set, divide the corresponding video sequence from the public medium and large RGBT target tracking data set for model training and testing. In this embodiment, the corresponding data is divided from the public LasHeR target tracking data set for model training and testing.
[0049] Step S2: Construct a target tracking model, including a feature extraction module, a modality perception module, a Transformer fusion module, a three-branch prediction head, and a template update module, as follows:
[0050] Step S2.1: Obtain the target template and the image of the search area from the pre-processed and registered visible light and thermal infrared images respectively, as the input of the target tracking model;
[0051] Step S2.2: construct a dual-channel feature extraction module to extract features of the target template and search area image of the visible light modality and thermal infrared modality respectively. In this embodiment, ResNet50 is used as the backbone network to extract templates and search area depth features of thermal infrared and visible light images;
[0052] Step S2.3: Construct a modal perception module. For the features extracted in step S2.2, first use the channel feature aggregation and distribution mechanism to remove the interference of redundant information and make full use of the attention mechanism to achieve multimodal robust feature learning, such as Figure 2 As shown, the formula is as follows:
[0053] d g =F g (GAP(f R +f T ))(1)
[0054]
[0055] In the above formula, represents a pair of RGBT target template features obtained from step S2.2. In formula (1), GAP represents the global average pooling operation, and F g represents the fully connected layer, d g Represents the global pooled vector feature. In the channel feature aggregation stage, the visible light feature f R With thermal infrared characteristics f T Then, the global average pooling operation is performed; finally, the channel attention weight is output through the fully connected layer. In the channel feature distribution stage, F in formula (2) i represents a dual-branch fully connected layer distribution feature, R represents the visible light modality, T represents the thermal infrared modality, σ represents the Sigmoid function, and finally the weight is combined with the original feature f i z Multiply to generate target template enhanced features Similarly, represents a pair of RGBT search region features obtained from step S2.2, based on the channel aggregation and distribution mechanism composed of f i x Get the search area enhanced features The channel feature aggregation and distribution mechanism realizes the feature enhancement of the target template and the search area, and the template fusion feature is obtained by superimposing the enhanced features of the target template under different modalities.
[0056] When fusing features, spatial feature similarity perception is performed. For the channel enhancement features of different modalities generated by the attention mechanism, spatial feature similarity calculation is used. The residual of the perception instance is generated through similarity learning to achieve feature screening, guide specific high-quality modal enhancement, and further strengthen more reliable modalities. The process is as follows: Figure 3 As shown, the details are as follows:
[0057]
[0058]
[0059]
[0060] In the above formula, S i Represents the similarity feature map, f conv represents a 3×3 convolution operation, up represents bilinear interpolation upsampling, represents the enhanced features of different modes of the target template and the search area, * represents the convolution operation, and σ represents the Sigmoid function. Formula (4) represents the final template feature It is obtained by adding the enhanced features of the visible light and thermal infrared modes. Formula (5) shows that the final search area fusion feature is obtained by adding the visible light and thermal infrared two modal features and the corresponding similarity map S i The residual map is generated by multiplying them respectively, and then added to the corresponding thermal infrared and visible light enhancement features.
[0061] Step S2.4: Perform feature flattening operation on the template fusion features and search area fusion features obtained in step S2.3, and use 1×1 convolution to convert the features into vectors for easy processing by subsequent fusion modules.
[0062] Step S2.5: Construct a Transformer fusion module to calculate the correlation between the target template and the search area features for the flattened features in step S2.4 to obtain a fusion vector. This module contains self-attention and cross-attention mechanisms, which are stacked four times to take into account the global dependency of features and fully fuse the features of the template and the search area.
[0063] Step S2.6: In order to overcome the problem of inconsistent predictions between classification and regression, the present invention constructs a three-branch prediction head. The fusion vector is passed through the three-branch prediction head to obtain the prediction result. The three-branch prediction head consists of three branches: classification, positioning and regression. It is trained through a constrained loss function to achieve information association between branches, which is used to suppress unreasonable tracking candidate areas. The three-branch prediction head is sorted according to the confidence score and outputs the target tracking result. The constraint loss function is as follows:
[0064] L cls =-∑ j ((y j log(p j )IoU+(1-y j )log(1-p j ))) (6)
[0065]
[0066]
[0067] Loss = n1L cls +n2L reg +n3L loc (9)
[0068] In the above formula, L cls Represents the classification loss. The present invention improves the classification loss by adopting the binary cross entropy loss based on IoU weighting. j Indicates the label of the jth sample, y j =1 indicates a positive sample, p jrepresents the probability of belonging to the foreground, and IoU represents the intersection of the predicted value and the true value; L reg represents the regression loss, including L1 norm loss and L CIoU Loss of two parts; b j represents the jth bounding box, p j Represents the classification confidence corresponding to the positive sample, λ1 and λ C Both represent regularization parameters; L loc represents the positioning loss, O j The IoU score calculated for the regression branch, Represents the predicted value of the positioning branch; Loss represents the overall loss, which is the weighted sum of the above three loss functions, where n1, n2 and n3 represent hyperparameters.
[0069] Step S2.7: According to the prediction results of step S2.6, the template update module classifies the target template state according to the confidence score, and chooses whether to update the target template according to the update strategy, so as to realize the adaptive update of the target template and maintain the reliability of the target template. The update strategy of the template update module is as follows: according to the confidence score output by the prediction head, the target tracking state is divided into steady state, temporary steady state and non-steady state. The steady state is defined as the state where the confidence score of M consecutive frames is greater than 0.9. Once the steady state is reached, the current template will replace the initial template; if the confidence score is between 0.7 and 0.9, the tracker is considered to be in a temporary steady state, and the template remains unchanged during this time period; if the confidence score is less than 0.7 and reaches N times cumulatively, the tracker is in a non-steady state, and the current template is reset to the initial target template of the first frame before subsequent tracking.
[0070] Step S3: Offline training, load the weights of the backbone network model pre-trained on the GOT10K dataset, repeat steps S2.1-S2.6, train the model using the AdamW optimizer until the loss converges, set the number of iterations to 40, and set the learning rate to 1e -4 , thereby obtaining the trained model parameters. In addition, the template update module does not need to be trained.
[0071] Step S4: Online tracking, as follows:
[0072] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, output the prediction value through the feature extraction module, modality perception module, Transformer fusion module, and three-branch prediction head, and then select the area with the highest confidence score to obtain the preliminary tracking result;
[0073] Step S4.2: Determine the target state according to the confidence score, and choose whether to update the template or reset the template according to the update conditions and the current state;
[0074] Step S4.3: Repeat steps S4.1-S4.2, gradually calculate the target tracking result of each frame, maintain and update the reliable target template, so as to achieve target tracking of the entire RGBT sequence.
[0075] In order to verify the effectiveness of the tracking model, a comparative experiment was conducted on the latest medium and large RGBT datasets. The experimental results are as follows: Figure 4 As shown, Figure 4 (a) is the accuracy score curve, Figure 4 (b) is the success rate score curve. Compared with the latest multimodal tracking method, the accuracy and success rate of the present invention on the LasHeR dataset are significantly improved, and it performs better than APFNet and the advanced DMCNet tracking method.
[0076] Finally, the present invention also selects representative challenge scenes as visualization examples for qualitative analysis of the performance of the tracking method. In complex environments, including background interference, partial occlusion, and scale changes, the method of the present invention can obtain relatively satisfactory results. In addition, the running rate of the method of the present invention is approximately 55 frames per second, which basically meets the real-time requirements.
Claims
1. A RGBT target tracking method based on modality-aware feature learning, Its characteristics are Building a target tracking model for RGBT target tracking tasks includes the following steps: Step S1: Construct a target tracking dataset and collect RGBT tracking video sequences for model training and testing; Step S2: Construct a target tracking model, including a feature extraction module, a modality perception module, a Transformer fusion module, a three-branch prediction head, and a template update module, as follows: Step S2.1: Obtain the target template and the image of the search area from the pre-processed and registered visible light and thermal infrared images respectively, as the input of the target tracking model; Step S2.2: The feature extraction module is a dual-channel feature extraction module, which extracts target template and search area features of visible light and thermal infrared images respectively, with a total of 4 groups of features; Step S2.3: The modal perception module first adopts the channel feature aggregation and distribution mechanism for the features extracted in step S2.2, and uses the attention mechanism to perform multimodal robust feature learning of visible light and thermal infrared respectively, to achieve feature enhancement of the target template and the search area, and obtains the template fusion feature by superimposing the enhanced features of the target template under different modalities, and then performs spatial feature similarity perception. For the channel enhanced features of different modalities generated by the attention mechanism, the spatial feature similarity calculation is used to perform feature screening, enhance the features of different modalities in the search area, and obtain the search area fusion feature after superposition; Step S2.4: performing feature flattening operation on the template fusion feature and the search area fusion feature, and converting the features into vectors using 1×1 convolution; Step S2.5: The Transformer fusion module calculates the correlation between the target template features and the search area features on the flattened feature vector to obtain a fusion vector; Step S2.6: The fused vector is passed through a three-branch prediction head to obtain a prediction result. The three-branch prediction head consists of three branches, namely classification, localization and regression branches. The three branches are interrelated through constraint loss, sorted according to confidence scores, and the target tracking result is output; Step S2.7: Based on the prediction result of step S2.6, the template update module classifies the target template state according to the confidence score, and chooses whether to update the target template according to the update strategy, so as to realize the adaptive update of the target template and maintain the reliability of the target template; Step S3: Offline training, using the AdamW optimizer to train the model until the loss converges, and obtain the trained model parameters, wherein the template update module does not need to be trained; Step S4: Online tracking: Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, output the prediction value through the target tracking model, and then select the area with the highest confidence score to obtain the preliminary tracking result; Step S4.2: Determine the current state of the target template based on the confidence score of the result, and choose whether to update or reset the template based on the update strategy and the current state; Step S4.3: Repeat steps S4.1-S4.2, gradually calculate the target tracking result of each frame, adaptively update the template to maintain the reliability of the target template, and complete the target tracking task of the entire RGBT sequence.
2. The RGBT target tracking method based on modality-aware feature learning according to claim 1, Its characteristics are In step S2.3, channel feature enhancement is achieved using the channel aggregation and distribution mechanism, as follows: d g =F g (GAP(f R +f T ))(1) In the above formula, represents a pair of RGBT target template features obtained from step S2.
2. In formula (1), GAP represents the global average pooling operation, and F g represents the fully connected layer, d g Represents the global pooled vector feature. In the channel feature aggregation stage, the visible light feature f R With thermal infrared characteristics f T Add, then perform global average pooling, and finally output channel attention weights through the fully connected layer; in the channel feature distribution stage, F in formula (2) i represents a dual-branch fully connected layer distribution feature, R represents the visible light modality, T represents the thermal infrared modality, σ represents the Sigmoid function, and finally the weight is combined with the original feature f i z Multiply to generate enhanced features by represents a pair of RGBT search region features obtained from step S2.2, and the same reason f i x get 3. The RGBT target tracking method based on modality-aware feature learning according to claim 1, Its characteristics are In step S2.3, spatial feature similarity perception uses a spatial similarity calculation mechanism to generate the residual of the perception instance through similarity learning, as follows: In the above formula, S i Represents the similarity feature map, f conv represents the convolution operation, up represents the bilinear interpolation upsampling, represents enhanced features, * represents convolution operation, σ represents Sigmoid function, and formula (4) represents the final template fusion feature Obtained by adding thermal infrared and visible light enhancement features, formula (5) represents the final search area fusion feature The two modal features of visible light and thermal infrared and the corresponding similarity graph S i The residual map is generated by multiplying them respectively, and then added to the corresponding thermal infrared and visible light enhancement features.
4. The RGBT target tracking method based on modality-aware feature learning according to claim 1, Its characteristics are In step S2.6, the three branches are correlated with each other through constraint loss, as follows: L cls =-∑ j ((y j log(p j )IoU+(1-y j )log(1-p j )))(6) Loss=n 1 L cls +n 2 L reg +n 3 L loc (9) In the above formula, L cls represents the classification loss, which is constructed based on the IoU-weighted binary cross entropy loss, y j Indicates the label of the jth sample, y j =1 indicates a positive sample, p j represents the probability of belonging to the foreground, and IoU represents the intersection of the predicted value and the true value; L reg represents the regression loss, including L 1 Norm loss and L CIoU Loss of two parts, b j represents the jth bounding box, p j Represents the classification confidence corresponding to the positive sample, λ 1 and λ C Both represent regularization parameters; L loc represents the positioning loss, O j The IoU score calculated for the regression branch, represents the predicted value of the positioning branch; Loss represents the overall loss, which is obtained by weighted summation of the above three loss functions, where n 1 、n 2 and n 3 represents a hyperparameter.
5. The RGBT target tracking method based on modality-aware feature learning according to claim 1, Its characteristics are The update strategy in step S4.2 is: according to the confidence score output by the three-branch prediction head, the target tracking state is divided into steady state, temporary steady state and non-steady state. The steady state is defined as the state where the confidence score of M consecutive frames is greater than 0.
9. Once the steady state is reached, the current target template is updated to replace the initial target template. If the confidence score is between 0.7 and 0.9, the tracker is considered to be in a temporary steady state, and the target template remains unchanged during this period. If the confidence is less than 0.7 and reaches N times in total, the tracker is in a non-steady state and the current target template is reset to the initial target template of the first frame.
Citation Information
Patent Citations
Cross-modal target tracking method and system based on modal perception feature learning
CN112949451A
RGBT target tracking method based on cross-modal sharing and specific representation form
CN113077491A
RGBT visual tracking method and system based on two-stage fusion structure search
CN113837296A
Real-time RGBT target tracking method based on multi-modal interaction and multi-stage optimization
CN115170605A
Cited By
RGBT target tracking method based on domain adaptation and spatio-temporal information fusion
CN119863491A
RGBT target tracking method based on convolution attention fusion
CN120088292A
Single-target tracking method based on view angle of unmanned aerial vehicle
CN122067145A
