RGB-T target tracking method based on multi-feature adaptive fusion and primary and auxiliary dynamic restoration

Abstract

The present invention relates to the field of computer vision technology and discloses an RGB-T target tracking method based on multi-feature adaptive fusion and primary-auxiliary dynamic recovery. By extracting multi-level features from RGB and thermal images, constructing a fusion response model, and dynamically calculating the importance of each modal feature, the method achieves adaptive fusion and tracking reliability assessment. When tracking is good, a short-term optimization strategy is used to quickly locate the target. When tracking is unreliable, a long-term recovery mechanism is triggered to recapture the target by switching between the primary and auxiliary modalities, ultimately achieving target position updates and continuous model optimization. By extracting multi-level features to construct a fusion response model, dynamically assigning modal weights, judging tracking reliability, and flexibly switching between short-term optimization and long-term recovery strategies, the method achieves accurate target tracking and continuous model updates, significantly enhancing the system's robustness and stability in complex environments.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an RGB-T target tracking method based on multi-feature adaptive fusion and primary-auxiliary dynamic restoration. Background Art

[0002] RGB-T tracking, a hot research area in multimodal visual fusion in recent years, has demonstrated tremendous potential in areas such as autonomous driving and intelligent surveillance. Its core advantage lies in combining the complementary properties of RGB and thermal infrared images, ensuring robustness under extreme lighting and occlusion conditions. However, existing technologies still have significant shortcomings in fusion methods and modality utilization mechanisms, and further breakthroughs are urgently needed.

[0003] On the one hand, most existing RGB-T tracking methods still use fixed weights or hierarchical splicing in their fusion strategies, making them difficult to adapt to the changing importance of modal features in different scenarios. Modal quality fluctuates dynamically in actual tracking, and single modality is particularly susceptible to failure in scenes with strong light, occlusion, or thermal infrared overexposure. This static fusion approach not only lacks specificity but also easily introduces redundant or misleading information, ultimately affecting tracking stability.

[0004] On the other hand, existing methods mostly rely on short-term tracking structures for specific modalities, and their ability to recover after tracking failures relies heavily on global redetection. While effective in some situations, this approach suffers from slow recovery and poor accuracy when the target is occluded for extended periods or when the modality is severely degraded, making it difficult to meet the requirements of continuous tracking. In particular, decision-level fusion methods often rely on manually set fixed thresholds to switch between primary and secondary modalities, lacking dynamic adaptive mechanisms to address sudden changes in modality availability.

[0005] Furthermore, existing technologies generally focus on modeling a single feature type. Most methods utilize only single-layer features output by convolutional networks, ignoring the complementary relationships between multiple layers of features. They also rarely combine deep semantic features with low-level handcrafted features such as edge textures. This limitation in feature representation significantly reduces the model's discriminative ability when the background is complex or the object is drastically deformed, making it prone to drift.

[0006] Overall, the current RGB-T tracking technology still has obvious shortcomings in the flexibility of feature fusion, dynamic modal switching capabilities, and long-term robustness. Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention provides an RGB-T target tracking method based on multi-feature adaptive fusion and primary and auxiliary dynamic restoration, which solves the problems of inflexible modal fusion, insufficient recovery capability and incomplete feature expression in existing RGB-T tracking.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: an RGB-T target tracking method based on multi-feature adaptive fusion and primary and auxiliary dynamic restoration, comprising the following steps:

[0009] S1. Perform multi-level feature extraction on the input RGB image and thermal imaging image to obtain the feature set of each modality;

[0010] S2. constructing a multi-feature response fusion model based on the feature set, and obtaining a fusion response graph and feature weight parameters through joint optimization calculation;

[0011] S3, calculating the tracking reliability evaluation value of the current frame according to the feature weight parameter;

[0012] S4. When the tracking reliability evaluation value meets a preset condition, performing short-term tracking optimization to determine the target position; when it does not meet the preset condition, performing long-term tracking recovery to recapture the target;

[0013] S5. Update the tracking model parameters according to the final determined target position and output the tracking result.

[0014] Preferably, the step S2 includes:

[0015] S21. Generate response maps for each of the M features of each modality, where M ≥ 2;

[0016] S22, establishing an optimization function with the goal of maximizing the peak significance of the response graph and minimizing the response fluctuation;

[0017] S23. An optimization strategy combining particle swarm initialization and nonlinear programming is used to solve the feature weight parameters.

[0018] Preferably, the optimization function is:

[0019] ;

[0020] Constraints include:

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] in, is the RGB modality feature weight vector, is the thermal imaging modal feature weight vector, is the multi-feature fusion response map, is the response graph size, is the average value of the fused response map, is an N×N matrix of all 1s, For RGB mode Feature response plots, Thermal imaging modality Feature response plots, is the number of feature types, For RGB mode The weight of the feature, Thermal imaging modality The weight of the feature, To represent the first Row, No. The response value at column, To express the difference between the response value of each pixel in the fusion response map and the overall average response value, Indicates that through optimization and The weighted coefficients of the modalities are used to find a set of optimal weights so that the fused response graph can achieve the maximum discriminative ability in the current frame. is the maximum response value in the entire fusion response graph.

[0026] Preferably, the tracking reliability evaluation value in step S3 is calculated by the following formula:

[0027] ;

[0028] in, and Calculated by HaarPSI algorithm, is the RGB modality template similarity, is the thermal imaging modality template similarity, is the joint modal reliability.

[0029] Preferably, the short-term tracking optimization in step S4 includes:

[0030] Generate a set of candidate regions associated with historical trajectories;

[0031] Filter the best candidates through spatial constraints and appearance similarity calculation;

[0032] Refine the object bounding box based on the filtering results.

[0033] Preferably, the screening of the best candidates adopts:

[0034] ;

[0035] in, is the overlap ratio, is the spatial distance, is the candidate box area, is the confidence of the candidate box, is the maximum distance, is the appearance similarity, is the reference target area.

[0036] Preferably, the long-term tracking recovery in step S4 includes:

[0037] Generate initial candidate boxes through the YOLOv4-tiny detector;

[0038] Select the top K=5 high-confidence candidates for multimodal response verification;

[0039] The candidate with the largest PSR response that exceeds the dynamic threshold is selected as the recovery target, where the dynamic threshold is the last valid PSR value before tracking failure.

[0040] The present invention provides an RGB-T target tracking method based on multi-feature adaptive fusion and primary and auxiliary dynamic restoration. It has the following beneficial effects:

[0041] 1. By adopting a multi-feature response adaptive fusion model, the present invention can dynamically identify the effectiveness of each modal feature in different scenarios, automatically adjust the fusion strategy, and ensure the maximization of feature complementarity. Unlike the existing fixed fusion ratio or simple splicing method, this solution breaks through the limitations of modal information redundancy and interference coexistence, and significantly improves the tracking and discrimination capabilities in complex environments.

[0042] 2. This invention introduces a primary-auxiliary dynamic selection recovery mechanism, which enables the tracking system to autonomously adjust modal priorities and recovery paths. It can recapture targets under thermal infrared interference or extreme RGB lighting without human intervention. Compared with traditional methods that rely on static modal switching strategies, this mechanism significantly improves the accuracy of target recovery after occlusion and effectively addresses the uncertainty risks brought about by modal degradation.

[0043] 3. The present invention utilizes the combined response of deep convolutional network output and manual features. This invention not only solves the problem of insufficient shallow feature expression, but also retains the complementary advantages of high-level semantics and low-level details. The existing technology often has a single feature expression, which leads to drift when the target size changes or the background is cluttered. The present technical solution maintains the stability and continuity of target positioning through multi-feature fusion and response guidance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2 This is the overall flow chart of the MFDSR of the present invention;

[0046] Figure 3 Performance evaluation on the GTOT dataset of this invention Figure 1 ;

[0047] Figure 4 Performance evaluation on the GTOT dataset of this invention Figure 2 ;

[0048] Figure 5 Performance evaluation on the RGBT234 dataset of the present invention Figure 1 ;

[0049] Figure 6 Performance evaluation on the RGBT234 dataset of the present invention Figure 2 ;

[0050] Figure 7 The performance evaluation on the LasHeR dataset of the present invention Figure 1 ;

[0051] Figure 8 The performance evaluation on the LasHeR dataset of the present invention Figure 2 ;

[0052] Figure 9 The performance evaluation of the present invention on the VTUAV-ST dataset Figure 1 ;

[0053] Figure 10 The performance evaluation of the present invention on the VTUAV-ST dataset Figure 2 ;

[0054] Figure 11 Performance evaluation on the VTUAV-LT dataset of the present invention Figure 1 ;

[0055] Figure 12 Performance evaluation on the VTUAV-LT dataset of the present invention Figure 2 ;

[0056] Figure 13 This is a visual comparison chart of the tracking results of the present invention;

[0057] Figure 14 This is the enlarged comparison image of frame 251 in the visual comparison image of the tracking results of the present invention;

[0058] Figure 15 This is the enlarged comparison image of frame 1751 in the visual comparison image of the tracking results of the present invention;

[0059] Figure 16 This is the enlarged comparison image of frame 5671 in the visual comparison image of the tracking results of the present invention;

[0060] Figure 17This is the enlarged comparison image of frame 451 in the visual comparison image of the tracking results of the present invention;

[0061] Figure 18 This is the enlarged comparison image of frame 601 in the visual comparison image of the tracking results of the present invention;

[0062] Figure 19 This is an enlarged comparison diagram of frame 4851 in the visual comparison diagram of the tracking results of the present invention. DETAILED DESCRIPTION

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the present specification. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0064] Please see the attached Figure 1 -Attached Figure 2 The embodiment of the present invention provides an RGB-T target tracking method based on multi-feature adaptive fusion and primary and auxiliary dynamic restoration, comprising the following steps:

[0065] S1. Perform multi-level feature extraction on the input RGB image and thermal imaging image to obtain the feature set of each modality;

[0066] Multi-level feature extraction utilizes a bimodal parallel processing architecture. For the input RGB image, a VGG-M network consisting of 13 convolutional layers and 3 fully connected layers is first used to extract the convolutional feature map of the third layer (conv3) and the fully connected feature map of the 14th layer (fc6). Histogram of Oriented Gradients (HOG) features and color attribute (CN) features are also calculated simultaneously. The same operation is performed simultaneously on the thermal image, forming a hybrid feature set containing four types of features.

[0067] The specific implementation of VGG-M network feature extraction is as follows: the input image is normalized to 125×125 pixel size, and after processing by the convolution layer group, the third layer outputs a 64-channel 28×28 spatial feature map, which is mathematically expressed as:

[0068] ;

[0069] in, is the three-dimensional feature tensor output by the third layer of RGB modality VGG-M, The 3D feature tensor output by the third layer of VGG-M for thermal imaging modality;

[0070] The 14th layer outputs a 4096-dimensional fully connected feature vector:

[0071] ;

[0072] in, is the fully connected feature vector output by the 14th layer of the RGB modality VGG-M, It is the fully connected feature vector output by the 14th layer of the VGG-M for thermal imaging modality;

[0073] The HOG feature calculation sets the cell size to 8×8 pixels and the block size to 16×16 pixels (including 2×2 cell units), generating a 31-dimensional feature descriptor:

[0074] ;

[0075] in, is a 31-dimensional descriptor of HOG features in RGB mode, It is a 31-dimensional descriptor of the HOG feature of the thermal imaging modality;

[0076] The CN feature uses color namespace mapping to output an 11-dimensional color attribute histogram:

[0077] ;

[0078] in, is the 11-dimensional color histogram of the RGB modality CN feature, is the 11-dimensional color histogram of CN features of thermal imaging modality;

[0079] The feature set construction satisfies:

[0080] ;

[0081] ;

[0082] Among them, the superscripts 1-4 correspond to the VGG-Mconv3 layer, fc6 layer, HOG and CN features respectively, and the number of features M=4. is the RGB modality feature set, is a feature set of thermal imaging modality. The size of each feature map is based on the width of the target bounding box. Dynamic calculation:

[0083] ;

[0084] The logical connection between the feature extraction process and the fusion model is as follows: HOG features provide edge structure information, CN features encode color distribution characteristics, VGG-M deep features capture semantic information, and shallow features retain spatial details. The four types of features are adaptively weighted to form a complementary representation, which is the basis for the subsequent optimization function:

[0085] ;

[0086] A differentiated feature substrate is provided, wherein is an N×N matrix of all 1s, is the mean of the fused response graph, is the RGB modality feature weight vector, is the thermal imaging modal feature weight vector, is the multi-feature fusion response map, is the response graph size, is the average value of the fused response map, To represent the first Row, No. The response value at column, To express the difference between the response value of each pixel in the fusion response map and the overall average response value, Indicates that through optimization and The weighted coefficients of the modalities are used to find a set of optimal weights so that the fused response graph can achieve the maximum discriminative ability in the current frame. is the maximum response value in the entire fusion response graph.

[0087] In this embodiment, when processing a 640×480 resolution input image, the following operations are performed: 125×125 pixel RGB and thermal dual-channel image patches are captured; a 64×28×28 feature tensor from the conv3 layer and a 4096-dimensional feature vector from the fc6 layer are extracted using a VGG-M network; 31-dimensional HOG features and 11-dimensional CN features are calculated; and these four feature classes are input into the optimization model for weight distribution. This implementation enables efficient multi-scale feature extraction, providing spatially and semantically complementary feature representations for subsequent adaptive fusion.

[0088] S2. Construct a multi-feature response fusion model based on the feature set, and obtain the fusion response graph and feature weight parameters through joint optimization calculation;

[0089] The construction of the multi-feature response fusion model includes the following technical processes. First, the response maps of M=4 features of each modality are generated separately, where the RGB modality response map is , the thermal imaging modal response atlas is . Response map size According to the target bounding box width Dynamic calculation: , where w is the target relative width normalized to the unit interval, Indicates a floor operation.

[0090] Establish an optimization function with the goal of maximizing the peak-to-sidelobe ratio (PSR):

[0091] ;

[0092] Constraints include:

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] in, is the RGB modality feature weight vector, is the thermal imaging modal feature weight vector, is the multi-feature fusion response map, is the response graph size, is the average value of the fused response map, For RGB mode Feature response plots, Thermal imaging modality Feature response plots, is the number of feature types, For RGB mode The weight of the feature, Thermal imaging modality The weight of the feature, is an N×N matrix of all 1s, satisfying , To represent the first Row, No. The response value at column, To express the difference between the response value of each pixel in the fusion response map and the overall average response value, Indicates that through optimization and The weighted coefficients of the modalities are used to find a set of optimal weights so that the fused response graph can achieve the maximum discriminative ability in the current frame. is the maximum response value in the entire fusion response graph.

[0098] The optimization strategy combining particle swarm initialization and interior point method is adopted, and the specific implementation is as follows:

[0099] Particle swarm initialization: Generate 60 particles, each particle contains The speed update formula of the dimensional parameters (M RGB weights + M TIR weights) is:

[0100] ;

[0101] in, , after 20 iterations, the optimal solution is retained , For the The particle In dimension speed, is a uniformly distributed random number, For particles In dimension The best historical position, For the The particle In dimension Current location, is the global optimal position of the group in dimension dd, To indicate the The particle in In terms of dimension, The speed value of the generation;

[0102] Interior point optimization: As a starting point, set the initial value of the barrier factor , the update strategy is:

[0103] ;

[0104] in, For the Iteration barrier factor, For the Sub-iteration barrier factor;

[0105] When the change of the objective function in adjacent iterations is less than Terminates when.

[0106] In this embodiment, when processing a 125×125 pixel ROI, the following operations are performed: bicubic interpolation is performed on the 28×28 response map output by the third layer of VGG-M to size; calculate the weighted sum of the four feature response maps; after obtaining the initial weights through particle swarm optimization, iteratively optimize the final output feature weight parameters using the interior point method. This implementation can achieve adaptive allocation of feature weights and improve the peak-to-sidelobe ratio of the fused response map.

[0107] S3, calculating the tracking reliability evaluation value of the current frame according to the feature weight parameter;

[0108] The calculation of the tracking reliability evaluation value is achieved through the following technical process. First, obtain the RGB template block of the current frame and thermal imaging template blocks , and the high confidence reference template block 、 Perform similarity calculation. Use HaarPSI algorithm to calculate similarity:

[0109] ;

[0110] HaarPSI ;

[0111] Among them, the HaarPSI algorithm calculates the local structural similarity through three-level wavelet decomposition, and the specific implementation is as follows:

[0112] Perform 3-layer Haar wavelet decomposition on the input module

[0113] Calculate the structural similarity (SSIM) and phase consistency (PC) of each layer

[0114] Weighted average similarity score of each layer, the weight coefficient is

[0115] Then the joint modal confidence is calculated by combining the feature weight parameters:

[0116] ;

[0117] Where, is the RGB modality feature weight vector, is the thermal imaging modal feature weight vector, the denominator To achieve weight normalization, and Calculated by HaarPSI algorithm, is the RGB modality template similarity, is the thermal imaging modality template similarity, is the joint modal reliability.

[0118] In the embodiment, when processing the occlusion scene, the following operations are performed: The frame extraction module performs HaarPSI similarity calculation; combined with the optimized weights and , calculate the joint modal reliability When the average confidence of 5 consecutive frames is lower than 0.35, the recovery mechanism is triggered.

[0119] S4. When the tracking reliability evaluation value meets the preset conditions, short-term tracking optimization is performed to determine the target position; when it does not meet the conditions, long-term tracking recovery is performed to recapture the target;

[0120] The implementation of the main-auxiliary dynamic recovery mechanism includes the following technical processes: When the average tracking reliability evaluation value of five consecutive frames is lower than the set threshold, long-term tracking recovery is performed; otherwise, short-term tracking optimization is performed.

[0121] The specific implementation of short-term tracking optimization is as follows:

[0122] Generate a set of candidate regions associated with the historical trajectory space

[0123] The candidate confidence is calculated by the following formula:

[0124] ;

[0125] in, is the overlap ratio, is the spatial distance, is the candidate box area, is the confidence of the candidate box, is the maximum distance, is the appearance similarity, is the reference target area.

[0126] The candidate box with the highest confidence is selected for bounding box refinement.

[0127] The specific implementation of long-term tracking recovery is as follows:

[0128] Generate initial candidate boxes through the YOLOv4-tiny detector;

[0129] Select the top K high-confidence candidates (K=5);

[0130] Conduct multimodal response verification on candidate regions;

[0131] Candidates whose PSR responses exceed a dynamic threshold are selected for target recovery.

[0132] In an embodiment, when long-term tracking recovery is triggered:

[0133] Use YOLOv4-tiny detector to generate candidate boxes;

[0134] Select the top 5 candidates;

[0135] During the verification process, the last valid PSR value before tracking failure is used as the dynamic threshold;

[0136] Reinitialize the tracker for candidates that meet the threshold conditions.

[0137] S5. Update the tracking model parameters according to the final determined target position and output the tracking result.

[0138] The model update and result output process is achieved through the following technical solutions. Based on the final target position, the ECO-related filter architecture is used to update the model parameters. The specific implementation includes:

[0139] Multi-feature template update:

[0140] Execute separately for RGB and thermal imaging modalities:

[0141] ;

[0142] in, is the updated feature template, VGG-M features extracted for the current frame (layers 3 and 14), is the learning rate parameter, is the smooth feature representation of the previous frame;

[0143] Related filter parameter update:

[0144] Adopt ECO standard update strategy:

[0145] ;

[0146] in, , is the numerator / denominator term, is the regularization coefficient, For the The filter / weight response function for the frame.

[0147] Weight parameter update: retain historical weight information:

[0148] ;

[0149] in, The feature weights obtained by optimizing this frame, For the forgetting factor, is the fusion weight vector of the current frame, indicating the In the frame, the fusion weight values corresponding to different modes (such as RGB, TIR, etc.) determine the proportion of multimodal features in the current frame fusion process. is the fusion weight vector of the previous frame.

[0150] In the embodiment, after the target position is determined:

[0151] Extract multimodal features of the target area;

[0152] Update VGG-M network feature template;

[0153] Adjust the histogram of oriented gradients and color feature weights;

[0154] Output target coordinates and bounding box parameters.

[0155] For test examples, see the attached Figure 3 -Attached Figure 13 :

[0156] Test environment configuration:

[0157] The system uses an Intel i9 processor (3.0 GHz), an NVIDIA RTX-3090 GPU, and 32 GB of memory, running in the Matlab-2022a environment. The detector uses YOLOv4-tiny with default parameters, and the correlation filter is implemented based on the ECO architecture.

[0158] Test method:

[0159] The algorithm is verified on five benchmark datasets: GTOT, RGBT234, LasHeR, VTUAV-ST and VTUAV-LT.

[0160] Set the model parameters:

[0161] The number of multi-features M=4 (VGG-M layer 3 / 14 + HOG + Color), where HOG is the gradient histogram feature and Color is the color feature;

[0162] Number of particles = 60, termination tolerance = 10 −6 , maximum number of iterations = 20;

[0163] The number of long-term recovery candidates K=5.

[0164] Evaluation Metrics:

[0165] Precision Rate (PR);

[0166] Success Rate (SR).

[0167] Performance evaluation index calculation:

[0168] Accuracy means the distance error between the center of the target bounding box and the center of the ground truth bounding box in each frame of tracking results. If the center distance error is less than 20 pixels, the tracking is considered correct.

[0169] Accuracy is calculated as the total number of frames with a center error less than 20 pixels divided by the total number of frames in the sequence.

[0170] Success rate means the overlap ratio between the target bounding box and the ground truth bounding box of each frame of tracking results. When the overlap ratio is greater than 0.5, the tracking is considered correct.

[0171] The success rate is calculated as the total number of frames with an overlap ratio greater than 0.5 divided by the total number of frames in the sequence.

[0172] For test results, please see the attached Figure 3 -Attached Figure 12 :

[0173] The following table summarizes the performance of the proposed algorithm on various benchmark datasets:

[0174]

[0175] Results Analysis: Based on the calculation method of accuracy and success rate, we can obtain the accuracy and success rate results of each tracker. The following is a detailed analysis of the test results on each dataset.

[0176] GTOT dataset, as attached Figure 3 and attached Figure 4 As shown. On the GTOT dataset, the precision rate (PR) of the MFDSR algorithm of the present invention is 0.911 and the success rate (SR) is 0.756. Compared with other methods, the performance of the method of the present application is the best on the GTOT dataset. This verifies the complementary fusion effect of deep features and manual features. The algorithms involved in the comparison include:

[0177] MFDSR: Multi-feature adaptive fusion and primary-auxiliary dynamic recovery target tracking algorithm;

[0178] DMCNet: Dual-gated mutual conditional target tracking algorithm;

[0179] HMFT: Hierarchical multimodal fusion target tracking algorithm;

[0180] JMMAC: Joint motion and appearance modeling for target tracking.

[0181] M5L: Multimodal Multi-Margin Metric Learning for Object Tracking.

[0182] MPT: target tracking algorithm that maximizes the peak-to-sidelobe ratio;

[0183] LSAR: A target tracking algorithm with online learning samples and adaptive recovery.

[0184] RGBT234 dataset, as attached Figure 5 and attached Figure 6 As shown in the figure, the MFDSR method achieved an accuracy of 0.837 and a success rate of 0.582 on the RGBT234 dataset, which is comparable to CAT++'s accuracy of 0.840 and success rate of 0.592 (with a small difference). In the field of object tracking, a performance difference of more than 1.0% is generally considered a significant performance gap. The algorithms involved in the comparison include:

[0185] CAT++: Challenging attribute-driven object tracking algorithm;

[0186] MFDSR: Multi-feature adaptive fusion and primary-auxiliary dynamic recovery target tracking algorithm;

[0187] M5L: Multimodal Multi-Margin Metric Learning for Object Tracking.

[0188] JTPMA: Multimodal multi-task feature fusion target tracking algorithm;

[0189] LSAR: a target tracking algorithm with online learning samples and adaptive recovery;

[0190] HMFT: Hierarchical multimodal fusion target tracking algorithm;

[0191] MPT: target tracking algorithm that maximizes the peak-to-sidelobe ratio;

[0192] DFAT: Adaptive decision-level fusion tracking algorithm.

[0193] LasHeR dataset, as attached Figure 7 and attached Figure 8 As shown. The PR / SR performance of MFDSR on the LasHeR dataset (0.475 / 0.395) significantly surpasses other state-of-the-art methods (compared with other methods, both the accuracy and success rate are higher than 1%). In particular, compared with the second-ranked CAT++ (a target tracking algorithm driven by challenging attributes), MFDSR achieved a performance advantage of 3.1% / 3.3% in PR / SR, respectively. In addition, the method of this application has achieved good performance in most challenges. It is particularly noteworthy that LasHeR contains a large number of long time series. In extremely challenging scenarios, the dynamic selection recovery mechanism can handle these situations involving long-term tracking well, with an SR of 0.395, which is significantly better than other methods. The algorithms involved in the comparison include:

[0194] MFDSR: Multi-feature adaptive fusion and primary-auxiliary dynamic recovery target tracking algorithm (this invention);

[0195] CAT++: Challenging attribute-driven object tracking algorithm;

[0196] APFNet: Attribute-based progressive fusion target tracking algorithm;

[0197] DMCNet: Dual-gated mutual conditional target tracking algorithm;

[0198] JTPMA: Multimodal multi-task feature fusion target tracking algorithm;

[0199] LSAR: a target tracking algorithm with online learning samples and adaptive recovery;

[0200] MPT: target tracking algorithm that maximizes the peak-to-sidelobe ratio;

[0201] DFAT: Adaptive decision-level fusion tracking algorithm.

[0202] VTUAV dataset (VTUAV-ST and VTUAV-LT), VTUAV-ST is attached Figure 9 and attached Figure 10 As shown, VTUAV-LT is as shown in the attached Figure 11 and attached Figure 12 shown.

[0203] MFDSR significantly outperforms other algorithms in PR / SR performance on both the VTUAV short-term (PR / SR: 0.773 / 0.626) and long-term (PR / SR: 0.530 / 0.453) tracking sub-datasets. Specifically:

[0204] In the VTUAV-ST (short-term) dataset, the success rate of MFDSR is basically the same as that of HMFT, but the accuracy is 1.5% higher than that of HMFT.

[0205] On the VTUAV-LT (Long-Term) dataset, MFDSR achieved a 9% higher success rate and 10.3% higher accuracy than LSAR. These results demonstrate the robustness of this approach in complex environments. Leveraging an adaptive fusion model and a dynamic selective recovery mechanism, MFDSR performs well on most short- and long-term tracking challenges.

[0206] The algorithms compared in the VTUAV-ST dataset include:

[0207] MFDSR: Multi-feature adaptive fusion and primary-auxiliary dynamic recovery target tracking algorithm (this invention);

[0208] HMFT: Hierarchical multimodal fusion target tracking algorithm;

[0209] MMMPT: Multimodal Mutual Information Transfer Target Tracking Algorithm;

[0210] LSAR: a target tracking algorithm with online learning samples and adaptive recovery;

[0211] mfDIMP: End-to-end fusion target tracking algorithm;

[0212] MPT: target tracking algorithm that maximizes the peak-to-sidelobe ratio;

[0213] ADRNet: An adaptive learning attribute-driven representation object tracking algorithm

[0214] DAFNet: Deep Adaptive Fusion Target Tracking Algorithm

[0215] The algorithms compared in the VTUAV-LT dataset include:

[0216] MFDSR: Multi-feature adaptive fusion and primary-auxiliary dynamic recovery target tracking algorithm (this invention);

[0217] HMFT: Hierarchical multimodal fusion target tracking algorithm;

[0218] LSAR: a target tracking algorithm with online learning samples and adaptive recovery;

[0219] mfDIMP: End-to-end fusion target tracking algorithm;

[0220] MPT: target tracking algorithm that maximizes the peak-to-sidelobe ratio;

[0221] ADRNet: Adaptive learning of attribute-driven representations for object tracking.

[0222] DAFNet: Deep Adaptive Fusion Object Tracking Algorithm;

[0223] FSRPN: A target tracking algorithm integrating Siamese network and attention mechanism.

[0224] Typical scenario verification:

[0225] In actual scenarios, we conducted comparative experiments on this algorithm with multiple traditional algorithms, and verified the effectiveness of this algorithm in actual use through tracking results in the following two typical scenarios. The specific tracking results are as follows:

[0226] Scenario 1:

[0227] Large-scale changes and occlusion scenes (corresponding to the attached Figure 13 The detailed comparison results are as follows: Figure 14 -Attached Figure 16 shown):

[0228] Large-scale changes refer to significant changes in the appearance of the tracked target, such as changes in size. Or when the shooting angle changes, the target's appearance will also change with the shooting angle.

[0229] Occlusion means that the target being tracked is blocked by other objects, resulting in the inability to find relevant information about the target in the image.

[0230] For the first scenario, the tracking target is the car in the video. Figure 14-16 The target is marked by a green box (the green box refers to the GT). This scenario is mainly for objects with large scale changes, beyond the field of view, severe occlusion, and poor thermal mode availability (thermal mode details are as follows Figure 14 -Attached Figure 16 The video sequence for the entire scene has 8107 frames. We extracted three typical frames from these 8107 frames for analysis, with the algorithm tracking markers explained below:

[0231] The results of ADRNet tracking are located in the attached Figure 14 -Attached Figure 16The pink box mark is shown in the figure, and the MPT tracking result position is shown in the figure. Figure 14 -Attached Figure 16 The orange box mark is shown in the figure, and the LSAR tracking result position is shown in the figure. Figure 14 -Attached Figure 16 The dark grey-green box mark is shown in the figure, and the result location of HMFT tracking is shown in the figure. Figure 14 -Attached Figure 16 The yellow box mark is shown in the figure, and the result location of MFDSR tracking is in the attached Figure 14 -Attached Figure 16 It is marked with a red box.

[0232] As attached Figure 14 As shown, when tracking to the 251st frame, the car has undergone serious deformation. The MFDSR algorithm completely marks the position of the car and can handle the appearance changes of the target very well. This is because our MFDSR algorithm has a built-in target detection algorithm, so it can accurately evaluate the size of the target. The green box is the real annotation box. Its main purpose is to mark the target we want to track, so that the tracking results of different algorithms can be directly compared with the standard results. Other algorithms have a constant height-to-width ratio of the target box output by their scale estimation module. For example, after scaling 1x2, it can only become a target box with a constant height-to-width ratio of 2x4 or 4x8. In actual tracking scenarios, the height and width of the target change in any proportion, so it cannot handle large-scale changes very well. When tracking to the 1751th frame, a short period of time before the 1751th frame, the car passed from behind the tree (the car was blocked by the tree during the passage). When the car appeared again, that is, the 1751st frame, the algorithm of this application is as shown in the attached Figure 15 As shown in the figure, the car to be tracked is accurately relocated. The other algorithms all failed to track. When tracking to the 5671st frame, the algorithm of this application is also very robust in tracking the target car, as shown in the figure. Figure 16 The red box position shown in the figure is the same as the position of the red box, while the other algorithms are invalid. Through the above comparison examples, it can be seen that the dynamic selection recovery mechanism of the present application can recapture the target and achieve continuous tracking.

[0233] Extreme lighting and complete occlusion scenes (corresponding to the attached Figure 13 The detailed comparison results are as follows: Figure 17 -Attached Figure 19 shown):

[0234] Extreme lighting and full occlusion scenes refer to low-light scenes plus occlusion scenes.

[0235] For the second scenario, the tracking target is a pedestrian in the video. Figure 14-16The target being tracked is marked by a green box (the green box represents the ground truth). This scenario primarily targets objects in low-light, occluded scenes, and where RGB modality availability is extremely poor. The video sequence for this scenario consists of 4987 frames. We have extracted three typical frames for analysis, with the algorithm's tracking markers explained below:

[0236] The results of ADRNet tracking are located in the attached Figure 17 -Attached Figure 19 The pink box mark is shown in the figure, and the MPT tracking result position is shown in the figure. Figure 17 -Attached Figure 19 The orange box mark is shown in the figure, and the LSAR tracking result position is shown in the figure. Figure 17 -Attached Figure 19 The dark grey-green box mark is shown in the figure, and the result location of HMFT tracking is shown in the figure. Figure 17 -Attached Figure 19 The yellow box mark is shown in the figure, and the result location of MFDSR tracking is in the attached Figure 17 -Attached Figure 19 It is marked with a red box.

[0237] As attached Figure 17 As shown in the figure, when tracking to the 451st frame, the tracked pedestrian is blocked by the tree, and when tracking to the 601st frame, the tracked pedestrian appears again (see the attached figure). Figure 18 As shown in the figure, all other algorithms except the MFDSR algorithm of this application fail to track the target, while the algorithm of this application tracks the target correctly. When tracking continues to frame 4851, the algorithm of this application is still tracking the target very accurately, while the other algorithms all fail and fail to accurately track the position of the target pedestrian. The above example shows that the adaptive fusion mechanism of the algorithm of this application can identify modal availability and optimize weight configuration, effectively maintaining tracking stability.

[0238] This application is attached Figure 13 -Attached Figure 19 The algorithms involved in the comparison include:

[0239] MFDSR: Multi-feature adaptive fusion and primary-auxiliary dynamic recovery target tracking algorithm;

[0240] HMFT: Hierarchical multimodal fusion target tracking algorithm;

[0241] LSAR: a target tracking algorithm with online learning samples and adaptive recovery;

[0242] MPT: target tracking algorithm that maximizes the peak-to-sidelobe ratio;

[0243] ADRNet: Adaptive learning of attribute-driven representations for object tracking.

[0244] GT: ground truth bounding box (label).

[0245] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. RGB-T target tracking method based on multi-feature adaptive fusion and primary and auxiliary dynamic restoration, characterized by: The following steps are involved: S1. Perform multi-level feature extraction on the input RGB image and thermal imaging image to obtain the feature set of each modality; S2. constructing a multi-feature response fusion model based on the feature set, and obtaining a fusion response graph and feature weight parameters through joint optimization calculation; The step S2 comprises: S21. Generate response maps for each of the M features of each modality, where M ≥ 2; S22, establishing an optimization function with the goal of maximizing the peak significance of the response graph and minimizing the response fluctuation; S23, using an optimization strategy combining particle swarm initialization and nonlinear programming to solve the feature weight parameters; The optimization function is: Constraints include: In RGB ∈[0,1] 1×M ;In TIR ∈[0,1] 1×M ; Among them, w RGB is the RGB modality feature weight vector, w TIR is the thermal imaging modal feature weight vector, is the multi-feature fusion response map, N is the response map size, is the average value of the fusion response map, I is an N×N all-1 matrix, is the i-th feature response map of RGB modality, is the i-th feature response diagram of the thermal imaging mode, M is the number of feature types, is the weight value of the i-th feature of the RGB modality, is the weight value of the i-th feature of the thermal imaging modality, represents the response value at row i and column j in the fusion response graph, To express the difference between the response value of each pixel in the fusion response map and the overall average response value, It means that by optimizing the weighted coefficients of RGB and TIR modalities, a set of optimal weights is found so that the fused response map can achieve the maximum discriminative ability in the current frame. is the maximum response value in the entire fusion response graph; S3, calculating the tracking reliability evaluation value of the current frame according to the feature weight parameter; S4. When the tracking reliability evaluation value meets a preset condition, performing short-term tracking optimization to determine the target position; when it does not meet the preset condition, performing long-term tracking recovery to recapture the target; S5. Update the tracking model parameters according to the final determined target position and output the tracking result.

2. The RGB-T target tracking method based on multi-feature adaptive fusion and primary-auxiliary dynamic restoration according to claim 1 is characterized in that: The tracking reliability evaluation value in step S3 is calculated by the following formula: in, and Calculated by HaarPSI algorithm, is the RGB modality template similarity, is the thermal imaging modality template similarity, ψ RGBT is the joint modal reliability.

3. The RGB-T target tracking method based on multi-feature adaptive fusion and primary-auxiliary dynamic restoration according to claim 1 is characterized in that: The short-term tracking optimization in step S4 includes: Generate a set of candidate regions associated with historical trajectories; Filter the best candidates through spatial constraints and appearance similarity calculation; Refine the object bounding box based on the filtering results.

4. The RGB-T target tracking method based on multi-feature adaptive fusion and primary-auxiliary dynamic restoration according to claim 3 is characterized in that: The screening of the best candidates adopts: in, is the overlap ratio, is the spatial distance, is the candidate box area, Φ i is the confidence of the candidate box, is the maximum distance, is the appearance similarity, is the reference target area.

5. The RGB-T target tracking method based on multi-feature adaptive fusion and primary-auxiliary dynamic restoration according to claim 1 is characterized in that: The long-term tracking recovery in step S4 includes: Generate initial candidate boxes through the YOLOv4-tiny detector; Select the top K = 5 high-confidence candidates for multimodal response verification; The candidate with the largest PSR response that exceeds the dynamic threshold is selected as the recovery target, where the dynamic threshold is the last valid PSR value before tracking failure.

Images