Trustedness-gated adaptive RGBT target tracking method, device and equipment and storage medium

Abstract

The application discloses a kind of based on credibility gate test time self-adaptive RGBT target tracking method, device, equipment and storage medium, first frame completes bimodal template initialization, saves the initial anchor point of predicted head batch normalization affine parameter;Frame by frame clip bimodal search area, whether the update of current frame BN statistics is controlled according to last frame credibility level gate;Quality estimation is carried out to bimodal and the difference of mode is calculated, and the update rate of statistics is adaptively configured;Response map and target prediction box are output after single forward inference;From response map, peak value, saliency and distribution entropy are extracted, credibility score is constructed and is divided into high, medium, low three grades grade;Only high credibility frame passes through self-supervision optimization target and executes single-step gradient update to prediction head BN affine parameter, the rest network parameters are frozen throughout, and anchor point is used Regular suppress parameter drift.The application realizes no label adaptive adaptation in test phase, and controllable inhibits error update cumulative drift.

Description

Technical Field

[0001] This invention belongs to the technical field of target tracking, specifically relating to an adaptive RGBT target tracking method, apparatus, device, and storage medium based on confidence-gated testing. Background Technology

[0002] RGBT target tracking utilizes complementary information from visible light and thermal infrared modalities to continuously locate targets in video sequences. In recent years, with the development of deep learning, especially the Transformer architecture, RGBT target tracking has gradually shifted from traditional manual feature extraction and simple weighted fusion to end-to-end modeling based on a template-search paradigm. These methods typically use a template region cropped from the first or historical frames, then crop the search region in the current frame, achieving target localization through feature encoding, cross-modal interaction, and matching decoding. While these methods have achieved good results on standard public datasets, most rely heavily on offline training to enhance generalization ability and lack proactive adaptation mechanisms to new distribution changes during the testing phase.

[0003] In real-world open environments, the distribution of test data often differs significantly from that of training data. For example, test videos may suffer from degradations such as noise, blurring, weather changes, and digital distortion. These degradations alter the statistical characteristics and discriminative capabilities of visible light and thermal infrared modes, causing mismatches in the feature representations, modality fusion methods, and classification / regression heads obtained through offline training during the testing phase. Direct consequences include reduced response peaks, jitter in target bounding boxes, localization shifts, template contamination, and consecutive frame drift.

[0004] To address the performance degradation caused by inconsistencies between the training and testing domains, test-time adaptation has gained increasing attention. The basic idea behind this approach is to adjust model parameters or normalized statistics online using only the observational information of the test samples themselves, without relying on manual annotation during the testing phase, to achieve rapid distribution compensation. However, most existing test-time adaptation methods are geared towards image classification or single-modal vision tasks, and directly transferring them to RGBT object tracking presents new challenges.

[0005] First, online updates in object tracking are highly dependent on the quality of pseudo-labels. During the testing phase, there are no human-generated ground truth values; the model's current output must be used as the basis for subsequent adaptation. Once the prediction for the current frame becomes unreliable, continuing to update model parameters or normalize statistics can easily infuse erroneous information into the model's internal state, accumulating over time and ultimately causing irreversible, catastrophic drift.

[0006] Secondly, the quality of the two modes in the RGBT scene is dynamic and asymmetrical. At certain times, the visible light mode is more reliable, while at other times the thermal infrared mode is more reliable, and different corrosion types and intensities will have different degrees of impact on the two modes. If a fixed update strategy, fixed statistical absorption rate, or fixed mode processing method is still used, the low-quality mode may continuously inject noise distribution into the model during the adaptation process of testing, due to the updating of dominant parameters or statistics.

[0007] Furthermore, existing adaptive methods often directly update large-scale network parameters or update multiple modules simultaneously, resulting in an excessively large adaptation space and a tendency to overfit the noise of the current single frame. For online tracking tasks, this not only incurs additional computational overhead but also significantly increases the risk of instability.

[0008] Therefore, a test-time adaptive method suitable for RGBT target tracking scenarios is needed. This method should first determine whether the prediction of the current frame is reliable without introducing manual annotation, and then selectively decide whether to update, where to update, and by how much. It should also combine bimodal quality differences to adjust the test-time adaptation process, thereby reducing the impact of domain offset while suppressing the cumulative drift caused by erroneous updates. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an adaptive RGBT target tracking method, device, equipment, and storage medium based on credibility gating during testing. This enables collaborative control of credibility perception, cross-frame gating, modal adaptation, and lightweight updates, thereby improving tracking robustness and real-time performance.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides an adaptive RGBT target tracking method based on confidence-gated testing, comprising the following steps:

[0012] Acquire a video sequence containing visible light RGB mode and thermal infrared TIR mode. In the first frame, complete the dual-modal template initialization based on the target initial bounding box, save the initial values ​​of the affine parameters of the batch normalization layer in the prediction head as anchor parameters, and initialize the confidence level of the previous frame to high confidence.

[0013] For frame t, the visible light search region and the thermal infrared search region are clipped based on the predicted bounding box of the previous frame. The current frame batch normalized statistics update is allowed based on the confidence level of the previous frame. If the previous frame is of low confidence, the statistics update is frozen; otherwise, the statistics update is allowed.

[0014] Modal quality estimation is performed on the visible light and thermal infrared search areas of the current frame, the modal quality difference is calculated, and the batch normalized statistics update rate is dynamically set accordingly. When statistics updates are prohibited, the update rate is set to zero.

[0015] Perform a single forward inference based on the current template and the bimodal search region, and output the response map, classification confidence information and target bounding box prediction results;

[0016] The response peak, peak prominence, and response distribution entropy statistics are extracted from the response map, and a confidence score is constructed by normalization or bounded mapping, and mapped to three confidence levels: high confidence, medium confidence, and low confidence.

[0017] Only when the current frame is of high confidence, a single-step gradient update is performed on the predicted first batch of normalized affine parameters based on the self-supervised optimization objective, and the remaining network parameters are kept frozen. Parameter updates are skipped for non-high confidence frames.

[0018] Save the current frame's confidence level and predicted bounding box, then proceed to the next frame loop until sequence tracking is complete.

[0019] As a preferred technical solution, gradient energy is used as a modal quality evaluation index, and the horizontal and vertical gradients are calculated for the visible light search region image and the thermal infrared search region image, respectively.

[0020] The sum of the first norms of the horizontal and vertical gradient features of the image is calculated using the first-order difference operator to obtain the single-mode gradient energy quality value.

[0021] By using the absolute difference between the visible light mode quality value and the thermal infrared mode quality value and the sum of the two values ​​to perform a ratio calculation, and introducing a minimum constant to prevent the denominator from being zero, a dual-mode quality difference quantity is constructed.

[0022] Based on the baseline update rate, a real-time statistical update rate is obtained by combining the linear correction of modal quality difference. The real-time statistical update rate is then constrained between a preset upper and lower bound to achieve adaptive adjustment of the update rate under bimodal asymmetric degradation.

[0023] As a preferred technical solution, before extracting the statistics of the response map, a two-dimensional Hanning window is used to perform element-wise multiplication with the original response map to complete the windowing filtering process;

[0024] In the windowed response graph, the global maximum response value is taken as the response peak.

[0025] The windowed response map is flattened into a one-dimensional vector, and the overall mean and standard deviation of the vector are calculated. The peak prominence is constructed by combining the response peaks to characterize the significance of the target main peak relative to the background response.

[0026] The windowed response map is summed along the horizontal and vertical dimensions to obtain two edge distribution vectors. The one-dimensional information entropy of the two edge distribution vectors is calculated and accumulated to form the response distribution entropy, which is used to characterize the degree of aggregation and dispersion of the target response.

[0027] As a preferred technical solution, the normalized response peak value and peak prominence are positively weighted, and the normalized response distribution entropy is negatively weighted, and the credibility score of the current frame is constructed by linear combination.

[0028] Two hierarchical thresholds, a fixed high threshold and a fixed low threshold, are pre-configured;

[0029] When the confidence score is greater than the high threshold, the current frame is determined to be of high confidence level;

[0030] When the confidence score is greater than the low threshold but not greater than the high threshold, the current frame is determined to be of medium confidence level.

[0031] When the confidence score is not greater than the low threshold, the current frame is determined to be of low confidence level;

[0032] Unsupervised confidence assessment can be completed solely by the tracker's own output response map, without the need for additional manual annotation or external supervision signals.

[0033] As a preferred technical solution, a cross-frame temporal gating mechanism is set up, and whether the batch normalized statistics of the current frame are updated is entirely determined by the confidence level of the previous frame.

[0034] If the previous frame is at a high or medium confidence level, then enable the current frame statistics update gating; if the previous frame is at a low confidence level, then disable the current frame statistics update gating.

[0035] When the gating is enabled, an exponential moving average iterative method is used to smoothly update the running mean and running variance of the batch normalization layer at the obtained adaptive update rate.

[0036] When the gating is closed, the current frame directly uses the batch normalized statistics that were fixed during the offline training phase of the model, without writing any test distribution information for the current frame, thus avoiding abnormal degenerate frames from polluting the internal statistical state of the model.

[0037] As a preferred technical solution, the self-supervised optimization objective consists of two parts:

[0038] The first term is the response distribution entropy constraint term, which minimizes the edge distribution entropy of the windowed response map, forcing the model to output a more concentrated and sharper target response peak, suppressing multi-peak interference and localization ambiguity;

[0039] The second term is the anchor point regularization constraint term, which calculates the L2 distance between the current batch normalized affine parameters and the anchor point parameters saved at the beginning of the test, and constrains the online parameter updates to not deviate from the parameter space of the initial pre-trained model.

[0040] The weighted fusion of the two constraints constitutes the self-supervised loss function during the final test. The optimization target can be constructed using only the forward inference result of the current frame, without the need for additional annotation and secondary forward inference.

[0041] As a preferred technical solution, the network parameters adopt a strategy of local lightweight updates and global full-process freezing.

[0042] Only the affine parameters consisting of the scaling factor γ and the offset factor β of the prediction head batch normalization layer are set as updatable parameters, while all other parameters, including those of the feature extraction backbone network, cross-modal interaction module, prediction head convolution and classification regression layer, are frozen throughout the process.

[0043] Only when the current frame is determined to be of high confidence level, a single-step stochastic gradient descent is used to iteratively update the BN affine parameters;

[0044] When the current frame is at a medium or low confidence level, the gradient backpropagation and parameter update process is skipped directly, thus eliminating the cumulative drift and template pollution caused by unreliable pseudo-labels.

[0045] Secondly, the present invention provides an adaptive RGBT target tracking device based on credibility gating during testing, which is applied to the aforementioned adaptive RGBT target tracking method based on credibility gating during testing. It includes an initialization module, a search pruning and statistical gating module, a modal quality estimation and rate control module, a forward inference module, a credibility modeling and level classification module, an adaptive parameter update module, and a loop iteration module.

[0046] The initialization module is used to acquire a video sequence containing visible light RGB mode and thermal infrared TIR mode, complete the dual-modal template initialization in the first frame according to the target initial bounding box, save the initial value of the affine parameter of the batch normalization layer in the prediction head as the anchor point parameter, and initialize the confidence level of the previous frame to high confidence.

[0047] The search cropping and statistical gating module is used to crop the visible light search region and the thermal infrared search region based on the predicted bounding box of the previous frame for the t-th frame, and to determine whether to allow the batch normalized statistics update of the current frame according to the confidence level of the previous frame. If the previous frame is of low confidence, the statistics update is frozen; otherwise, the statistics update is allowed.

[0048] The modal quality estimation and rate control module is used to perform modal quality estimation on the visible light and thermal infrared search areas of the current frame, calculate the modal quality difference, and dynamically set the batch normalized statistics update rate accordingly. When the statistics update is prohibited, the update rate is set to zero.

[0049] The forward inference module is used to perform a single forward inference based on the current template and the bimodal search region, and output the response map, classification confidence information and target bounding box prediction results.

[0050] The credibility modeling and classification module is used to extract the response peak, peak prominence and response distribution entropy statistics from the response map, construct credibility scores through normalization or bounded mapping, and map them into three credibility levels: high credibility, medium credibility and low credibility.

[0051] The adaptive parameter update module is used to perform a single-step gradient update on the predicted first batch of normalized affine parameters based on the self-supervised optimization objective only when the current frame is high confidence, while keeping the other network parameters frozen, and skipping parameter updates for non-high confidence frames.

[0052] The loop iteration module is used to save the current frame confidence level and predicted bounding box, and then enter the next frame loop until the sequence tracking is completed.

[0053] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0054] At least one processor; and,

[0055] A memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to execute the adaptive RGBT target tracking method based on confidence-gated testing.

[0057] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the aforementioned adaptive RGBT target tracking method based on confidence-gated testing.

[0058] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0059] (1) This invention extracts statistical signals such as peak value, salience and distribution entropy from the response map to construct low-overhead confidence scores and confidence levels, so that the model can obtain quantitative basis for "whether it is suitable to update" during the testing phase, avoiding blind online updates.

[0060] (2) The present invention restricts parameter updates to be triggered by high-confidence frames and controls statistical updates by the confidence state of the previous frame, so that abnormal frames are not likely to immediately pollute the internal state of the model, thereby effectively suppressing the chain propagation of erroneous pseudo-labels in the time dimension.

[0061] (3) This invention only updates the affine parameters of the batch normalization layer in the prediction head and directly completes the single-step test optimization based on the forward result of the current frame. It does not require full model backpropagation and multiple forward inferences, thus taking into account both adaptive benefits and online efficiency.

[0062] (4) The present invention adaptively adjusts the update rate of the statistical quantity based on the current quality difference between the visible light mode and the thermal infrared mode, so that the amplitude of the statistical absorption test distribution changes dynamically with the modal state during operation, thereby reducing the risk of low-quality modes injecting noise into the internal state of the model.

[0063] (5) The method of the present invention has achieved stable performance improvement on the RGBT corrosion benchmark, which includes multiple degradation conditions such as noise, blur, weather and digital distortion, and shows consistent gain on different base trackers, indicating that it has good versatility and engineering adaptability.

[0064] (6) The present invention adopts a single forward main path, local parameter update and lightweight gating design, which improves performance while maintaining good end-to-end operating efficiency, making it more suitable for online deployment.

[0065] The aforementioned beneficial effects are supported by the following technical solutions in specific implementations: credibility score From windowed response plot, peak intensity Peak sidelobe prominence and marginal entropy Composition; statistical updates are based on the confidence level of the previous frame. The generated switch Control, and update rate Write the mean and variance of the gated batch normalization layer; parameter updates only apply to the predicted first batch of normalized affine parameters. And based on the current frame's trust level Gating; Modal adaptive fusion module according to , and adjust The above formulaic implementation is shown in steps S3, S5, and S6 below. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart of an adaptive RGBT target tracking method based on confidence-gated testing according to an embodiment of the present invention;

[0068] Figure 2 This is a block diagram of an adaptive RGBT target tracking device based on confidence-gated testing according to an embodiment of the present invention.

[0069] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0070] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0071] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0072] Key term definitions:

[0073] RGBT: Refers to a dual-modal target tracking task that uses both visible light (RGB) and thermal infrared (TIR) ​​modes. The RGB mode provides texture, color, and edge information, while the TIR mode provides a more stable thermal response in low-light and nighttime scenes.

[0074] Test-time adaptation: refers to the online adjustment of the model during the testing phase without relying on manual annotation, using only the observation information of the test sequence itself, in order to alleviate the distribution shift between the test domain and the training domain.

[0075] Domain offset: refers to the difference between the distribution of training data and the distribution of test data. For example, changes in the statistical characteristics of the input caused by noise, fuzziness, weather disturbances, compression distortion, etc., can cause the performance of the model trained offline to degrade during testing.

[0076] The Confidence Modeling Module (CMM) is a module that extracts statistical quantities such as response peak, peak prominence, and distribution entropy from the prediction output of the current frame, and constructs the numerical confidence score and confidence level of the current frame to measure whether the prediction of the current frame is reliable.

[0077] Online Self-Adaptation Module (OSAM): This module performs statistical and parameter updates based on confidence level during the testing phase. It is used for lightweight adaptation on reliable frames and freezing or weakening updates on unreliable frames.

[0078] Modality-Adaptive Fusion Module (MAFM): This module dynamically adjusts the normalized statistical absorption rate or modal influence based on the current quality difference between the visible light mode and the thermal infrared mode, in order to reduce the interference of low-quality modes on the adaptation process during testing.

[0079] Gated batch normalization: This refers to adding gating and variable rate control to the update process of mean and variance during runtime, based on conventional batch normalization. This allows the updating of statistics and the rate at which they are updated to be dynamically determined based on confidence level or modal quality.

[0080] BN affine parameters: refer to the learnable scaling and offset parameters in the batch normalization layer, usually denoted as γ and β. In this invention, gradient updates are only allowed when testing these low-dimensional parameters, while the remaining network parameters remain frozen.

[0081] Anchor point regularization: refers to constraining the currently updatable parameters to the vicinity of the initial parameters saved before the test started during the update process, in order to suppress parameter drift caused by cumulative erroneous updates.

[0082] Response graph: refers to the target response distribution map output by the prediction head, used to represent the matching confidence level of the target at each position in the current search region. This invention utilizes response graph statistics to construct confidence scores and self-supervised optimization objectives.

[0083] This invention is based on the RGBT template-search-tracking framework. During the testing phase, it processes the visible light and thermal infrared dual-modal input sequences frame by frame in the form of a video stream. For the t-th frame, based on the prediction result of the previous frame, the search region is cropped in both the visible light image and the thermal infrared image, and then combined with the current template to complete forward inference, outputting the response map, classification confidence information, and target bounding box prediction result.

[0084] After obtaining the forward output of the current frame, this invention first constructs the current frame confidence score based on the statistical features of the response graph, and divides it into three confidence levels: high, medium, and low, to quantify the reliability of the current prediction result, and at the same time serve as the basis for online adaptive control in the subsequent testing phase.

[0085] Building upon this foundation, this invention implements a controllable design for the test-time update process, separating statistical updates from parameter updates and applying gating constraints based on cross-frame reliability information to prevent unreliable frames from contaminating the model's internal state. Simultaneously, by considering the real-time quality differences between the visible light mode and the thermal infrared mode, the test-time update process is adaptively adjusted to reduce the interference of low-quality modes on online adaptive stability.

[0086] To balance tracking stability and real-time performance, this invention performs lightweight updates on only a few key parameters while keeping the rest of the network parameters frozen. Furthermore, the update process is directly based on the forward result of the current frame, without the need for additional complex inference, thereby achieving stable online adaptation of the RGBT target tracking model in complex scenarios.

[0087] like Figure 1 As shown, an embodiment of the present invention provides an adaptive RGBT target tracking method based on confidence-gated testing, specifically including the following steps:

[0088] Step S1: Input sequence and initialization, specifically:

[0089] The system acquires the video sequence to be tracked, where each frame includes a visible light image and a thermal infrared image. In the first frame, an initial bounding box of the target is input, and the bimodal template region is cropped based on this bounding box to complete template initialization. Simultaneously, the initial values ​​of the batch normalization layer affine parameters in the prediction head are saved as anchor point parameters for subsequent testing; and the confidence level of the previous frame is initialized to high confidence.

[0090] Step S2: Search region cropping and current frame forward inference preparation, specifically:

[0091] For the Each frame crops the bimodal search region in both the visible light and thermal infrared images based on the predicted bounding box from the previous frame. Before performing forward inference for the current frame, the confidence level of the previous frame is used to determine whether batch normalized statistics updates are allowed. If the confidence level of the previous frame is not low confidence, statistics updates for the current frame are allowed; otherwise, the writing of statistics for the current frame is frozen.

[0092] Step S3: Setting the modal quality estimation and statistical update rate based on the modal adaptive fusion module, specifically:

[0093] This step corresponds to the Modality Adaptive Fusion Module (MAFM). It involves searching the visible light region of the current frame. and thermal infrared search area Rapid mass estimation is performed separately to obtain the mass values ​​of the two modes. and The modal quality difference of this frame is calculated based on the relative difference between the two. Then, based on Dynamically set the update rate of the batch normalized statistics for the current frame. If the update of statistics for this frame is prohibited, the effective statistics update rate will be set to zero.

[0094] Specifically, record The set of spatial indexes for the search region is used, and gradient energy is employed as a lightweight quality estimation metric. ,in and These represent the horizontal and vertical difference operators, respectively. The two-modal mass estimation results are as follows: , The relative difference between the two modes is defined as: ,in To prevent extremely small constants with a denominator of zero, Used to characterize the degree of bimodal degradation asymmetry in the current frame.

[0095] The update rate of statistics is set as follows: ,in Based on the update rate, This is the modal difference adjustment coefficient. and These are the lower and upper bounds of the rate, respectively. The switch is updated in conjunction with the statistics from step S2. The final effective update rate is ;when When the statistics are frozen, When using It participates in subsequent gating batch normalized statistics updates. It should be noted that MAFM does not change the explicit bimodal feature fusion structure in the forward inference of the base tracker. Instead, it achieves indirect adaptive control at the modality level by adjusting the normalized statistics write rate in the test-time adaptation, thereby reducing the interference of low-quality modalities on the distribution alignment process during test.

[0096] Step S4: Perform a single forward inference for the current frame, specifically:

[0097] After setting the gating state and statistical update rate, a regular forward inference is performed using the current template and the bimodal search region to obtain the current frame response map, classification output, and target bounding box prediction results. In terms of specific network structure, the basic tracker includes a bimodal template—search input, a feature extraction backbone network, a cross-modal interaction or encoding module, and a prediction head. The first frame template region is denoted as... , , No. The frame search region is denoted as , Template features are obtained after backbone network and cross-modal coding. and search features Predictor head output response graph Classification confidence information and target bounding box prediction results The adaptive testing mechanism of this invention only applies to the gated batch normalization layer in the prediction head, without changing the main structure of the backbone network and cross-modal interaction module.

[0098] Step S5: Credibility modeling, specifically:

[0099] Based on the current frame response map, extract at least one or more of the following statistics: response peak value; peak prominence formed by the response peak value, the overall mean, and the standard deviation; and response map edge distribution entropy. Then, normalize or boundedly map the statistics to construct a numerical confidence score, and map the score to three levels—high confidence, medium confidence, and low confidence—according to a preset threshold.

[0100] Specifically, let the response graph output by the prediction head in frame t be... First, a two-dimensional Hanning window is used. Perform windowed processing: ,in This indicates element-wise multiplication. Windowing is used to ensure that the confidence calculation uses a consistent response distribution with the target localization process.

[0101] The peak intensity of the windowed response plot is defined as: .Will Flattened into a vector And let their mean and standard deviation be denoted as follows: and The peak-to-side-lobe salience ratio is defined as: .

[0102] To characterize the central tendency of the response distribution, The edge vector is obtained by summing along the horizontal and vertical directions respectively. and Marginal entropy is defined as: ,in , When the response distribution is more concentrated, The smaller the value; when the response distribution is more dispersed, The larger.

[0103] Bounded normalization is performed on the above statistics: , , $, where As a scale factor, This is the upper bound of the normalized marginal entropy.

[0104] The final confidence score is obtained by a linear combination of three types of normalized statistics: .in , and These represent the weights corresponding to peak intensity, peak prominence, and edge entropy, respectively. The credibility level is mapped as follows: when... hour, ;when hour, ;when hour, .

[0105] Furthermore, credibility modeling primarily addresses the problem of "the inability to determine the reliability of the current frame prediction during the testing phase." Its core idea is to extract statistics reflecting localization stability solely from the tracker's own output response map for the current frame, without introducing additional manual annotations or external supervision signals, and to construct a unified numerical credibility score.

[0106] In the specific implementation, the current frame response map is first windowed to ensure that the confidence calculation and target localization process use the same response distribution; then, three types of statistical signals are extracted from the windowed response map, specifically:

[0107] The first type of statistical signal is the response peak. The higher the response peak, the more concentrated the model's matching of a certain position is in the current search area, and the more likely the prediction result is to be reliable.

[0108] The second type of statistical signal is peak prominence, which reflects the significance of the target peak relative to the background response by combining the peak value of the response with the mean and standard deviation of the overall response distribution. The greater the peak prominence, the more distinct the main response peak, and the easier it is to distinguish the target from the background.

[0109] The third type of statistical signal is the response distribution entropy, which is calculated by normalizing the edge distribution of the response map along different directions. It is used to reflect whether the response is concentrated. The lower the entropy value, the more concentrated the response and the more stable the predicted location; the higher the entropy value, the more dispersed the response, and there may be risks of occlusion, noise, or drift.

[0110] Since the statistical values ​​mentioned above have different ranges, this invention further performs bounded mapping or normalization on each statistical value, and constructs a final credibility score using a weighted combination method. Then, this credibility score is compared with preset high and low thresholds to obtain the credibility level of the current frame.

[0111] Trustworthiness levels are divided into three tiers:

[0112] High confidence: This indicates that the prediction of the current frame is relatively stable, allowing for parameter updates during testing;

[0113] Medium reliability: This indicates that the prediction for the current frame is basically usable, but it is not suitable for performing gradient updates;

[0114] Low confidence: This indicates that the current frame may have significant drift or severe degradation, requiring freezing or weakening of the update process.

[0115] Through the above design, this module can provide quantifiable, low-overhead, and reliable evidence directly related to the tracking output for update decisions during the testing phase.

[0116] Step S6: Update the parameters of the credibility gating, specifically:

[0117] If the current frame has a high confidence level, a test-time self-supervised optimization objective is constructed based on the first forward result of the current frame, and only one gradient update is performed on the affine parameters of the batch normalization layer in the prediction header. If the current frame does not have a high confidence level, the parameter update is skipped. The test-time self-supervised optimization objective includes at least: an objective term for reducing the uncertainty of the response distribution; and a regularization term for constraining the current affine parameters to be close to the initial anchor parameters.

[0118] Furthermore, the reliability-gated parameter update primarily addresses the issue of "how to stably execute online updates during the testing phase." Its core idea is to divide test-time updates into two categories: statistical updates and parameter updates, and apply different gating conditions to each, thereby reducing the cumulative propagation of erroneous updates over time. Specifically, this includes the following:

[0119] Step S61: Update statistics;

[0120] The statistics update applies to the runtime mean and variance of the gated batch normalization layer in the prediction head. Unlike conventional testing where training statistics are used consistently, this invention allows the model to gradually incorporate the current test distribution into the runtime statistics on a subset of test frames, enabling the normalization layer to compensate for the test environment online.

[0121] remember For the statistics update switch, then ,in For indicator functions, This represents the confidence level of the previous frame. If the previous frame was of low confidence, then... If the previous frame was of high or medium confidence, then .

[0122] For any gated batch normalization layer in the prediction head, let the current input feature be... The current frame batch statistics are and The runtime mean and variance are written using exponential moving averages: , .when When, the above equation degenerates into , This means that no new statistics are written to this frame.

[0123] However, this invention does not allow the statistics to be updated in any frame. Instead, it adopts a cross-frame gating method: whether the statistics are allowed to be updated in frame t is not directly determined by frame t itself, but by the confidence level of frame (t-1).

[0124] If the previous frame is of low confidence, it means that the model has just experienced a potentially unstable prediction. In this case, freezing the update of statistics in the current frame can prevent abnormal distributions from being written into the runtime statistics immediately. If the previous frame is of high or medium confidence, the statistics in the current frame are allowed to be updated at a certain rate.

[0125] The statistics are updated using an exponential moving average, writing the current batch statistics into the historical runtime statistics. Compared to a complete replacement, this method is smoother and helps reduce drastic disturbances caused by single-frame noise.

[0126] Step S62: Parameter update;

[0127] Parameter updates only apply to the affine parameters of the batch normalized layers in the prediction head, namely the scaling and offset parameters. All parameters in the backbone network, modal interaction module, and prediction head are frozen.

[0128] Specifically, the set of updatable parameters is denoted as ,in Scaling parameters for batch normalization layers. This is the offset parameter. Only allowed during the testing phase. While participating in gradient updates, the feature extraction backbone network, cross-modal interaction module, and other parameters in the prediction head are all kept frozen.

[0129] The reason for this is that batch-normalized affine parameters have low dimensionality and clear positions of action. Updating them can achieve small corrections to the channel-level feature distribution, while not significantly destroying the overall representation space obtained from source domain training as a whole-network update.

[0130] Parameter updates are triggered only when the current frame's confidence level is high. In other words, the model is only allowed to use the output of the current frame to correct itself if the prediction of the current frame is sufficiently reliable; if the current frame is of medium or low confidence, parameter updates are skipped.

[0131] The parameter update adopts a single-step gradient descent method and constructs the optimization target directly based on the first forward result of the current frame, without performing secondary forward inference, thereby reducing additional computational overhead.

[0132] Step S63: Optimize the target during testing;

[0133] To ensure that parameter updates both compensate for domain offset and do not deviate excessively from the initial model of the source domain, the test-time optimization objective designed in this invention consists of two parts:

[0134] In this embodiment, the optimization objective during testing is defined as follows: ,in This refers to the uncertainty term in the response distribution constructed from the marginal distribution of the response map in step S5. Anchor points for BN affine parameters saved before the test begins. This is the anchoring coefficient.

[0135] A single-step gradient update is performed only when the current frame's confidence level is high confidence. The update formula is as follows: ,in Let be the learning rate. If... If it is of medium or low confidence, then This skips parameter updates. The update directly reuses the existing forward results from the current frame to construct the optimization objective, without performing secondary forward inference or multi-step iterations. One aspect is the response distribution uncertainty term. This term can use response map margin entropy or other functions that reflect the degree of response dispersion to encourage the model to output a more concentrated and clear response distribution.

[0136] The second is the anchor point regularization term. This term constrains the currently updatable affine parameters to be close to the initial affine parameters saved before the test started, avoiding cumulative drift caused by continuous erroneous updates during the test.

[0137] By combining the design of "updating only high-confidence frames, updating only low-dimensional affine parameters, and anchor point constraints", it is possible to stably absorb environmental change information during the testing phase while suppressing catastrophic drift.

[0138] Step S7: Save the current frame's trusted state and proceed to the next frame, specifically:

[0139] The confidence level of the current frame is saved and used as the gating basis for statistical updates in the next frame. At the same time, the predicted bounding box of the current frame is used as the reference box for cropping the search region in the next frame, and the process continues until the target tracking of the entire sequence is completed.

[0140] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0141] Based on the same idea as the confidence-gated adaptive RGBT target tracking method during testing in the above embodiments, the present invention also provides a confidence-gated adaptive RGBT target tracking device during testing. This device can be used to execute the aforementioned confidence-gated adaptive RGBT target tracking method during testing. For ease of explanation, the structural schematic diagram of the embodiment of the confidence-gated adaptive RGBT target tracking device during testing only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0142] Please see Figure 2 In another embodiment of this application, an adaptive RGBT target tracking device 100 based on credibility gating test is provided. The device includes an initialization module 101, a search pruning and statistical gating module 102, a modal quality estimation and rate control module 103, a forward inference module 104, a credibility modeling and level classification module 105, an adaptive parameter update module 106, and a loop iteration module 107.

[0143] The initialization module 101 is used to acquire a video sequence containing visible light RGB mode and thermal infrared TIR mode, complete the dual-mode template initialization in the first frame according to the target initial bounding box, save the initial value of the affine parameter of the batch normalization layer in the prediction head as the anchor point parameter, and initialize the confidence level of the previous frame to high confidence.

[0144] The search cropping and statistical gating module 102 is used to crop the visible light search region and the thermal infrared search region based on the predicted bounding box of the previous frame for the t-th frame, and to determine whether to allow the batch normalized statistics update of the current frame according to the confidence level of the previous frame. If the previous frame is of low confidence, the statistics update is frozen; otherwise, the statistics update is allowed.

[0145] The modal quality estimation and rate control module 103 is used to perform modal quality estimation on the visible light and thermal infrared search areas of the current frame, calculate the modal quality difference, and dynamically set the batch normalized statistics update rate accordingly. When the statistics update is prohibited, the update rate is set to zero.

[0146] The forward inference module 104 is used to perform a single forward inference based on the current template and the bimodal search region, and output a response map, classification confidence information and target bounding box prediction results.

[0147] The credibility modeling and classification module 105 is used to extract the response peak, peak prominence and response distribution entropy statistics from the response map, construct credibility scores through normalization or bounded mapping, and map them into three credibility levels: high credibility, medium credibility and low credibility.

[0148] The adaptive parameter update module 106 is used to perform a single-step gradient update on the predicted first batch of normalized affine parameters based on the self-supervised optimization objective only when the current frame is high confidence, while keeping the other network parameters frozen, and skipping parameter updates for non-high confidence frames.

[0149] The loop iteration module 107 is used to save the current frame confidence level and the predicted bounding box, and enter the next frame loop until the sequence tracking is completed.

[0150] It should be noted that the RGBT target tracking device based on confidence-gated testing and adaptive during testing of the present invention corresponds one-to-one with the RGBT target tracking method based on confidence-gated testing of the present invention. The technical features and beneficial effects described in the embodiments of the RGBT target tracking method based on confidence-gated testing and adaptive during testing of the above invention are applicable to the embodiments of RGBT target tracking based on confidence-gated testing and adaptive during testing of the present invention. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0151] Furthermore, in the embodiments of the RGBT target tracking device based on confidence-gated testing that are adaptive during the above-described embodiments, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the RGBT target tracking device based on confidence-gated testing that is adaptive during the above-described embodiments can be divided into different program modules to complete all or part of the functions described above.

[0152] Please see Figure 3 In one embodiment, an electronic device is provided that implements an adaptive RGBT target tracking method based on confidence-gated testing. The electronic device 200 may include a first processor 201, a first memory 202, and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an adaptive RGBT target tracking program 203 based on confidence-gated testing.

[0153] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the adaptive RGBT target tracking program 203 during trust-based gating testing, but also to temporarily store data that has been output or will be output.

[0154] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0155] Figure 3 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 3 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0156] The first memory 202 in the electronic device 200 stores an adaptive RGBT target tracking program 203 based on confidence-gated testing, which is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0157] Acquire a video sequence containing visible light RGB mode and thermal infrared TIR mode. In the first frame, complete the dual-modal template initialization based on the target initial bounding box, save the initial values ​​of the affine parameters of the batch normalization layer in the prediction head as anchor parameters, and initialize the confidence level of the previous frame to high confidence.

[0158] For frame t, the visible light search region and the thermal infrared search region are clipped based on the predicted bounding box of the previous frame. The current frame batch normalized statistics update is allowed based on the confidence level of the previous frame. If the previous frame is of low confidence, the statistics update is frozen; otherwise, the statistics update is allowed.

[0159] Modal quality estimation is performed on the visible light and thermal infrared search areas of the current frame, the modal quality difference is calculated, and the batch normalized statistics update rate is dynamically set accordingly. When statistics updates are prohibited, the update rate is set to zero.

[0160] Perform a single forward inference based on the current template and the bimodal search region, and output the response map, classification confidence information and target bounding box prediction results;

[0161] The response peak, peak prominence, and response distribution entropy statistics are extracted from the response map, and a confidence score is constructed by normalization or bounded mapping, and mapped to three confidence levels: high confidence, medium confidence, and low confidence.

[0162] Only when the current frame is of high confidence, a single-step gradient update is performed on the predicted first batch of normalized affine parameters based on the self-supervised optimization objective, and the remaining network parameters are kept frozen. Parameter updates are skipped for non-high confidence frames.

[0163] Save the current frame's confidence level and predicted bounding box, then proceed to the next frame loop until sequence tracking is complete.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An adaptive RGBT target tracking method based on confidence-gated testing, characterized in that, Includes the following steps: Acquire a video sequence containing visible light RGB mode and thermal infrared TIR mode. In the first frame, complete the dual-modal template initialization based on the target initial bounding box, save the initial values ​​of the affine parameters of the batch normalization layer in the prediction head as anchor parameters, and initialize the confidence level of the previous frame to high confidence. For frame t, the visible light search region and the thermal infrared search region are clipped based on the predicted bounding box of the previous frame. The current frame batch normalized statistics update is allowed based on the confidence level of the previous frame. If the previous frame is of low confidence, the statistics update is frozen; otherwise, the statistics update is allowed. Modal quality estimation is performed on the visible light and thermal infrared search areas of the current frame, the modal quality difference is calculated, and the batch normalized statistics update rate is dynamically set accordingly. When statistics updates are prohibited, the update rate is set to zero. Perform a single forward inference based on the current template and the bimodal search region, and output the response map, classification confidence information and target bounding box prediction results; The response peak, peak prominence, and response distribution entropy statistics are extracted from the response map, and a confidence score is constructed by normalization or bounded mapping, and mapped to three confidence levels: high confidence, medium confidence, and low confidence. Only when the current frame is of high confidence, a single-step gradient update is performed on the predicted first batch of normalized affine parameters based on the self-supervised optimization objective, and the remaining network parameters are kept frozen. Parameter updates are skipped for non-high confidence frames. Save the current frame's confidence level and predicted bounding box, then proceed to the next frame loop until sequence tracking is complete.

2. The adaptive RGBT target tracking method based on confidence-gated testing according to claim 1, characterized in that, Gradient energy is used as a modal quality evaluation index, and the horizontal and vertical gradients are calculated for the visible light search region image and the thermal infrared search region image, respectively. The sum of the first norms of the horizontal and vertical gradient features of the image is calculated using the first-order difference operator to obtain the single-mode gradient energy quality value. By using the absolute difference between the visible light mode quality value and the thermal infrared mode quality value and the sum of the two values ​​to perform a ratio calculation, and introducing a minimum constant to prevent the denominator from being zero, a dual-mode quality difference quantity is constructed. Based on the baseline update rate, a real-time statistical update rate is obtained by combining the linear correction of modal quality difference. The real-time statistical update rate is then constrained between a preset upper and lower bound to achieve adaptive adjustment of the update rate under bimodal asymmetric degradation.

3. The adaptive RGBT target tracking method based on confidence-gated testing according to claim 1, characterized in that, Before extracting the statistics from the response map, a two-dimensional Hanning window is used to perform element-wise multiplication with the original response map to complete the windowing filtering process. In the windowed response graph, the global maximum response value is taken as the response peak. The windowed response map is flattened into a one-dimensional vector, and the overall mean and standard deviation of the vector are calculated. The peak prominence is constructed by combining the response peaks to characterize the significance of the target main peak relative to the background response. The windowed response map is summed along the horizontal and vertical dimensions to obtain two edge distribution vectors. The one-dimensional information entropy of the two edge distribution vectors is calculated and accumulated to form the response distribution entropy, which is used to characterize the degree of aggregation and dispersion of the target response.

4. The adaptive RGBT target tracking method based on confidence-gated testing according to claim 1, characterized in that, The credibility score of the current frame is constructed by positively weighting the normalized response peak value and peak prominence, and negatively weighting the normalized response distribution entropy. Two hierarchical thresholds, a fixed high threshold and a fixed low threshold, are pre-configured. When the confidence score is greater than the high threshold, the current frame is determined to be of high confidence level; When the confidence score is greater than the low threshold but not greater than the high threshold, the current frame is determined to be of medium confidence level. When the confidence score is not greater than the low threshold, the current frame is determined to be of low confidence level; Unsupervised confidence assessment can be completed solely by the tracker's own output response map, without the need for additional manual annotation or external supervision signals.

5. The adaptive RGBT target tracking method based on confidence-gated testing according to claim 1, characterized in that, A cross-frame timing gating mechanism is set up, and whether the batch normalized statistics of the current frame are updated is entirely determined by the confidence level of the previous frame. If the previous frame is at a high or medium confidence level, then enable the current frame statistics update gating; if the previous frame is at a low confidence level, then disable the current frame statistics update gating. When the gating is enabled, an exponential moving average iterative method is used to smoothly update the running mean and running variance of the batch normalization layer at the obtained adaptive update rate. When the gating is closed, the current frame directly uses the batch normalized statistics that were fixed during the offline training phase of the model, without writing any test distribution information for the current frame, thus avoiding abnormal degenerate frames from polluting the internal statistical state of the model.

6. The adaptive RGBT target tracking method based on confidence-gated testing according to claim 1, characterized in that, The self-supervised optimization objective consists of two parts: The first term is the response distribution entropy constraint term, which minimizes the edge distribution entropy of the windowed response map, forcing the model to output a more concentrated and sharper target response peak, suppressing multi-peak interference and localization ambiguity; The second term is the anchor point regularization constraint term, which calculates the L2 distance between the current batch normalized affine parameters and the anchor point parameters saved at the beginning of the test, and constrains the online parameter updates to not deviate from the parameter space of the initial pre-trained model. The weighted fusion of the two constraints constitutes the self-supervised loss function during the final test. The optimization target can be constructed using only the forward inference result of the current frame, without the need for additional annotation and secondary forward inference.

7. The adaptive RGBT target tracking method based on confidence-gated testing according to claim 1, characterized in that, Network parameters employ a strategy of local lightweight updates and global full-process freezing. Only the affine parameters consisting of the scaling factor γ and the offset factor β of the prediction head batch normalization layer are set as updatable parameters, while all other parameters, including those of the feature extraction backbone network, cross-modal interaction module, prediction head convolution and classification regression layer, are frozen throughout the process. Only when the current frame is determined to be of high confidence level, a single-step stochastic gradient descent is used to iteratively update the BN affine parameters; When the current frame is at a medium or low confidence level, the gradient backpropagation and parameter update process is skipped directly, thus eliminating the cumulative drift and template pollution caused by unreliable pseudo-labels.

8. An adaptive RGBT target tracking device based on confidence-gated testing, characterized in that, The adaptive RGBT target tracking method based on credibility gating test applied to any one of claims 1-7 includes an initialization module, a search pruning and statistical gating module, a modal quality estimation and rate control module, a forward inference module, a credibility modeling and level classification module, an adaptive parameter update module, and a loop iteration module. The initialization module is used to acquire a video sequence containing visible light RGB mode and thermal infrared TIR mode, complete the dual-modal template initialization in the first frame according to the target initial bounding box, save the initial value of the affine parameter of the batch normalization layer in the prediction head as the anchor point parameter, and initialize the confidence level of the previous frame to high confidence. The search cropping and statistical gating module is used to crop the visible light search region and the thermal infrared search region based on the predicted bounding box of the previous frame for the t-th frame, and to determine whether to allow the batch normalized statistics update of the current frame according to the confidence level of the previous frame. If the previous frame is of low confidence, the statistics update is frozen; otherwise, the statistics update is allowed. The modal quality estimation and rate control module is used to perform modal quality estimation on the visible light and thermal infrared search areas of the current frame, calculate the modal quality difference, and dynamically set the batch normalized statistics update rate accordingly. When the statistics update is prohibited, the update rate is set to zero. The forward inference module is used to perform a single forward inference based on the current template and the bimodal search region, and output the response map, classification confidence information and target bounding box prediction results. The credibility modeling and classification module is used to extract the response peak, peak prominence and response distribution entropy statistics from the response map, construct credibility scores through normalization or bounded mapping, and map them into three credibility levels: high credibility, medium credibility and low credibility. The adaptive parameter update module is used to perform a single-step gradient update on the predicted first batch of normalized affine parameters based on the self-supervised optimization objective only when the current frame is high confidence, while keeping the other network parameters frozen, and skipping parameter updates for non-high confidence frames. The loop iteration module is used to save the current frame confidence level and predicted bounding box, and then enter the next frame loop until the sequence tracking is completed.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the RGBT target tracking method based on confidence-gated testing as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the adaptive RGBT target tracking method based on confidence-gated testing as described in any one of claims 1-7.

Images