Adaptive rgbt target tracking method and device based on quality perception and space-frequency fusion, equipment and storage medium

Abstract

The application discloses a kind of based on quality perception and space-frequency fusion adaptive RGBT target tracking method, device, equipment and storage medium, comprising: first complete first frame initialization and template memory set construction, frame-by-frame cutting search area and extracting double-mode Token feature;Through dynamic quality modeling and re-calibration module, from modal level and Token level double-layer evaluation feature reliability;Parallelly construct spatial domain and frequency domain double-expert branch, space domain retains local texture edge, frequency domain extracts frequency statistical characteristics, and adaptive fusion space-frequency representation is realized by gated routing;Then, in combination with first frame template and historical template memory, target matching positioning is completed, candidate template quality is evaluated in combination with prediction confidence and modal quality score, and quality gating strategy is used to realize safe update of fixed capacity template memory set.The application can adaptively perceive the dynamic change of double-mode quality, effectively suppress template drift and pollution, and significantly improve the accuracy and stability of long-time RGBT target tracking.

Description

Technical Field

[0001] This invention belongs to the technical field of target tracking, specifically relating to an adaptive RGBT target tracking method, device, equipment, and storage medium based on quality perception and space-frequency fusion. Background Technology

[0002] RGBT bimodal target tracking aims to continuously estimate target position in complex environments by utilizing complementary information from visible light and thermal infrared images. Compared to single-modal tracking, the RGB mode provides richer texture, color, and boundary information, while the TIR mode exhibits a more stable thermal response under low light, nighttime, shadow occlusion, and complex background conditions. Therefore, RGBT tracking has broad application prospects in fields such as nighttime surveillance, unmanned system perception, security early warning, autonomous driving assistance, and complex scene search.

[0003] Existing RGBT tracking methods have generally evolved from traditional feature fusion to deep network fusion, and then to Transformer-based end-to-end modeling. Early methods often achieved bimodal fusion through manually designed features, saliency modeling, or simple weighting strategies, but their adaptability to complex interference scenarios was limited. Subsequently, convolutional neural networks gradually became the mainstream framework for RGBT tracking, and mechanisms such as channel attention, spatial attention, and modal attention were introduced to achieve more effective multimodal interactions at the feature layer, decision layer, or template layer, thereby improving tracking accuracy and robustness. In recent years, Transformer-based end-to-end tracking frameworks have further emerged. With self-attention mechanisms at their core, they demonstrate greater potential in target localization and cross-modal fusion through unified token representation of templates and search regions and global dependency modeling.

[0004] However, existing RGBT tracking technology still faces several common difficulties in real-world complex scenarios.

[0005] On the one hand, the effectiveness of RGB and TIR modalities is not fixed but dynamically changes with time and scene. For example, in low-light scenes, the texture and edges of the RGB modal may degrade significantly, while the TIR modal is more reliable; in scenes with thermal crossover or small background temperature differences, the TIR modal may lose its discriminative power, while the RGB modal provides more stable target cues. In existing methods, many fusion strategies still use fixed fusion ratios, single attention mechanisms, or weakly constrained feature overlays, making it difficult to accurately identify the dominant modality and low-quality modalities at the current moment. This easily introduces noise responses, erroneous edges, or invalid thermal information into the final representation, causing matching offsets and localization drift.

[0006] On the other hand, complexity degradation, in particular, can damage the texture statistics and local edge information in the spatial domain. For example, noise amplifies random perturbations, blurring weakens edge sharpness, and compression distortion introduces blocky artifacts, all of which can cause deviations in the local details extracted by the tracker in the spatial domain. If matching is based solely on spatial domain features, attention shifts, target boundary jitter, and even tracking failures can easily occur. While some existing methods attempt to enhance representation robustness, they typically do not explicitly incorporate frequency domain statistics and lack mechanisms to dynamically adjust the spatial and frequency domain contributions based on the current scene content.

[0007] Furthermore, in long-term tracking scenarios, the appearance of the target changes gradually over time, such as pose changes, scale changes, partial occlusion recovery, and viewpoint switching. Relying solely on the first frame template is insufficient to maintain matching capability over the long term. Therefore, many methods have introduced online template update mechanisms. However, existing update strategies are often either too aggressive or too conservative: the former easily writes incorrectly predicted appearances into the template, leading to template contamination and cumulative drift; the latter struggles to adapt to changes in the target's actual appearance, causing mismatches in later stages. Especially in RGBT scenarios, without a reliable basis for writing based on dual-modal quality information, template updates can easily be mistakenly triggered by low-quality frames.

[0008] Therefore, how to dynamically perceive modal quality during RGBT tracking, improve the robustness of representation under complex degradation conditions, and achieve safe and effective template updates during long-term tracking remains an important problem to be solved by existing technologies. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an adaptive RGBT target tracking method, device, equipment and storage medium based on quality perception and space-frequency fusion, so as to realize dual-modal explicit quality perception, adaptive fusion of space-frequency features and safe and controllable template updates, thereby improving the stability and accuracy of tracking in complex scenes and long-term tracking.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides an adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion, comprising the following steps:

[0012] S1. Obtain the RGB image sequence and TIR image sequence, and input the target initial bounding box in the first frame. Based on the initial bounding box, crop the template region of the first frame in the RGB mode and TIR mode respectively, extract the first frame bimodal template features, and initialize the template memory set with the first frame bimodal template features.

[0013] S2. For frame t, the search region is cropped in the RGB image and TIR image respectively based on the predicted bounding box of the previous frame, and the RGB features and TIR features of the search region in the current frame are extracted by the patch embedding and encoding network.

[0014] S3. Utilize the intramodal correlation between the current frame template features and the search features to calculate the evidence strength and uncertainty information for each modality, and perform soft recalibration on the search features based on the uncertainty information estimation results; at the same time, further calculate the per-token quality score for the search tokens to suppress low-quality tokens, thereby obtaining the quality-calibrated RGB modal search features and TIR modal search features.

[0015] S4. Input the quality-calibrated bimodal search features into the spatial domain expert branch and the frequency domain expert branch respectively. The spatial domain expert branch performs spatial-level fusion of the bimodal features, preserving local structure, edge, and texture information. The frequency domain expert branch performs frequency domain transformation on the bimodal spatial features, extracts frequency statistics of predefined frequency bands, converts the frequency statistics into frequency domain enhancement biases, and injects them into the features in the form of residuals to obtain frequency domain enhancement features. The gated routing network generates dynamic weights for the spatial domain expert branch and the frequency domain expert branch based on the global statistics of the current input features, and performs weighted fusion of the outputs of the two branches to obtain the fused representation of the search region.

[0016] S5. Construct the current template representation based on the template features of the first frame and the template memory set. Input the current template representation and the search region fusion representation into the matching decoder and prediction head, and output the target existence confidence and target prediction bounding box of the current frame.

[0017] S6. Based on the predicted bounding box of the current frame, crop the candidate template region in the RGB image and TIR image and extract the candidate template features. Combine the target existence confidence and bimodal quality information of the current frame to calculate the template update quality score of the current frame.

[0018] S7. Determine whether the quality score of the updated template is higher than the preset threshold. If it is higher, write the candidate template into the template memory set. If the template memory set has not reached the capacity limit, insert a new template directly. If it is full, replace the template with the lowest quality score in the set with the current candidate template. If it is not higher than the threshold, keep the template memory set unchanged.

[0019] S8. Use the predicted bounding box of the current frame as the basis for cropping the search region of the next frame until the target tracking of the entire image sequence is completed.

[0020] As a preferred technical solution, step S3 involves performing soft recalibration and calculating the quality score per token, specifically as follows:

[0021] A modal correlation matrix is ​​constructed by utilizing the correlation between template tokens and search tokens, and the modal evidence strength is obtained by statistically analyzing the correlation matrix.

[0022] Based on the modal evidence strength, a modal uncertainty index is constructed. A monotonic mapping function is used to map the modal uncertainty into a recalibration intensity, and the residual is applied to the search features to complete the modal-level soft recalibration.

[0023] By calculating per-token quality scores using lightweight normalization mapping or lightweight networks, and restricting the per-token quality scores to the range of 0 to 1, the features of the corresponding search tokens are weighted using per-token quality scores, thereby achieving fine-grained quality screening and suppression.

[0024] As a preferred technical solution, in step S3, the characterization index of the uncertainty information is selected from at least one of template-search relevance, response peak, entropy value, consistency score, similarity statistics, and attention distribution stability.

[0025] As a preferred technical solution, in step S4, the frequency domain expert branch specifically includes:

[0026] Perform two-dimensional real Fourier transform, discrete Fourier transform, wavelet transform, or discrete cosine transform on the dual-modal spatial features to extract amplitude spectrum information;

[0027] Within a predefined frequency band, perform mean statistics, maximum statistics, weighted statistics, or multi-band splicing on the amplitude spectrum to obtain a channel-level frequency domain vector;

[0028] The channel-level frequency domain vectors are converted into frequency domain enhancement biases using a lightweight mapping network, and then applied to the original features using residual injection to obtain frequency domain enhanced features.

[0029] As a preferred technical solution, in step S4, the frequency statistics information is converted into a frequency domain enhancement bias and injected into the features in the form of residuals, specifically as follows:

[0030] The channel-level frequency domain statistical vector is input into a lightweight mapping network to generate a frequency domain enhancement bias. The frequency domain enhancement bias is then added to the original spatial features in the form of a residual to obtain a frequency domain enhancement feature that incorporates frequency domain information.

[0031] As a preferred technical solution, in step S5, the current template representation is constructed as follows:

[0032] The first frame template features are concatenated with all historical template features in the template memory set in the sequence dimension to form a joint template sequence;

[0033] Alternatively, the features of historical templates can be weighted and summed based on the quality scores of each historical template in the template memory set, and then concatenated with the features of the first frame template to construct the current template representation.

[0034] As a preferred technical solution, in step S6, the current frame template update quality score is calculated, specifically as follows:

[0035] Predict bounding boxes based on the current frame. Candidate template regions are cropped from the RGB and TIR images respectively, and represented as follows: ,in This indicates that the prediction box and preset template size are used. The clipping operator executed;

[0036] The candidate templates are processed by an encoder that shares parameters with the search branch to extract features, resulting in... ;

[0037] Then, the target confidence score output from the current frame classification header is combined. And bimodal quality score, calculate the template update quality score for the current frame. ,in and This is the balance coefficient.

[0038] Secondly, the present invention provides an adaptive RGBT target tracking device based on quality perception and space-frequency fusion, which is applied to the aforementioned adaptive RGBT target tracking method based on quality perception and space-frequency fusion, including a sequence initialization module, an inter-frame feature extraction module, a dynamic quality modeling recalibration module, a dual-expert hybrid module, a matching prediction module, a quality score update module, a template memory update module, and a loop module.

[0039] The sequence initialization module is used to acquire RGB image sequences and TIR image sequences, input the target initial bounding box in the first frame, crop the template region of the first frame in the RGB mode and TIR mode respectively according to the initial bounding box, extract the first frame bimodal template features, and initialize the template memory set with the first frame bimodal template features.

[0040] The inter-frame feature extraction module is used to crop the search region in the RGB image and TIR image respectively based on the predicted bounding box of the previous frame for the t-th frame, and extract the RGB features and TIR features of the search region of the current frame through a patch embedding and encoding network.

[0041] The dynamic quality modeling recalibration module is used to calculate the evidence strength and uncertainty information of each modality by utilizing the intramodal correlation between the current frame template features and the search features, and to perform soft recalibration on the search features based on the uncertainty information estimation results; at the same time, it further calculates the per-token quality score for the search token to suppress low-quality tokens, thereby obtaining the quality-calibrated RGB modal search features and TIR modal search features.

[0042] The dual-expert hybrid module is used to input the quality-calibrated dual-modal search features into the spatial domain expert branch and the frequency domain expert branch, respectively. The spatial domain expert branch performs spatial-level fusion of the dual-modal features, preserving local structure, edge, and texture information. The frequency domain expert branch performs frequency domain transformation on the dual-modal spatial features, extracts frequency statistics of predefined frequency bands, converts the frequency statistics into frequency domain enhancement biases, and injects them into the features in the form of residuals to obtain frequency domain enhancement features. The gated routing network generates dynamic weights for the spatial domain expert branch and the frequency domain expert branch based on the global statistics of the current input features, and performs weighted fusion of the outputs of the two branches to obtain the fused representation of the search region.

[0043] The matching prediction module is used to construct the current template representation based on the template features of the first frame and the template memory set, input the current template representation and the search region fusion representation into the matching decoder and the prediction head, and output the target existence confidence and the target prediction bounding box of the current frame.

[0044] The quality score update module is used to crop candidate template regions in the RGB and TIR images based on the predicted bounding boxes of the current frame and extract candidate template features, and calculate the template update quality score of the current frame by combining the target existence confidence and bimodal quality information of the current frame.

[0045] The template memory update module is used to determine whether the quality score of the updated template is higher than a preset threshold. If it is higher, the candidate template is written into the template memory set. If the template memory set has not reached its capacity limit, a new template is directly inserted. If it is full, the template with the lowest quality score in the set is replaced with the current candidate template. If it is not higher than the threshold, the template memory set remains unchanged.

[0046] The loop module uses the predicted bounding box of the current frame as the basis for cropping the search region of the next frame until the target tracking of the entire image sequence is completed.

[0047] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0048] At least one processor; and,

[0049] A memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to execute the adaptive RGBT target tracking method based on quality perception and space-frequency fusion.

[0051] Fourthly, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the aforementioned adaptive RGBT target tracking method based on quality perception and space-frequency fusion.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] (1) The present invention introduces a dynamic quality modeling and recalibration mechanism before feature interaction, which can estimate the reliability of the current observation at both the modal and token levels, thereby suppressing unreliable regions before fusion and matching, effectively reducing noise pollution in low-light, occlusion, heat cross and complex background scenes.

[0054] (2) This invention introduces two complementary modeling methods in parallel: spatial domain and frequency domain. The spatial domain is used to preserve local structural details, while the frequency domain is used to enhance the overall frequency statistical stability. Dynamic combination is achieved through gated routing, so that the tracker can still maintain relatively stable discrimination ability and positioning consistency in complex degradation scenarios such as noise, ambiguity, and compression distortion.

[0055] (3) The present invention designs a quality-gated template update mechanism, which only allows writing to the template memory set when the prediction quality of the current frame is high enough, thereby avoiding writing erroneous appearances to historical templates. At the same time, through fixed capacity memory and quality-first replacement strategy, the template set maintains high representativeness and reliability.

[0056] (4) This invention does not simply rely on the first frame template, nor does it aggressively cover frame by frame. Instead, it constructs a closed-loop mechanism of “candidate template generation - quality assessment - secure writing - online selection”, which makes it easier to maintain long-term stable tracking in scenarios with changes in target pose, scale, occlusion recovery and viewpoint switching.

[0057] (5) The quality modeling module, space-frequency mixing module, and template update module proposed in this invention can all be plugged into existing template-search type RGBT tracking frameworks, and have good engineering adaptability and expansion potential. They also have strong comprehensive performance improvement capabilities. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart of an adaptive RGBT target tracking method based on quality perception and space-frequency fusion according to an embodiment of the present invention;

[0060] Figure 2 is a schematic diagram of the dynamic quality modeling and recalibration module of the present invention;

[0061] Figure 3 is a schematic diagram of the space-frequency dual-expert hybrid module structure of the present invention;

[0062] Figure 4 This is a block diagram of an adaptive RGBT target tracking device based on quality perception and space-frequency fusion according to an embodiment of the present invention.

[0063] Figure 5 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0064] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0065] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0066] Definitions of relevant terms:

[0067] RGBT: refers to the dual-mode of visible light (RGB) and thermal infrared (TIR).

[0068] Template-search paradigm: This refers to a common processing method in target tracking. It typically uses the target region in the first frame or historical frames as a template, crops the search region in the current frame, and achieves target localization by matching the template with the search region.

[0069] DQMRM (Dynamic Quality Modeling and Recalibration Module): Its function is to estimate the reliability of each modality in the current frame before the bimodal features are incorporated into subsequent interactions and matching, and to suppress or recalibrate unreliable responses, thereby outputting a more reliable feature representation.

[0070] SFDEM (Spatial-Frequency Dual-Expert Mixture Module): This module models features in both the spatial and frequency domains separately, and adaptively adjusts the ratio of each domain using a gated routing mechanism to improve representation stability in degraded scenarios such as noise, ambiguity, and compression distortion.

[0071] ATUM (Adaptive Template Update Module): Its function is to maintain a candidate template memory set during the testing phase and decide whether to write a new template based on the prediction quality of the current frame, thereby maintaining adaptation as the target appearance gradually changes, while reducing the risk of template contamination.

[0072] Quality perception refers to estimating feature quality based on clues such as intramodal matching consistency, response strength, and feature uncertainty, and using this quality information for subsequent feature recalibration, fusion, or update decisions.

[0073] Spatial-frequency fusion refers to modeling features from both spatial domain representation and frequency domain statistics, and adaptively combining the two types of information to take into account local details, edge textures, overall structure and degradation stability.

[0074] Template memory set: refers to a set of multiple historical candidate templates maintained during the testing phase, used to replace the static representation method that only relies on the first frame template, in order to enhance the appearance adaptation capability in long-term tracking.

[0075] This invention is based on the template-search paradigm of RGBT target tracking framework. It takes the temporally input visible light (RGB) image sequence and thermal infrared (TIR) ​​image sequence as the processing objects, completes the tracking initialization with the target bounding box of the first frame, and finally outputs the target localization results of each frame in the video sequence.

[0076] For the current frame to be processed, this invention first determines and crops the search region in the RGB and TIR modal images respectively. Simultaneously, it constructs the current template region by combining the first frame template and the historical template memory set, and extracts the dual-modal template features and search features respectively. Subsequently, a dynamic quality modeling and recalibration module suppresses and calibrates low-reliability modalities and low-quality feature regions. Then, a spatial-frequency dual-expert hybrid module jointly models the search features in the spatial and frequency domains and completes adaptive fusion.

[0077] Subsequently, the template representation and the search representation fused with space and frequency are fed into the matching decoding stage to obtain the target existence confidence and predicted bounding box of the current frame. Finally, based on the prediction results and bimodal quality assessment information, candidate template screening and dynamic updating of the template memory set are performed, thereby realizing a safe, stable, and controllable template maintenance mechanism during long-term tracking.

[0078] In order to make Figures 1 to 3 The letter parameters, encoders, and processing units in this embodiment correspond one-to-one with the descriptions in the specification. In this embodiment, the visible light mode is uniformly referred to as the superscript. Thermal infrared mode is indicated by a superscript. , Given an RGBT video sequence The target bounding box in the first frame is . Indicates the template area. Indicates the search area. This indicates that the patch is embedded. Indicates a shared feature encoder. and These represent the template token and the search token, respectively. This represents the joint mapping between the matching decoder and the prediction head. Represents a template memory set. and This represents the bimodal mass fraction.

[0079] like Figure 1 As shown in the figure, this embodiment provides an adaptive RGBT target tracking method based on quality perception and space-frequency fusion, which includes the following steps:

[0080] Step S1: Input sequence and first frame initialization, specifically:

[0081] Obtain RGB image sequence and TIR image sequences And input the initial bounding box of the target in the first frame. .according to Crop the first frame template area in both RGB and TIR modes. and The first frame template token is obtained through patch embedding and shared feature encoder, and is represented as ,in The first frame template is retained as a stable reference template, and the template memory set is initialized simultaneously. .

[0082] Step S2: Crop and feature extraction of the current frame search region, specifically:

[0083] For the Frame, and At that time, based on the predicted bounding box of the previous frame... In RGB images and TIR images Medium crop search area and For each modality's search region, patch embedding and shared encoding are performed to obtain the current frame's search token, represented as... .in, Corresponding to the patch embedding module in the attached diagram, The corresponding feature extraction coding network is shown in the attached figure.

[0084] Step S3: Dynamic quality modeling and recalibration, specifically:

[0085] Let the first The first frame involved in the matching The modal template token and the search token are respectively and First, construct the intramodal correlation matrix. This is used to characterize the matching consistency between the template region and the search region; then, the row mean is used to calculate the first... Evidence strength of a template token Based on this, modal uncertainty is constructed. The modal mass fraction was further obtained. The lower the quality score, the less reliable the current observations of that mode are.

[0086] Furthermore, the dynamic quality modeling and recalibration module adopts a two-level processing mechanism of "modal-level soft recalibration + token-level fine-grained suppression" to solve the problem that existing methods are unable to explicitly perceive the dynamic changes of bimodal quality over time.

[0087] Specifically, such as Figure 2 As shown, the modal-level recalibration branch first outputs the overall quality score of the two modes based on the correlation matrix. and This quality score is then used as a unified quality prior for subsequent feature calibration and template updates.

[0088] To avoid abrupt information gaps caused by simple hard thresholding, this invention employs residual soft recalibration, that is, using a monotonic mapping function g(·) to map modal-level quality scores to recalibration intensities, and then performing recalibration on the search features. , where λ is the recalibrated strength coefficient. The above notation corresponds to... Figure 2 The modal-level calibration branch in the circuit is used to suppress low-reliability modal responses while maintaining the continuity of information transmission.

[0089] After completing the modal-level soft recalibration, the first... Each search token is further calculated to determine its token-level quality weight. Then perform token-by-token relabeling. Where LN(·) denotes layer normalization, These are learnable parameters. This allows for the suppression of low-quality tokens and the preservation of high-quality tokens at the local region level.

[0090] The dynamic quality modeling and recalibration module ultimately outputs the quality-calibrated RGB features. and TIR characteristics Simultaneously output modal mass fraction and The modal quality score is not only used for recalibration within this module, but also serves as an important input for the candidate template update quality assessment in steps S6 and S7.

[0091] Step S4: Space-frequency dual-expert hybrid modeling, specifically as follows:

[0092] The quality-recalibrated bimodal search features are fed into the spatial domain expert branch and the frequency domain expert branch, respectively. The spatial domain expert branch focuses on preserving local structure, edge, and texture information, while the frequency domain expert branch focuses on introducing more stable frequency statistics. Then, the outputs of the two branches are dynamically weighted through a gated routing network to obtain the final fused representation of the search region.

[0093] Furthermore, the spatial-frequency dual-expert hybrid module mainly addresses the problem of insufficient robustness when relying solely on spatial domain features in scenarios with noise, ambiguity, and compression distortion. Its core idea is to configure complementary expert branches for the spatial and frequency domains respectively, and to dynamically adjust the proportion of the two types of information by the gated routing network according to the current input state.

[0094] Specifically, such as Figure 3 As shown, spatial domain experts directly fuse the bimodal search features after quality recalibration, and the spatial domain response can be expressed as: This branch corresponds to the spatial domain expert branch in the attached diagram, and is used to preserve the target's local texture, edges, and salient structural information.

[0095] Frequency domain experts first use rearrangement operators Restore the token to a two-dimensional feature map and perform a frequency domain transformation. ,get Then in the predefined frequency band Statistical aggregation is performed to obtain channel-level frequency domain vectors. Then through a lightweight mapping network Generate frequency domain enhanced bias And inject the original features in the form of residuals to obtain The frequency domain enhancement features of the two modes are added together to form the frequency domain response. .

[0096] In obtaining spatial domain response and frequency domain response Then, the route input is constructed through global aggregation. Then, dynamic weights are generated by the gated routing network. ,in The final search region fusion is characterized as follows: When the scene texture is clear, Relative increase; when noise, blur, or compression distortion are present, Relative increase.

[0097] Step S5: Template construction and target matching prediction, specifically:

[0098] Construct the current template representation based on the features of the first template and the historical template memory set. The template representation is fused with the search region representation obtained in step S4. The common input matches the decoder and the prediction head, and their joint mapping can be represented as follows: ,in This indicates a match between the decoder and the prediction head. There is a confidence level for the target in the current frame. Predict bounding boxes for the target in the current frame.

[0099] Step S6: Candidate template generation and update quality assessment, specifically:

[0100] Predict bounding boxes based on the current frame. Candidate template regions are cropped from the RGB and TIR images respectively, and represented as follows: ,in This indicates that the prediction box and preset template size are used. The pruning operator is executed. Candidate templates are processed by an encoder that shares parameters with the search branch to extract features, resulting in... Then, the target confidence score output from the current frame classification header is combined. And using the bimodal quality score obtained in step S3, calculate the template update quality score for the current frame. ,in and This is the balance coefficient.

[0101] Step S7: Update the template memory for quality gating, specifically:

[0102] If the template updates its quality score Higher than the preset threshold Then, candidate templates are allowed to be written into the template memory set; if If the value is not higher than this threshold, the template memory set remains unchanged. To achieve fixed capacity management, the modality is recorded. Online template memory for When the memory capacity is not full, insert a new entry directly. When the memory capacity is full, first determine the entry with the lowest quality. Only when When the current candidate template is used, the corresponding lowest quality template is replaced.

[0103] Furthermore, the adaptive template update module mainly addresses the contradiction between long-term appearance changes and template contamination. Its core idea is to maintain a fixed-capacity candidate template memory set during the testing phase and use a quality gating strategy to control the template writing, replacement, and subsequent template construction processes.

[0104] Once the predicted bounding box of the current frame is obtained, the candidate template features are preferably obtained through an encoder that shares parameters with the backbone feature extraction network, so as to ensure that the candidate template representation, the first frame template representation, and the search representation are located in a unified feature space.

[0105] Then, the template updates the quality score. It simultaneously characterizes both "whether the current frame prediction is reliable" and "whether the current frame observation is suitable as a future template." Therefore, only when... Above the threshold Only when a template is written to the template memory set is a template allowed to be written, thus avoiding template pollution caused by low-quality frames.

[0106] The template memory set adopts a fixed capacity design, with the upper limit of capacity denoted as . .when New templates can be inserted directly at this time; when Replacement is only performed when the quality of the current candidate template is higher than the lowest quality template in memory, so that a set of high-quality and representative candidate appearances is always retained in the historical templates.

[0107] When constructing the template representation for the next frame, this module provides at least two implementation methods. The first is the sequence concatenation method, i.e. The second method is the quality-weighted fusion method, namely... ,in The former enhances template diversity, while the latter highlights a highly reliable historical appearance through quality weighting.

[0108] Step S8: Enter the next frame loop, specifically:

[0109] Predict the bounding box of the current frame. As a reference box for cropping the search region in the next frame, repeat steps S2 to S7 until the target tracking of the entire sequence is completed.

[0110] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0111] Based on the same idea as the adaptive RGBT target tracking method based on quality perception and space-frequency fusion in the above embodiments, the present invention also provides an adaptive RGBT target tracking device based on quality perception and space-frequency fusion. This device can be used to execute the aforementioned adaptive RGBT target tracking method based on quality perception and space-frequency fusion. For ease of explanation, the structural schematic diagram of the embodiment of the adaptive RGBT target tracking device based on quality perception and space-frequency fusion only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0112] Please see Figure 4 In another embodiment of this application, an adaptive RGBT target tracking device 100 based on quality perception and space-frequency fusion is provided. The device includes a sequence initialization module 101, an inter-frame feature extraction module 102, a dynamic quality modeling recalibration module 103, a dual expert hybrid module 104, a matching prediction module 105, a quality score update module 106, a template memory update module 107, and a loop module 108.

[0113] The sequence initialization module 101 is used to acquire RGB image sequences and TIR image sequences, input the target initial bounding box in the first frame, crop the template region of the first frame in the RGB mode and TIR mode respectively according to the initial bounding box, extract the first frame bimodal template features, and initialize the template memory set with the first frame bimodal template features.

[0114] The inter-frame feature extraction module 102 is used to, for the t-th frame, crop the search region in the RGB image and TIR image respectively according to the predicted bounding box of the previous frame, and extract the RGB features and TIR features of the search region of the current frame through a patch embedding and encoding network.

[0115] The dynamic quality modeling recalibration module 103 is used to calculate the evidence strength and uncertainty information of each modality by utilizing the intramodal correlation between the template features of the current frame and the search features, and to perform soft recalibration on the search features based on the uncertainty information estimation results; at the same time, it further calculates the per-token quality score for the search token to suppress low-quality tokens, thereby obtaining the quality-calibrated RGB modal search features and TIR modal search features;

[0116] The dual-expert hybrid module 104 is used to input the quality-calibrated dual-modal search features into the spatial domain expert branch and the frequency domain expert branch, respectively. The spatial domain expert branch performs spatial-level fusion of the dual-modal features, preserving local structure, edge, and texture information. The frequency domain expert branch performs frequency domain transformation on the dual-modal spatial features, extracts frequency statistics of predefined frequency bands, converts the frequency statistics into frequency domain enhancement biases, and injects them into the features in the form of residuals to obtain frequency domain enhancement features. The gated routing network generates dynamic weights for the spatial domain expert branch and the frequency domain expert branch based on the global statistics of the current input features, and performs weighted fusion of the outputs of the two branches to obtain the fused representation of the search region.

[0117] The matching prediction module 105 is used to construct the current template representation based on the template features of the first frame and the template memory set, input the current template representation and the search region fusion representation into the matching decoder and the prediction head, and output the target existence confidence and the target prediction bounding box of the current frame.

[0118] The quality score update module 106 is used to crop the candidate template region in the RGB image and TIR image according to the predicted bounding box of the current frame and extract the candidate template features, and calculate the template update quality score of the current frame by combining the target existence confidence and bimodal quality information of the current frame.

[0119] The template memory update module 107 is used to determine whether the template update quality score is higher than a preset threshold. If it is higher, the candidate template is written into the template memory set. When the template memory set has not reached its capacity limit, a new template is directly inserted. When it is full, the template with the lowest quality score in the set is replaced by the current candidate template. If it is not higher than the threshold, the template memory set remains unchanged.

[0120] The loop module 108 uses the predicted bounding box of the current frame as the basis for cropping the search region of the next frame until the target tracking of the entire image sequence is completed.

[0121] It should be noted that the adaptive RGBT target tracking device based on quality perception and spatial-frequency fusion of the present invention corresponds one-to-one with the adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion of the present invention. The technical features and beneficial effects described in the embodiments of the adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion are applicable to the embodiments of the adaptive RGBT target tracking based on quality perception and spatial-frequency fusion. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0122] Furthermore, in the embodiments of the adaptive RGBT target tracking device based on quality perception and space-frequency fusion described above, the logical division of each program module is merely an example. In actual applications, the above functions can be assigned to different program modules as needed, for example, for the sake of corresponding hardware configuration requirements or the convenience of software implementation. That is, the internal structure of the adaptive RGBT target tracking device based on quality perception and space-frequency fusion can be divided into different program modules to complete all or part of the functions described above.

[0123] Please see Figure 5 In one embodiment, an electronic device is provided for implementing an adaptive RGBT target tracking method based on quality perception and space-frequency fusion. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as an adaptive RGBT target tracking program 203 based on quality perception and space-frequency fusion.

[0124] The first memory 202 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as the portable hard drive of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 200. Furthermore, the first memory 202 can include both internal storage units and external storage devices of the electronic device 200. The first memory 202 can be used not only to store application software and various types of data installed on the electronic device 200, such as the code of the adaptive RGBT target tracking program 203 based on quality perception and space-frequency fusion, but also to temporarily store data that has been output or will be output.

[0125] In some embodiments, the first processor 201 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the first memory 202 and calls data stored in the first memory 202 to perform various functions of the electronic device 200 and process data.

[0126] Figure 5 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 5 The structure shown does not constitute a limitation on the electronic device 200, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0127] The adaptive RGBT target tracking program 203 based on quality perception and space-frequency fusion, stored in the first memory 202 of the electronic device 200, is a combination of multiple instructions. When run in the first processor 201, it can achieve the following:

[0128] S1. Obtain the RGB image sequence and TIR image sequence, and input the target initial bounding box in the first frame. Based on the initial bounding box, crop the template region of the first frame in the RGB mode and TIR mode respectively, extract the first frame bimodal template features, and initialize the template memory set with the first frame bimodal template features.

[0129] S2. For frame t, the search region is cropped in the RGB image and TIR image respectively based on the predicted bounding box of the previous frame, and the RGB features and TIR features of the search region in the current frame are extracted by the patch embedding and encoding network.

[0130] S3. Utilizing the intra-modal correlation between the current frame template features and the search features, calculate the evidence strength and uncertainty information for each modality, and perform soft recalibration on the search features based on the uncertainty information estimation results. Simultaneously, further calculate per-token quality scores for the search tokens to suppress low-quality tokens, thereby obtaining the quality-calibrated RGB modal search features and TIR modal search features.

[0131] S4. Input the quality-calibrated bimodal search features into the spatial domain expert branch and the frequency domain expert branch respectively. The spatial domain expert branch performs spatial-level fusion of the bimodal features, preserving local structure, edge, and texture information. The frequency domain expert branch performs frequency domain transformation on the bimodal spatial features, extracts frequency statistics of predefined frequency bands, converts the frequency statistics into frequency domain enhancement biases, and injects them into the features in the form of residuals to obtain frequency domain enhancement features. The gated routing network generates dynamic weights for the spatial domain expert branch and the frequency domain expert branch based on the global statistics of the current input features, and performs weighted fusion of the outputs of the two branches to obtain the fused representation of the search region.

[0132] S5. Construct the current template representation based on the template features of the first frame and the template memory set. Input the current template representation and the search region fusion representation into the matching decoder and prediction head, and output the target existence confidence and target prediction bounding box of the current frame.

[0133] S6. Based on the predicted bounding box of the current frame, crop the candidate template region in the RGB image and TIR image and extract the candidate template features. Combine the target existence confidence and bimodal quality information of the current frame to calculate the template update quality score of the current frame.

[0134] S7. Determine whether the quality score of the updated template is higher than the preset threshold. If it is higher, write the candidate template into the template memory set. If the template memory set has not reached the capacity limit, insert a new template directly. If it is full, replace the template with the lowest quality score in the set with the current candidate template. If it is not higher than the threshold, keep the template memory set unchanged.

[0135] S8. Use the predicted bounding box of the current frame as the basis for cropping the search region of the next frame until the target tracking of the entire image sequence is completed.

[0136] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion, characterized in that, Includes the following steps: S1. Obtain the RGB image sequence and TIR image sequence, and input the target initial bounding box in the first frame. Based on the initial bounding box, crop the template region of the first frame in the RGB mode and TIR mode respectively, extract the first frame bimodal template features, and initialize the template memory set with the first frame bimodal template features. S2. For frame t, the search region is cropped in the RGB image and TIR image respectively based on the predicted bounding box of the previous frame, and the RGB features and TIR features of the search region in the current frame are extracted by the patch embedding and encoding network. S3. Utilize the intramodal correlation between the current frame template features and the search features to calculate the evidence strength and uncertainty information for each modality, and perform soft recalibration on the search features based on the uncertainty information estimation results; at the same time, further calculate the per-token quality score for the search tokens to suppress low-quality tokens, thereby obtaining the quality-calibrated RGB modal search features and TIR modal search features. S4. Input the quality-calibrated bimodal search features into the spatial domain expert branch and the frequency domain expert branch respectively. The spatial domain expert branch performs spatial-level fusion on the bimodal features, preserving local structure, edge and texture information. The frequency domain expert branch performs frequency domain transformation on the bimodal spatial features, extracts frequency statistics of predefined frequency bands, converts the frequency statistics into frequency domain enhancement bias and injects them into the features in the form of residuals to obtain frequency domain enhanced features. The gated routing network generates dynamic weights for spatial domain expert branches and frequency domain expert branches based on global statistical information of the current input features. The outputs of the two branches are then weighted and fused to obtain a fused representation of the search region. S5. Construct the current template representation based on the template features of the first frame and the template memory set. Input the current template representation and the search region fusion representation into the matching decoder and prediction head, and output the target existence confidence and target prediction bounding box of the current frame. S6. Based on the predicted bounding box of the current frame, crop the candidate template region in the RGB image and TIR image and extract the candidate template features. Combine the target existence confidence and bimodal quality information of the current frame to calculate the template update quality score of the current frame. S7. Determine whether the template update quality score is higher than the preset threshold. If it is higher, write the candidate template into the template memory set. When the template memory set has not reached its capacity limit, a new template is inserted directly; when it is full, the template with the lowest quality score in the set is replaced with the current candidate template. If the value is not higher than the threshold, the template memory set remains unchanged. S8. Use the predicted bounding box of the current frame as the basis for cropping the search region of the next frame until the target tracking of the entire image sequence is completed.

2. The adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion according to claim 1, characterized in that, In step S3, soft recalibration and per-token quality score calculation are performed, specifically as follows: A modal correlation matrix is ​​constructed by utilizing the correlation between template tokens and search tokens, and the modal evidence strength is obtained by statistically analyzing the correlation matrix. Based on the modal evidence strength, a modal uncertainty index is constructed. A monotonic mapping function is used to map the modal uncertainty into a recalibration intensity, and the residual is applied to the search features to complete the modal-level soft recalibration. By calculating per-token quality scores using lightweight normalization mapping or lightweight networks, and restricting the per-token quality scores to the range of 0 to 1, the features of the corresponding search tokens are weighted using per-token quality scores, thereby achieving fine-grained quality screening and suppression.

3. The adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion according to claim 1, characterized in that, In step S3, the characterization index of the uncertainty information is selected from at least one of template-search relevance, response peak, entropy value, consistency score, similarity statistics, and attention distribution stability.

4. The adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion according to claim 1, characterized in that, In step S4, the frequency domain expert branch specifically includes: Perform two-dimensional real Fourier transform, discrete Fourier transform, wavelet transform, or discrete cosine transform on the dual-modal spatial features to extract amplitude spectrum information; Within a predefined frequency band, perform mean statistics, maximum statistics, weighted statistics, or multi-band splicing on the amplitude spectrum to obtain a channel-level frequency domain vector; The channel-level frequency domain vectors are converted into frequency domain enhancement biases using a lightweight mapping network, and then applied to the original features using residual injection to obtain frequency domain enhanced features.

5. The adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion according to claim 1, characterized in that, In step S4, the frequency statistics are converted into a frequency domain enhancement bias and injected into the features in the form of residuals, specifically as follows: The channel-level frequency domain statistical vector is input into a lightweight mapping network to generate a frequency domain enhancement bias. The frequency domain enhancement bias is then added to the original spatial features in the form of a residual to obtain a frequency domain enhancement feature that incorporates frequency domain information.

6. The adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion according to claim 1, characterized in that, In step S5, the current template representation is constructed as follows: The first frame template features are concatenated with all historical template features in the template memory set in the sequence dimension to form a joint template sequence; Alternatively, the features of historical templates can be weighted and summed based on the quality scores of each historical template in the template memory set, and then concatenated with the features of the first frame template to construct the current template representation.

7. The adaptive RGBT target tracking method based on quality perception and spatial-frequency fusion according to claim 1, characterized in that, In step S6, the current frame template update quality score is calculated, specifically as follows: Predict bounding boxes based on the current frame. Candidate template regions are cropped from the RGB and TIR images respectively, and represented as follows: ,in This indicates that the prediction box and preset template size are used. The clipping operator executed; The candidate templates are processed by an encoder that shares parameters with the search branch to extract features, resulting in... ; Then, the target confidence score output from the current frame classification header is combined. And bimodal quality score, calculate the template update quality score for the current frame. ,in and This is the balance coefficient.

8. An adaptive RGBT target tracking device based on quality perception and space-frequency fusion, characterized in that, The adaptive RGBT target tracking method based on quality perception and space-frequency fusion, applied to any one of claims 1-7, includes a sequence initialization module, an inter-frame feature extraction module, a dynamic quality modeling and recalibration module, a dual-expert hybrid module, a matching prediction module, a quality score update module, a template memory update module, and a loop module; The sequence initialization module is used to acquire RGB image sequences and TIR image sequences, input the target initial bounding box in the first frame, crop the template region of the first frame in the RGB mode and TIR mode respectively according to the initial bounding box, extract the first frame bimodal template features, and initialize the template memory set with the first frame bimodal template features. The inter-frame feature extraction module is used to crop the search region in the RGB image and TIR image respectively based on the predicted bounding box of the previous frame for the t-th frame, and extract the RGB features and TIR features of the search region of the current frame through a patch embedding and encoding network. The dynamic quality modeling recalibration module is used to calculate the evidence strength and uncertainty information of each modality by utilizing the intramodal correlation between the current frame template features and the search features, and to perform soft recalibration on the search features based on the uncertainty information estimation results; at the same time, it further calculates the per-token quality score for the search token to suppress low-quality tokens, thereby obtaining the quality-calibrated RGB modal search features and TIR modal search features. The dual-expert hybrid module is used to input the quality-calibrated dual-modal search features into the spatial domain expert branch and the frequency domain expert branch respectively. The spatial domain expert branch performs spatial-level fusion on the dual-modal features, preserving local structure, edge and texture information; the frequency domain expert branch performs frequency domain transformation on the dual-modal spatial features, extracts frequency statistics of predefined frequency bands, converts the frequency statistics into frequency domain enhancement bias and injects them into the features in the form of residuals to obtain frequency domain enhanced features. The gated routing network generates dynamic weights for spatial domain expert branches and frequency domain expert branches based on global statistical information of the current input features. The outputs of the two branches are then weighted and fused to obtain a fused representation of the search region. The matching prediction module is used to construct the current template representation based on the template features of the first frame and the template memory set, input the current template representation and the search region fusion representation into the matching decoder and the prediction head, and output the target existence confidence and the target prediction bounding box of the current frame. The quality score update module is used to crop candidate template regions in the RGB and TIR images based on the predicted bounding boxes of the current frame and extract candidate template features, and calculate the template update quality score of the current frame by combining the target existence confidence and bimodal quality information of the current frame. The template memory update module is used to determine whether the template update quality score is higher than a preset threshold. If it is higher, the candidate template is written into the template memory set. When the template memory set has not reached its capacity limit, a new template is inserted directly; when it is full, the template with the lowest quality score in the set is replaced with the current candidate template. If the value is not higher than the threshold, the template memory set remains unchanged. The loop module uses the predicted bounding box of the current frame as the basis for cropping the search region of the next frame until the target tracking of the entire image sequence is completed.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the adaptive RGBT target tracking method based on quality perception and space-frequency fusion as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the adaptive RGBT target tracking method based on quality perception and spatial frequency fusion as described in any one of claims 1-7.

Images