A Visible-Infrared Target Tracking Method and System Based on Learnable Gated Fusion

Abstract

This invention discloses a visible-infrared target tracking method and system based on learnable gating fusion, belonging to the fields of computer vision and embedded artificial intelligence. The method includes: spatial alignment of unregistered modalities based on mode matching of response peaks using a decision-level response guidance module; evaluation of modal quality and dynamic selection of single-modal or fusion tracking paths using a learnable dataflow gating module; channel-weighted fusion when the gating decision is fusion using a feature-level fusion module; and outputting target bounding boxes using a lightweight tracking head with shared weights. The embedded deployment employs a four-level optimization scheme, including multi-core NPU task parallelism, asynchronous pipeline, circular buffer memory reuse, and INT8 quantization solidification. This invention achieves robust tracking in unregistered scenarios and achieves real-time performance on embedded platforms, offering advantages such as high accuracy, high efficiency, and low resource consumption.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a visible light-infrared target tracking method and system based on learnable gating fusion. Background Technology

[0002] Existing visible-infrared (RGBT) fusion target tracking algorithms, while pursuing high performance, typically face two major bottlenecks: First, the model computational complexity is high, relying on large backbone networks and dense fusion operations, resulting in poor real-time performance and difficulty in deployment on resource-constrained edge devices; second, the algorithm has strict pixel-level spatial registration requirements for input data, but in practical applications, dual-mode sensors often cause image misalignment due to hardware differences, which drastically reduces the performance of such algorithms.

[0003] Existing work often struggles to strike a balance between accuracy, speed, and robustness (adaptability to unregistration). Some methods improve robustness by preserving the original single-modal information, but introduce additional computational overhead; others attempt to simplify the model, sacrificing accuracy or adaptability to complex scenarios. Furthermore, few studies systematically address the efficient deployment of algorithms on embedded multi-core NPU platforms, such as task scheduling, memory management, and model quantization. Therefore, there is an urgent need for a lightweight, adaptive RGBT tracking scheme at the algorithm level, which can be efficiently deployed at the system level, to promote its application in practical edge computing scenarios such as security, automotive, and drones. Summary of the Invention

[0004] The purpose of this invention is to address the technical challenges of existing RGBT tracking algorithms, such as bloated models, low computational efficiency, over-reliance on data registration, and difficulty in real-time deployment on embedded platforms. By designing an algorithm architecture integrating "decision-level spatial reconstruction, dynamic gating modality selection, and lightweight feature fusion," and combining it with heterogeneous computing optimization strategies for embedded NPUs, this invention aims to achieve the following objectives: 1) Robust spatial semantic alignment on unregistered data through high-level response guidance; 2) Dynamically selecting the most reliable single-modality or fused modality for tracking based on the scene through a learnable gating mechanism, avoiding invalid computation and significantly improving efficiency; 3) Designing complete lightweight network components to achieve high frame rates while maintaining high tracking accuracy; 4) Providing an embedded deployment optimization scheme from algorithm to system, including multi-core NPU task scheduling, asynchronous pipelines, memory reuse, and model quantization, to achieve efficient and stable operation of the algorithm on edge devices.

[0005] The technical solution of the present invention: According to a first aspect of the present invention, a visible light-infrared target tracking method based on learnable gating fusion is proposed, comprising the following steps: Step S1: Through the decision-level response guidance module, feature extraction and deep cross-correlation are performed on the search map and template map of visible light and infrared modes to obtain the response map of each mode; the matching center is calculated based on the mode of the response peak coordinates of all channels in the response map, and the visible light response map is spatially registered with the matching center of the infrared mode as the reference to obtain the registered response map. Step S2: Using the learnable data stream gating module, extract the modal quality feature vectors of the registered response map and predict the reliability score of each modality; dynamically select a single-modality path or a fusion path based on the comparison result of the reliability score and a preset threshold; when the reliability score of any modality is greater than or equal to the threshold, select the original features of that modality for target tracking; when the reliability scores of both modalities are less than the threshold, activate the feature-level fusion module. Step S3: Through the feature-level fusion module, in the fusion path, based on the channel weights decoupled from the modal quality feature vector, the original search features and template features are channel-weighted and concatenated to obtain the fused search features and template features; Step S4: Input the original features in the single-modal path or the fused features in the fusion path into the lightweight tracking head with shared weights. The lightweight tracking head includes a convolutional layer with a shared base as well as regression branches, classification branches and centrality branches, and outputs the target bounding box parameters.

[0006] In one possible embodiment, in step S1, the mode matching based on response peaks specifically includes: calculating the mode of the response peak coordinates of all channels in the visible light response map and the infrared response map, and using them as the visible light matching center and the infrared matching center, respectively; subtracting the coordinates of the visible light matching center from the infrared matching center as the origin to obtain a translation vector; and using the translation vector to perform translation or edge filling operations on the visible light response map to achieve spatial alignment with the infrared response map.

[0007] In one possible embodiment, in step S2, the learnable data stream gating module includes: performing channel dimensionality reduction on the registered response map through 1×1 convolution, and then compressing the spatial dimension through global average pooling to obtain a modal quality feature vector; inputting the modal quality feature vector into a classification network consisting of three fully connected layers, and using the Sigmoid activation function to output a reliability score.

[0008] In one possible embodiment, in step S2, the preset threshold is 0.5; when either the reliability score p_RGB of the visible light mode or the reliability score p_TIR of the infrared mode is greater than or equal to 0.5, the original search feature and template feature corresponding to the high confidence mode are selected for single-mode tracking; when both p_RGB and p_TIR are less than 0.5, a fusion path is selected.

[0009] In one possible embodiment, in step S3, the channel weighted fusion specifically includes: decoupling the fusion weight scalar of each channel from the modal quality feature vector; multiplying the original search feature and original template feature of each modality by the fusion weight scalar of its corresponding channel; and concatenating the weighted visible light and infrared features along the channel dimension to obtain the fused search feature and template feature.

[0010] In one possible embodiment, in step S4, the lightweight tracking head with shared weights includes: a shared base consisting of four 3×3 convolutional layers, with the first three layers followed by a ReLU activation function and the fourth layer having no activation function; the regression branch, classification branch, and centrality branch share the features extracted by the first three convolutional layers, and output a classification heatmap, a centrality heatmap, and bounding box regression parameters; wherein, the centrality heatmap and the classification heatmap are multiplied pixel by pixel to obtain a joint confidence heatmap, which is used to determine the center position of the target.

[0011] In one possible embodiment, the method further includes a training process for a total loss function, which is a weighted sum of a gating loss L_gate and a tracking loss L_track; the gating loss L_gate includes a modality priority loss L_p, a decision boundary reinforcement loss L_b, and a modality competition regularization loss L_c; wherein the modality priority loss employs a weighted binary cross-entropy loss calculated based on the number of high-quality visible light samples and the number of high-quality infrared samples to alleviate data imbalance.

[0012] In one possible embodiment, the training of the total loss function adopts a phased progressive training strategy: the first phase freezes the gating module and the tracking head, and only fine-tunes the backbone network; the second phase freezes the backbone network and the tracking head, and only updates the gating module parameters; the third phase unfreezes all network modules and performs end-to-end joint fine-tuning.

[0013] According to a second aspect of the present invention, a learnable lightweight unregistered visible-infrared target tracking system based on dynamic gating is proposed to implement the aforementioned learnable lightweight unregistered visible-infrared target tracking method based on dynamic gating, comprising: The decision-level response guidance module is used to extract features and perform deep cross-correlation on the search maps and template maps of visible light and infrared modes to obtain response maps for each mode. Based on the mode of the response peak coordinates of all channels in the response map, the matching center is calculated, and the visible light response map is spatially registered with the matching center of the infrared mode as the reference to obtain the registered response map. This module aims to solve the spatial non-registration problem of the input image and avoid semantic misalignment caused by direct fusion at the bottom layer.

[0014] The learnable data flow gating module is used to extract the modal quality feature vectors of the registered response map and predict the reliability score of each modality; based on the comparison result of the reliability score and the preset threshold, it dynamically selects a single-modal path or a fusion path; when the reliability score of any modality is greater than or equal to the threshold, the original features of that modality are selected for target tracking; when the reliability scores of both modalities are less than the threshold, the feature-level fusion module is activated; this module acts as the system's "intelligent scheduler," evaluating the quality of each modality and determining the most efficient tracking path.

[0015] The feature-level fusion module is used to perform channel weighting and concatenation on the original search features and template features based on the channel weights decoupled from the modal quality feature vector in the fusion path, so as to obtain the fused search features and template features. This module is activated only when the LDG module determines that fusion is needed, and performs lightweight and effective feature integration.

[0016] The shared-weight lightweight tracking head is used to input the original features from the single-modal path or the fused features from the fusion path into the shared-weight lightweight tracking head and output the target bounding box parameters.

[0017] In one possible embodiment, for an embedded multi-core NPU platform, the system adopts the following embedded deployment optimization scheme: Task-level parallel scheduling: The forward inference of the backbone network of the visible light and infrared search graphs and template graphs is assigned to the first and second NPU cores, and the computation of the gating module and the tracking head is assigned to the third NPU core; Asynchronous data flow pipeline: It adopts a producer-consumer model and uses lock-free atomic flags to synchronously achieve decoupling between NPU cores and full pipeline operation; Memory reuse and management: A circular buffer is pre-allocated in shared memory to cache intermediate feature data output by each NPU core, and memory reuse is achieved by cyclically moving the read and write pointers; Model quantization and solidification: The trained model is quantized using INT8, and the quantized weights are solidified into constants during the compilation phase and embedded into the on-chip cache of the NPU.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: The lightweight adaptive architecture of "Decision Reconstruction - Dynamic Gating" is the first to achieve non-registration space reconstruction at the decision level through mode matching of response graphs, avoiding noise in the underlying fusion. The core innovation lies in the Learnable Gating (LDG) module, which can evaluate modal quality based on the real-time scene and dynamically select single modality or fusion path, realizing true "on-demand computation". While maintaining high accuracy, it significantly improves tracking efficiency (especially in simple scenarios).

[0019] For gating collaborative training mechanisms for imbalanced data, a collaborative training objective is designed that integrates modality priority loss, decision boundary reinforcement loss, and modality competition regularization loss. The distribution of training samples is balanced by manually labeled modality quality labels, which effectively solves the learning problem of gating networks under complex data distributions and makes their decisions more accurate and decisive.

[0020] An end-to-end deployment optimization system for multi-core NPUs: Going beyond single algorithm design, this system proposes a complete embedded deployment optimization scheme encompassing task scheduling, asynchronous pipelines, circular buffer memory reuse, and model quantization. This scheme fully leverages the hardware characteristics of embedded heterogeneous computing platforms, enabling efficient and low-latency migration of algorithms from high-performance servers to resource-constrained edge devices, making real-time, high-performance RGBT tracking possible at the edge. Attached Figure Description

[0021] Figure 1 This is a flowchart of a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of the network structure of LGFT according to a preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the embedded deployment architecture of a preferred embodiment of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] like Figure 1 As shown, a visible light-infrared target tracking method based on learnable gating fusion includes the following steps: Step S1: Through the decision-level response guidance module, feature extraction and deep cross-correlation are performed on the search map and template map of visible light and infrared modes to obtain the response map of each mode; the matching center is calculated based on the mode of the response peak coordinates of all channels in the response map, and the visible light response map is spatially registered with the matching center of the infrared mode as the reference to obtain the registered response map. Step S1 is specifically implemented through the following process: Step D1: Lightweight feature extraction and deep cross-correlation: Lightweight MobileNet is used as the backbone network with shared weights to extract features from the visible light and infrared search maps and template maps respectively; Subsequently, a deep cross-correlation operation is performed independently for each modality: the search features of each channel are cross-correlated with the template features to generate a set of preliminary multi-channel response maps; Step D2: Generate the final response maps: Concatenate the preliminary multi-channel response maps of each modality along the channel dimension, and then fuse and reduce the dimensions using a 1x1 convolutional layer to obtain the final visible light response map f_RGB and the final infrared response map f_TIR. These response maps characterize the spatial distribution probability of the target in each modality; Step D3: Spatial registration based on response peaks: Calculate the mode of the peak coordinates of all channels in f_RGB and f_TIR to obtain the visible light matching center M_RGB and the infrared matching center M_TIR, respectively. Using the more stable infrared response as the central reference, calculate the translation vector; This vector is used to perform translation or edge-filling operations on the visible light response map f_RGB, aligning it spatially with the infrared response map f_TIR, resulting in registered response maps F_RGB and F_TIR. This step reconstructs the spatial correspondence between unregistered modes at the decision level.

[0024] Step S2: Using the learnable data stream gating module, extract the modal quality feature vectors of the registered response map and predict the reliability score of each modality; dynamically select a single-modality path or a fusion path based on the comparison result of the reliability score and a preset threshold; when the reliability score of any modality is greater than or equal to the threshold, select the original features of that modality for target tracking; when the reliability scores of both modalities are less than the threshold, activate the feature-level fusion module. Step S2 is specifically implemented through the following process: Step L1: Modal Quality Feature Vector Extraction: The registered response maps F_RGB and F_TIR output from the DRG module are reduced in dimensionality by 1x1 convolution, and then compressed using global average pooling (GAP) to obtain 1x1xC feature vectors Q_RGB and Q_TIR representing modal quality; C is the number of channels, which varies depending on the image. For example, color images have 3 channels, and grayscale images have 1 channel.

[0025] Step L2: Reliability score prediction: Input Q_RGB and Q_TIR into a small classification network consisting of three fully connected layers, and finally output a scalar between 0 and 1 through the Sigmoid function, which is the reliability score p_RGB and p_TIR of this mode; Step L3: Dynamic Path Selection: Set an adjustable hyperparameter as a threshold; perform intelligent judgment during inference: If either p_RGB or p_TIR is greater than or equal to the threshold, the current scene is determined to be simple and the modality is reliable enough. The system will skip the subsequent fusion module and directly select the original template and search features corresponding to the high-confidence modality, and send them to the tracking head for single-modality tracking. If both p_RGB and p_TIR are less than the threshold, the scene is considered complex and a single modality is unreliable. The system will activate the fusion path, send Q_RGB, Q_TIR and the original features to the feature-level fusion module (FLF) for information fusion, and then send the fused features to the tracking head.

[0026] Step S3: Through the feature-level fusion module, in the fusion path, based on the channel weights decoupled from the modal quality feature vector, the original search features and template features are channel-weighted and concatenated to obtain the fused search features and template features; Step S3 is specifically implemented through the following process: Step F1: Weight decoupling: Decouple the fusion weight scalars Q_i^RGB and Q_i^TIR corresponding to each channel from the modal quality feature vectors Q_RGB and Q_TIR extracted from the LDG module; Step F2: Channel-weighted fusion: Obtain the original search features X_i^RGB / X_i^TIR and template features T_i^RGB / T_i^TIR before the DRG module; multiply the feature map of each modality with the weight scalar of its corresponding channel to achieve adaptive weighting in the channel dimension; finally, concatenate the weighted visible light and infrared features in the channel dimension to obtain the fused search features X^F and template features T^F.

[0027] Step S4 involves inputting the original features from the single-modal path or the fused features from the fusion path into a lightweight tracking head with shared weights. This lightweight tracking head employs a minimalist shared base structure containing four 3x3 convolutional layers. The regression, classification, and centrality evaluation branches share the features extracted from the previous three convolutional layers, significantly reducing the number of parameters. The final output includes a classification heatmap, a centrality heatmap, and bounding box regression parameters, which are then post-processed to obtain the target bounding box.

[0028] In one embodiment, the method further includes a training process for the total loss function: Loss function: The total system loss is a weighted sum of the gating loss L_gate and the tracking loss L_track.

[0029] Gating loss L_gate: Used for training the LDG module, and consists of three terms: Modal priority loss L_p: A weighted binary cross-entropy loss is used. The weights ω_v and ω_t are calculated based on the number of manually labeled "high-quality visible light" samples N_v and "high-quality infrared" samples N_t in the training set to alleviate the problem of imbalanced data distribution.

[0030] Decision boundary reinforcement loss L_b: penalizes fuzzy decisions with output probabilities close to 0.5, and encourages the network to make high-confidence judgments.

[0031] Modal competition regularization loss L_c: In a single-modal scenario, it widens the difference in output probabilities between two modalities, thereby enhancing the discriminative power between modalities.

[0032] The total gating loss is: L_gate = α·L_p + β·L_b + γ·L_c, where α, β, and γ are balancing weights, which can be set to 1, 0.6, and 0.4, respectively.

[0033] The tracking loss L_track is a weighted sum of the classification loss (cross-entropy), regression loss (GIoU Loss), and centrality loss (binary cross-entropy), i.e., L_track = L_cls + λ1·L_reg + λ2·L_cen. The hyperparameters λ1 and λ2 need to be determined through a parameter grid search.

[0034] In one embodiment, the scenario is summarized as follows: This embodiment is designed for continuous target tracking by a drone under complex weather conditions.

[0035] System hardware configuration: Rockchip RK3588 and other edge computing platforms.

[0036] System software framework: Ubuntu 20.04, PyTorch 1.12, Python 3.10 LGFT uses lightweight pure convolutional networks such as MobileNet as its backbone, and further reduces computation by using weight sharing in the visible light and infrared branches. After obtaining the visible light and infrared search maps and template maps, it enters MobileNet to extract features.

[0037] S1: Decision-level response guidance module The Decision Response Guide Module (DRG) performs a deep cross-correlation operation on the search maps and template feature maps of the visible and infrared modes to obtain the response maps for each mode. The specific calculation method for the response maps is as follows: 1) Search for feature vectors of visible light and infrared modes. and and template image feature vector and Divided by depth along the channel direction 1 eigenvector; 2) Search the feature vector of each pair of visible light modes and infrared modes. and With template feature vector and Perform cross-correlation operations separately to generate response feature maps. and The length and width of the response feature map are represented by x and y, respectively.

[0038] 3) In the depth direction and After connection, the final response feature map of each mode is generated separately. and .

[0039] The above process can be represented as:

[0040]

[0041] in It's a 1×1 convolution, used for cross-correlation operations. express, It's a splicing operation. It refers to the number of channels.

[0042] Next, the mode of the response peak coordinates of all layers in the visible light response feature map and the infrared response feature map will be calculated respectively. and The mode, rather than the mean, is chosen as the matching center. The mode can more effectively filter out some erroneous peaks caused by noise or other interference factors, thereby improving the stability and accuracy of spatial reconstruction.

[0043] After obtaining the matching center, use the matching center corresponding to the infrared image. Using the origin as the reference point, subtract the coordinates of the matching center of the visible light image. This yields a translation vector. Using this translation vector to analyze the visible light characteristic response map and By performing appropriate translation operations or edge padding, the two objects are finally registered, resulting in the registered feature response map. and .

[0044] S2: Learnable dataflow gating module The Learnable Data-flow Gating Module (LDG) is a mechanism that evaluates the quality of each modality based on the response distribution output by the decision-level response guidance module. It also automatically extracts discriminative cues from the feature response map through joint learning of convolutional neural networks and fully connected layers, thereby enabling intelligent judgment and selection of the reliability of different modalities.

[0045] The specific implementation process of LDG is as follows: 1) The characteristic response map output by the decision-level response guidance module. and The inputs are fed into 1×1 convolutions for channel dimensionality reduction, followed by GAP compression of the spatial dimension to reduce subsequent computational complexity. After these two processing steps, modal quality feature vectors corresponding to the visible and infrared modes are extracted. and ; 2) Combine the two feature vectors mentioned above and The data are fed into a classification network consisting of three fully connected layers, and the output is normalized using a sigmoid activation function. The network is then trained using a binary cross-entropy loss function to obtain the reliability score for each modality. and ; 3) Determine the tracking path based on the obtained probability values: when and If either of these values ​​is greater than or equal to 0.5, the single modality with the corresponding high confidence level is selected and used directly for target tracking. Conversely, if both values ​​are less than 0.5, it indicates that the current scenario is highly complex or challenging, and a single modality is not robust enough to guarantee the tracking performance. In this case, the system automatically switches to the fused modality to perform the tracking task.

[0046] S3: Feature-level fusion The Feature Level Fusion Module (FLF) takes as input the modal quality feature vector extracted by the LDG module. and It can reflect the ability of visible light and infrared modes to distinguish target and background information in the feature space. Based on this characteristic, in the feature-level fusion stage, the above two quality vectors are used as weight guidance signals in the channel dimension to adjust the feature interaction between visible light and infrared modes, thereby realizing cross-modal adaptive feature fusion.

[0047] The specific implementation process is as follows: 1) Obtain the search graph feature vectors of visible light and infrared modes from the decision-level response guidance module. and and the corresponding template feature vector and ; 2) From and The fusion weights of the corresponding layers are decoupled respectively. and ; 3) Multiply the search graph features and template features of each modality by the channel weights of their corresponding layers, and then concatenate them channel by channel to obtain the fused search graph feature vector. and template feature vector .

[0048] The above process can be represented as:

[0049]

[0050] S4: Tracking Head To reduce the number of parameters and improve computational efficiency, LGFT efficiently implements weight sharing in the tracking head. Specifically, it's based on a shared pedestal constructed from four 3×3 convolutional layers with 256 channels each. In this shared pedestal, the first three layers are followed by a ReLU activation function, while the last layer does not use an activation function. Regression, classification, and centrality branches are all implemented based on this shared pedestal, reducing redundant feature extraction.

[0051] Specifically, the classification branch outputs a classification heatmap representing the probability of the target's presence, while the centrality evaluation branch generates a centrality heatmap reflecting the confidence level of the target's center location. These two heatmaps are then fused pixel-by-pixel through multiplication to form the final joint confidence heatmap. Simultaneously, the regression branch predicts the distance offset from each spatial location on the feature map to the four sides of the target bounding box, as well as a scaling factor. During the inference phase, the system first identifies the response point with the highest confidence level in the final heatmap and, combined with the corresponding offset and scaling parameters from the regression branch, decodes the final target bounding box.

[0052] Since LGFT uses infrared imagery as its processing baseline, it employs a decision-level response guidance module to correct inconsistencies between the visible and infrared modalities in unregistered data. Therefore, the generated bounding box coordinates are based on infrared image localization, and when these bounding boxes are displayed on the visible image, they must be adjusted according to specific correction parameters. This design not only improves the accuracy of cross-modal target tracking but also ensures the high efficiency of the system's real-time processing performance.

[0053] The total loss function of LGFT consists of two main parts: the splitting loss from the learnable data stream gating and the loss from the target tracking task itself. The splitting loss is further decomposed into three sub-terms: modality priority loss, decision boundary reinforcement loss, and modality competition regularization loss. The modality priority loss is designed using weighted binary cross-entropy to address the data bias caused by the uneven distribution of effective samples between visible light and infrared modalities during the training phase. This loss term allows the model to more accurately measure the credibility of each modality in different scenarios, thereby enhancing the discriminative power and rationality of the gating mechanism in modality selection.

[0054] Modal priority loss The definition is as follows:

[0055]

[0056]

[0057] in, This indicates the number of sequences where the infrared modes are of high quality. This indicates the number of sequences where the infrared modes are of high quality. and The weights for visible light and infrared images are respectively. and They are the first Sample labels and These are the outputs of the LGD module. The probabilities of visible light mode preference and infrared mode preference in a sample. It refers to the batch size.

[0058] Fuzzy decisions with a confidence level around 0.5 need to be penalized to enhance the network's discriminative ability. The incentive model makes explicit decisions regarding visible light priority scenarios and infrared priority scenarios. Decision boundary reinforcement loss is used. It consists of two parts: fusion scene loss and single-modal scene loss, which are defined as follows:

[0059]

[0060] in It is a logical OR operation, when the first... The value is 1 if the sample is selected based on visible light priority or infrared priority, and 0 otherwise. For the first The dominant mode probability of each sample.

[0061] The modality competition regularization loss is designed to enhance the model's ability to distinguish between the "visible light priority" and "infrared priority" decision paths. When the system selects the fused mode as the final output, this loss term is automatically set to zero and does not participate in the optimization. Its mathematical definition is as follows: Modal competition regularization loss The definition is as follows:

[0062] in It is a symbolic function.

[0063] Total loss of learnable data stream gating module as follows:

[0064] in , and The weights are used to balance the three losses; in this chapter, we select the following weights respectively. , and .

[0065] The tracking loss consists of three parts: classification loss, regression loss, and centrality loss. The classification loss uses the standard cross-entropy loss function to supervise the confidence prediction of the target center point. However, when dealing with weak targets or cases with large localization errors, the intersection between the predicted and ground truth boxes may be extremely small or even zero. This causes the regression loss based on traditional IoU to fail to provide an effective gradient signal, leading to the vanishing gradient problem and hindering model optimization.

[0066] To address this issue, LGFT introduces the Generalized Intersection over Union (GIoU) ​​loss into the regression loss. The GIoU is calculated as follows:

[0067] Where A is the predicted bounding box, B is the ground truth bounding box, and C is the smallest rectangle that can completely contain both A and B. The calculation method is as follows:

[0068] in and These are the predicted value and the actual value, respectively. Centrality loss. The binary cross-entropy loss is used for calculation, and the calculation method is as follows:

[0069] in, This represents the predicted value output by the centrality branch. The actual value represents the data set during annotation, and its calculation method is as follows:

[0070] in, , , and Indicates respectively The distance of the point from the left, right, top, and bottom bounding boxes of the real target.

[0071] Total loss function It can be represented as:

[0072] in, and These are hyperparameters used for various losses, and are obtained through network search for parameters.

[0073] During network training, a phased, progressive training strategy is adopted to improve model convergence stability and optimize the collaborative performance of each module.

[0074] like Figure 2 As shown, firstly, visible light and infrared templates, along with the search image, are input into a MobileNet backbone network with shared weights to extract their respective features. These features undergo deep cross-correlation in the Decision-Level Response Guidance (DRG) module to generate a response map reflecting the target location. This module calculates the mode of the response peaks to align the response maps of the two modalities, resolving the spatial mismatch problem.

[0075] Next, the aligned response map is fed into the Learnable Data Stream Gated (LDG) module. The LDG evaluates and outputs reliability scores for both the visible and infrared modalities. This is a dynamic decision center that determines, based on the scores, which modality is more reliable in the current scene, or whether it is necessary to fuse the information from both modalities.

[0076] Finally, based on the LDG's decision, the system selects the optimal path: if a modality's score is high enough, the original features of that modality are used directly; if both scores are low, the Feature-Level Fusion (FLF) module is activated to fuse the original features of the two modalities according to their weights. Ultimately, the selected features are input into a shared-weights lightweight tracking head (HEAD), which decodes the final target bounding box through three branches: classification, regression, and centrality.

[0077] The first stage is the backbone network warm-up stage: the learnable dataflow gating module and the tracking head are frozen, and only the backbone network is fine-tuned. A stochastic gradient descent (SGD) optimizer is used with an initial learning rate of 10⁻⁴, and training is performed for 10 epochs to allow the backbone network to acquire preliminary adaptive feature representations for the target task. The second stage focuses on training the gating module: at this stage, the backbone network and tracking head are frozen, and only the parameters of the learnable dataflow gating module are updated. The optimizer is switched to AdamW, with weight decay set to 10⁻⁵ and a learning rate of 10⁻⁵, and training is performed for 30 epochs to fully learn modality quality assessment and dynamic selection capabilities. The third stage performs end-to-end joint fine-tuning: all network modules, including the backbone network, gating module, and tracking head, are unfrozen, and the SGD optimizer is used for overall optimization with an initial learning rate of 5-6. Training continues for 10 epochs, allowing all components to collaboratively optimize under a unified goal, ultimately obtaining a high-performance and efficient RGBT tracking model.

[0078] like Figure 3 As shown, for embedded multi-core NPU platforms such as Rockchip RK3588, this invention proposes the following four-level optimization scheme: 1) At the task deployment level, the forward inference tasks of the search area image and the template image in the backbone network are handled independently by NPU0 and NPU1 respectively; at the same time, NPU2 is dedicated to the computation of the learnable data stream gating module (LDG) and the subsequent tracking head.

[0079] 2) In terms of data flow architecture design, the system adopts a dual-stream asynchronous pipeline mechanism: NPU0 and NPU1 act as feature generation units, processing visible light and infrared modal feature extraction tasks in parallel, respectively; while NPU2 acts as a downstream consumer, responsible for receiving these two feature streams and performing subsequent operations such as fusion, decision-making, and bounding box regression. To ensure efficient collaboration among multi-cores, a lock-free atomic synchronization scheme based on buffer readiness flags is introduced. This mechanism allows NPU0 and NPU1 to immediately switch to the next frame after completing feature calculation for the current frame, without waiting for NPU2; simultaneously, NPU2 always reads valid data from the buffer marked as "ready". This achieves non-blocking, high-throughput continuous data flow, significantly improving overall pipeline efficiency.

[0080] 3) At the memory management level, the system writes the feature maps output by NPU0 and NPU1 into a pre-allocated shared circular memory buffer. This buffer allocates a fixed-size contiguous physical memory during initialization, with sufficient capacity to hold the intermediate feature data required for processing several frames. Through a carefully designed read / write pointer mechanism, the system can reuse this fixed memory area cyclically during the processing of different video frames: once the feature map of a frame has been consumed, the space it occupies is immediately marked as overwriteable for subsequent frames.

[0081] 4) Regarding the deployment of model weights, the system uses the INT8 quantization strategy to compress the original network, and embeds the quantized weights directly into the NPU's on-chip cache as constants during the compilation phase. This significantly reduces the model size from the original 30MB to only 6MB.

[0082] Compared with existing technologies / conventional solutions, the embodiments of the present invention exhibit outstanding beneficial technical effects in the following aspects: (1) Algorithm accuracy and efficiency Comprehensive testing on four public benchmark datasets—GTOT, RGBT210, RGBT234, and LasHeR—demonstrates that LGFT achieves a competitive level of tracking accuracy (success rate and precision) comparable to current state-of-the-art algorithms such as TBSI and QAT. Its speed performance is particularly outstanding: on an NVIDIA RTX 3080Ti GPU, it achieves 135.5 FPS on the RGBT234 dataset, significantly outperforming similar precision algorithms. On a self-built spatial unregistered dataset, LGFT achieves a success rate of 52.9% and a precision of 63.7%, significantly outperforming comparative algorithms without a specific design, demonstrating its strong robustness to unregistered data.

[0083] (2) Real-time performance of embedded deployment On the Rockchip RK3588 embedded platform, after the aforementioned four-level optimization deployment, LGFT achieved a real-time tracking speed of up to 29.6 FPS. The average utilization of the three NPU cores exceeded 90%, and the model size was compressed to approximately 6MB. This demonstrates that LGFT and its deployment scheme can effectively meet the stringent requirements of edge devices for low power consumption and high real-time performance.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visible-infrared target tracking method based on learnable gating fusion, characterized in that, Includes the following steps: Step S1: Through the decision-level response guidance module, feature extraction and deep cross-correlation are performed on the search map and template map of visible light and infrared modes to obtain the response map of each mode; The matching center is calculated based on the mode of the response peak coordinates of all channels in the response map, and the visible light response map is spatially registered with the matching center of the infrared mode as the reference to obtain the registered response map. Step S2: Using the learnable data stream gating module, extract the modal quality feature vectors of the registered response map and predict the reliability score of each modality; dynamically select a single-modality path or a fusion path based on the comparison result of the reliability score and a preset threshold; when the reliability score of any modality is greater than or equal to the threshold, select the original features of that modality for target tracking; when the reliability scores of both modalities are less than the threshold, activate the feature-level fusion module. Step S3: Through the feature-level fusion module, in the fusion path, based on the channel weights decoupled from the modal quality feature vector, the original search features and template features are channel-weighted and concatenated to obtain the fused search features and template features; Step S4: Input the original features in the single-modal path or the fused features in the fusion path into the lightweight tracking head with shared weights. The lightweight tracking head includes a convolutional layer with a shared base as well as regression branches, classification branches and centrality branches, and outputs the target bounding box parameters.

2. The visible-infrared target tracking method based on learnable gating fusion according to claim 1, characterized in that, In step S1, the mode matching based on response peak values ​​specifically includes: calculating the mode of the response peak coordinates of all channels in the visible light response map and the infrared response map, and using them as the visible light matching center and the infrared matching center, respectively; subtracting the coordinates of the visible light matching center from the infrared matching center as the origin to obtain a translation vector; and using this translation vector to perform translation or edge filling operations on the visible light response map to achieve spatial alignment with the infrared response map.

3. The visible-infrared target tracking method based on learnable gating fusion according to claim 1, characterized in that, Step S2 specifically includes the following process: channel dimensionality reduction of the registered response map is performed by 1×1 convolution, and then the spatial dimension is compressed by global average pooling to obtain the modal quality feature vector; the modal quality feature vector is input into a classification network consisting of three fully connected layers, and a reliability score is output using the Sigmoid activation function.

4. The visible-infrared target tracking method based on learnable gating fusion according to claim 1, characterized in that, In step S2, the preset threshold is 0.

5.

5. The visible-infrared target tracking method based on learnable gating fusion according to claim 1, characterized in that, In step S3, the channel weighted fusion specifically includes: decoupling the fusion weight scalar of each channel from the modal quality feature vector; multiplying the original search feature and original template feature of each modality by the fusion weight scalar of its corresponding channel; and concatenating the weighted visible light and infrared features along the channel dimension to obtain the fused search feature and template feature.

6. The visible-infrared target tracking method based on learnable gating fusion according to claim 1, characterized in that, Step S4 specifically includes the following process: a shared base consisting of four 3×3 convolutional layers, with the first three layers followed by a ReLU activation function and the fourth layer having no activation function; the regression branch, classification branch, and centrality branch share the features extracted by the first three convolutional layers, and output a classification heatmap, a centrality heatmap, and bounding box regression parameters; wherein, the centrality heatmap and the classification heatmap are multiplied pixel by pixel to obtain a joint confidence heatmap, which is used to determine the center position of the target.

7. The visible-infrared target tracking method based on learnable gating fusion according to claim 1, characterized in that, The method also includes a training process for the total loss function, which is composed of a weighted gating loss L_gate and a tracking loss L_track. The gating loss L_gate includes a modality priority loss L_p, a decision boundary reinforcement loss L_b, and a modality competition regularization loss L_c. The modality priority loss uses a weighted binary cross-entropy loss calculated based on the number of high-quality visible light samples and the number of high-quality infrared samples to alleviate the imbalance in data distribution.

8. A visible-infrared target tracking method based on learnable gating fusion according to claim 7, characterized in that, The training of the total loss function adopts a phased progressive training strategy: in the first phase, the gating module and the tracking head are frozen, and only the backbone network is fine-tuned; The second phase freezes the backbone network and tracking head, updating only the gating module parameters; the third phase unfreezes all network modules for end-to-end joint fine-tuning.

9. A learnable lightweight unregistered visible-infrared target tracking system based on dynamic gating, used to implement the learnable lightweight unregistered visible-infrared target tracking method based on dynamic gating as described in any one of claims 1-8, characterized in that, include: The decision-level response guidance module is used to extract features and perform deep cross-correlation on the search graph and template graph of visible light and infrared modes to obtain the response graph of each mode; The matching center is calculated based on the mode of the response peak coordinates of all channels in the response map, and the visible light response map is spatially registered with the matching center of the infrared mode as the reference to obtain the registered response map. A learnable data stream gating module is used to extract modal quality feature vectors from the registered response map and predict the reliability score of each modality; based on the comparison result of the reliability score and a preset threshold, a single-modality path or a fusion path is dynamically selected; when the reliability score of any modality is greater than or equal to the threshold, the original features of that modality are selected for target tracking; when the reliability scores of both modalities are less than the threshold, the feature-level fusion module is activated. The feature-level fusion module is used to perform channel weighting and concatenation on the original search features and template features based on the channel weights decoupled from the modal quality feature vector in the fusion path, so as to obtain the fused search features and template features. This module is activated only when the LDG module determines that fusion is needed, and performs lightweight and effective feature integration. The shared-weight lightweight tracking head is used to input the original features from the single-modal path or the fused features from the fusion path into the shared-weight lightweight tracking head and output the target bounding box parameters.

10. A learnable, lightweight, unregistered visible-infrared target tracking system based on dynamic gating according to claim 9, characterized in that, For the embedded multi-core NPU platform, the system adopts the following embedded deployment optimization scheme: Task-level parallel scheduling: The forward inference of the backbone network of the visible light and infrared search graphs and template graphs is assigned to the first and second NPU cores, and the computation of the gating module and the tracking head is assigned to the third NPU core; Asynchronous data flow pipeline: It adopts a producer-consumer model and uses lock-free atomic flags to synchronously achieve decoupling between NPU cores and full pipeline operation; Memory reuse and management: A circular buffer is pre-allocated in shared memory to cache intermediate feature data output by each NPU core, and memory reuse is achieved by cyclically moving the read and write pointers; Model quantization and solidification: The trained model is quantized using INT8, and the quantized weights are solidified into constants during the compilation phase and embedded into the on-chip cache of the NPU.

Images