An RGBT target tracking training system, a training method and an inference method

Abstract

This application discloses an RGBT target tracking training system, training method, and inference method. The system includes a multi-layer teacher network, a lightweight student network, a first distillation branch for extracting feature statistics and task response maps to calculate alignment loss, a second distillation branch for extracting inter-word relation matrices to calculate relation loss, a relation control module for controlling the visibility of inter-word relations based on dynamically generated masks during training, and an auxiliary supervision module including a fused tracking head and a modality-specific tracking head for multi-head supervision of student features. The system avoids forced alignment at the feature level of heterogeneous networks through a decoupled dual-branch distillation strategy, ensures optimization stability by dynamically controlling relation complexity through a progressive relation exposure mechanism, and preserves single-modality discriminative features through a modality-preserving learning strategy. This invention can significantly improve inference speed while maintaining high tracking accuracy, achieving an effective balance between model accuracy and runtime efficiency.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to an RGBT target tracking method, training method, and system that uses knowledge distillation to train a lightweight network. Background Technology

[0002] Visual single-object tracking is an important task in the field of computer vision, aiming to continuously locate a target in subsequent video sequences based on its position given in an initial frame. Visible light (RGB) and thermal infrared (TIR) ​​modalities have natural complementary advantages: the RGB modality can capture rich color texture and geometric information and has strong discrimination ability in well-lit environments; the TIR modality utilizes the difference in thermal radiation between the target and the background for detection and performs stably in low-visibility scenes such as nighttime and smoke. Therefore, RGBT (Visible Light-Temperature Infrared) dual-modal target tracking technology has broad application prospects in fields such as drone monitoring, autonomous driving, and security monitoring.

[0003] However, existing RGBT tracking algorithms mostly rely on deep neural networks with a large number of parameters. Their high computational cost and storage requirements make it difficult to deploy these models on edge devices with limited computing power. To address this issue, lightweight RGBT tracking methods have gradually become a research hotspot. Current lightweight solutions mainly focus on two directions: one is to use lightweight backbone networks as feature extractors, such as compact network architectures like Mobile Vision Transformers; the other is to transfer the discriminative power of high-performance large models to lightweight small models through knowledge distillation techniques.

[0004] In the field of knowledge distillation, Chinese patent application CN120807580A discloses a fast RGBT target tracking method based on cross-modal interaction enhancement and knowledge distillation. This method constructs a student model consisting of a visible light feature extraction network, a thermal infrared feature extraction network, a cross-modal interaction enhancement network, and a multimodal tracking head. It then selects a teacher model for inter-modal and intra-modal distillation, and uses a loss function constructed from the real labels and tracking head output for training. However, when performing knowledge distillation between heterogeneous backbone networks, this method often requires the introduction of additional adapters or projection modules to align the dimensional differences between the teacher and student models. These mapping layers may not only distort the original feature distribution but also introduce additional optimization complexity and computational overhead when the architectural differences are significant.

[0005] Chinese patent application CN118887255A discloses a multimodal target tracking method based on coupled knowledge distillation. This method performs style distillation between style features of two student branches, content distillation between teacher and student branches with the same modal input, and normalizes features using instance normalization. Simultaneously, the method randomly masks 25% of the pixels in the input image of the student branch for mask modeling. However, this method uses a fixed distillation strategy and does not consider the receptiveness of the student model at different learning stages. It requires the model to learn complex global dependencies early in the training process, which may lead to instability in the optimization process and affect the final model performance.

[0006] Furthermore, existing single-stream architecture RGBT tracking methods typically perform modal fusion in the early stages of feature extraction. While this can improve computational efficiency, it can also lead to the masking of modal-specific cues, resulting in the loss of texture details in the RGB modal and thermal radiation information in the TIR modal, thus affecting tracking performance.

[0007] Therefore, a new lightweight RGBT target tracking training method is needed to achieve efficient knowledge transfer between heterogeneous backbone networks, avoid feature distortion and computational overhead caused by additional projection layers, control the complexity of relational dependencies through a progressive learning strategy to ensure the stability of the training process, and effectively preserve the discriminative features of the two modes in a single-stream architecture, thereby achieving ultra-fast inference while maintaining high tracking accuracy. Summary of the Invention

[0008] The technical problem to be solved by this invention is that in the prior art, knowledge distillation between heterogeneous backbone networks requires an additional projection layer, which leads to feature distortion and computational overhead; early fusion in single-stream architecture masks modality-specific cues, resulting in information loss; and directly exposing all relational dependencies imposes an excessive learning burden on lightweight models, leading to training instability.

[0009] The present invention solves the above-mentioned technical problems through the following technical means:

[0010] An RGBT target tracking training system includes a teacher network and a student network for feature extraction, wherein the capabilities of the teacher network are transferred to the student network through knowledge distillation, and further includes:

[0011] The teacher network is a multi-layer Vision Transformer network used to receive visible light modal images and thermal infrared modal images respectively, and extract dual-modal features;

[0012] The student network is a lightweight Vision Transformer network used to receive the fusion input of the visible light modal image and the thermal infrared modal image and extract the fusion features;

[0013] The first distillation branch is used to extract the feature statistics and task response map of the teacher network, extract the feature statistics and task response map of the student network, calculate the difference in feature statistics and task response map between the teacher network and the student network, and generate the first distillation loss.

[0014] The second distillation branch is used to extract the feature word relation matrix of the teacher network, extract the feature word relation matrix of the student network, calculate the difference between the word relation matrices of the teacher network and the student network, and generate the second distillation loss.

[0015] The relation control module is used to generate a dynamic mask based on the current training stage, and to control the visibility range of relations between terms in the second distillation branch through the dynamic mask;

[0016] The auxiliary supervision module includes a fusion tracking head and at least two modality-specific tracking heads. The fusion tracking head is used to track and supervise the fusion features of the student network, and the modality-specific tracking head is used to track and supervise the modality splitting features of the student network to generate a modality preservation loss.

[0017] Preferably, the teacher network includes a cross-modal interaction module, which is located in the middle layer of the teacher network and is used to interactively fuse the features of the visible light modal image and the features of the thermal infrared modal image.

[0018] Preferably, the teacher network has 8 to 16 layers, and the student network has 4 to 8 layers.

[0019] Preferably, the first distillation branch extracts feature statistics by: calculating the standard deviation of the channel dimension for each word in the feature matrix, and aggregating the standard deviations of all words to obtain a word aggregated channel descriptor.

[0020] Preferably, the auxiliary supervision module includes a fusion tracking head and two modality-specific tracking heads, wherein the two modality-specific tracking heads are a visible light modality tracking head and a thermal infrared modality tracking head, respectively.

[0021] Preferably, the teacher network is a 12-layer Vision Transformer network, and the student network is a 6-layer Vision Transformer network.

[0022] Preferably, the teacher network and the auxiliary supervision module are removed during the inference phase, leaving only the student network to perform the target tracking task.

[0023] An RGBT target tracking training method includes the steps of constructing a teacher network and a student network, and inputting an image into the network to extract features for knowledge distillation training. The method is characterized by further including the following steps:

[0024] Construct a multi-layered Vision Transformer teacher network and a lightweight Vision Transformer student network;

[0025] The visible light modal image and the thermal infrared modal image are input into the teacher network and the student network, and teacher features and student features are extracted respectively;

[0026] Extract the statistics and task response maps of the teacher features, extract the statistics and task response maps of the student features, calculate the statistical differences and task response map differences, and generate the first distillation loss.

[0027] Extract the word relation matrix of the teacher features, extract the word relation matrix of the student features, calculate the difference between the word relation matrices, and generate the second distillation loss;

[0028] A dynamic mask is generated based on the current training progress, and the second distillation loss is weighted by the mask to dynamically control the learning complexity of the relationship between words.

[0029] Modality segmentation is performed on the student features, and joint supervision is conducted by fusing tracking heads and modality-specific tracking heads to generate modality preservation loss;

[0030] The student network is trained by jointly optimizing the first distillation loss, the second distillation loss, the mode preservation loss, and the tracking task loss.

[0031] Preferably, the step of extracting feature statistics includes: calculating the standard deviation of the channel dimension for each word in the feature matrix, and aggregating the standard deviations of all words to obtain a word aggregated channel descriptor.

[0032] Preferably, the learning complexity of the dynamically controlled inter-lexical relationships includes: in the early stage of training, the dynamic mask only exposes intra-frame inter-lexical relationships, which include intra-modal inter-lexical relationships and cross-modal inter-lexical relationships; in the later stage of training, the dynamic mask expands to expose inter-frame inter-lexical relationships, which include time-series intra-modal inter-lexical relationships and time-series cross-modal inter-lexical relationships.

[0033] Preferably, in the calculation of the first distillation loss, the weighting coefficient of the statistical difference is 0.0001 to 0.001, and the weighting coefficient of the task response map difference is 50 to 200.

[0034] Preferably, the statistic is the characteristic standard deviation.

[0035] Preferably, the training dataset includes the LasHeR dataset, and the optimizer is the AdamW optimizer.

[0036] An RGBT target tracking inference method includes the steps of acquiring a trained target tracking network and inputting the image of the current frame into the network to determine the target location, characterized by further including the following steps:

[0037] Obtain a lightweight Vision Transformer target tracking network trained by knowledge distillation;

[0038] The visible light modal image and thermal infrared modal image of the current frame are input into the target tracking network to extract fused features;

[0039] The fused features are processed by the tracking head of the target tracking network to generate a tracking response map;

[0040] The target location and bounding box are determined based on the tracking response map, and the target tracking result is output.

[0041] Preferably, the inference speed of the inference method is greater than or equal to 500 frames per second.

[0042] The beneficial effects of this invention are:

[0043] (1) By using the dual-branch distillation strategy of architecture decoupling, the teacher supervision is decomposed into architecture-independent branches (based on feature statistics and response graphs) and architecture-related branches (based on word relation matrix), which avoids the additional projection layer required for forced alignment of features between heterogeneous networks, eliminates feature distortion and computational overhead caused by projection layer, and significantly improves the inference speed of student model compared with teacher model.

[0044] (2) By using a progressive relationship exposure mechanism and a course learning strategy to dynamically control the visibility of relationship dependencies, only intra-frame relationships (intra-modal and cross-modal interactions) are exposed in the early stage of training, and then extended to inter-frame temporal relationships in the later stage. This ensures the stability of the lightweight model optimization process. Compared with the baseline method that directly exposes all relationships, the training stability is significantly improved, the convergence speed is faster, and the final accuracy is higher.

[0045] (3) By using a modality-preserving learning strategy, an auxiliary tracking head is introduced during the training phase to force the preservation of modality-specific features of RGB and TIR, which effectively solves the problem of modality information loss caused by early fusion in single-stream architecture. The auxiliary component is removed during inference to achieve zero additional overhead.

[0046] (4) Comprehensive tests on multiple mainstream RGBT tracking benchmark datasets show that the student model of the present invention achieves ultra-fast inference while maintaining high tracking accuracy. Compared with similar lightweight methods, the inference speed is greatly improved and the accuracy loss is small, achieving the optimal balance between accuracy and speed, and providing an effective solution for real-time RGBT tracking on edge devices. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall architecture of the RGBT target tracking training system in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the progressive relationship exposure mechanism in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This embodiment provides an RGBT target tracking training system and training method, such as Figure 1 As shown, it includes a teacher network, a student network, a first distillation branch (architecture-independent branch), a second distillation branch (architecture-dependent branch), a relationship control module (progressive relationship exposure), and an auxiliary supervision module (modality preservation learning).

[0051] The teacher network employs a 12-layer Vision Transformer-Based (ViT-Base) two-stream architecture. Each layer includes a multi-head self-attention mechanism and a feedforward neural network, with a hidden dimension of 768 and 12 attention heads. The teacher network receives visible light modal images (RGB images) and thermal infrared modal images (TIR images) as input, both with a size of 256×256 pixels. The RGB and TIR images are first processed through a patch embedding layer, dividing the image into 16×16 patches. Each patch is mapped to a 768-dimensional word vector, generating a total of 256 words (including 1 classification word and 255 image patch words). The teacher network introduces a cross-modal interaction module in the middle layer (layer 6), fusing RGB and TIR features through a cross-attention mechanism, allowing the features of the two modalities to mutually enhance each other. The output of the teacher network is the fused bimodal features, with a feature dimension of [missing information - likely a missing element]. B × N ×C ,in B For batch size, N The number of lexical units (256). C The number of feature channels is 768.

[0052] The student network employs a 6-layer Vision Transformer-Tiny (ViT-Tiny) single-stream architecture, with each layer containing a multi-head self-attention mechanism and a feedforward neural network. The hidden dimension is 192, and the number of attention heads is 3. The student network receives a fused input of visible light modal images and thermal infrared modal images; that is, the RGB and TIR images are concatenated along the channel dimension before being input into the network. The input image size for the student network is also 256×256 pixels, which, after processing by the patched embedding layer, generates 256 192-dimensional word vectors. The output of the student network is the fused feature, with a feature dimension of [missing information]. B × N × C ,in B For batch size, N The number of lexical units (256). C The number of feature channels is 192.

[0053] The training process used the LasHeR dataset, which contains 1224 sequences (over 1.4 million frames). During training, visible light modal images and thermal infrared modal images were randomly sampled from the dataset. Each image pair contained a template frame and a search frame. The template frame provided initial appearance information of the target, and the search frame was used to locate the target's position in subsequent frames. The AdamW optimizer was used for training, with a learning rate of 1×10^-4, a batch size of 64, and 35 training epochs. The first 25 epochs were the first stage, and the last 10 epochs were the second stage, used to complement the gradual relationship exposure mechanism.

[0054] The first distillation branch (architecture-independent branch) extracts feature statistics and task response maps from the teacher and student networks, and calculates the alignment loss. Feature statistics are calculated using token-aggregated channel descriptors. Specifically, the steps involve fusing features from the teacher network... ,in B For batch size, N The number of lexical units, C To determine the number of feature channels, first calculate the standard deviation of each word in the channel dimension. For the th... i Each word element is used to calculate the channel average. = , Indicating the convergence characteristics of teacher networks The i The first word element k The values ​​for each channel are then calculated, and the variance is then calculated. σ i 2 = Finally, the standard deviation is obtained from the variance of all lexical units, and then aggregated to obtain the statistical measure. ,in k Indicates the channels, from the first channel to the second. C One channel, C The total number of channels is 768. This relates to the integration characteristics of the student network. The statistic std( is obtained using the same calculation method) Task Response Diagram and The response maps are output by the tracking heads of the teacher network and student network, respectively. The response map size is H'×W' (e.g., 16×16), and the value at each location represents the confidence that location is the center of the target. First distillation loss. The calculation formula is:

[0055]

[0056] in L For network layers, For the current layer, and These represent the first and second parts of the teacher network and student network, respectively. Layer fusion characteristics, and As a balance coefficient, this embodiment sets and .

[0057] The second distillation branch (architecture-related branch) is used to extract the inter-lexical relation matrix between the teacher and student networks and calculate the relation loss. The inter-lexical relation matrix is ​​obtained through autocorrelation calculation, specifically by fusing features from the teacher network. ,in B For batch size, N The number of lexical units, C Calculate the relation matrix for the number of feature channels. relation matrix Dimensions ,in B For batch size, N This represents the number of lexical units. Each element... The first characteristic representing the convergence of teacher networks i The word element and the first j Semantic affinity of shared channels among lexical units. Fusion characteristics of student networks. The relation matrix is ​​obtained using the same calculation method. Second distillation loss The calculation formula is: .

[0058] The relation control module (progressive relation exposure) is used to generate dynamic masks based on the current training stage, controlling the visibility range of relations between terms in the second distillation branch. For example... Figure 2 As shown, the progressive relation exposure mechanism divides the training process into two phases: batch size 64, training 35 rounds, with the first 25 rounds in the first phase and the last 10 rounds in the second phase. Dynamic mask M It is a binary matrix. N M is the number of lexical units, where M {i,j} =1 indicates the first i The word element and the first j The relationships between individual lexical units are exposed, M {i,j} =0 indicates that the relation is masked. In the first stage, the mask only assigns a value of 1 to intra-frame relations, which include intra-modal lexical relations (between template frame lexicals of the same modality, and between search frame lexicals of the same modality) and cross-modal lexical relations (between RGB template frame lexicals and TIR template frame lexicals, and between RGB search frame lexicals and TIR search frame lexicals). In the second stage, the mask is extended to inter-frame relations, which include cross-temporal intra-modal lexical relations (between template frame lexicals and search frame lexicals of the same modality) and cross-temporal cross-modal lexical relations (between RGB template frame lexicals and TIR search frame lexicals, and between TIR template frame lexicals and RGB search frame lexicals). The second distillation loss is redefined as the mask-weighted relation difference: , where ⊙ represents the Hadamard product (element-by-element product).

[0059] The auxiliary supervision module (modality preservation learning) includes one fusion tracking head and two modality-specific tracking heads (a visible light modality tracking head and a thermal infrared modality tracking head). The fusion tracking head operates on the fusion features of the student network. Output fused response map R fused Modality-specific tracking head applied to modality segmentation features of student networks. and Modality splitting features are obtained by dividing the fused features along the lexical dimension, i.e. For fusion features The former N / 2 word units, For the Empress N / 2 words. Visible light modal tracking head output RGB response diagram R rgb TIR response diagram of thermal infrared mode tracking head output R tirAll tracking heads use the standard tracking loss function. Supervision is performed, and the tracking loss function includes classification loss (cross-entropy loss) and regression loss ( Loss). Mode preservation loss. The calculation formula is:

[0060]

[0061] Indicates ground-truth annotation. and These represent the outputs of the student's fused, RGB, and TIR tracking heads, respectively. Through joint supervision of the fused representation and its modality-specific subspace, MPL guides the backbone network to retain discriminative cues from each modality. During inference, all modality-specific tracking heads are removed, thus avoiding additional computational overhead. The training process jointly optimizes the student network's parameters by minimizing the overall loss function. After training, the student network performs well on the LasHeR test set and also achieves high accuracy and success rate on the RGBT234 test set.

[0062] As shown in Table 1, the performance comparison data of the AKD-Student model in this embodiment with other mainstream RGBT tracking methods shows that the present invention achieves ultra-fast inference while maintaining high tracking accuracy.

[0063] Table 1 Comparison of Method Results

[0064]

[0065] Note: The test conditions were based on the single pass evaluation (OPE) protocol, and the test platform was an NVIDIA RTX4090 GPU, PyTorch 1.12 framework, and CUDA 11.6.

[0066] As can be seen from Table 1, the AKD student model in this embodiment achieved good performance on multiple datasets. Compared with other methods that are also lightweight backbone networks, both accuracy and success rate were improved.

[0067] As shown in Table 2, the test conditions are the same as in Table 1. The performance comparison on the LasHeR dataset further verifies the technical advantages of the present invention.

[0068] Table 2 Comparison of Method Results

[0069]

[0070] As shown in Table 2, the AKD student model in this embodiment achieves high inference speed on the LasHeR dataset while maintaining good tracking accuracy. Compared to other methods using lightweight backbone networks, the inference speed is significantly improved. Compared to the teacher model, the student model exhibits a substantial increase in inference speed, with accuracy loss remaining within an acceptable range.

[0071] The mechanisms underlying the aforementioned technical effects are analyzed as follows: First, the architecture-decoupled dual-branch distillation strategy decomposes teacher supervision into architecture-independent branches (based on feature statistics and response maps) and architecture-related branches (based on the lexical relation matrix), avoiding the additional projection layers required for forced feature-level alignment between heterogeneous networks and eliminating feature distortion and computational overhead caused by projection layers. Second, the progressive relation exposure mechanism uses a course learning strategy to dynamically control the visibility of relation dependencies. In the early stages of training, only intra-frame relations are exposed, and later, inter-frame relations are expanded, ensuring the stability of the lightweight model optimization process and avoiding gradient oscillations caused by premature exposure to complex relations. Third, the modality-preserving learning strategy introduces an auxiliary tracking head during the training phase to forcibly preserve the modality-specific features of RGB and TIR, effectively solving the problem of modality information loss caused by early fusion in single-stream architectures. During inference, the auxiliary components are removed to achieve zero additional overhead.

[0072] Example 2

[0073] This embodiment, based on embodiment 1, further describes in detail the statistical calculation method for architecture-independent branches, the method for extracting the relation matrix of architecture-related branches, the stage division for progressive relation exposure, and the configuration of the tracking head for modality-preserving learning.

[0074] The difference from Embodiment 1 lies in that this embodiment details the specific implementation of the cross-modal interaction module of the teacher network. The cross-modal interaction module is located at layers 3, 6, and 9 of the teacher network, employing a cross-attention mechanism to fuse RGB and TIR features. Specifically, RGB features are used as queries, and TIR features as keys and values, respectively, to obtain RGB enhanced features through multi-head attention computation; similarly, TIR features are used as queries, and RGB features as keys and values, to obtain TIR enhanced features. The RGB and TIR enhanced features are residually concatenated with the original RGB and TIR features, respectively, and then concatenated to form the fused features.

[0075] This embodiment details the steps for extracting statistics for architecture-independent branches. For the fusion features of the teacher network, statistics extraction uses word aggregation and channel descriptor calculation to determine the feature standard deviation. The specific steps are: first, calculate the mean of the channel dimension for each word; second, calculate the channel standard deviation for each word; third, aggregate the standard deviations of all words to obtain the statistics. The same calculation method is used to obtain statistics for the fusion features of the student network. The statistics reflect the activation intensity distribution of features along the channel dimension and are insensitive to the semantic and embedding dimensions of specific channels, thus enabling stable knowledge transfer between heterogeneous backbone networks.

[0076] This embodiment details the phased division of progressive relationship exposure. For example... Figure 2 As shown, the progressive relation exposure mechanism divides the training process into two phases: a batch size of 64 and 35 training rounds. The first 25 rounds constitute the first phase, and the last 10 rounds constitute the second phase, expanding the exposure of inter-frame word relationships. Intra-frame word relationships include intra-modal word relationships and cross-modal word relationships. Intra-modal word relationships refer to the relationships between template frame words of the same modality and the relationships between search frame words of the same modality. Cross-modal word relationships refer to the relationships between RGB template frame words and TIR template frame words, and the relationships between RGB search frame words and TIR search frame words. Inter-frame word relationships include cross-temporal intra-modal word relationships and cross-temporal cross-modal word relationships. Cross-temporal intra-modal word relationships refer to the relationships between template frame words and search frame words of the same modality. Cross-temporal cross-modal word relationships refer to the relationships between RGB template frame words and TIR search frame words, and the relationships between TIR template frame words and RGB search frame words.

[0077] This embodiment details the tracking head configuration of the auxiliary supervision module. The auxiliary supervision module includes one fusion tracking head and two modality-specific tracking heads: a visible light modality tracking head and a thermal infrared modality tracking head. The fusion tracking head operates on the fusion features of the student network, mapping these features to a response map through a fully connected layer. Each position in the response map represents the confidence that the position is the center of the target. The visible light modality tracking head operates on the modality splitting features of the student network, mapping them to an RGB response map through a fully connected layer. The thermal infrared modality tracking head operates on the modality splitting features of the student network, mapping them to a TIR response map through a fully connected layer.

[0078] The technical effects of this embodiment are the same as those of Embodiment 1. By using the architecture-decoupled dual-branch distillation strategy, the progressive relation exposure mechanism, and the modality-preserving learning strategy, the optimal balance between high accuracy and ultra-fast inference is achieved.

[0079] Example 3

[0080] Based on Example 1, this embodiment further compares the performance under different network layer configurations, verifying that a 12-layer teacher network and a 6-layer student network are the optimal configurations.

[0081] The difference from Example 1 is that this example tested different combinations of teacher network layers (8, 10, 12, and 16 layers) and student network layers (4, 6, and 8 layers). Each configuration was trained using the training method described in Example 1 for 35 epochs, with other training parameters remaining consistent. Test results show that:

[0082] When the teacher network has 8 layers and the student network has 4 layers, the student model has a faster inference speed, but relatively lower accuracy and success rate. When the teacher network has 10 layers and the student network has 6 layers, the performance of the student model improves. When the teacher network has 12 layers and the student network has 6 layers (i.e., the configuration in Example 1), the student model achieves an optimal balance between accuracy, success rate, and inference speed. When the teacher network has 16 layers and the student network has 8 layers, the accuracy and success rate of the student model further improve, but the inference speed decreases.

[0083] The comparative data above shows that as the number of layers in the teacher network increases, the accuracy and success rate of the student model gradually improve, but the inference speed gradually decreases. When the teacher network has 12 layers and the student network has 6 layers, the student model achieves the optimal balance between accuracy, success rate, and inference speed. Therefore, a 12-layer teacher network and a 6-layer student network are the optimal configuration.

[0084] This embodiment further tests the performance under different loss weight coefficient configurations, verifying that a reasonable weight configuration can improve model performance. The performance of λ was tested. std Within the ranges of 0.0001, 0.001, and 0.01, λ score Different combinations within the ranges of 50, 100, and 200 show that: when λ std When λ is too small, the contribution of characteristic statistic distillation is relatively small; when λ is too small... std When λ is moderate, the model performs well; when λ is moderate, the model performs well. std When λ is too large, the weight of feature statistics distillation becomes excessive, leading to optimization instability. score When λ is small, the contribution of distillation to the response diagram is relatively small; when λ is small... score When λ is moderate, the model performs well; when λ is moderate, the model performs well. score When the weight is too large, the weight of the response map distillation becomes excessive, which may affect the optimization of other losses. Therefore, appropriately selecting the loss weight coefficients can improve model performance.

[0085] The technical effect of this embodiment shows that by reasonably selecting the network layer configuration and loss weight coefficient, ultra-fast inference can be achieved while maintaining high tracking accuracy, verifying the effectiveness and robustness of the method of the present invention.

[0086] Example 4

[0087] This embodiment, based on embodiment 1, describes in detail the inference deployment process of the student network after training.

[0088] The difference from Embodiment 1 is that this embodiment focuses on describing the operational steps of the inference phase. During the inference phase, the teacher network and auxiliary supervision module (including the visible light modal tracking head and the thermal infrared modal tracking head) are removed, leaving only the student network and the fusion tracking head to perform the target tracking task.

[0089] The inference method includes the following steps: First, obtain a lightweight VisionTransformer target tracking network trained with knowledge distillation, i.e., a 6-layer ViT-Tiny student network. Second, input the visible light modal image and thermal infrared modal image of the current frame into the student network. The input image size is 256×256 pixels. The visible light modal image and thermal infrared modal image are concatenated along the channel dimension before being input into the network. Third, extract fusion features through the 6-layer VisionTransformer structure of the student network. Fourth, process the fusion features through the fusion tracking head to generate a tracking response map. The response map size is 16×16, and the value at each position represents the confidence that the position is the target center. Fifth, determine the target location and bounding box based on the tracking response map. The target center is determined by finding the peak position in the response map, and the width and height of the target bounding box are predicted through a regression branch. The target tracking result (target center coordinates and bounding box size) is output.

[0090] The inference method in this embodiment achieves an inference speed of over 500 frames per second (FPS) on an NVIDIA RTX 4090 GPU, significantly reducing the computational complexity of the inference phase. The number of parameters and floating-point operations in the student network are greatly reduced compared to the teacher network.

[0091] The technical effects of this embodiment demonstrate that by removing the teacher network and auxiliary supervision module, the student network achieves ultra-fast inference with zero additional overhead during the inference phase, meeting the real-time RGBT tracking requirements on edge devices. Simultaneously, the student network maintains high tracking accuracy, verifying the optimal balance between accuracy and speed achieved by the method of this invention.

[0092] Comparative Example 1 (Comparison without gradual relationship exposure)

[0093] This comparative example is used to verify the impact of the progressive relation exposure (PRE) mechanism on training stability and final accuracy.

[0094] The difference from Example 1 is that this comparative example employs a training strategy without progressive relation exposure, meaning that all inter-terminal relations (including intra-frame and inter-frame relations) are directly exposed during training, without using dynamic masking to control relation visibility. Specifically, the calculation of the second distillation loss does not involve masking weighting, where all elements of the relation matrix participate in the loss calculation. Other training parameters remain consistent with Example 1: batch size 64, 35 training epochs (first 25 epochs for the first stage, last 10 epochs for the second stage), learning rate 1×10^-4, and the optimizer AdamW.

[0095] The loss curves during training show that the training strategy without progressive relation exposure exhibits loss oscillations in the early stages, indicating training instability. In contrast, the loss curve of Example 1, which uses progressive relation exposure, decreases smoothly. Regarding convergence speed, the training strategy without progressive relation exposure takes longer to converge to a stable state, while Example 1, which uses progressive relation exposure, converges much faster.

[0096] The final test results show that the student model with the training strategy without progressive relation exposure performs worse than Example 1 which uses progressive relation exposure.

[0097] The comparative data above demonstrates that the progressive relation exposure mechanism, by dynamically controlling the visibility of inter-lexical relations—exposing only intra-frame relations in the early stages of training and expanding to inter-frame relations later—effectively avoids gradient oscillations and training instability issues caused by overly complex relation dependencies in the early optimization phase of the lightweight student model. The progressive relation exposure mechanism significantly improves training stability, shortens convergence time, and ultimately enhances the tracking performance of the student model, validating the necessity and effectiveness of this mechanism.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An RGBT target tracking training system comprising a teacher network and a student network for extracting features, wherein the capabilities of the teacher network are transferred to the student network through knowledge distillation, characterized in that, Also includes: The teacher network is a multi-layer Vision Transformer network used to receive visible light modal images and thermal infrared modal images respectively, and extract dual-modal features; The student network is a lightweight Vision Transformer network used to receive the fusion input of the visible light modal image and the thermal infrared modal image and extract the fusion features; The first distillation branch is used to extract the feature statistics and task response map of the teacher network, extract the feature statistics and task response map of the student network, calculate the difference in feature statistics and task response map between the teacher network and the student network, and generate the first distillation loss. The first distillation branch extracts feature statistics by calculating the standard deviation of the channel dimension for each word in the feature matrix and aggregating the standard deviations of all words to obtain a word aggregation channel descriptor. The second distillation branch is used to extract the feature word relation matrix of the teacher network, extract the feature word relation matrix of the student network, calculate the difference between the word relation matrices of the teacher network and the student network, and generate the second distillation loss. The relation control module is used to generate a dynamic mask based on the current training stage, and to control the visibility range of relations between terms in the second distillation branch through the dynamic mask; The auxiliary supervision module includes a fusion tracking head and at least two modality-specific tracking heads. The fusion tracking head is used to track and supervise the fusion features of the student network, and the modality-specific tracking head is used to track and supervise the modality splitting features of the student network to generate a modality preservation loss. The first distillation loss, the second distillation loss, the modality preservation loss, and the tracking task loss are jointly optimized to train the student network. 2.The RGBT target tracking training system of claim 1, wherein, The teacher network includes a cross-modal interaction module, which is located in the middle layer of the teacher network and is used to interactively fuse the features of the visible light modal image and the features of the thermal infrared modal image. 3.The RGBT target tracking training system of claim 1, wherein, The teacher network has 8 to 16 layers, and the student network has 4 to 8 layers.

4. The RGBT target tracking training system according to claim 1, characterized in that, The auxiliary supervision module includes a fusion tracking head and two modality-specific tracking heads, namely a visible light modality tracking head and a thermal infrared modality tracking head.

5. The RGBT target tracking training system according to claim 3, characterized in that, The teacher network is a 12-layer Vision Transformer network, and the student network is a 6-layer Vision Transformer network.

6. The RGBT target tracking training system according to claim 1, characterized in that, During the inference phase, the teacher network and the auxiliary supervision module are removed, leaving only the student network to perform the target tracking task.

7. An RGBT target tracking training method, comprising the steps of constructing a teacher network and a student network, and inputting an image into the network to extract features for knowledge distillation training, characterized in that, It also includes the following steps: Construct a multi-layered Vision Transformer teacher network and a lightweight Vision Transformer student network; The visible light modal image and the thermal infrared modal image are input into the teacher network and the student network, and teacher features and student features are extracted respectively; Extract the statistics and task response map of the teacher features, extract the statistics and task response map of the student features, calculate the difference in statistics and the difference in task response map, and generate the first distillation loss; the first distillation branch extracts the feature statistics by calculating the standard deviation of the channel dimension for each word in the feature matrix, and aggregating the standard deviations of all words to obtain the word aggregation channel descriptor; Extract the word relation matrix of the teacher features, extract the word relation matrix of the student features, calculate the difference between the word relation matrices, and generate the second distillation loss; A dynamic mask is generated based on the current training progress, and the second distillation loss is weighted by the mask to dynamically control the learning complexity of the relationship between words. Modality segmentation is performed on the student features, and joint supervision is conducted by fusing tracking heads and modality-specific tracking heads to generate modality preservation loss; The student network is trained by jointly optimizing the first distillation loss, the second distillation loss, the mode preservation loss, and the tracking task loss.

8. The RGBT target tracking training method according to claim 7, characterized in that, The steps for extracting feature statistics include: calculating the standard deviation of the channel dimension for each word in the feature matrix, and aggregating the standard deviations of all words to obtain a word aggregated channel descriptor; the learning complexity of dynamically controlling the relationships between words includes: in the early stage of training, the dynamic mask only exposes intra-frame word relationships, which include intra-modal word relationships and cross-modal word relationships; in the later stage of training, the dynamic mask expands to expose inter-frame word relationships, which include inter-temporal intra-modal word relationships and inter-temporal cross-modal word relationships.

9. An RGBT target tracking inference method, comprising the steps of training a target tracking network using the method described in claim 7 or 8, and inputting an image of the current frame into the network to determine the target location, characterized in that, It also includes the following steps: Obtain a lightweight Vision Transformer target tracking network trained by knowledge distillation; The visible light modal image and thermal infrared modal image of the current frame are input into the target tracking network to extract fused features; The fused features are processed by the tracking head of the target tracking network to generate a tracking response map; The target location and bounding box are determined based on the tracking response map, and the target tracking result is output.

Images