A lightweight visible-infrared target detection method based on multi-modal knowledge distillation

Abstract

This application relates to the field of computer vision technology, and specifically to a lightweight visible-infrared target detection method based on multimodal knowledge distillation. The method includes: constructing and combining a dual-branch backbone network, a multimodal efficient hybrid encoder, a multimodal cross-attention fusion module, and a decoder to obtain a target detection model; performing two-stage pre-training on the target detection model using a multimodal pre-training strategy based on visible-light and infrared modal training sets to obtain a visible-light modal pre-trained model and a multimodal pre-trained model; employing a self-distillation strategy, using the visible-light modal pre-trained model and the multimodal pre-trained model as the student model and teacher model respectively, and performing infrared frequency-sensing distillation training to obtain a trained target detection model; inputting the image to be detected into the trained target detection model to obtain the target detection result. This method improves the speed, accuracy, and robustness of target detection in complex scenes.

Description

Technical Field

[0001] The embodiments of this application relate to the field of computer vision technology, and in particular to a lightweight visible and infrared target detection method based on multimodal knowledge distillation. Background Technology

[0002] Object detection is one of the fundamental tasks in computer vision technology. Its goal is to detect objects of interest in a given image and provide their location and category information. Existing visible light object detection algorithms struggle to accurately detect objects in low-light or occluded environments. Therefore, researchers have attempted to combine visible light and infrared images. By combining the rich color and texture details of visible light images with the unaffected lighting conditions of infrared images, they aim to improve the adaptability of object detection in complex scenes and under complex conditions.

[0003] In visible and infrared target detection tasks, in addition to high requirements for detection accuracy, robustness, and all-weather perception capabilities, computational efficiency and real-time performance are also crucial. Since these tasks are typically deployed on edge devices with limited computing power, such as drones and unmanned vehicles, and applied in high-real-time scenarios such as autonomous driving and unmanned exploration, designing an algorithm that maintains both excellent detection performance and efficient inference speed has significant research value and practical implications.

[0004] Current research has focused on achieving lightweight detection of visible and infrared modalities. Most approaches have made progress in lightweight fusion modules and fine-tuning based on high-performance pre-trained models, but further optimization is still possible. For example, most existing methods rely on pre-trained weights for visible modes, failing to effectively incorporate infrared modal information. Furthermore, they rarely utilize techniques like knowledge distillation to further enhance model performance. Traditional cross-attention mechanisms, which use the multi-head self-attention structure of Transformers, introduce high computational overhead, contradicting the goal of lightweight design. Summary of the Invention

[0005] In view of this, embodiments of this application propose a lightweight visible and infrared target detection method based on multimodal knowledge distillation. The aim is to extract and fuse deep features of the visible and infrared modalities through multi-source collaborative learning, thereby making full use of useful information in different modalities to improve the speed, accuracy and robustness of target detection in complex scenes and under complex conditions.

[0006] To achieve the above objectives, embodiments of this application propose a lightweight visible-infrared target detection method based on multimodal knowledge distillation, implemented using an RT-DETR target detection model. The method includes the following steps: constructing a dual-branch backbone network composed of visible-light and infrared modal branches, where the visible-light and infrared modal branches are used to extract multi-layer features of the visible-light and infrared modes, respectively; constructing a multimodal efficient hybrid encoder composed of visible-light and infrared modal coding units, where the visible-light and infrared modal coding units are used to enhance the feature representation of the multi-layer features of the visible-light and infrared modes, respectively, to obtain enhanced features of the visible-light and infrared modes; constructing a multimodal cross-attention fusion module and an RT-DETR decoder, where the multimodal cross-attention fusion module performs cross-attention fusion of the enhanced features of the visible-light and infrared modes based on a kernel-function linear attention mechanism to obtain fused features, and the RT-DETR decoder decodes based on the fused features to obtain the target detection result; and using a dual-branch backbone network... A target detection model is constructed by combining a network, a multimodal efficient hybrid encoder, a multimodal cross-attention fusion module, and an RT-DETR decoder. Based on visible light and infrared modal training sets, a two-stage pre-training strategy is employed to train the target detection model. In the first stage, both the visible light and infrared modal branches receive visible light training samples as input, resulting in a visible light modal pre-trained model. In the second stage, the visible light and infrared modal branches are used as the basis for the multimodal pre-trained model, receiving infrared training samples as input. A self-distillation strategy is used, with the visible light and multimodal pre-trained models serving as the student and teacher models, respectively, for infrared frequency-sensing distillation training. The infrared frequency-sensing distillation loss is incorporated into the original RT-DETR loss, ultimately yielding the trained target detection model. Pairs of visible light and infrared images to be detected are then input into the trained target detection model to obtain the target detection results output by the model.

[0007] To achieve the above objectives, embodiments of this application also propose an electronic device, including: a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement a lightweight visible-infrared target detection method based on multimodal knowledge distillation as described above.

[0008] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a lightweight visible-infrared target detection method based on multimodal knowledge distillation as described above.

[0009] In some optional embodiments, the basic backbone of the dual-branch backbone network is MobileNetV4, whose convolutional architecture is the most compact MobileNetV4ConvSmall. MobileNetV4ConvSmall reduces the number of parameters and computational complexity while maintaining feature representation capabilities through depthwise separable convolutions and a general inverse bottleneck structure. The visible light modal branch and the infrared modal branch are both sequentially connected five-layer structures. The features of the corresponding modal extracted from the third, fourth and fifth layers are output to the multimodal efficient hybrid encoder.

[0010] In some optional embodiments, the visible light mode coding unit and the infrared mode coding unit of the multimodal high-efficiency hybrid encoder are independent of each other, and the computational overhead is reduced by reducing the number of intermediate feature channels. The visible light mode coding unit performs corresponding sampling, feature expression enhancement and splicing on the multi-layer features of the visible light mode to obtain the enhanced features of the visible light mode, and the infrared mode coding unit performs corresponding sampling, feature expression enhancement and splicing on the multi-layer features of the infrared mode to obtain the enhanced features of the infrared mode.

[0011] In some optional embodiments, the multimodal cross-attention fusion module first performs layer normalization on the enhanced features of the visible light mode and the enhanced features of the infrared mode. It generates query vectors, key vectors, and value vectors for each mode through independent linear projection layers. Then, it calculates the attention in two directions: from the visible light mode to the infrared mode and from the infrared mode to the visible light mode, to achieve more complete feature fusion. The multimodal cross-attention fusion module adopts a ConvNeXt-style feedforward network design and uses depthwise separable convolutions with a large 5×5 kernel size to ensure a balance between computation and performance, ensuring that the enhanced features of the visible light mode and the enhanced features of the infrared mode can be better fused through a larger receptive field.

[0012] In some optional embodiments, the computation process based on the kernel-functionalized linear attention mechanism is as follows: First, the query vector and key vector Perform layer normalization and apply a temperature scaling factor. By stabilizing the training process, layer-normalized query vectors are obtained. and layer normalized key vectors ; ; ; in, This indicates that layer normalization is being performed. Subsequently, a kernel function with modified linear units and small constants was used to... and The query vector is corrected to ensure the non-negativity and non-linearity of the features. and the corrected key vector ; ; ; in, Indicates a corrected linear unit. This is a small constant used to avoid gradient vanishing; Finally, change the order of calculations and focus on... Decompose into two linear operations; ; in, The shape is , For batch size, The number of heads that attract self-attention. For sequence length, For dimension size, Representing a vector of all 1s with the last dimension being 1 is equivalent to... Summing the second to last dimension This is a numerically stable term used to prevent division by zero.

[0013] In some optional embodiments, in the first stage of pre-training, both the visible light modal branch and the infrared modal branch are input with visible light modal training samples to obtain initial feature representations aligned with the distribution of targets in the natural scene. After the first stage of pre-training is completed, a visible light modal pre-trained model is obtained. In the second stage of pre-training, based on the visible light modal pre-trained model, both the visible light modal branch and the infrared modal branch are input with infrared modal training samples to learn feature representations of thermal radiation information. After the second stage of pre-training is completed, a multimodal pre-trained model is obtained.

[0014] In some optional embodiments, a self-distillation strategy is employed, using the visible light modal pre-trained model and the multimodal pre-trained model as the student model and teacher model, respectively, for infrared frequency-aware distillation training. An infrared frequency-aware distillation loss is designed and added to the original RT-DETR loss. This includes: using a self-distillation strategy, the visible light modal pre-trained model is used as the student model, with its visible light modal branch and infrared modal branch inputting visible light modal training samples and infrared modal training samples, respectively; the infrared modal branch of the multimodal pre-trained model is used as the teacher model, inputting infrared modal training samples; the outputs of the teacher model and the student model are decomposed into low-frequency and high-frequency components using Haar wavelet transform, with low-frequency component alignment as the primary focus, and horizontal information from the high-frequency components introduced as a regularization constraint; an infrared frequency-aware distillation loss is designed, and finally, the infrared frequency-aware distillation loss is added to the original RT-DETR loss.

[0015] In some optional embodiments, infrared frequency sensing of distillation loss Represented as: ; ; ; in, To balance the weights, For low-frequency loss, For high-frequency level loss, The total number of feature pixels. and These are the low-frequency components of the outputs of the teacher model and the student model, respectively. and These are the high-frequency components of the outputs of the teacher model and the student model, respectively.

[0016] This application proposes a lightweight visible-infrared target detection method based on multimodal knowledge distillation. Through lightweight architecture design, efficient multimodal feature fusion, and the synergistic innovation of two-stage pre-training and infrared frequency sensing knowledge distillation, it achieves lightweight, high-precision, and high real-time performance in visible-infrared target detection. Compared with traditional multimodal target detection methods, this application has the following advantages.

[0017] First, it significantly improves inference efficiency. This application employs a lightweight dual-branch backbone network based on MobileNetV4, combined with a multimodal high-efficiency hybrid encoder with reduced coding channels, and kernel-based linear attention, reducing computational complexity from... Reduce to The number of model parameters is significantly reduced, and the inference speed is increased by at least two times. It can be directly deployed on edge devices with limited computing resources, such as drones and unmanned vehicles.

[0018] Second, it achieves more comprehensive multimodal feature fusion. The lightweight multimodal cross-attention fusion module realizes bidirectional cross-attention fusion of visible light and infrared modes. Combined with 5×5 large kernel depthwise separable convolution to expand the receptive field, it solves the problems of single fusion mode, coarse fusion granularity and high computational cost of traditional feature fusion methods, maximizes the preservation of unique features of the two modes and achieves efficient complementarity.

[0019] Third, it effectively improves the robustness of target detection. This application effectively integrates the texture detail advantages of the visible light mode with the infrared mode's unaffected by illumination and occlusion, overcoming the problem of single-modal target detection failing in low-light and occluded scenes, and improving the all-weather perception and complex scene adaptation capabilities of target detection.

[0020] Fourth, it achieves efficient cross-modal knowledge transfer and reduces training costs. This application enables the model to quickly align the data distributions of the visible light modality and the infrared modality through two-stage pre-training. The infrared frequency-aware self-distillation mainly uses low-frequency semantic alignment and is supplemented by high-frequency horizontal detail constraints, which achieves cross-modal knowledge transfer at low cost and avoids the defects of traditional distillation, such as loss of details and interference with end-to-end learning. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0022] Figure 1 This is a flowchart of a lightweight visible and infrared target detection method based on multimodal knowledge distillation provided in one embodiment of this application; Figure 2 This is a structural diagram of the target detection model provided in one embodiment of this application; Figure 3 This is a structural diagram of a multimodal cross-attention fusion module provided in one embodiment of this application; Figure 4 This is a schematic diagram illustrating the performance comparison of the method provided in one embodiment of this application with other mainstream methods on multiple datasets; Figure 5 This is a structural diagram of an electronic device provided in another embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0024] One embodiment of this application proposes a lightweight visible-infrared target detection method based on multimodal knowledge distillation, implemented by a target detection model with an RT-DETR structure. The implementation details of the lightweight visible-infrared target detection method based on multimodal knowledge distillation proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0025] The specific process of the lightweight visible-infrared target detection method based on multimodal knowledge distillation proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 11: Construct a dual-branch backbone network consisting of a visible light mode branch and an infrared mode branch. The visible light mode branch and the infrared mode branch are used to extract multi-layer features of the visible light mode and the infrared mode, respectively.

[0026] In its implementation, the lightweight dual-branch backbone network is the most fundamental component of the RT-DETR target detection model. The two branches of the dual-branch backbone network are the visible light mode branch and the infrared mode branch, which are used to extract multi-layer features of the visible light mode and the infrared mode, respectively.

[0027] In one example, the basic backbone of the dual-branch backbone network is MobileNetV4, and its convolutional architecture is the most compact MobileNetV4ConvSmall. MobileNetV4ConvSmall can reduce the number of parameters and computational complexity while maintaining feature representation capabilities through depthwise separable convolutions and a general inverse bottleneck structure.

[0028] In one example, the specific structure of the RT-DETR target detection model is as follows: Figure 2 As shown, both the visible light mode branch and the infrared mode branch are sequentially connected five-layer structures. The features of the corresponding modes extracted from the third, fourth and fifth layers are output to the multimodal high-efficiency hybrid encoder.

[0029] Step 12: Construct a multi-modal high-efficiency hybrid encoder consisting of a visible light mode coding unit and an infrared mode coding unit. The visible light mode coding unit and the infrared mode coding unit are used to enhance the feature representation of the multi-layer features of the visible light mode and the infrared mode, respectively, to obtain the enhanced features of the visible light mode and the infrared mode.

[0030] In practical implementation, the task of the lightweight multimodal high-efficiency hybrid encoder is to enhance feature representation. The multimodal high-efficiency hybrid encoder is equipped with independent coding units for different modes, namely visible light mode coding units and infrared mode coding units. These two units are used to enhance the feature representation of multi-layer features of the visible light mode and multi-layer features of the infrared mode, respectively, to obtain enhanced features of the visible light mode and enhanced features of the infrared mode.

[0031] In one example, such as Figure 2 As shown, the visible light mode coding unit and the infrared mode coding unit of the multimodal high-efficiency hybrid encoder are independent of each other, reducing computational overhead by decreasing the number of intermediate feature channels. The visible light mode coding unit performs corresponding sampling, feature representation enhancement, and concatenation on the multi-layer features of the visible light mode to obtain the enhanced features of the visible light mode. The infrared mode coding unit performs corresponding sampling, feature representation enhancement, and concatenation on the multi-layer features of the infrared mode to obtain the enhanced features of the infrared mode.

[0032] Step 13: Construct a multimodal cross-attention fusion module and an RT-DETR decoder. The multimodal cross-attention fusion module is used to perform cross-attention fusion on the enhanced features of the visible light mode and the infrared mode based on the kernel-function linear attention mechanism to obtain fused features. The RT-DETR decoder is used to decode based on the fused features to obtain the target detection results.

[0033] In its implementation, the lightweight multimodal cross-attention fusion module provides the object detection model with the ability to fully leverage the advantages of different modalities. By placing feature fusion before the RT-DETR decoder and employing parameterless addition operations to integrate bimodal features, the number of layers and queries in the RT-DETR decoder can be reduced, further lowering the complexity of the object detection model. The lightweight multimodal cross-attention fusion module performs cross-attention fusion of enhanced features from the visible light and infrared modalities based on a kernel-based linear attention mechanism to obtain fused features. The RT-DETR decoder then decodes based on these fused features to obtain the object detection result.

[0034] In one example, such as Figure 2As shown, the multimodal cross-attention fusion module (ECFM) first performs layer normalization on the enhanced features of the visible light mode and the infrared mode. It then generates query vectors, key vectors, and value vectors for each modality through independent linear projection layers. Next, it calculates attention in two directions: from the visible light mode to the infrared mode and from the infrared mode to the visible light mode, to achieve more comprehensive feature fusion. The multimodal cross-attention fusion module reduces the computational complexity of traditional attention from... Reduce to It is particularly suitable for processing high-resolution image features.

[0035] In one example, the multimodal cross-attention fusion module adopts a ConvNeXt-style feedforward network design, using a depthwise separable convolution with a large 5×5 kernel size to ensure a balance between computation and performance, and to ensure that the enhanced features of the visible light mode and the enhanced features of the infrared mode can be better and more fully fused through a larger receptive field.

[0036] In one example, the computation process based on the kernel-function-based linear attention mechanism is as follows (see reference). Figure 3 ).

[0037] First, for the query vector and key vector Perform layer normalization and apply a temperature scaling factor. By stabilizing the training process, layer-normalized query vectors are obtained. and layer normalized key vectors .

[0038] ; ; in, This indicates that layer normalization is being performed.

[0039] Subsequently, a kernel function with modified linear units and small constants was used to... and The query vector is corrected to ensure the non-negativity and non-linearity of the features. and the corrected key vector .

[0040] ; ; in, Indicates a corrected linear unit. This is a small constant used to avoid gradient vanishing.

[0041] Finally, change the order of calculations and focus attention. It can be decomposed into two linear operations.

[0042] ; in, The shape is , For batch size, The number of heads that attract self-attention. For sequence length, For dimension size, Representing a vector of all 1s with the last dimension being 1 is equivalent to... Summing the second to last dimension This is a numerically stable term used to prevent division by zero.

[0043] Step 14: Based on the combination of a dual-branch backbone network, a multimodal efficient hybrid encoder, a multimodal cross-attention fusion module, and an RT-DETR decoder, a target detection model is constructed.

[0044] In the specific implementation, after constructing the dual-branch backbone network, the multimodal efficient hybrid encoder, the multimodal cross-attention fusion module, and the RT-DETR decoder in sequence, the target detection model with the RT-DETR structure can be constructed by combining the dual-branch backbone network, the multimodal efficient hybrid encoder, the multimodal cross-attention fusion module, and the RT-DETR decoder.

[0045] Step 15: Based on the visible light modal training set and the infrared modal training set, a multimodal pre-training strategy is adopted to perform two-stage pre-training on the target detection model. In the first stage of pre-training, both the visible light modal branch and the infrared modal branch are input with visible light modal training samples to obtain the visible light modal pre-trained model. In the second stage of pre-training, based on the visible light modal pre-trained model, both the visible light modal branch and the infrared modal branch are input with infrared modal training samples to obtain the multimodal pre-trained model.

[0046] In the specific implementation, after constructing the target detection model with the RT-DETR structure, a two-stage pre-training strategy is adopted based on the visible light modal training set and the infrared modal training set to obtain better initial weights. In the first stage of pre-training, both the visible light modal branch and the infrared modal branch are input with visible light modal training samples to obtain the visible light modal pre-trained model. In the second stage of pre-training, based on the visible light modal pre-trained model, both the visible light modal branch and the infrared modal branch are input with infrared modal training samples to obtain the multi-modal pre-trained model.

[0047] In one example, the visible light modal training set can be the COCO dataset. In the first stage of pre-training, both the visible light modal branch and the infrared modal branch are input with visible light modal training samples from the COCO dataset to quickly obtain initial feature representations aligned with the distribution of targets in the natural scene. After the first stage of pre-training is completed, the visible light modal pre-trained model is obtained.

[0048] In one example, the infrared modal training set can be the FLIR_ADAS_V2 dataset. In the second stage of pre-training, based on the visible light modal pre-trained model, both the visible light modal branch and the infrared modal branch are input with infrared modal training samples from the FLIR_ADAS_V2 dataset to learn the feature representation of thermal radiation information. After the second stage of pre-training is completed, a multimodal pre-trained model is obtained.

[0049] Step 16: Using a self-distillation strategy, the visible light modal pre-trained model and the multimodal pre-trained model are used as the student model and the teacher model, respectively, for infrared frequency sensing distillation training. The infrared frequency sensing distillation loss is designed and added to the original RT-DETR loss to finally obtain the trained target detection model.

[0050] In the specific implementation, after completing the two-stage pre-training, we need to adopt a self-distillation strategy, using the visible light modality pre-trained model and the multimodal pre-trained model as the student model and teacher model, respectively, for infrared frequency sensing distillation training. The infrared frequency sensing distillation loss is designed and added to the original RT-DETR loss, ultimately obtaining the trained target detection model. Infrared frequency sensing distillation training aims to achieve cross-modal knowledge transfer with limited training costs. Considering that the visible light modality pre-trained model and the multimodal pre-trained model have already been sufficiently pre-trained on the visible light modality training set, the distillation process is only performed on the infrared modality branch.

[0051] We employ a self-distillation strategy, using the visible light modal pre-trained model as the student model, with its visible light and infrared modal branches inputting into the visible light and infrared modal training samples, respectively. The infrared modal branch of the multimodal pre-trained model is used as the teacher model, inputting into the infrared modal training samples. Haar wavelet transform is used to decompose the outputs of the teacher and student models into low-frequency and high-frequency components, respectively. The low-frequency components carry the main image structure and semantic information, while the high-frequency components contain edges and details. We primarily align the low-frequency components and introduce horizontal information from the high-frequency components as a regularization constraint to design an infrared frequency-aware distillation loss. Finally, this infrared frequency-aware distillation loss is added to the original loss of RT-DETR.

[0052] It is important to note that in core knowledge distillation, traditional methods often employ mean squared error loss or cosine similarity loss for feature alignment. However, mean squared error may interfere with the model's end-to-end learning of the target task, while cosine similarity is prone to losing detailed information. Considering the dominance of low-frequency information in infrared images, this embodiment chooses to perform infrared frequency-sensing distillation. Its core is to align teacher and student features in the frequency domain, focusing on matching low-frequency components while supplementing with controllable high-frequency constraints.

[0053] In one example, infrared frequency sensing distillation loss Represented as: ; ; ; in, To balance the weights, For low-frequency loss, For high-frequency level loss, The total number of feature pixels. and These are the low-frequency components of the outputs of the teacher model and the student model, respectively. and These are the high-frequency components of the outputs of the teacher model and the student model, respectively.

[0054] Step 17: Input the paired visible light image and infrared image to be detected into the trained target detection model to obtain the target detection result output by the trained target detection model.

[0055] In practice, after obtaining the trained target detection model, it can be deployed in the required scene. Then, the paired visible light and infrared images to be detected are acquired. The paired visible light and infrared images to be detected are input into the trained target detection model to obtain the target detection result output by the trained target detection model, thus achieving high-precision target detection.

[0056] In summary, this embodiment proposes a lightweight visible-infrared target detection method based on multimodal knowledge distillation. Through lightweight architecture design, efficient multimodal feature fusion, and the synergistic innovation of two-stage pre-training and infrared frequency sensing knowledge distillation, it achieves lightweight, high-precision, and high real-time performance in visible-infrared target detection. Compared with traditional multimodal target detection methods, this method has the following advantages.

[0057] First, it significantly improves inference efficiency. This embodiment uses a lightweight dual-branch backbone network based on MobileNetV4, combined with a multimodal high-efficiency hybrid encoder with reduced coding channels, and kernel-based linear attention, reducing the computational complexity from O(N^2) to O(N). The number of model parameters is significantly reduced, and the inference speed is improved by at least two times. It can be directly deployed on edge devices with limited computing resources, such as drones and unmanned vehicles.

[0058] Second, it achieves more comprehensive multimodal feature fusion. The lightweight multimodal cross-attention fusion module realizes bidirectional cross-attention fusion of visible light and infrared modes. Combined with 5×5 large kernel depthwise separable convolution to expand the receptive field, it solves the problems of single fusion mode, coarse fusion granularity and high computational cost of traditional feature fusion methods, maximizes the preservation of unique features of the two modes and achieves efficient complementarity.

[0059] Third, it effectively improves the robustness of target detection. This embodiment combines the advantages of visible light mode in terms of texture detail with the characteristics of infrared mode, which is unaffected by illumination and occlusion. It overcomes the problem of single-modal target detection failing in low-light and occluded scenes, and improves the all-weather perception and complex scene adaptation capabilities of target detection.

[0060] Fourth, it achieves efficient cross-modal knowledge transfer and reduces training costs. This embodiment enables the model to quickly align the data distributions of the visible light modality and the infrared modality through two-stage pre-training. Infrared frequency-aware self-distillation primarily uses low-frequency semantic alignment and secondarily uses high-frequency horizontal detail constraints, achieving cross-modal knowledge transfer at low cost and avoiding the shortcomings of traditional distillation, such as loss of details and interference with end-to-end learning.

[0061] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0062] In the specific implementation, to verify the effectiveness of the lightweight visible-infrared target detection method based on multimodal knowledge distillation proposed in this application (hereinafter referred to as this method or OURS), we trained and tested the RT-DETR structured target detection model using the public dataset FLIR, and compared its performance with other target detection methods. The comparison results are as follows: Figure 4As shown, CFT is a classic multi-source object detection model, LRAF-NET uses a cross-attention mechanism to improve the model's labels, and MiPa uses multimodal training and input segmentation, achieving good performance with only a single backbone network.

[0063] The evaluation metric for multi-source algorithms is mAP (Mean Average Precision), which is the mean of the average precision (AP). Common performance metrics include mAP50 and mAP50-95. The mAP50 metric calculates the mean AP when the IOU threshold is 0.50, while the mAP50-95 metric calculates the average AP over an IOU threshold ranging from 0.50 to 0.95. On the FLIR dataset, compared to the mainstream algorithm MiPa, the OURS method outperforms the OURS method by 1.2% and 1.4% in mAP50 and mAP50-95, respectively, confirming the effectiveness of our proposed method.

[0064] Another embodiment of this application provides an electronic device, such as Figure 5 As shown, it includes a processor 21 and a memory 22. The memory 22 stores instructions that the processor 21 can execute. When the processor 21 is configured to execute the instructions, the electronic device can realize a lightweight visible-infrared target detection method based on multimodal knowledge distillation as described in the above method embodiment.

[0065] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges. The bus can connect various circuits of one or more processors and memories, as well as other circuits such as peripherals, voltage regulators, and power management circuits—all well-known in the art and therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which also receives and transmits data to the processor.

[0066] The processor manages the bus and handles general processing, providing various functions, including but not limited to timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory, on the other hand, is used to store data used by the processor during operation.

[0067] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a lightweight visible-infrared target detection method based on multimodal knowledge distillation as described in the above method embodiments.

[0068] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0069] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A lightweight visible-infrared target detection method based on multi-modal knowledge distillation, realized by a target detection model of an RT-DETR structure, characterized in that, The method includes: A dual-branch backbone network consisting of a visible light mode branch and an infrared mode branch is constructed. The visible light mode branch and the infrared mode branch are used to extract multi-layer features of the visible light mode and the infrared mode, respectively. A multimodal high-efficiency hybrid encoder composed of visible light mode coding units and infrared mode coding units is constructed. The visible light mode coding units and infrared mode coding units are used to enhance the feature representation of the multi-layer features of the visible light mode and the infrared mode, respectively, to obtain the enhanced features of the visible light mode and the infrared mode. A multimodal cross-attention fusion module and an RT-DETR decoder are constructed. The multimodal cross-attention fusion module is used to perform cross-attention fusion on the enhanced features of the visible light mode and the infrared mode based on the kernel-function linear attention mechanism to obtain fused features. The RT-DETR decoder is used to decode based on the fused features to obtain the target detection results. A target detection model is constructed by combining a dual-branch backbone network, a multimodal high-efficiency hybrid encoder, a multimodal cross-attention fusion module, and an RT-DETR decoder. Based on the visible light modal training set and the infrared modal training set, a multimodal pre-training strategy is adopted to perform two-stage pre-training on the target detection model. In the first stage of pre-training, both the visible light modal branch and the infrared modal branch are input with visible light modal training samples to obtain the visible light modal pre-trained model. In the second stage of pre-training, based on the visible light modal pre-trained model, both the visible light modal branch and the infrared modal branch are input with infrared modal training samples to obtain the multimodal pre-trained model. A self-distillation strategy was adopted, using the visible light modal pre-trained model and the multimodal pre-trained model as the student model and the teacher model, respectively, for infrared frequency sensing distillation training. The infrared frequency sensing distillation loss was designed and added to the original RT-DETR loss, and finally the trained target detection model was obtained. The paired visible light and infrared images to be detected are input into the trained target detection model to obtain the target detection results output by the trained target detection model.

2. The lightweight visible-infrared target detection method based on multi-modal knowledge distillation according to claim 1, characterized in that, The basic backbone of the dual-branch backbone network is MobileNetV4, and its convolutional architecture is the most compact MobileNetV4ConvSmall. MobileNetV4ConvSmall reduces the number of parameters and computational complexity while maintaining feature representation capabilities through depthwise separable convolutions and a general inverse bottleneck structure. Both the visible light mode branch and the infrared mode branch are five-layer structures connected in sequence. The features of the corresponding modes extracted from the third, fourth and fifth layers are output to the multi-modal high-efficiency hybrid encoder.

3. The method of claim 2, wherein the method is characterized by, The visible light mode coding unit and the infrared mode coding unit of the multimodal high-efficiency hybrid encoder are independent of each other, and the computational overhead is reduced by reducing the number of intermediate feature channels; The visible light mode coding unit performs corresponding sampling, feature representation enhancement, and splicing on the multi-layer features of the visible light mode to obtain the enhanced features of the visible light mode. The infrared mode coding unit performs corresponding sampling, feature representation enhancement, and splicing on the multi-layer features of the infrared mode to obtain the enhanced features of the infrared mode.

4. The method of claim 3, wherein, The multimodal cross-attention fusion module first performs layer normalization on the enhanced features of the visible light mode and the enhanced features of the infrared mode. It then generates query vectors, key vectors, and value vectors for each mode through independent linear projection layers. Finally, it calculates the attention in two directions: from the visible light mode to the infrared mode and from the infrared mode to the visible light mode, in order to achieve more complete feature fusion. The multimodal cross-attention fusion module adopts a ConvNeXt-style feedforward network design, using a depthwise separable convolution with a large 5×5 kernel size to ensure a balance between computation and performance, and to ensure that the enhanced features of the visible light mode and the enhanced features of the infrared mode can be better fused through a larger receptive field.

5. The method of claim 4, wherein, The computation process based on the kernel-based linear attention mechanism is as follows: First, the query vector and the key vector are layer normalized, and a temperature scaling coefficient is applied to stabilize the training process, to obtain the layer-normalized query vector and the layer-normalized key vector ; ； ； in, This indicates that layer normalization is being performed. Subsequently, a kernel function with modified linear units and small constants was used to... and The query vector is corrected to ensure the non-negativity and non-linearity of the features. and the corrected key vector ; ； ； in, Indicates a modified linear unit. This is a small constant used to avoid gradient vanishing; Finally, change the order of calculations and focus on... Decompose into two linear operations; ； in, The shape is , For batch size, The number of heads that attract self-attention. For sequence length, For dimension size, Representing a vector of all 1s with the last dimension being 1 is equivalent to having... Summing the second to last dimension This is a numerically stable term used to prevent division by zero.

6. The lightweight visible-infrared target detection method based on multimodal knowledge distillation according to claim 1, characterized in that, In the first stage of pre-training, both the visible light modal branch and the infrared modal branch are input with visible light modal training samples to obtain initial feature representations aligned with the distribution of targets in the natural scene. After the first stage of pre-training is completed, the visible light modal pre-trained model is obtained. In the second stage of pre-training, based on the visible light modal pre-trained model, both the visible light modal branch and the infrared modal branch are input with infrared modal training samples to learn the feature representation of thermal radiation information. After the second stage of pre-training is completed, a multimodal pre-trained model is obtained.

7. A lightweight visible-infrared target detection method based on multimodal knowledge distillation according to claim 6, characterized in that, A self-distillation strategy is employed, using the visible light modal pre-trained model and the multimodal pre-trained model as the student model and teacher model, respectively, for infrared frequency sensing distillation training. An infrared frequency sensing distillation loss is designed and incorporated into the original RT-DETR loss, including: A self-distillation strategy is adopted, using the visible light modal pre-trained model as the student model, with its visible light modal branch and infrared modal branch inputting visible light modal training samples and infrared modal training samples, respectively, and using the infrared modal branch of the multimodal pre-trained model as the teacher model, with infrared modal training samples input. The outputs of the teacher model and the student model are decomposed into low-frequency and high-frequency components respectively by Haar wavelet transform. The low-frequency components are aligned as the main component, and the horizontal information in the high-frequency components is introduced as a regularization constraint. The infrared frequency sensing distillation loss is designed and finally added to the original loss of RT-DETR.

8. A lightweight visible-infrared target detection method based on multimodal knowledge distillation according to claim 7, characterized in that, Infrared frequency sensing distillation loss Represented as: ； ； ； in, To balance the weights, For low-frequency loss, For high-frequency level loss, The total number of feature pixels. and These are the low-frequency components of the outputs of the teacher model and the student model, respectively. and These are the high-frequency components of the outputs of the teacher model and the student model, respectively.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a lightweight visible-infrared target detection method based on multimodal knowledge distillation as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a lightweight visible-infrared target detection method based on multimodal knowledge distillation as described in any one of claims 1 to 8.

Images