A dual-modality imaging method based on knowledge distillation

Abstract

This invention discloses a bimodal imaging method based on knowledge distillation, relating to the field of image processing technology. The method first acquires and preprocesses a degraded infrared image to be reconstructed, a visible light reference image, and an infrared reference image simultaneously acquired from the same scene. Then, it constructs a bimodal knowledge distillation network comprising a visible light teacher network, an infrared teacher network, and a lightweight Y-shaped student network, and freezes the teacher network parameters after training. The degraded infrared image is then input into the frozen teacher and student networks, and distillation training is performed based on a piecewise dynamic weighted joint loss function with staged constraints on each loss term. Finally, the lightweight Y-shaped student network, using only the degraded infrared image as input, simultaneously outputs both the visible light reconstructed image and the infrared reconstructed image. This invention reduces the number of model parameters and computational complexity while maintaining the accuracy of bimodal reconstruction, making it suitable for edge computing and lightweight imaging scenarios.

Description

Technical Field

[0001] This invention relates to the fields of image processing, computational imaging, and deep learning, and particularly to a dual-modal imaging method based on knowledge distillation. Background Technology

[0002] In applications such as edge computing, embedded visual perception, unmanned platform perception, security monitoring, and miniaturized imaging systems, imaging systems typically need to complete image acquisition and real-time processing under limited volume, power consumption, and computing power. However, due to factors such as optical system structure, imaging device size, focal length limitations, optical aberrations, environmental noise, and complex lighting conditions, the acquired images often suffer from problems such as blurriness, loss of detail, insufficient contrast, and information degradation, thus affecting the accuracy of subsequent target recognition, scene understanding, and visual analysis.

[0003] Infrared imaging can reflect the thermal radiation information of a target and has strong target detection capabilities in scenarios such as nighttime, low light, smoke, obstruction, or adverse weather conditions. However, infrared images typically suffer from insufficient texture detail, low spatial resolution, and missing color information. Visible light imaging can provide richer texture, edge, and color information, but it is prone to image quality degradation or even failure under low illumination, no light, or complex environmental interference conditions. Therefore, combining the advantages of infrared and visible light images to achieve dual-modal reconstruction of infrared and visible light modes is an important technical approach to improve imaging quality and perception robustness in complex environments.

[0004] In existing technologies, the restoration and dual-modal reconstruction of degraded infrared images typically employs deep convolutional neural networks such as U-Net and ResNet to process different modalities separately. While these methods can improve reconstruction quality to some extent, in practical applications, multiple network models often need to be built and run separately, leading to a significant increase in the number of model parameters and computational load, consuming substantial GPU memory and computing resources. In embedded chip or mobile NPU environments with limited computing power, these methods suffer from high inference latency, making it difficult to meet the demands of real-time imaging and online processing.

[0005] Knowledge distillation can leverage high-performance teacher networks to guide student networks with fewer parameters, balancing model performance and inference efficiency to some extent. However, in bimodal imaging tasks, how to simultaneously utilize the reconstruction knowledge of infrared and visible light teacher networks, how to construct a lightweight student network capable of simultaneously outputting infrared and visible light reconstructed images, and how to reasonably balance pixel constraints, distillation constraints, perceptual constraints, frequency domain constraints, and adversarial constraints at different training stages remain technical challenges that need to be addressed.

[0006] Therefore, how to reduce model complexity while ensuring the quality of dual-modal reconstruction and achieve efficient imaging processing suitable for edge computing environments has become a pressing technical problem in the field of image processing and computational imaging. Summary of the Invention

[0007] The purpose of this invention is to address the problems of existing bimodal imaging methods, such as large model parameter count, high computational complexity, insufficient reconstruction quality of lightweight networks, and difficulty in balancing the optimization needs of different training stages with fixed loss weights. This invention provides a bimodal imaging method based on knowledge distillation. This method constructs a bimodal knowledge distillation network comprising a visible light teacher network, an infrared teacher network, and a lightweight Y-type student network. The dual teacher network supervises the distillation of the lightweight Y-type student network, enabling it to simultaneously output both visible light and infrared reconstructed images from only the degraded infrared image to be reconstructed, thereby reducing model complexity while maintaining bimodal reconstruction quality.

[0008] To achieve the above objectives, the present invention employs the following technical solution:

[0009] Step 1: Acquire the degraded infrared image to be reconstructed, the visible light reference image, and the infrared reference image simultaneously acquired in the same scene, and preprocess the above three types of images to obtain the preprocessed degraded infrared image to be reconstructed, the preprocessed visible light reference image, and the preprocessed infrared reference image data set. Divide the data set into a training set and a test set in a 9:1 ratio.

[0010] Step 2: Construct a bimodal knowledge distillation network, including a visible light teacher network, an infrared teacher network, and a lightweight Y-shaped student network; train the visible light teacher network using the degraded infrared image to be reconstructed and the visible light reference image in the training set; train the infrared teacher network using the degraded infrared image to be reconstructed and the infrared reference image in the training set; after training, freeze the network parameters of the visible light teacher network and the infrared teacher network to obtain the frozen visible light teacher network and the frozen infrared teacher network.

[0011] Step 3: Input the degraded infrared images to be reconstructed from the training set into the frozen visible light teacher network, the frozen infrared teacher network, and the lightweight Y-type student network, respectively. Based on the reconstruction results of the frozen visible light teacher network and the frozen infrared teacher network, construct a piecewise dynamic weighted joint loss function, and use the piecewise dynamic weighted joint loss function to perform distillation training on the lightweight Y-type student network to obtain the trained lightweight Y-type student network.

[0012] Step 4: Input the degraded infrared images to be reconstructed from the test set into the trained lightweight Y-type student network, and output the visible light reconstructed image and the infrared reconstructed image.

[0013] In step one, the degraded infrared image to be reconstructed refers to an infrared image that is blurred, lacks detail, or has degraded information, including but not limited to infrared superlens images and degraded infrared images affected by imaging system limitations; the degraded infrared image to be reconstructed, the infrared reference image, and the visible light reference image are acquired synchronously for the same scene; the preprocessing includes normalization and cropping, normalization is used to linearly map the image pixel values ​​to the 0-1 range, and cropping is used to adjust the normalized image to a uniform size of 256×256.

[0014] In step two, the visible light teacher network and the infrared teacher network have the same structure, both employing an encoder, bottleneck layer, and decoder. The encoder includes a reflection fill layer and a downsampling convolutional layer to extract multi-scale features from the degraded infrared image to be reconstructed. The bottleneck layer includes nine cascaded residual attention blocks to enhance the representation of deep features. The decoder includes an upsampling layer and a convolutional layer to restore the image's spatial resolution and output the reconstructed image of the corresponding modality. The residual attention block has 4.75 × 10^6 parameters. 6 Its structure includes a residual connection structure, a channel attention unit, and a spatial attention unit.

[0015] In step three, the lightweight Y-type student network includes a shared encoder, a lightweight bottleneck layer, and visible light decoder and infrared decoder paths connected to the lightweight bottleneck layer. The shared encoder is used to extract common features from the degraded infrared image to be reconstructed that can simultaneously serve visible light reconstruction and infrared reconstruction; the lightweight bottleneck layer is used to further extract deep abstract features; the visible light decoder path is used to output the visible light reconstructed image, and the infrared decoder path is used to output the infrared reconstructed image. Through the structural design of the shared encoder and dual decoder paths, the feature differences of different modal reconstruction tasks can be preserved while reducing the computational cost of repetitive feature extraction.

[0016] In step three, the shared encoder includes a standard convolutional layer and four cascaded lightweight downsampling blocks. Each lightweight downsampling block includes a depthwise separable convolutional layer, a normalization layer, an activation layer, and a lightweight attention module. The lightweight bottleneck layer includes four cascaded lightweight feature extraction blocks. The infrared decoder path and the visible light decoder path have the same structure, both consisting of four cascaded lightweight upsampling blocks. Each lightweight upsampling block includes an upsampling layer, a normalization layer, an activation layer, and a lightweight attention module. The lightweight attention module includes a global average pooling layer, a one-dimensional convolutional layer, and an activation layer, with a parameter count of 3, used to enhance important channel features and suppress redundant features under low parameter count conditions.

[0017] In step three, the piecewise dynamic weighted joint loss function Due to pixel loss Distillation loss Perceived loss Frequency domain amplitude loss and combating losses The weighted summation is expressed as follows:

[0018]

[0019] in, , , , and represents the weight coefficients that change with the training phase t. Pixel loss is used to constrain the pixel-level differences between the reconstructed image and the reference image; distillation loss is used to constrain the student network to learn the output of the corresponding teacher network; perceptual loss is used to improve the texture and semantic consistency of the reconstructed image; frequency domain amplitude loss is used to constrain the consistency of the frequency domain amplitude distribution of the reconstructed image; adversarial loss is used to enhance the visual realism of the reconstructed image.

[0020] In step three, the weight coefficients are dynamically adjusted according to the training phase. At the initial training stage (t=1), =0.35、 =0.35、 , , Pixel loss and distillation loss are used as primary constraints to prioritize the global structure, brightness distribution, and teacher network output distribution of the reconstructed image. Training enters the mid-stage when the sum of pixel loss and distillation loss falls below 50% of its initial value. At t=2 in the mid-stage training... =0.2、 =0.2、 , , This allows five losses to jointly constrain the student network, optimizing texture detail, edge sharpness, frequency domain consistency, and visual realism while maintaining global structural stability. When the sum of the perceptual loss, frequency domain amplitude loss, and adversarial loss decreases to below 70% of the initial weighted sum of these three losses (perceptual loss, frequency domain amplitude loss, and adversarial loss) at the mid-stage of training, the system enters the late-stage training phase. At t=3 in the late-stage training phase... =0.10、 =0.15 =0.30、 =0.25 =0.20, making perceptual loss, frequency domain amplitude loss, and adversarial loss the main optimization terms, which are used to further improve the texture details, frequency domain consistency, and visual realism of the reconstructed image.

[0021] In step three, the distillation training includes visible light branch distillation supervision and infrared branch distillation supervision. For the visible light branch, the frozen visible light teacher network is used to perform distillation supervision on the visible light decoder path of the lightweight Y-type student network; for the infrared branch, the frozen infrared teacher network is used to perform distillation supervision on the infrared decoder path of the lightweight Y-type student network. The distillation loss... Including characteristic distillation losses and output distillation loss In this context, feature distillation loss is used to calculate the difference between the branch features of the student network and the corresponding layer features of the teacher network, while output distillation loss is used to calculate the difference between the output image of the student network and the output image of the teacher network. The frequency domain amplitude loss The error is calculated by performing a two-dimensional fast Fourier transform on the output images of the student network branches and the corresponding output images of the teacher network, and then performing logarithmic smoothing on the amplitude features.

[0022] Compared with existing technologies, this invention has at least the following beneficial effects: By constructing a visible light teacher network and an infrared teacher network, and training a lightweight Y-shaped student network using a dual-teacher knowledge distillation supervision method, the student network can simultaneously output high-quality visible light reconstructed images and infrared reconstructed images with only degraded infrared images as input, achieving dual-modal joint reconstruction; by employing depthwise separable convolution and a lightweight attention module in the lightweight Y-shaped student network, the number of model parameters and computational complexity are significantly reduced, improving operational efficiency; at the same time, a piecewise dynamic weighted joint loss function is used to constrain pixel loss, distillation loss, perceptual loss, frequency domain amplitude loss, and adversarial loss in stages, effectively enhancing texture details, edge sharpness, frequency domain consistency, and visual realism while ensuring the global structure and brightness distribution of the reconstructed image, thus improving the effect of the reconstructed image.

[0023] This invention has a clear structure, is easy to implement, and is suitable for lightweight imaging scenarios such as edge computing and embedded visual perception, and has good engineering application value. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0025] Figure 2 This is a schematic diagram of the lightweight Y-shaped student network of the present invention;

[0026] Figure 3 This is a schematic diagram of the lightweight attention module of the present invention;

[0027] Figure 4 This is a training framework diagram for the knowledge distillation and segmented dynamic weighted joint loss of this invention;

[0028] Figure 5 This is a schematic diagram of the degraded infrared image to be reconstructed and the visible light reconstructed image and infrared reconstructed image output by the lightweight Y-shaped student network in this invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention.

[0030] like Figure 1 As shown, this invention provides a bimodal imaging method based on knowledge distillation, which mainly includes data acquisition and preprocessing, construction of a bimodal knowledge distillation network and training of the teacher network, training of a lightweight Y-shaped student network distillation, and bimodal image reconstruction output.

[0031] In step one, the degraded infrared image to be reconstructed, the visible light reference image, and the infrared reference image are acquired synchronously in the same scene. The above three types of images are preprocessed to obtain the preprocessed degraded infrared image to be reconstructed, the preprocessed visible light reference image, and the preprocessed infrared reference image data set.

[0032] The degraded infrared image to be reconstructed refers to an infrared image that is blurred, lacks detail, or has degraded information, including but not limited to infrared superlens images and degraded infrared images affected by imaging system limitations. The visible light reference image serves as the supervision target for the visible light teacher network and the lightweight Y-type student network's visible light decoder path, while the infrared reference image serves as the supervision target for the infrared teacher network and the lightweight Y-type student network's infrared decoder path. The three types of images are acquired synchronously for the same scene, maintaining a correspondence in spatial location and temporal acquisition, forming a set of training samples.

[0033] The preprocessing includes normalization and cropping. Normalization linearly maps image pixel values ​​to the 0-1 range to reduce the impact of differences in grayscale or brightness ranges of different images on network training. Cropping adjusts the normalized image to a uniform size of 256×256 to ensure consistent input and output sizes for the network. After preprocessing, the data set is divided into training and testing sets in a 9:1 ratio. The training set is used to train the visible light teacher network, infrared teacher network, and lightweight Y-type student network, while the testing set is used to verify the bimodal reconstruction performance of the trained lightweight Y-type student network.

[0034] In step two, a bimodal knowledge distillation network is constructed. This network comprises a visible light teacher network, an infrared teacher network, and a lightweight Y-shaped student network. The visible light teacher network is trained using the degraded infrared image to be reconstructed and a visible light reference image from the training set, enabling it to learn the mapping relationship from the degraded infrared image to the visible light reference image. Similarly, the infrared teacher network is trained using the degraded infrared image to be reconstructed and an infrared reference image from the training set, enabling it to learn the mapping relationship from the degraded infrared image to the infrared reference image. After training, the network parameters of the visible light and infrared teacher networks are frozen, so that they only provide distillation supervision information during subsequent student network training and are no longer updated.

[0035] In this embodiment, the visible light teacher network and the infrared teacher network have the same structure, both employing an encoder, a bottleneck layer, and a decoder. The encoder is used to extract multi-scale features from the degraded infrared image to be reconstructed, the bottleneck layer is used to enhance the representation capability of deep features, and the decoder is used to restore the spatial resolution of the image and output the reconstructed image of the corresponding modality.

[0036] Specifically, the encoder includes a reflection fill layer and a downsampling convolutional layer. In this embodiment, the encoder first reduces edge artifacts caused by convolution operations at image boundaries through the reflection fill layer, then extracts shallow features through a convolutional layer with a kernel size of 7×7 and a stride of 1, and then progressively reduces the feature map space size and expands the receptive field through three downsampling convolutional layers with a kernel size of 3×3 and a stride of 2, thereby obtaining multi-scale encoded features.

[0037] The bottleneck layer comprises nine cascaded residual attention blocks. Each residual attention block includes a residual connection structure, a channel attention unit, and a spatial attention unit, with a parameter count of 4.75 × 10⁻⁶. 6 The residual connection structure is used to alleviate the gradient degradation problem during the training of deep networks, enabling features to be stably transmitted in the network; the channel attention unit is used to adaptively weight the importance of different channels, enhancing the effective channel features related to image reconstruction; the spatial attention unit is used to highlight key spatial locations such as edges, textures, and target regions, thereby improving the teacher network's ability to express image structure and detail information.

[0038] The decoder includes upsampling layers and convolutional layers. In this embodiment, the decoder includes three sets of upsampling layers and convolutional layers connected in sequence. Each set recovers the feature map spatial size through upsampling operations, and the upsampled features are integrated through convolutional layers to finally output the reconstructed image of the corresponding modality. Specifically, the visible light teacher network outputs the visible light teacher reconstructed image, and the infrared teacher network outputs the infrared teacher reconstructed image.

[0039] In step three, the degraded infrared images to be reconstructed from the training set are input into the frozen visible light teacher network, the frozen infrared teacher network, and the lightweight Y-type student network, respectively. Based on the reconstruction results of the frozen visible light teacher network and the frozen infrared teacher network, a piecewise dynamic weighted joint loss function is constructed, and the lightweight Y-type student network is distilled and trained using the piecewise dynamic weighted joint loss function to obtain the trained lightweight Y-type student network.

[0040] like Figure 2 As shown, the lightweight Y-type student network includes a shared encoder, a lightweight bottleneck layer, a visible light decoder path, and an infrared decoder path. The shared encoder extracts common features from the degraded infrared image to be reconstructed that can simultaneously serve both visible light and infrared reconstruction; the lightweight bottleneck layer further extracts deep abstract features; the visible light decoder path outputs the visible light reconstructed image, and the infrared decoder path outputs the infrared reconstructed image. Through the structural design of the shared encoder and dual decoder paths, the repetitive feature extraction computation caused by constructing two independent networks can be reduced, while preserving the differences in texture, brightness, thermal radiation information, and color representation between different modal reconstruction tasks.

[0041] In this embodiment, the shared encoder first performs shallow feature extraction on the degraded infrared image to be reconstructed using a standard convolutional layer, followed by multi-scale feature encoding through four cascaded lightweight downsampling blocks. Each lightweight downsampling block includes a depthwise separable convolutional layer, a normalization layer, an activation layer, and a lightweight attention module. The depthwise separable convolutional layer replaces the ordinary convolutional layer to reduce the number of parameters and computational cost of convolution operations; the normalization layer stabilizes the network training process; the activation layer introduces non-linear expressive power; and the lightweight attention module adaptively weights the channel features to enhance important channel features and suppress redundant channel features.

[0042] The lightweight bottleneck layer comprises four cascaded lightweight feature extraction blocks, used to further extract deep semantic features and global contextual information in a feature space with lower spatial resolution. The visible light decoder path and the infrared decoder path have the same structure, both consisting of four cascaded lightweight upsampling blocks. Each lightweight upsampling block includes an upsampling layer, a normalization layer, an activation layer, and a lightweight attention module. Through progressive upsampling, the visible light decoder path outputs a visible light reconstructed image, and the infrared decoder path outputs an infrared reconstructed image.

[0043] In a preferred embodiment, the visible light decoder path and the infrared decoder path are connected to the feature maps of the corresponding scale in the shared encoder during the upsampling process to fuse shallow spatial details and deep semantic information, thereby improving the edge sharpness and texture restoration effect of the reconstructed image.

[0044] like Figure 3 As shown, the lightweight attention module includes a global average pooling layer, a one-dimensional convolutional layer, and an activation layer. Specifically, the global average pooling layer compresses the input feature map in the spatial dimension to obtain a global description vector for each channel; the one-dimensional convolutional layer models the local dependencies between adjacent channels; and the activation layer generates channel weight coefficients. Subsequently, the channel weight coefficients are multiplied with the input feature map channel by channel to enhance important feature channels and suppress redundant feature channels. In this embodiment, the one-dimensional convolutional kernel size of the lightweight attention module is 3, therefore its parameter count is 3. This lightweight attention module does not contain a multi-layer fully connected structure, which can further reduce model complexity while maintaining channel feature selection capabilities.

[0045] During student network training, the same degraded infrared image to be reconstructed is simultaneously input into a frozen visible light teacher network, a frozen infrared teacher network, and a lightweight Y-type student network. The frozen visible light teacher network outputs the visible light teacher reconstructed image and its intermediate layer features, while the frozen infrared teacher network outputs the infrared teacher reconstructed image and its intermediate layer features. The lightweight Y-type student network extracts common features through a shared encoder and then outputs the visible light student reconstructed image and the infrared student reconstructed image through the visible light decoder path and the infrared decoder path, respectively.

[0046] For the visible light branch, pixel loss and perceptual loss are calculated between the visible light student reconstructed image and the visible light reference image. Output distillation loss and frequency domain amplitude loss are calculated between the visible light student reconstructed image and the visible light teacher reconstructed image. Simultaneously, feature distillation loss is calculated between the intermediate features of the visible light student branch and the corresponding layer features of the visible light teacher network. For the infrared branch, pixel loss and perceptual loss are calculated between the infrared student reconstructed image and the infrared reference image. Output distillation loss and frequency domain amplitude loss are calculated between the infrared student reconstructed image and the infrared teacher reconstructed image. Simultaneously, feature distillation loss is calculated between the intermediate features of the infrared student branch and the corresponding layer features of the infrared teacher network. Through this method, the lightweight Y-shaped student network is constrained by both the reference image and the teacher network output during training. It can learn the pixel distribution and visual features of the real image while inheriting the feature representation capabilities of the teacher network in bimodal reconstruction tasks.

[0047] In step three, the piecewise dynamic weighted joint loss function Due to pixel loss Distillation loss Perceived loss Frequency domain amplitude loss and combating losses The weighted summation is expressed as follows:

[0048]

[0049] in, , , , and The weight coefficients vary with the training phase t.

[0050] The pixel loss This is used to constrain the pixel-level differences between the student network output image and the corresponding reference image, so that the reconstructed image approximates the real reference image in terms of overall brightness, spatial structure, and target contour. The distillation loss... This is used to constrain the intermediate features and output results of the corresponding teacher network in the student network's learning. The distillation loss includes feature distillation loss. and output distillation loss Its expression is:

[0051]

[0052] in, This represents the difference between the branch features of the student network and the corresponding layer features of the teacher network. This represents the difference between the student network output image and the corresponding teacher network output image. For the visible light branch, and The results were calculated based on the visible light student branch and the visible light teacher network, respectively; for the infrared branch, and The results were obtained from the infrared student branch and the infrared teacher network, respectively.

[0053] The perceived loss This is used to constrain the difference between the student network's output image and the corresponding reference image in a high-level feature space, making the reconstructed image closer to the reference image in terms of texture structure, edge information, and visual perception quality. In this embodiment, the perceptual loss can be obtained by extracting high-level features of the reconstructed image and the reference image through a pre-trained feature extraction network and calculating the feature difference between the two.

[0054] The frequency domain amplitude loss This method is used to constrain the consistency of the frequency domain amplitude distribution between the student network output image and the corresponding teacher network output image. Specifically, two-dimensional fast Fourier transforms are performed on the student network branch output image and the corresponding teacher network output image to obtain their frequency domain representations; then, their amplitude spectra are taken and logarithmically smoothed; finally, the error between the two is calculated to obtain the frequency domain amplitude loss, which is used to enhance the consistency between the student network output image and the teacher network output image in terms of edges, texture, and high-frequency details.

[0055] The resistance loss This is used to enhance the visual realism of the images output by the student network. In this embodiment, a discriminator is set up to distinguish the reconstructed images output by the lightweight Y-shaped student network from the corresponding reference images, so that the student network generates reconstruction results that are closer to the distribution of the corresponding reference images during training.

[0056] In step three, the weighting coefficients are dynamically adjusted according to the training phase. The training phase includes the initial training phase, the middle training phase, and the later training phase.

[0057] At the initial stage of training, t=1 =0.35、 =0.35、 , , The pixel loss and distillation loss are used as the main constraints to prioritize the global structure, brightness distribution and teacher network output distribution of the reconstructed image; when the sum of pixel loss and distillation loss drops to less than 50% of the initial value, the training enters the middle stage.

[0058] At t=2 during the middle of training =0.2、 =0.2、 , , This allows five losses to jointly constrain the student network, optimizing texture details, edge sharpness, frequency domain consistency, and visual realism while maintaining global structural stability. Training enters its later stages when the sum of the perceptual loss, frequency domain amplitude loss, and adversarial loss falls below 70% of the initial weighted sum of these three losses (perceptual loss, frequency domain amplitude loss, and adversarial loss) at the mid-stage.

[0059] At t=3 in the later stage of training =0.10、 =0.15 =0.30、 =0.25 =0.20, making perceptual loss, frequency domain amplitude loss, and adversarial loss the main optimization terms, which are used to further improve the texture details, frequency domain consistency, and visual realism of the reconstructed image.

[0060] In step four, the degraded infrared images to be reconstructed from the test set are input into the trained lightweight Y-type student network. These images pass sequentially through a shared encoder and a lightweight bottleneck layer before entering the visible light decoder path and the infrared decoder path, respectively. The visible light decoder path outputs a visible light reconstructed image, and the infrared decoder path outputs an infrared reconstructed image. After training, the lightweight Y-type student network does not require running visible light and infrared teacher networks during the inference phase; dual-modal image reconstruction can be completed using only a single lightweight student network.

[0061] Figure 5 The output diagrams show the degraded infrared image to be reconstructed, the visible light reconstructed image, and the infrared reconstructed image. Figure 5 It is understood that the present invention can simultaneously obtain a visible light reconstructed image and an infrared reconstructed image when only a degraded infrared image to be reconstructed is input.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the technical concept of the present invention, and all such modifications and improvements should fall within the scope of protection of the present invention.

Claims

1. A dual-modal imaging method based on knowledge distillation, characterized in that, The method includes the following steps: Step 1: Acquire the degraded infrared image to be reconstructed, the visible light reference image, and the infrared reference image simultaneously acquired in the same scene, and preprocess the above three types of images to obtain the preprocessed degraded infrared image to be reconstructed, the preprocessed visible light reference image, and the preprocessed infrared reference image data set. Divide the data set into training set and test set in a 9:1 ratio. Step 2: Construct a bimodal knowledge distillation network, including a visible light teacher network, an infrared teacher network, and a lightweight Y-shaped student network; train the visible light teacher network using the degraded infrared image to be reconstructed and the visible light reference image in the training set; train the infrared teacher network using the degraded infrared image to be reconstructed and the infrared reference image in the training set; after training, freeze the network parameters of the visible light teacher network and the infrared teacher network to obtain the frozen visible light teacher network and the frozen infrared teacher network. Step 3: Input the degraded infrared images to be reconstructed from the training set into the frozen visible light teacher network, the frozen infrared teacher network, and the lightweight Y-type student network, respectively. Based on the reconstruction results of the frozen visible light teacher network and the frozen infrared teacher network, construct a piecewise dynamic weighted joint loss function, and use the piecewise dynamic weighted joint loss function to distill and train the lightweight Y-type student network to obtain the trained lightweight Y-type student network. Step 4: Input the degraded infrared images to be reconstructed from the test set into the trained lightweight Y-type student network, and output the visible light reconstructed image and the infrared reconstructed image.

2. The dual-modal imaging method based on knowledge distillation according to claim 1, characterized in that, In step one, the degraded infrared image to be reconstructed refers to an infrared image that is blurred, lacks detail, or has degraded information, including but not limited to infrared superlens images and degraded infrared images affected by imaging system limitations; the degraded infrared image to be reconstructed, the infrared reference image, and the visible light reference image are acquired synchronously for the same scene; the preprocessing includes normalization and cropping, normalization is used to linearly map the image pixel values ​​to the 0-1 range, and cropping is used to adjust the normalized image to a uniform size of 256×256.

3. The dual-modal imaging method based on knowledge distillation according to claim 1, characterized in that, In step two, the visible light teacher network and the infrared teacher network have the same structure, both adopting an encoder, bottleneck layer and decoder structure. The encoder includes a reflection filling layer and a downsampling convolutional layer, the bottleneck layer includes nine cascaded residual attention blocks, and the decoder includes an upsampling layer and a convolutional layer.

4. The dual-modal imaging method based on knowledge distillation according to claim 3, characterized in that, The parameter count of the residual attention block is 4.75 × 10⁻⁶. 6 Its structure includes a residual connection structure, a channel attention unit, and a spatial attention unit.

5. The dual-modal imaging method based on knowledge distillation according to claim 1, characterized in that, In step three, the lightweight Y-shaped student network includes a shared encoder, a lightweight bottleneck layer, and a visible light decoder path and an infrared decoder path connected to the lightweight bottleneck layer. The shared encoder includes a standard convolutional layer and four cascaded lightweight downsampling blocks. Each lightweight downsampling block includes a depthwise separable convolutional layer, a normalization layer, an activation layer, and a lightweight attention module. The lightweight bottleneck layer includes four cascaded lightweight feature extraction blocks. The infrared decoder path and the visible light decoder path have the same structure, both consisting of four cascaded lightweight upsampling blocks. Each lightweight upsampling block includes an upsampling layer, a normalization layer, an activation layer, and a lightweight attention module.

6. The dual-modal imaging method based on knowledge distillation according to claim 5, characterized in that, The lightweight attention module has 3 parameters and its structure includes a global average pooling layer, a one-dimensional convolutional layer, and an activation layer.

7. The dual-modal imaging method based on knowledge distillation according to claim 1, characterized in that, In step three, the piecewise dynamic weighted joint loss function Due to pixel loss Distillation loss Perceived loss Frequency domain amplitude loss and combating losses The weighted summation is expressed as follows:

8. Among them, , , , and The weight coefficients vary with the training phase t.

9. The dual-modal imaging method based on knowledge distillation according to claim 7, characterized in that, The weighting coefficients Dynamically adjusted based on training phase t: Initial training phase t=1: =0.35、 =0.35、 , , When the sum of pixel loss and distillation loss decreases to below 50% of the initial value, the training enters the mid-stage; mid-stage training t=2: =0.2、 =0.2、 , , When the sum of perceptual loss, frequency domain amplitude loss, and adversarial loss decreases to below 70% of the initial value of the weighted sum of these three losses (perceptual loss, frequency domain amplitude loss, and adversarial loss) at the mid-stage, the training enters the late stage; late-stage training t=3: =0.10、 =0.15 =0.30、 =0.25 =0.

20.

10. The dual-modal imaging method based on knowledge distillation according to claim 1, characterized in that, The distillation training includes visible light branched distillation supervision and infrared branched distillation supervision, and the distillation loss... The calculation method is as follows: The difference between the branch features of the student network and the corresponding layer features of the teacher network is calculated to construct the feature distillation loss. The difference between the output images of the student network and the teacher network is calculated to construct the output distillation loss. ,in The frequency domain amplitude loss It is obtained by performing a two-dimensional fast Fourier transform on the branch output image and then performing logarithmic smoothing on the amplitude features before calculating the error.

Images