Image defogging method, device and equipment based on Transform-CNN heterogeneous knowledge distillation

Abstract

The invention discloses an image defogging method and device based on Transform-CNN heterogeneous knowledge distillation, equipment and a storage medium. The method comprises the following steps: acquiring a training data set; the image pairs in the training data set comprise fog images and corresponding clear images; using knowledge distillation to take Transform as a teacher, guiding a student model CNN, extracting and enhancing image features, and obtaining global image features and regional image features, namely feature details; seamlessly integrating the global modeling capability of Transformers and the local representation advantage of CNN by using knowledge distillation, and obtaining a restoration result which retains details and is globally defogged; feedback comparison is carried out according to the fog image and the restored image, so that the correctness of an enhancement result is proved; and inputting the enhanced image into a multi-scale feature discrimination network, optimizing the model, and further optimizing the network model. According to the method, clear and accurate defogged image details are obtained, and a solid foundation is laid for subsequent image processing and application.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and digital image processing technology, specifically to an image dehazing method, and more particularly to a method, apparatus and related equipment for achieving efficient and high-quality image dehazing by utilizing heterogeneous knowledge distillation between the Transformer model and the Convolutional Neural Network (CNN). Background Technology

[0002] In severe weather conditions such as fog and haze, suspended particles in the atmosphere scatter and absorb light, leading to degradation in captured images, including decreased contrast, color distortion, and blurred details. This severely impacts the performance of subsequent computer vision tasks (such as object detection, autonomous driving, and video surveillance). Therefore, image dehazing technology has become an important research direction in the field of image restoration and enhancement.

[0003] Traditional image dehazing methods are mostly based on physical models, such as the widely used Dark Channel Prior (DCP) model. These methods rely on hand-designed prior knowledge, have poor generalization ability in complex and variable real-world scenes, and are prone to color distortion and halo artifacts. With the development of deep learning, data-driven image dehazing methods have made significant progress. These methods have mainly evolved along two technical routes: one is based on Convolutional Neural Networks (CNNs), such as DehazeNet and FFA-Net. CNNs, with their local connectivity and weight sharing characteristics, can efficiently extract local detail features of images, but their receptive field is limited, making it difficult to establish long-distance dependencies and insufficient for modeling the global haze distribution. The second is based on Transformers, such as DeHamer and Uformer. The self-attention mechanism of Transformers can effectively capture global contextual information of images and performs well in handling uniform haze and complex degradation scenes. However, its computational complexity is proportional to the square of the image size, resulting in a large number of model parameters and high computational cost, making it difficult to deploy on devices with limited computing resources.

[0004] To combine the locality advantage of CNNs with the globality advantage of Transformers, researchers have proposed hybrid architectures. However, these architectures typically require designing complex network structures from scratch, making development difficult, and they lack "plug-and-play" characteristics, making them unsuitable for directly enhancing existing, mature, lightweight CNN models.

[0005] Knowledge distillation offers a clever solution to the aforementioned problem. It distills the knowledge from the teacher model into the student model by having a lightweight "student" model learn and mimic the behavior of a powerful but cumbersome "teacher" model. However, in heterogeneous distillation scenarios from Transformer (teacher) to CNN (student), the two differ significantly in feature representation, structure, and semantic abstraction. Direct distillation can lead to inefficient knowledge transfer and even performance degradation. Summary of the Invention

[0006] This project aims to design an image dehazing method based on Transformer-CNN heterogeneous knowledge distillation, termed the TCKD network. The teacher model employs a hierarchical Transformer architecture to capture degradation patterns caused by global weather, while the student network uses a CNN-based encoder-decoder structure, focusing on effectively restoring local details. To ensure functional consistency across the heterogeneous architectures, a dual-end feature adapter aligns the features of the student and teacher models into the same latent space, thereby coordinating their feature distribution and achieving cross-architecture compatibility. A front-to-back feature comparison module enhances feature consistency by performing distillation through feature comparison at the beginning and end stages. During the comparison distillation process, the distillation intensity is dynamically adjusted using learnable parameters, allowing flexible control over the teacher's involvement during feature extraction. Therefore, the proposed method integrates the global modeling capabilities of Transformers with the local representation advantages of CNNs through knowledge distillation. This hybrid design retains the detail-preserving advantages of CNNs while leveraging the global context awareness of Transformers to improve dehazing performance.

[0007] The first objective of this invention is to provide an image dehazing method based on heterogeneous knowledge distillation of Transformer-CNN.

[0008] The second objective of this invention is to provide an image dehazing device based on Transformer-CNN heterogeneous knowledge distillation.

[0009] The third objective of this invention is to provide an image dehazing device based on Transformer-CNN heterogeneous knowledge distillation.

[0010] A fourth objective of this invention is to provide a storage medium.

[0011] The first objective of this invention can be achieved by adopting the following technical solution: An image dehazing method based on heterogeneous knowledge distillation of Transformer-CNN, the method comprising: S1. Obtain the training dataset, which contains fog images and their corresponding fog-free clear images; S2. Construct a teacher-student knowledge distillation framework: The teacher model is a pre-trained image dehazing network based on the Transformer architecture, used to extract global contextual features of the input hazy image; the student model is an image dehazing network based on the CNN architecture, serving as the target lightweight model to be enhanced. S3. Feature Space Alignment: The shallow input features and deep output features of the student model are transformed by the dual-end feature adapter module to align them with the corresponding features of the teacher model in terms of channel dimension and semantic space. S4. Multi-level knowledge distillation: Through the head and tail feature comparison distillation module, KL divergence is used to calculate the distribution differences of the teacher model and the student model on shallow features and deep features respectively, and this is used as the distillation loss. S5. Joint optimization: Combining image reconstruction loss, edge preservation loss and distillation loss, a joint loss function is constructed to optimize the parameters of the student model end-to-end, while the parameters of the teacher model remain frozen during training. S6. Model Deployment: Deploy the trained student model on the application terminal to perform dehazing on the input foggy image and output a clear image.

[0012] Furthermore, the dual-ended feature adapter module includes: Head Feature Adapter: Embedded in the front end of the student model encoder, it consists of two 3×3 convolutional layers and a ReLU activation function, and is used to receive the original input image and output features aligned with the number of shallow feature channels of the teacher model. Tail Feature Adapter: Embedded at the end of the student model decoder, it consists of a 1×1 convolutional layer, a 3×3 convolutional layer, and a ReLU activation function, and is used to transform the output features of the student model into features aligned with the deep feature distribution of the teacher model.

[0013] Furthermore, the head and tail feature comparison distillation loss of the distillation module The calculation is as follows: ; in, Let KL divergence be the KL divergence. and These are the superficial (head) features of students and teachers, respectively. and These are the deep (tail) features of students and teachers, respectively. For temperature coefficient, and These are learnable weight parameters.

[0014] Furthermore, the joint loss function for:

[0015] in, For reconstruction loss based on L1 norm, For edge preservation loss based on multi-directional gradient difference, Weighting coefficients to balance the various losses.

[0016] Furthermore, the teacher model is GridFormer, and the student model is one of FAM-Net, DEA-Net, or ConvIR.

[0017] The second objective of this invention can be achieved by adopting the following technical solution: An image dehazing device based on Transformer-CNN heterogeneous knowledge distillation, the device comprising: The data acquisition module is used to acquire the input foggy image; The feature extraction and alignment module is used to run the student model and the dual-end feature adapter to extract and align features. A knowledge distillation module is used to calculate the head and tail feature contrast distillation loss during the training phase; The image restoration module is used to output the final dehazed image.

[0018] The third objective of this invention can be achieved by adopting the following technical solution: An image enhancement device includes a processor and a memory for storing a processor-executable program, wherein when the processor executes the program stored in the memory, it implements the image dehazing method described above.

[0019] The fourth objective of this invention can be achieved by adopting the following technical solution: A storage medium storing a program that, when executed by a processor, implements the image enhancement method described above.

[0020] The present invention has the following advantages over the prior art: Significant Performance Improvement: Through heterogeneous knowledge distillation, the powerful global perception and contextual modeling capabilities of the Transformer teacher model are transferred to the lightweight CNN student model, enabling it to approach or surpass the original teacher model in objective metrics such as PSNR and SSIM, as well as subjective visual quality. Extremely High Computational Efficiency: This invention is a "plug-and-play" enhancement framework that does not require changes to the core structure of the student model. Performance improvements are achieved simply by adding a very lightweight adapter (e.g., adding only 0.01M parameters) and distillation loss. Inference speed is almost identical to the student model, making it ideal for deployment on mobile and edge devices. High Versatility: The framework is decoupled from specific CNN model structures and can be widely applied to various mainstream CNN dehazing models (such as FAM-Net, DEA-Net, ConvIR, etc.), demonstrating high versatility. Stable Optimization Process: The dual-ended feature adapter effectively solves the feature mismatch problem between heterogeneous architectures, and the distillation intensity is dynamically adjusted through learnable weights, ensuring the effectiveness of knowledge transfer and the stability of the training process. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the implementation of the overall image dehazing method based on heterogeneous knowledge distillation of Transformer-CNN in Embodiment 1 of the present invention.

[0023] Figure 2 This is a flowchart of the detailed module method for image dehazing based on heterogeneous knowledge distillation of Transformer-CNN in Embodiment 1 of the present invention.

[0024] Figure 3 This is a schematic diagram of the dual-ended feature adapter structure of the network model in Embodiment 1 of the present invention.

[0025] Figure 4 This is a schematic diagram of the head and tail feature comparison distillation module of the network model in Embodiment 1 of the present invention.

[0026] Figure 5 This is a visual comparison diagram of the front and back end feature maps of Embodiment 1 of the present invention.

[0027] Figure 6 This is a comparison of the image evaluation metrics of unprocessed and processed images in Embodiment 1 of the present invention.

[0028] Figure 7 This is a schematic diagram of the hardware structure of the electronic device according to Embodiment 2 of the present invention.

[0029] Figure 8 This is a schematic diagram of the hardware structure of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be understood that the specific embodiments described are merely used to explain this application and are not intended to limit this application.

[0031] Example 1: This embodiment is based on the TensorFlow framework and the PyCharm development environment. The TensorFlow framework is a Python-based development architecture that allows for the convenient and rapid construction of reasonable deep learning network models. It also boasts excellent cross-platform interoperability and enables GPU training and validation, improving computational efficiency. TensorFlow provides interfaces to numerous wrapper functions in deep learning architectures and various image processing functions, including OpenCV-related image processing functions.

[0032] The PyCharm development environment (IDE) on Windows or Linux platforms is currently one of the preferred choices for deep learning network design and development. PyCharm provides customers with new templates, design tools, and testing and debugging tools, and also allows customers to directly call interfaces on remote servers.

[0033] like Figure 2 As shown, this embodiment provides an image dehazing method based on the Transformer-CNN heterogeneous knowledge distillation framework. This embodiment uses GridFormer as the teacher model and FAM-Net as the student model, and trains and tests it on the Haze4K dataset. The main steps are as follows: S1. Data Preparation: The Haze4K dataset was used, which contains fogged images and their corresponding clear images. During training, the images were randomly cropped to 512x512 pixels.

[0034] S2, Model Building: Teacher model: Load the GridFormer model pre-trained on the image dehazing task and freeze its parameters.

[0035] Student model: FAM-Net is used as the base model.

[0036] The dual-ended feature adapter includes a head adapter and a tail adapter, specifically: The head adapter consists of two 3×3 convolutional layers and a ReLU activation function. The specific structure is Conv3×3 -> ReLU -> Conv3×3 -> ReLU, which converts the 3 channels of the input image into the number of channels that match the shallow features of the teacher.

[0037] The tail adapter consists of a 1×1 convolutional layer, a 3×3 convolutional layer, and two ReLU activation functions. The specific structural flow is Conv1x1 -> ReLU -> Conv3x3 -> ReLU, which converts the number of channels of the student's output features into the number of channels that match the teacher's deep features.

[0038] S3. Training Strategy: The Adam optimizer is used with an initial learning rate of 1e-4, adjusted using cosine annealing. The batch size is set to 16, and training lasts for 300 epochs. The loss function weights are set as follows: =0.5, =0.1, =1.0. The temperature coefficient τ is set to 1. and Initialize to 1.0 and participate in training updates.

[0039] S4. Distillation and Optimization: For each training image, input both the teacher model and the student model simultaneously. The teacher model provides shallow features. and deep features The features of the student model are first adapted using a dual-ended feature adapter to obtain... and Calculate reconstruction loss Edge loss and distillation loss Summary After calculating the total loss, only backpropagation is used to optimize the parameters of the student model and the two-ended feature adapter.

[0040] S5, Testing and Deployment: After training, only the enhanced student model (including the integrated dual-end feature adapter) is retained. During inference, given a hazy image as input, the student model can directly output a high-quality dehazed image. On the Haze4K test set, this embodiment improved FAM-Net's PSNR from 31.27dB to 31.76dB, while the number of parameters only increased from 47.33M to 47.34M, and the increase in FLOPs was negligible, achieving a perfect balance between performance and efficiency. The network model was trained on a high-performance GPU, and the specific training parameters are as follows: The Adam optimizer was used with parameters set to 0.9 / 0.999; the required epochs for network model training were set to 500, and the initial learning rate was set to 0.001. To ensure the model reaches its optimal value during training and avoids suboptimal values, the learning rate was continuously adjusted and decreased using a cosine annealing strategy as the epoch increased. When the deployed GPU memory was fixed, the batch size should be set as large as possible, regardless of the number of training image dataset samples (but not exceeding the deployed GPU memory, otherwise training will fail).

[0041] Training the network model using the training dataset specifically includes: (1) Set the loss function as follows: ; in, Let KL divergence be the KL divergence. and These are the superficial (head) features of students and teachers, respectively. and These are the deep (tail) features of students and teachers, respectively. For temperature coefficient, and These are learnable weight parameters.

[0042] Furthermore, the joint loss function is:

[0043] in, For reconstruction loss based on L1 norm, For edge preservation loss based on multi-directional gradient difference, Weighting coefficients to balance the various losses.

[0044] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware, and the corresponding program can be stored in a computer-readable storage medium.

[0045] It should be noted that although the method operations of the above embodiments are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the order of execution of the described steps may be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0046] Example 2: like Figure 7 As shown, this embodiment provides an image dehazing device based on Transformer-CNN heterogeneous knowledge distillation, the device comprising: The data interface unit is used to receive foggy images from a camera or an image database.

[0047] The computing unit is loaded with the student model trained as in Example 1, which is based on the Transformer-CNN heterogeneous knowledge distillation enhancement.

[0048] The processing unit executes the forward propagation process of the student model to complete the image dehazing calculation.

[0049] The output unit outputs the processed, clear image to a display, storage device, or subsequent analysis system.

[0050] The specific implementation of each module in this embodiment 1 can be found in the above embodiment 1, and will not be repeated here. It should be noted that the device provided in this embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0051] Example 3: This embodiment provides an image dehazing device based on heterogeneous knowledge distillation of Transformer-CNN. This image restoration device can be a computer, such as... Figure 8 As shown, the processor 802, memory, input device 803, display 804, and network interface 805 are connected via system bus 801. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 806 and internal memory 807. The non-volatile storage medium 806 stores the operating system, computer programs, and database. The internal memory 807 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When the processor 802 executes the computer program stored in the memory, it implements the image dehazing method of Embodiment 1 described above, as follows: Obtain the training dataset; use the Haze4K dataset, which contains foggy images and their corresponding clear images; The foggy image is subjected to feature extraction and feature modulation using the dual-ended feature adapter module in the network model to obtain the head and tail features of the student model. The head-tail feature contrast distillation module in the network model is used to calculate the head-tail feature contrast distillation loss during the training phase. The student model learns the teacher features to obtain the restored image. Based on the restored image and the high-quality clear images in the corresponding training dataset, the loss function of the network model is optimized to achieve convergence of the network model, thereby obtaining a well-trained network model. The image to be tested is input into the trained network model to generate a restored image with clear details.

[0052] Example 4: This embodiment provides a storage medium, which is a computer-readable storage medium, storing a computer program. When the computer program is executed by a processor, it implements the image dehazing method of Embodiment 1 above, as follows: Obtain the training dataset; use the Haze4K dataset, which contains foggy images and their corresponding clear images; The foggy image is subjected to feature extraction and feature modulation using the dual-ended feature adapter module in the network model to obtain the head and tail features of the student model. The head-tail feature contrast distillation module in the network model is used to calculate the head-tail feature contrast distillation loss during the training phase. The student model learns the teacher features to obtain the restored image. Based on the restored image and the high-quality clear images in the corresponding training dataset, the loss function of the network model is optimized to achieve convergence of the network model, thereby obtaining a well-trained network model. The image to be tested is input into the trained network model to generate a restored image with clear details.

[0053] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0054] In summary, this invention addresses the processing of images affected by haze. The constructed network model includes a dual-end feature adapter and a head-to-tail contrastive distillation module. The dual-end feature adapter comprises a head feature adapter and a tail feature adapter. Through the dual-end feature adapter, degenerate features are extracted from both the teacher and student models. The head-to-tail contrastive module performs contrastive distillation on the feature maps of the teacher model and the student model, enabling the student model to learn the teacher's feature processing and achieve the ability to combine global and detailed feature extraction, thus improving the image's global and detailed edges. The dehazing features guide the generation of image detail features, while the loss function calculation process further refines the semantic information of the image features, thereby extracting more accurate image features after dehazing. This provides more effective information for subsequent image reconstruction, generating a restored image with clear details and laying a solid foundation for subsequent image processing and applications.

[0055] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. An image dehazing method based on heterogeneous knowledge distillation of Transformer-CNN, characterized in that, include: Obtain the training dataset; the image pairs in the training dataset include a foggy image and its corresponding fog-free image; A knowledge distillation framework is constructed, in which the teacher model is a pre-trained Transformer dehazing network and the student model is a CNN dehazing network; The shallow and deep features extracted by the student model are transformed using a dual-ended feature adapter module to align with the corresponding features extracted by the teacher model in the feature space. The knowledge distillation loss is obtained by calculating the distribution differences of shallow and deep features between the teacher model and the student model through the head and tail feature comparison distillation module. A joint loss function is constructed by combining the knowledge distillation loss, image reconstruction loss, and edge preservation loss, and the student model is optimized based on the joint loss function; The optimized student model is used to dehaze the input foggy image.

2. The image dehazing method according to claim 1, characterized in that, The dual-ended feature adapter module includes: A head feature adapter, embedded in the front end of the student model encoder, is used to perform feature transformation and alignment on the input image; A tail feature adapter, embedded at the end of the student model decoder, is used to perform feature transformation and alignment on the output features of the student model.

3. The image dehazing method according to claim 2, characterized in that, The head feature adapter consists of two 3×3 convolutional layers and a ReLU activation function connected in sequence; the tail feature adapter consists of a 1×1 convolutional layer, a 3×3 convolutional layer, and a ReLU activation function connected in sequence.

4. The image dehazing method according to claim 1, characterized in that, The head-tail feature comparison distillation module calculates the distribution difference using KL divergence, and its knowledge distillation loss... The calculation formula is: ； in, Let KL divergence be the KL divergence. and These are the superficial (head) features of students and teachers, respectively. and These are the deep (tail) features of students and teachers, respectively. For temperature coefficient, and These are learnable weight parameters.

5. The image dehazing method according to claim 1, characterized in that, The joint loss function for: in, For reconstruction loss based on L1 norm, For edge preservation loss based on multi-directional gradient difference, Weighting coefficients to balance the various losses.

6. The image dehazing method according to claim 1, characterized in that, The teacher model is GridFormer, and the student model is one of FAM-Net, DEA-Net, or ConvIR.

7. An image dehazing device based on heterogeneous knowledge distillation of Transformer-CNN, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 6.

8. An image dehazing device based on heterogeneous knowledge distillation of Transformer-CNN, characterized in that, Includes the image dehazing device as described in claim 7.

9. A computer-readable storage medium for image dehazing based on Transformer-CNN heterogeneous knowledge distillation, wherein a computer program is stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

10. The computer-readable storage medium according to claim 9, characterized in that, The storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

Images