An unsupervised real-world image super-resolution method based on knowledge distillation
By employing an unsupervised method based on knowledge distillation, utilizing a knowledge distillation network of teacher and student models and an adversarial generative network, the problem of complex image degradation in real-world scenarios is solved, achieving efficient image super-resolution reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2023-01-17
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies struggle to effectively adapt to the complex degradation of images in real-world scenarios, resulting in poor reconstruction performance of image super-resolution networks in the real world.
An unsupervised method based on knowledge distillation is adopted, which utilizes the knowledge distillation network of the teacher model and the student model, combined with the adversarial generative network for domain adaptation, and guides the training of the student model through a high-frequency extractor and a discriminator to reduce the image domain distance.
Significant results have been achieved in real-world image super-resolution reconstruction, improving the subjective visual quality and objective quantitative measurement performance of images.
Smart Images
Figure CN116029902B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, and more specifically, relates to an unsupervised real-world image super-resolution method based on knowledge distillation. Background Technology
[0002] Image super-resolution aims to reconstruct a sharp, high-resolution image from a blurry, low-resolution image. It has broad application prospects in fields such as medicine and remote sensing. Early work relied on the powerful feature extraction capabilities of convolutional neural networks (CNNs), and existing CNN-based methods achieved quite good reconstruction results. However, it should be noted that most of these methods use synthetic methods to form HR-LR image pairs to train the network. Although they show good performance on synthetic datasets, these methods struggle to adapt to real-world images.
[0003] The main reason why supervised training methods perform poorly in reconstructing real-world images is the domain gap between real and synthetic images. This domain gap is primarily reflected in the complexity of image degradation. Typically, synthetic LR images exhibit a simple degradation pattern, such as bicubic interpolation. However, real-world images often suffer from diverse, unknown, and complex degradation (a combination of noise, blur, compression, etc.) due to factors such as sensor noise and camera shake.
[0004] To address this issue, some works have obtained datasets of paired real images by adjusting the camera focal length, and then trained the network in a supervised manner. However, collecting these datasets is difficult and costly. Other works have proposed blind super-resolution algorithms, assuming that LR images are derived from HR images through some form of degradation. These blind super-resolution methods have better generalization than models trained on synthetic data, but this fixed degradation estimation is not effective for images with complex and unknown degradation. Recently, some works have implemented unsupervised super-resolution methods using unpaired HR-LR images. Degraded images with the same distribution as real-world LR images are generated from existing HR images, and then the generated pseudo-HR-LR image pairs are used to supervise the training of the super-resolution network. While this approach can reduce the domain gap between the generated degraded images and the real-world LR images to some extent, the domain distance between them still exists.
[0005] A search revealed numerous existing solutions for super-resolution of real-world images, such as patent application number 202110635893.7, filed on June 8, 2021. The invention is titled: "Real-world Image Super-resolution Method and System Based on Degradation Variational Autoencoder." This application designs a degradation variational autoencoder and a super-resolution network. A low-resolution image is input into the super-resolution network to obtain a high-resolution image; the high-resolution image is input into the degradation variational autoencoder network to obtain a low-resolution image; and the low-resolution image is input into the super-resolution network to obtain a reconstructed high-resolution image. A cycle consistency loss is constructed to simultaneously train the super-resolution network and the degradation variational autoencoder network. It should be noted that the application uses a degradation variational autoencoder to extract the degradation patterns of real low-resolution images and then guides the generation of low-resolution images with similar degradation distributions from high-resolution images. Although this method can generate images with degradation patterns similar to real-world low-resolution images to some extent, the degradation patterns extracted by the encoder are not complete, resulting in differences in the degradation distribution between the generated low-resolution images and real-world low-resolution images, which affects the performance of the super-resolution network.
[0006] Based on the above analysis, existing technologies require an image super-resolution method that can adapt to the real world. Summary of the Invention
[0007] 1. The technical problem that the invention aims to solve
[0008] To overcome the problem that existing super-resolution networks cannot well adapt to complex and degraded images in reality, this invention proposes an unsupervised real-world image super-resolution method based on knowledge distillation. This invention utilizes a knowledge distillation model and an adversarial generative network to adapt to different domains of images, enabling it to process real-world images and meet complex real-world requirements.
[0009] 2. Technical Solution
[0010] To achieve the above objectives, the technical solution provided by the present invention is as follows:
[0011] The present invention provides an unsupervised real-world image super-resolution method based on knowledge distillation, the steps of which are as follows:
[0012] Step 1: Obtain unpaired high-resolution and low-resolution image datasets from the real world, and preprocess the images in the dataset to obtain the training set;
[0013] Step 2: Construct a knowledge distillation network based on the teacher model and student model for network model training;
[0014] Step 3: Based on the training set preprocessed in Step 1 and the model constructed in Step 2, adjust the model parameters, train the teacher model for a certain number of rounds, and then train the teacher model and the student model simultaneously.
[0015] Step 4: Use a low-resolution real-world image as input to the student model, and reconstruct a clear, high-resolution image as output using the parameters learned in Step 3.
[0016] Furthermore, the image preprocessing in step 1 is as follows: unpaired high-resolution and low-resolution images from the AIM2019 and DIV2K datasets are randomly flipped, rotated, and cropped to obtain unpaired high-resolution and low-resolution image patches, which are used as inputs to the teacher model and student model in the knowledge distillation network, respectively.
[0017] Furthermore, the knowledge distillation model constructed in step 2 includes a teacher model, a student model, a high-frequency extractor, and a discriminator. Both the teacher model and the student model contain two modules: a degenerate network and a reconstruction network.
[0018] Furthermore, in the aforementioned teacher model,
[0019] Degradation networks degrade input high-resolution images into blurry low-resolution images;
[0020] A reconstruction network is used to reconstruct low-resolution images generated by a degraded network into clear, high-resolution images.
[0021] In the degradation network used, randomly generated Gaussian noise is used as a degradation factor. The input high-resolution image is first fused with the generated Gaussian noise of the same size, and then passed through four downsampling modules and two upsampling modules to obtain a low-resolution degradation image with a quarter of the size of the original high-resolution image.
[0022] In the reconstruction network used, the low-resolution degraded image generated by the degradation network first passes through a convolutional layer to map the original image to a higher-dimensional feature map; then it passes through 23 layers of residual dense blocks to extract image features layer by layer; then it passes through an upsampling layer to obtain a high-resolution image magnified four times; finally, it passes through two convolutional layers to restore the feature map to an RGB image.
[0023] Furthermore, the reconstruction network and degradation network included in the student model have the same structure as the two modules in the teacher model. The low-resolution image input to the student model is processed by the reconstruction network to obtain a high-resolution image. Then, the high-resolution image is processed by two branches. One branch is processed by the degradation network to obtain a degradation image that is consistent with the content of the input image of the student model. The other branch is sent to the high-frequency extractor along with the output of the teacher model. The high-frequency extractor extracts the high-frequency information of the two images respectively and then sends them to the discriminator.
[0024] Furthermore, in the high-frequency extractor, the input image undergoes Haar wavelet transform to obtain four frequency components: one low-frequency component and three high-frequency components. The three high-frequency components correspond to the high-frequency components of the image in the horizontal, vertical, and diagonal directions, respectively. The three high-frequency components obtained above are fused to obtain the final high-frequency information of the image.
[0025] Furthermore, the discriminator comprises a 10-layer network. The high-frequency information feature map of the input image first passes through a 5x5 convolutional layer, increasing the number of feature maps from 9 to 64. Then it passes through a LeakyReLU activation function layer, followed by two sets of "5x5 convolutional layer-InstanceNorm2d regularization layer-LeakyReLU activation function layer", increasing the number of feature maps from 64 to 128, and then to 256. Next, it passes through a 1x1 convolutional layer, reducing the number of feature maps from 256 to 1. Finally, it passes through a sigmoid activation function layer, mapping each value in the feature map to 0-1.
[0026] Furthermore, the model's operation process in step 3 is as follows: First, the teacher model is trained for a certain number of rounds. The parameters of the teacher model's reconstruction network pre-trained are used to initialize the student model's reconstruction network. Then, the teacher model and the student model are trained simultaneously. The high-frequency extractor is used to extract the high-frequency information of the output images of the teacher model and the student model and feed it into the discriminator. The reconstruction network and discriminator loss are used to guide the generation of high-resolution images with more realistic details.
[0027] Furthermore, in step 3, the Adam optimization algorithm is used to optimize the teacher model. The loss function of the teacher model consists of three parts: pixel reconstruction loss. Perceived loss and loss of identity During training, the above losses are weighted and summed for optimization.
[0028] Furthermore, in step 3, the Adam optimization algorithm is used to optimize the student model, employing a learning rate decay strategy. The loss function of the student model consists of three parts: pixel reconstruction loss. Generator loss and loss of identity During training, network parameters are updated by specifying the number of iterations.
[0029] 3. Beneficial effects
[0030] Compared with existing known technologies, the technical solution provided by this invention has the following significant advantages:
[0031] (1) The present invention provides an unsupervised real-world image super-resolution method based on knowledge distillation. Considering the unknown and complex degradation process of images in real-world scenes, the method utilizes a knowledge distillation model and an adversarial generative network to adapt to different images in the domain. At the same time, an unsupervised model is constructed to deal with the complex degradation process in real-world images. The method has achieved good results in real-world image super-resolution.
[0032] (2) The present invention provides an unsupervised real-world image super-resolution method based on knowledge distillation, which constructs a teacher model and a student model, and uses the reconstruction capability of the teacher model to guide the training of the student model to achieve better real-world image super-resolution effect.
[0033] (3) The present invention provides an unsupervised real-world image super-resolution method based on knowledge distillation, which uses a high-frequency extractor to extract high-frequency information of the image and then sends it to a discriminator for discrimination. This method effectively reduces the domain distance between the reconstructed image and the real-world image and has broad application prospects in the field of real-world image super-resolution. Attached Figure Description
[0034] Figure 1 This is a flowchart of the unsupervised real-world image super-resolution method based on knowledge distillation of the present invention;
[0035] Figure 2 This is a schematic diagram of a neural network model based on knowledge distillation learning constructed in this invention;
[0036] Figure 3 This is a diagram of the degenerate network structure constructed in this invention;
[0037] Figure 4 This is a structural diagram of the high-frequency information extractor constructed in this invention. Detailed Implementation
[0038] To further understand the content of this invention, a detailed description of the invention will be provided in conjunction with the accompanying drawings and embodiments.
[0039] Example 1
[0040] Combination Figure 1 This embodiment of an unsupervised real-world image super-resolution method based on knowledge distillation specifically includes the following steps:
[0041] Step 1: First, a PyTorch environment needs to be deployed to facilitate subsequent model building, training, and testing. Common datasets such as AIM2019 and DIV2K are used. Unpaired high-resolution and low-resolution images from the AIM2019 and DIV2K datasets are randomly flipped, rotated, and cropped to obtain unpaired high-resolution and low-resolution image patches, which are then used as inputs to the teacher and student models in the knowledge distillation network, respectively.
[0042] Step 2: Construct a knowledge distillation learning network model for model training;
[0043] 2-1. The network model for knowledge distillation learning is constructed, with the following specific structure: Figure 2 As shown, the model includes a teacher model, a student model, a high-frequency extractor, and a discriminator. Both the teacher and student models contain two modules: a degradation network and a reconstruction network. The model's operation flow is as follows: First, the teacher model is trained for a certain number of epochs. The pre-trained parameters of the teacher model's reconstruction network are used to initialize the student model's reconstruction network. Then, the teacher and student models are trained simultaneously. The high-frequency extractor extracts the high-frequency information from the output images of the teacher and student models and feeds it into the discriminator. The reconstruction network and discriminator loss are used to guide the generation of high-resolution images with more realistic details.
[0044] 2-2. The teacher model is divided into two modules:
[0045] Degradation networks degrade input high-resolution images into blurry low-resolution images;
[0046] A reconstruction network is used to reconstruct low-resolution images generated by a degraded network into clear, high-resolution images.
[0047] In the degradation network used, randomly generated Gaussian noise is used as the degradation factor. The input high-resolution image is first fused with the generated Gaussian noise of the same size, and then passed through four downsampling modules and two upsampling modules to obtain a low-resolution degradation image with a size of one-quarter of the original high-resolution image. The specific process can be represented by formula (1):
[0048] f = Concat(HR) T ,z)
[0049] Down = Conv_BN_LRelu(f)
[0050] Up=Deconv_BN_LRelu(D(f)) (1)
[0051] Among them HR TRepresents a high-resolution image, z represents randomly generated Gaussian noise, Concat() represents the fusion operation, and f represents the high resolution (HR). T The feature map generated after fusing with z, where Down represents the downsampling module, Up represents the upsampling module, Conv_BN_LRelu() represents the convolution-regularization-activation operation, and Deconv_BN_LRelu() represents the deconvolution-regularization-activation operation.
[0052] The reconstruction network used employs multi-level residual dense blocks to extract image features. Specifically, the low-resolution degraded image generated by the degradation network first passes through a convolutional layer to map the original image to a higher-dimensional feature map. Then, it passes through 23 layers of residual dense blocks to extract image features layer by layer. Subsequently, it passes through an upsampling layer to obtain a high-resolution image magnified four times. Finally, it passes through two convolutional layers to restore the feature map to an RGB image. The specific process can be represented by formula (2):
[0053]
[0054] in, rrdb represents a low-resolution image generated by a degenerate network. 23 Represents a 23-layer residual dense block, SR T This represents the reconstructed high-resolution image.
[0055] 2-3. The student model contains two modules with the same structure as the teacher model: a reconstruction network and a degradation network. The specific process is as follows: the input low-resolution image is processed by the reconstruction network to obtain a high-resolution image. Then, the high-resolution image goes through two branches. One branch goes through the degradation network to obtain a degradation image that is consistent with the content of the input image of the student model, which is used to ensure the cyclic consistency of the network. The other branch is sent to the high-frequency extractor along with the output of the teacher model. The high-frequency extractor extracts the high-frequency information of the two images respectively, and then sends them to the discriminator.
[0056] 2-4. The high-frequency extractor uses Haar wavelet transform. The specific process is as follows: After the input image undergoes Haar wavelet transform, four frequency components are obtained: one low-frequency component and three high-frequency components. These three high-frequency components correspond to the high-frequency components of the image in the horizontal, vertical, and diagonal directions, respectively. The three high-frequency components obtained above are fused to obtain the final high-frequency information of the image, which can be expressed by formula (3):
[0057] g, h, v, d = haar(I)
[0058] I hf =concat(h,v,d) (3)
[0059] Where haar() represents the Haar wavelet transform, g, h, v, and d represent the low-frequency components of the image and the high-frequency components in the horizontal, vertical, and diagonal directions, respectively. I represents the input image, I hf This represents the high-frequency information of the fused image.
[0060] 2-5. The discriminator uses a relative discriminator to distinguish between real and fake samples. The discriminator consists of 10 layers: the high-frequency feature maps of the input image first pass through a 5x5 convolutional layer, increasing the number of feature maps from 9 to 64. Then, they pass through a LeakyReLU activation function layer, followed by two sets of "5x5 convolutional layer - InstanceNorm2d regularization layer - LeakyReLU activation function layer," increasing the number of feature maps from 64 to 128, and then to 256. Next, a 1x1 convolutional layer reduces the number of feature maps from 256 to 1. Finally, a sigmoid activation function layer maps each value in the feature maps to 0-1.
[0061] Step 3: Based on the training set obtained in Step 1 and the network constructed in Step 2, train the network;
[0062] 3-1. Use the PyTorch deep learning framework deployed in step 1 to train the network. For the knowledge distillation learning network built in step 2, first initialize the super-resolution network using the default Xavier method, and initialize all biases to 0.
[0063] 3-2. The general Adam optimization algorithm is used to optimize the network model. The loss function of the teacher model consists of three parts: pixel reconstruction loss. Perceived loss and loss of identity The loss function for training the teacher model is shown in Equation (4):
[0064]
[0065]
[0066]
[0067] Where Φ() represents the VGG feature extractor of the convolutional neural network, specifically the conv5_3 layer of VGG19. LR bic Represents high-resolution image HR T The low-resolution image is obtained through bicubic downsampling. During training, we optimize the loss by weighted summation, and the overall loss of the teacher model is shown in Equation (5):
[0068]
[0069] The loss function of the student model consists of three parts: pixel reconstruction loss. Generator loss and loss of identity As shown in formula (6):
[0070]
[0071]
[0072]
[0073] Among them, LR S This represents a low-resolution image of the input student model. The reconstructed high-resolution image SR S Low-resolution degraded image obtained through a degradation network. hf T hf represents the high-frequency information of the teacher model's output image. S This represents the high-frequency information of the student model's output image. D() represents the discriminator operation, σ() represents the sigmoid activation function, and E represents the mean of all data in a batch. G() represents the reconstruction network. The overall loss of the student model can be expressed by formula (7):
[0074]
[0075] During training, network parameters are updated by specifying the number of iterations.
[0076] Step 4: Use a low-resolution real-world image as input to the network, and reconstruct a high-resolution image using the parameters learned in Step 3 as output.
[0077] This embodiment presents an unsupervised real-world image super-resolution method based on knowledge distillation. Considering the unknown and complex degradation process of images in real-world scenes, it utilizes a knowledge distillation model and a generative adversarial network to adapt different images to their respective domains. This method outperforms current advanced methods in both subjective visual quality and objective quantitative measurement, and has broad application prospects in fields such as image processing.
[0078] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs should fall within the protection scope of the present invention.
Claims
1. An unsupervised real-world image super-resolution method based on knowledge distillation, characterized in that, The steps are as follows: Step 1: Obtain unpaired high-resolution and low-resolution image datasets from the real world, and preprocess the images in the dataset to obtain the training set; Step 2: Construct a knowledge distillation network based on a teacher model and a student model for network model training; the knowledge distillation network includes a teacher model, a student model, a high-frequency extractor, and a discriminator. Both the teacher model and the student model contain two modules: a degradation network and a reconstruction network. In the teacher model described, Degradation networks degrade input high-resolution images into blurry low-resolution images; A reconstruction network is used to reconstruct low-resolution images generated by a degraded network into clear, high-resolution images. In the degradation network used, randomly generated Gaussian noise is used as a degradation factor. The input high-resolution image is first fused with the generated Gaussian noise of the same size, and then passed through four downsampling modules and two upsampling modules to obtain a low-resolution degradation image with a quarter of the size of the original high-resolution image. In the reconstruction network used, the low-resolution degraded image generated by the degradation network first passes through a convolutional layer to map the original image to a higher-dimensional feature map; then it passes through 23 layers of residual dense blocks to extract image features layer by layer; then it passes through an upsampling layer to obtain a high-resolution image magnified four times; finally, it passes through two convolutional layers to restore the feature map to an RGB image. The student model includes a reconstruction network and a degradation network, which have the same structure as the two modules in the teacher model. The low-resolution image input to the student model is processed by the reconstruction network to obtain a high-resolution image. Then, the high-resolution image is processed by two branches. One branch is processed by the degradation network to obtain a degradation image that is consistent with the content of the input image of the student model. The other branch is fed into the high-frequency extractor along with the output of the teacher model. The high-frequency extractor extracts the high-frequency information of the two images respectively and then sends it to the discriminator. Step 3: Based on the training set preprocessed in Step 1 and the model constructed in Step 2, adjust the model parameters. First, train the teacher model, and then train both the teacher and student models simultaneously. The model's operation flow is as follows: First, train the teacher model. Use the pre-trained parameters of the teacher model's reconstruction network to initialize the student model's reconstruction network. Then, train the teacher and student models simultaneously. Use a high-frequency extractor to extract high-frequency information from the output images of the teacher and student models and feed it into the discriminator. Use the reconstruction network and discriminator loss to guide the generation of high-resolution images with more realistic details. Step 4: Use a low-resolution real-world image as input to the student model, and reconstruct a clear, high-resolution image as output using the parameters learned in Step 3.
2. The unsupervised real-world image super-resolution method based on knowledge distillation according to claim 1, characterized in that: The image preprocessing process in step 1 is as follows: unpaired high-resolution and low-resolution images in the AIM2019 and DIV2K datasets are randomly flipped, rotated, and cropped to obtain unpaired high-resolution and low-resolution image patches, which are used as inputs to the teacher model and student model in the knowledge distillation network, respectively.
3. The unsupervised real-world image super-resolution method based on knowledge distillation according to claim 2, characterized in that: In the high-frequency extractor, the input image undergoes Haar wavelet transform to obtain four frequency components: one low-frequency component and three high-frequency components. The three high-frequency components correspond to the high-frequency components of the image in the horizontal, vertical and diagonal directions, respectively. The three high-frequency components obtained above are fused to obtain the final high-frequency information of the image.
4. The unsupervised real-world image super-resolution method based on knowledge distillation according to claim 3, characterized in that: The discriminator comprises a 10-layer network. The high-frequency information feature map of the input image first passes through a 5x5 convolutional layer, increasing the number of feature maps from 9 to 64. Then it passes through a LeakyReLU activation function layer, followed by two sets of "5x5 convolutional layer-InstanceNorm2d regularization layer-LeakyReLU activation function layer", increasing the number of feature maps from 64 to 128, and then to 256. Next, it passes through a 1x1 convolutional layer, reducing the number of feature maps from 256 to 1. Finally, it passes through a sigmoid activation function layer, mapping each value in the feature map to 0-1.
5. The unsupervised real-world image super-resolution method based on knowledge distillation according to claim 4, characterized in that: Step 3 uses the Adam optimization algorithm to optimize the teacher model. The loss function of the teacher model consists of three parts: pixel reconstruction loss. ， Perceived loss and loss of identity During training, the above losses are weighted and summed for optimization.
6. The unsupervised real-world image super-resolution method based on knowledge distillation according to claim 5, characterized in that: In step 3, the Adam optimization algorithm is used to optimize the student model, employing a learning rate decay strategy. The loss function of the student model consists of three parts: pixel reconstruction loss. Generator loss and loss of identity During training, network parameters are updated by specifying the number of iterations.