LR image generation method and real image super-resolution method based on uncertainty

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to an LR image generation method and a real image super-resolution method based on uncertainty. It overcomes the problem that the existing deep SISR network trained based on a fixed operator synthetic dataset is difficult to apply to real scenes, and the problem that the existing method of constructing a dataset based on unsupervised learning ignores degradation uncertainty, resulting in unsatisfactory results when applied to real scenes. The present invention simultaneously learns real degraded LR images and corresponding degradation uncertainties, and samples multiple LR images from the learned LR images and degradation uncertainties. Any super-resolution network that requires supervised training can be trained using the constructed LR-HR data pairs. The results show that the method proposed in the present invention stabilizes the training of the model and restores higher visual quality while improving objective performance, and is suitable for production scenarios where super-resolution of real images is required.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an LR image generation method and a real image super-resolution method based on uncertainty. Background Art

[0002] As people's quality of life improves, the visual task of obtaining higher resolution images has attracted widespread attention. The Single Image Super Resolution (SISR) task is to restore a high-resolution image (HR) from a degraded low-resolution image (LR) and improve its clarity. In recent years, deep learning-based methods have achieved remarkable results in terms of fidelity performance (such as Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184–199, 2014. Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132–1140, 2017. Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018.). However, most of these methods use a fixed bicubic downsampling operation to construct training data pairs. While this allows for simple and effective benchmarking and the generation of virtually unlimited training data pairs, this approach does not generalize well to real-world scenarios.

[0003] To alleviate this situation, some works have obtained image pairs of different resolutions by modifying the focal length of the camera to construct real datasets to meet the actual needs (such as Jianrui Cai, Hui Zeng, Hongwei Yong, Zisheng Cao, and Lei Zhang. Toward real-world single image superresolution: A new benchmark and a new model. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 3086–3095, 2019. Chang Chen, Zhiwei Xiong, Xinmei Tian, ​​Zheng-Jun Zha, and Feng Wu. Camera lens super-resolution. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 1652–1660, 2019. Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. DSLR-quality photos on mobile devices with deep convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 3277–3285, 2017.). However, obtaining corresponding LR-HR pairs on a large scale is a tedious and labor-intensive task.Recently, some works have used unpaired datasets to generate LR images from HR training images in an unsupervised manner (e.g., Manuel Fritsche, Shuhang Gu, and Radu Timofte. Frequency separation for real-world super-resolution. In 2019 IEEE / CVF International Conference on Computer Vision Workshop (ICCVW), pages 3599–3608. IEEE, 2019. Sanghyun Son, Jaeha Kim, Wei-Sheng Lai, Ming-Hsuan Yang, and Kyoung Mu Lee. Toward real-world super-resolution via adaptive downsampling models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.). Specifically, in order to obtain realistic LR images, the content loss of the bicubic downsampled LR images is used to maintain the same content, and the adversarial loss of the real LR images is used to obtain realistic textures and details. The generated LR-HR pairs can then be used to train the super-resolution network. FSSR proposes a frequency separation strategy that applies the adversarial loss only to high-frequency components (see Manuel Fritsche, Shuhang Gu, and Radu Timofte. Frequency separation for real-world super-resolution. In 2019 IEEE / CVF International Conference on Computer Vision Workshop (ICCVW), pages 3599–3608. IEEE, 2019.).In addition, DASR proposes to reconstruct more realistic images through domain gap aware training (see Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longcun Jin, and Hengjie Song. Unsupervised real-world image super resolution via domain-distance aware training. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 13385–13394, 2021.); Deflow proposes to model the degradation process with conditional flows (see Valentin Wolf, Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Deflow: Learning complex image degradations from unpaired data with conditional flows. In 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 94–103, 2021.).

[0004] While these unsupervised methods have achieved promising results, they only learn a single deterministic mapping model, generating only one LR image from a single HR image, rather than multiple LR images. This completely ignores the uncertainty of degradation of real LR images due to the inevitable randomness of real-world degradation. This is clearly inconsistent with reality when considering more influencing factors, such as random noise. Therefore, how to generate multiple distinct yet similar LR images from a single HR image remains a research topic. Summary of the Invention

[0005] The purpose of the present invention is to provide an LR image generation method and a real image super-resolution method based on uncertainty, so as to overcome the problem that the existing deep SISR network trained based on fixed operator synthetic data sets is difficult to apply to real scenes, and the problem that the existing method of constructing data sets based on unsupervised learning ignores degradation uncertainty, resulting in unsatisfactory results when applied to real scenes.

[0006] The concept of the present invention is:

[0007] This method simultaneously learns real degraded LR images and the corresponding degradation uncertainty, and samples multiple LR images from the learned LR image (mean) and degradation uncertainty (variance). This constructed LR-HR data pair can be used to train any super-resolution network requiring supervised training. Results show that the proposed method improves objective performance while stabilizing model training and restoring higher visual quality, making it suitable for production scenarios requiring super-resolution of real images.

[0008] The technical solution of the present invention is:

[0009] A LR image generation method, which is special in that it includes the following steps:

[0010] Step 1: Obtain the true degraded LR image and the corresponding degradation uncertainty;

[0011] Step 2: Sample multiple LR images from the obtained true degraded LR image and the corresponding degraded uncertainty.

[0012] Furthermore, in step 1, based on the DSN network, the real degraded LR image and the corresponding degradation uncertainty are obtained.

[0013] Furthermore, step 1 is specifically as follows:

[0014] Step 1.1, build DSN network structure;

[0015] Step 1.2: Train the DSN network to learn the real degraded LR image and the corresponding degradation uncertainty to complete the training;

[0016] Step 1.3: Based on the trained DSN network, obtain the real degraded LR image and the corresponding degradation uncertainty.

[0017] Furthermore, the degradation uncertainty in step 1 is the pixel domain degradation uncertainty θ.

[0018] Furthermore, step 1.2 trains the DSN network using the frequency domain separation training strategy;

[0019] The true degraded LR image y is learned based on the following formula g And the corresponding pixel domain degradation uncertainty θ:

[0020]

[0021] Where, Represents the learning of the real degraded LR image y g and the corresponding pixel domain degradation uncertainty θ loss function, y b is the LR image after bicubic downsampling of the HR image x, To calculate the loss mean of all pixels in the entire image, To calculate y g The average loss of all pixels in the entire image.

[0022] Furthermore, in order to better generate the real degraded LR image y g , and also learn the corresponding feature domain degradation uncertainty σ based on the following formula:

[0023]

[0024] Where, represents the loss function of the degradation uncertainty of the learning feature domain, φ(·) represents the feature extractor, and exploring the uncertainty of the feature domain helps to better generate the real LR image y g .

[0025] Furthermore, in order to obtain more stable training and visually pleasing results, a frequency domain separation training strategy is used to learn the true degraded LR image y based on the following formula: g And the corresponding pixel domain degradation uncertainty θ:

[0026]

[0027] In the formula, α1, α2, and α3 are the weights of each loss function, Represents the adversarial loss function of the generator and the adversarial loss function of the discriminator

[0028] Furthermore, and Calculated by the following formula:

[0029]

[0030]

[0031] Where D(·) represents the discriminator, F(·) represents a Gaussian high-frequency filter of size 5×5; F(y g ) indicates the value of y g Perform high frequency filtering, F(y r ) indicates the value of y r Perform high-frequency filtering, To calculate y r The average loss of all pixels in the entire image, y r is an unpaired real scene LR image.

[0032] Furthermore, the DSN network structure constructed in step 1.1 is:

[0033] The network structure of DSN includes a first convolution module, a residual module, a second convolution module and an output module;

[0034] The first convolution module is used to receive the HR image and input it into the residual module after convolution operation;

[0035] The residual module is used to extract feature information from the HR image;

[0036] The second convolution module is used to reduce the spatial resolution of feature information;

[0037] The output module includes three output heads, which are used to convert features into real degraded LR images, as well as output pixel domain degradation uncertainty and feature domain degradation uncertainty.

[0038] Furthermore, the residual module mainly consists of 8 residual blocks, each of which contains two convolutional layers and a PReLU activation in the middle.

[0039] Furthermore, in step 2, multiple LR images are sampled from the learned real degraded LR image and the corresponding pixel-domain degradation uncertainty θ. The sampling process can be expressed as:

[0040]

[0041] Where ∈ = L(0, I) is the standard Laplace distribution. When a HR image x is given, multiple real LR images can be generated.

[0042] The present invention also provides a real image super-resolution method based on uncertainty, which is special in that it includes the following steps:

[0043] Step 1: Generate multiple real degraded LR images from a single HR image using the steps of the above LR image generation method. In different training batches, select different real degraded LR images to construct paired training data.

[0044] Step 2: Use paired training data to train the SR network.

[0045] Furthermore, during step 2 training, the following loss function is used for training:

[0046]

[0047] Where β1, β2, and β3 are the weights of each loss function. is the content loss function, is the perceptual loss function, Represents the adversarial loss function of the generator and the adversarial loss function of the discriminator

[0048] The present invention also provides an uncertainty-based real image super-resolution system, including a memory and a processor, wherein a computer program is stored in the memory. The system is special in that when the computer program runs in the processor, the steps of the above-mentioned uncertainty-based real image super-resolution method are executed.

[0049] The beneficial effects of the present invention are:

[0050] 1. The present invention uses the degradation uncertainty learned in unsupervised learning for real image SR. Given unpaired data, the real degraded LR image and the corresponding degradation uncertainty are first learned simultaneously. Then, multiple LR images are sampled from the learned LR image (mean) and the estimated degradation uncertainty (variance) to construct more robust paired training data to train the SR network. Any super-resolution network that requires supervised training can be trained using the constructed LR-HR data pairs. Compared with other competing methods, the method proposed in the present invention stabilizes the training of the model and restores higher visual quality while improving the objective performance. It can obtain clearer images with fewer artifacts and is suitable for production scenarios where super-resolution of real images is required.

[0051] 2. In the process of learning real LR images, the present invention explores the uncertainty of the feature domain, which helps to better generate real LR images.

[0052] 3. In the process of learning the real degraded LR images and the corresponding pixel domain degradation uncertainty, the present invention uses the frequency domain separation training strategy to train the DSN network, achieving more stable training and visually pleasing results.

[0053] 4. This paper proposes to use gradient information to calculate the adversarial loss in the SR stage, which further stabilizes the training performance and achieves better visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Statistical histogram of the difference between the real image and the bicubic LR image and the corresponding fitted distribution.

[0055] Figure 2 This is the framework of the super-resolution network training method of the present invention.

[0056] Figure 3 This is the strategy diagram for training the DSN network of the present invention.

[0057] Figure 4 This is the DSN network structure of the present invention and the framework of KL-based learning degradation uncertainty.

[0058] Figure 5This is a strategy diagram for training the SRN network using the generated data in the present invention.

[0059] Figure 6 Comparison of SISR visual quality of NTIRE 2020 Track 1 test images using different methods; (a) is the LR image; (b) is the HR image; (c) is the ZSSR restored image; (d) is the PTESRGAN restored image; (e) is the ZSSR+KernelGAN restored image; (f) is the Impressionism restored image; (g) is the DeFlow restored image; and (h) is the restored image of the present invention.

[0060] Figure 7 Comparison of SISR visual quality of NTIRE 2020 Track 2 test images using different methods; (a) is the LR image; (b) is the ZSSR restored image; (c) is the PTESRGAN restored image; (d) is the ZSSR+KernelGAN restored image; (e) is the Impressionism restored image; (f) is the DeFlow restored image; and (g) is the restored image of the present invention.

[0061] Figure 8 HR images and corresponding generated LR images in NTIRE 2020 Track 1; (a) is the HR image; (b) is the real LR image; (c) is the LR image generated by the present invention; (d) is the learned uncertainty.

[0062] Figure 9 These are the HR images and the corresponding generated LR images in NTIRE 2020 Track 2; (a) and (b) are both HR images; (c) is the LR image generated by the present invention; and (d) is the learned uncertainty. DETAILED DESCRIPTION

[0063] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0064] Due to the difficulty in obtaining paired real degraded images and their corresponding high-definition images, the development of unsupervised real-world image super-resolution has been promoted. In FSSR (see Manuel Fritsche, Shuhang Gu, and Radu Timofte. Frequency separation for real-world super-resolution. In 2019 IEEE / CVF International Conference on Computer Vision Workshop (ICCVW), pages 3599–3608. IEEE, 2019.) and DASR (see Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longcun Jin, and Hengjie Song. Unsupervised real-world image super resolution via domain-distance aware training. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 13385–13394, 2021.), a down-sampled adversarial generative network is first used to generate LR images that are similar to the real degradation, thereby obtaining paired LR-HR data pairs. The generated paired data is then used to train the SR network. However, both works only learn a single deterministic mapping model, ignoring the uncertainty of degradation in real scenes. DeFlow (see Valentin Wolf, Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Deflow: Learning complex image degradations from unpaired data with conditional flows. In 2021 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), pages 94–103, 2021.) proposes to use conditional flows through a reversible network to learn the real degraded LR image. Although multiple LR images can be sampled by giving different random initial Gaussian noise, this method is difficult to learn complex degradations due to the limitation of the representation capacity of the reversible network.

[0065] Generally speaking, when training downsampling networks, three types of losses are used, namely content loss, perceptual loss, and adversarial loss, to generate realistic degraded LR images. Used to maintain content consistency. Taking the loss as an example, the likelihood of the generated degraded LR image can be expressed as:

[0066]

[0067] Among them, f (W) (·) is the parameterized degenerate network of W, c is the variance which is a space-invariant constant, y b is the LR image after bicubic downsampling of the HR image x, y g To generate a real degraded LR image, the degradation process is modeled as a deterministic mapping, ignoring the degradation uncertainty of the degraded LR image, which means that when a clean HR image x is given, only a real degraded LR image y can be generated. g .

[0068] Data uncertainty describes the inherent noise in observational data and has been widely studied in computer vision because the performance and robustness of deep networks can be improved by modeling the uncertainty in the observational data. In the paper Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 5710–5719, 2020. the uncertainty of the data is modeled using the mean and variance estimated in face recognition, achieving stronger performance on noisy training data. In the paper Qian Ning, Weisheng Dong, Xin Li, Jinjian Wu, and Guangming Shi. Uncertainty-driven loss for single image super-resolution. Advances in Neural Information Processing Systems, 34, 2021. an adaptive weighted loss SISR is proposed, which gives higher weights to pixels with higher uncertainty during training. These works on modeling data uncertainty mainly focus on supervised training.

[0069] Different from these works, this paper models the degradation uncertainty in unpaired data and proposes a method to simultaneously learn the true degraded LR image and the corresponding degradation uncertainty. Figure 1 As shown in Figure 2, the difference between the true degraded LR image and the LR image after bicubic downsampling of the HR image can be well fitted by a Laplace distribution (while a Gaussian distribution cannot). Based on this, the present invention proposes to learn the true degraded LR image and the corresponding degradation uncertainty by minimizing the KL divergence of two Laplace distributions. Then, multiple LR images are sampled from the learned true degraded LR image (mean) and degradation uncertainty (variance), constructing more complete and robust paired training data to train the SR network, thereby achieving better true SR performance.

[0070] In unsupervised SR, only unpaired real-scene LR images and HR images are provided for training. That is, the content of LR images and HR images is different and cannot be used for supervised training. X represents the real HR image dataset, and x represents the sample. Similarly, the real-scene LR image dataset can be represented by Y r Indicates that the sample can be expressed as y r Note that the real scene LR image y r The corresponding HR image is not provided. In unsupervised SR, the training process can be divided into two steps. The goal of the first step is to generate a real degraded LR image y from the HR image x g In this way, paired training data can be constructed, and any existing SR network that requires paired data training can be trained. Therefore, the key to unsupervised SR is to generate a real degraded LR image. In the first step, an LR image with the same content as the HR image x is required as a content reference. In this invention, the HR image x is bicubic downsampling to generate the LR image y b As the content reference image. Under the above conditions, the method proposed by the present invention for real-world image super-resolution using the learned degradation uncertainty is introduced. The framework of this method is as follows Figure 2 The method consists of two steps: first, learning the real degraded LR image and the corresponding degradation uncertainty; then, sampling multiple LR images from the learned real degraded LR image and the corresponding degradation uncertainty to construct paired training data, and using the paired training data to train the SR network.

[0071] First, combine Figure 3 , introduces how to train a down sampling network (DSN) to learn the true degraded LR image and the corresponding degradation uncertainty.

[0072] By explicitly constraining L(y g,θ) is close to the Laplace distribution L(y b ,I), minimize the Kullbacklebler (KL) divergence of two Laplace distributions to learn the true degraded image and its corresponding degradation uncertainty, which can be expressed as:

[0073]

[0074] Where, Represents the learning of the real degraded LR image y g And the corresponding loss function of pixel domain degradation uncertainty θ, In order to calculate the loss mean of all pixels in the entire image, by learning the degradation uncertainty, for an HR image, the real degraded LR image y can be learned g (mean) and the learned degradation uncertainty θ (variance) are used to sample multiple different but similar LR images.

[0075] The above studies the degradation uncertainty in the pixel domain. Similarly, there is uncertainty in the feature domain, which can be explored in the perceptual loss. By explicitly constraining the feature distribution L(φ(y g ),σ) is close to the Laplace distribution L(φ(y b ),I), minimize the Kullbacklebler (KL) divergence of two Laplace distributions to learn uncertainty, which can be expressed as:

[0076]

[0077] Where φ(·) represents the feature extractor and σ represents the uncertainty of the feature domain. Exploring the uncertainty of the feature domain helps to better generate realistic degraded LR images y g .

[0078] In order to obtain more stable training and visually pleasing results, a frequency domain separation training strategy is adopted, which only calculates the adversarial loss of high-frequency information filtered by the Gaussian blur kernel. The adversarial loss of training the generator can be expressed as:

[0079]

[0080] The loss for training the discriminator can be expressed as:

[0081]

[0082] Where D(·) represents the discriminator, F(·) represents a Gaussian high-frequency filter of size 5×5, and F(y g ) indicates the value of y g Perform high frequency filtering, F(y r ) indicates the value of y rPerform high-frequency filtering. Calculating the adversarial loss from high-frequency information ignores low-frequency information that is less relevant to the degradation, allowing more attention to the degradation details of the image. In addition, this training strategy reduces the difficulty of adversarial training.

[0083] In one embodiment of the present invention, the generator network structure of DSN is as follows: Figure 4 As shown in the figure, it includes a first convolutional module, a residual module, a second convolutional module, and an output module. The first convolutional module is used to receive the HR image and input it into the residual module after convolution operation. The residual module is used to extract feature information from the HR image and is mainly composed of 8 residual blocks, each of which contains two convolutional layers and an intermediate PReLU activation. The second convolutional module has a stride of 2 and is used to reduce the spatial resolution of the feature information. The output module includes three output heads, which are used to convert features into LR degraded images, pixel domain degradation uncertainty, and feature domain degradation uncertainty before output.

[0084] In addition, U-net is used as the discriminator of DSN. The entire DSN is trained by combining three losses:

[0085]

[0086] Among them, α1=1,α2=0.01,α3=0.01 are the weights of each loss function. Represents the learning of the real degraded LR image y g And the corresponding loss function of pixel domain degradation uncertainty θ, The loss function that represents the degradation uncertainty of the learned feature domain, Represents the adversarial loss function of the generator and the adversarial loss function of the discriminator 16 RGB HR image patches of size 256×256 were randomly selected as batch input. The model was trained for 500 epochs with an initial learning rate of 0.0001 and reduced by half every 100 epochs.

[0087] Secondly, combined Figure 5 , introduces how to use the learned real degraded LR images and corresponding uncertainties to train the SRN network.

[0088] Different from previous works that only learn a single deterministic mapping, this paper learns the real degraded LR image y g (mean) and degradation uncertainty (variance) are sampled from multiple degraded LR images to generate multiple real degraded LR images from a single HR image The sampling process can be expressed as:

[0089]

[0090] Where ∈ = L(0, I) is the standard Laplace distribution. When a HR image x is given, multiple real degraded LR images can be generated. Therefore, in different training batches, different true degraded LR images can be adaptively generated from the learned degradation uncertainty for a single HR image.

[0091] Inspired by the fact that DSN only calculates the adversarial loss for high-frequency information, this paper proposes to calculate the adversarial loss for gradient information in the SRN stage to achieve stable training performance and better visual quality. By calculating the adversarial loss using gradient information, the network can ignore low-frequency information that is less relevant to the visual effect, thereby focusing more on image texture and edge details. The adversarial loss for training the generator in SRN can be expressed as:

[0092]

[0093] The loss for training the discriminator can be expressed as:

[0094]

[0095] Where D(·) is the discriminator and G(·) is the gradient calculation, which can be expressed as:

[0096] G(i,j)=‖I(i+1,j)-I(i,j),I(i,j+1)-I(i,j)‖2 (10)

[0097] Where I is the pixel value, and i and j are the pixel positions. Compared to the Gaussian high-frequency filter F(·), the gradient calculator G(·) has a stronger high-frequency filtering effect and can explore more texture and edge details. Therefore, the present invention uses the gradient calculator G(·) in the SRN to achieve better SR performance and uses the Gaussian high-frequency filter F(·) in the DSN to preserve more degradation information.

[0098] The entire SRN is trained by combining three losses:

[0099]

[0100] Among them, β1=0.01, β2=1, β3=0.005 are the weights of each loss function. is the content loss function, is the perceptual loss function, Represents the adversarial loss function of the generator and the adversarial loss function of the discriminator 16 RGB LR image patches of size 64 × 64 are randomly selected as batch input. The model is trained for 1000 epochs with an initial learning rate of 0.0001, and the exponential moving average (EMA) is used for more stable training and better performance.

[0101] It is worth noting that the method of the present invention can be applied to any existing supervised training SR network architecture.

[0102] The effect of the present invention is verified as follows:

[0103] The NTIRE 2020 Real-World Super-Resolution Challenge provides two tracks for unsupervised SR. HR images from DIV2K are used as HR images for both tracks. In Track 1, the synthetically degraded Flickr2K dataset is treated as real degraded LR images. In addition, Track 1 provides a validation dataset for quantitative comparison, which contains 100 training LR images with the same degree of degradation. In Track 2, the real degraded LR images come from the DPED dataset, which consists of real low-quality images from an iPhone 3. Note that reference images (i.e., HR images corresponding to LR images) for evaluation are not provided in the test set of Track 2.

[0104] For datasets that provide reference images in the test set, the present invention uses reference-based evaluation indicators for evaluation, such as PSNR (peak signal-to-noise ratio) and SSIM (structural similarity). In addition, the learned perceptual image patch similarity (LPIPS) is used as a perceptual quality evaluation indicator. Since LPIPS is closely related to human perceptual quality, it is regarded as the most important indicator among the three reference indicators. For the Track 2 dataset that does not provide reference images, NIQE is used as the evaluation indicator.

[0105] 1. Ablation experiment

[0106] To verify the effectiveness of the proposed method, we conducted an ablation experiment on the Track 1 dataset and compared the PSNR / SSIM / LPIPS results of the SRN under different ablation conditions. The results are shown in Table 1.

[0107] Table 1 Ablation experiment results

[0108]

[0109] θ and σ represent the learning degradation uncertainty in the pixel domain and feature domain of the DSN stage, respectively. Sampling refers to the application of sampling strategy based on the learned degradation uncertainty θ, and Gradient refers to the calculation of the adversarial loss in the SRN stage based on the gradient information. It is worth noting that Represents sampling of the real degraded LR image Because uncertainty is not learned in the DSN stage, the results of the ablation experiment can be divided into the following two parts for analysis:

[0110] The effectiveness of learning degradation uncertainty in pixel and feature domains. First, comparing Case 1 and Case 3, learning degradation uncertainty in the pixel domain and sampling LR images using learned uncertainty can improve the performance of LPIPS. Comparing Cases 1, 3, and 5, learning degradation uncertainty in both pixel and feature domains (labeled as 5) can improve the performance in terms of PSNR, SSIM, and LPIPS. In addition, in order to study the impact of sampling strategy on degradation uncertainty, an experiment as shown in Case 2 was conducted, where the variance was set to I and the sampling process can be expressed as Comparing cases 1 and 2, it can be seen that adding spatially invariant random variance cannot improve the performance and robustness of the SR network, while using the learned degradation uncertainty as the spatially adaptive variance (cases 3 / 5) can improve the performance and robustness of the SR network.

[0111] The effectiveness of using gradient information to calculate adversarial loss is shown in Case 4 and Case 5 in Table 1. Using gradient information to calculate adversarial loss has good performance in both SSIM and LPIPS, verifying the effectiveness of the proposed method.

[0112] 2. SOTA comparison

[0113] The proposed method is compared with other SR methods on two datasets of the NTIRE 2020 challenge. The compared methods include Zero-Shot SR (ZSSR), ZSSR+KernelGAN, and unsupervised real-world SR methods Impressionism (winner of the NTIRE 2020 RWSR Challenge), Deflow, and ESRGAN (PTESRGAN) trained on the bicubic degradation dataset.

[0114] The quantitative comparison results of Track 1 and Track 2 are shown in Table 2. In general, the method proposed in this paper achieves the best performance in all indicators. Figure 6 As shown, the visual effect of track 2 is as follows Figure 7 Although no GT image is provided as a reference for the visual results of track 2, it can be found that the restored image produced by the present invention has fewer artifacts.

[0115] Table 2 Results on two datasets of the NTIRE 2020 Challenge

[0116]

[0117] exist Figure 8 and Figure 9 Figure 3 shows the generated LR images and the learned degradation uncertainty for the Track 1 and Track 2 datasets, respectively. The degradation levels of the generated LR images are similar to those of the real LR images, with higher degradation uncertainties in edge and texture regions, corresponding to the degradation uncertainties shown in the figure. This further demonstrates the superiority of the proposed method.

[0118] In summary, the present invention proposes a new method to use the unsupervised learned degradation uncertainty for real image SR. Given unpaired data, the real LR image and degradation uncertainty are first learned simultaneously. Then, multiple LR images are sampled from the learned LR image (mean) and the estimated degradation uncertainty (variance) to construct LR-HR data pairs for training the SR network. In addition, the present invention proposes to use gradient information to calculate the adversarial loss in the SR stage to stabilize the training performance and better visual quality. Experimental results on real image datasets show that the method of the present invention is effective in real-world SR, and can obtain clearer images with fewer artifacts compared with other competing methods.

Claims

1. A LR image generation method, characterized in that: The following steps are involved: Step 1: Obtain the real degraded LR image and the corresponding degradation uncertainty; specifically: Step 1.1, build DSN network structure; Step 1.2: Train the DSN network using the frequency domain separation training strategy. The true degraded LR image y is learned based on the following formula g And the corresponding pixel domain degradation uncertainty θ: Where, Represents the learning of the real degraded LR image y g and the corresponding pixel domain degradation uncertainty θ loss function, y b is the LR image after bicubic downsampling of the HR image x, To calculate the loss mean of all pixels in the entire image, To calculate y g The average loss of all pixels in the entire image The corresponding feature domain degradation uncertainty σ is learned based on the following formula: Where, represents the loss function of the degradation uncertainty of the learned feature domain, and φ(·) represents the feature extractor; Using the frequency domain separation training strategy, we learn the true degraded LR image y based on the following formula g And the corresponding pixel domain degradation uncertainty θ: In the formula, α1, α2, and α3 are the weights of each loss function, Represents the adversarial loss function of the generator and the adversarial loss function of the discriminator and Calculated by the following formula: Where D(·) represents the discriminator, F(·) represents a Gaussian high-frequency filter of size 5×5; F(y g ) indicates the value of y g Perform high frequency filtering, F(y r ) indicates the value of y r Perform high-frequency filtering, To calculate y r The average loss of all pixels in the entire image, y r is an unpaired real scene LR image; Complete training; Step 1.3: Based on the trained DSN network, obtain the real degraded LR image and the corresponding degradation uncertainty; Step 2: Sample multiple LR images from the obtained true degraded LR image and the corresponding degraded uncertainty.

2. The LR image generation method according to claim 1, characterized in that The DSN network structure constructed in step 1.1 is: The network structure of DSN includes a first convolution module, a residual module, a second convolution module and an output module; The first convolution module is used to receive the HR image and input it into the residual module after convolution operation; The residual module is used to extract feature information from the HR image; The second convolution module is used to reduce the spatial resolution of feature information; The output module includes three output heads, which are used to convert features into real degraded LR images, as well as output pixel domain degradation uncertainty and feature domain degradation uncertainty.

3. The LR image generation method according to claim 2, wherein: The residual module mainly consists of 8 residual blocks, each of which contains two convolutional layers and a PReLU activation in the middle.

4. The LR image generation method according to claim 3, characterized in that In step 2, multiple LR images are sampled from the learned real degraded LR image and the corresponding pixel domain degradation uncertainty θ The sampling process can be expressed as: Wherein, ∈=L(0, I) is the standard Laplace distribution.

5. A real image super-resolution method based on uncertainty, characterized in that: The following steps are involved: Step 1: Generate multiple real degraded LR images from a single HR image using the steps of the LR image generation method according to any one of claims 1 to 4. In different training batches, select different real degraded LR images to construct paired training data. Step 2: Use paired training data to train the SR network.

6. The uncertainty-based real image super-resolution method according to claim 5, characterized in that: During step 2 training, the following loss function is used for training: Where β1, β2, and β3 are the weights of each loss function respectively; is the content loss function, is the perceptual loss function, Represents the adversarial loss function of the generator and the adversarial loss function of the discriminator 7. A real image super-resolution system based on uncertainty, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the computer program runs in a processor, it executes the steps of the uncertainty-based real image super-resolution method according to claim 5 or 6.

Images