A real-world image super-resolution method based on active sampling adversarial learning

Abstract

The present invention provides a real-world image super-resolution method based on active sampling adversarial learning, including: using existing high-resolution images and corresponding low-resolution images degraded by bicubic interpolation to train a super-resolution model in a supervised manner; first denoising the real-world image through an unsupervised denoising network; inputting the real-world image that has undergone a round of denoising and the existing and paired high-resolution image-low-resolution image into a network framework based on a generative adversarial network, and using an active sampling strategy to sample image blocks to train the entire model. The final model can directly perform super-resolution reconstruction on the low-resolution image of the real world to obtain a high-resolution output image. Compared with the prior art, the super-resolution method for real-world images proposed in the present invention reduces the training cost and can achieve better image reconstruction effects.

Description

Technical Field

[0001] The present invention relates to a method for improving the resolution of real-world images, and a real-world image super-resolution method based on active sampling adversarial learning. Background Art

[0002] In recent years, with the development of deep learning technology and the upgrade of computer hardware, image processing technology has become more and more perfect. Due to the performance gap between different shooting devices and the large space occupied by high-resolution images, it is not easy to store them in large quantities on the Internet. Therefore, most of the images we can obtain are low-resolution, which not only makes the visual experience poor, but also loses a lot of image information. Therefore, it is of great significance to improve the resolution of images.

[0003] The existing technologies for improving image resolution include traditional methods and deep learning-based methods, but most of the current technologies assume that the input image and the high-resolution image are degraded by predefined degradation parameters (such as bicubic interpolation). However, images in the real world may be closely related to the environment, equipment, and the photographer's operation, which often makes the images in the real world have unknown blur and noise, and there is no corresponding high-resolution image pairing for supervised training. The models trained by the existing supervised super-resolution methods cannot process low-resolution images in the real world. Even some popular blind super-resolution methods in recent years can only estimate limited degradation parameters and are difficult to adapt to images in the real world.

[0004] With the development of generative adversarial networks (GAN, reference: Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances inneural information processing systems, 2014, 27.) and related technologies, adversarial-based image reconstruction technology has attracted people's attention.

[0005] When adversarial image reconstruction technology is applied to image super-resolution tasks, there is no need to consider the degradation parameters of real-world images. Adversarial learning is used to convert noisy and blurred real-world images into clean and clear images. In this way, the existing super-resolution model can be used to process them, and the resulting high-resolution images are clear and noise-free, providing a better viewing experience. Summary of the invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a real-world image super-resolution method based on active sampling adversarial learning in view of the shortcomings of the prior art.

[0007] In order to solve the above technical problems, the present invention discloses a real-world image super-resolution method based on active sampling adversarial learning, comprising the following steps:

[0008] Step 1: Select the data set and super-resolution backbone network;

[0009] Step 2, use the high-resolution image dataset DIV2K and its bicubic-degraded low-resolution image dataset DIV2K bicubic to train the super-resolution network in a supervised manner;

[0010] Step 3, using an unsupervised denoising network to perform a first denoising preprocessing on the real-world low-resolution image in the data set described in step 1;

[0011] Step 4: Use edge detection algorithm to perform edge detection on each image to obtain edge information graphs of all images;

[0012] Step 5: Sample 400 random image blocks for each edge information map; the low-resolution image is 32×32 and the high-resolution image is 128×128. Count and sort the edge pixels of each image block to obtain an edge block pool and a flat slider pool, and the size of each pool is 50.

[0013] Step 6, input the real-world low-resolution image after step 3 and the DIV2K bicubic image into the GAN-based network framework, and use global random sampling and random sampling of the edge block pool and the flat slider pool in step 4 to combine the sampling data for training;

[0014] Step 7, adding the supervised super-resolution network trained in step 2, fixing the relevant module parameters trained in step 6, adding the DIV2K high-resolution image fine-tuning super-resolution network, and obtaining the trained super-resolution network model;

[0015] Step 8: Test the trained super-resolution network model on the real low-resolution image and reconstruct the final high-resolution image.

[0016] The data set described in step 1 of the present invention includes: a real-world low-resolution image data set, a high-resolution image data set DIV2K and a low-resolution image data set DIV2K bicubic; wherein the real-world low-resolution image data set is a DIV2K mild data set degraded by ×4 times, and the DIV2K bicubic low-resolution image also has a degradation factor of ×4 times. The super-resolution model selects the EDSR model (a supervised super-resolution network model, reference: Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 136-144.), and the unsupervised denoising network selects the Neighbor2Neighbor model (an unsupervised denoising network model based on a self-supervised strategy, reference: Huang T, Li S, Jia X, et al. Neighbor2neighbor: Self-supervised denoising from single noisy images [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:14781-14790.).

[0017] In the present invention, step 2 comprises:

[0018] The EDSR structured super-resolution network is used as the training model, and the paired DIV2K high-resolution images and DIV2K bicubic low-resolution images are used as training datasets. The ADAM optimizer (an optimizer, reference: KingmaD P, Ba J.Adam:A method for stochastic optimization[J].arXiv preprint arXiv:1412.6980,2014.) is used, and the batch size is set to 16 and the initial learning rate is 10. -4 , every 2×10 5 After a small batch, it decays to half of the original value; the EDSR super-resolution network is trained in this supervised way until it is fitted.

[0019] In the present invention, step 3 comprises:

[0020] Use the unsupervised denoising method Neighbor2Neighbor based on self-supervised strategy as the denoising network;

[0021] The DIV2K mild low-resolution image dataset is used as the training set, and random flipping and random 90° rotation data augmentation operations are performed. The compromise parameter γ is set to 1, and the denoising network is trained to fit.

[0022] The trained denoising network is used to process the DIV2K mild low-resolution image to obtain the DIV2K mild low-resolution image after the first denoising process.

[0023] In the present invention, step 4 comprises:

[0024] The Canny operator (an edge detection operator, reference: Canny JA computational approach to edge detection [J]. IEEE Transactions on pattern analysis and machine intelligence, 1986 (6): 679-698.) is used as an edge detection tool, where the gradient double thresholds for determining the edge are set to 60 and 180 respectively; edge detection is performed on each low-resolution image in DIV2K mild to obtain the corresponding edge information map; edge detection is performed on each DIV2K high-resolution image to obtain the corresponding edge information map.

[0025] In the present invention, step 5 comprises:

[0026] Each edge information map is randomly sampled with a number of 400. For low-resolution images, the size of each image block is 32×32, and for high-resolution images, the size of each image block is 128×128.

[0027] The 400 image blocks were processed by the non-maximum suppression algorithm (NMS, reference: Neubeck A, Van Gool L. Efficient non-maximum suppression [C] / / 18th International Conference on Pattern Recognition (ICPR'06). IEEE, 2006, 3: 850-855.) to reduce duplication, and the intersection-over-union ratio (IOU) threshold was set to 0.2; the number of edge pixels of all processed image blocks was counted and sorted from large to small, and the first 50 image blocks and the last 50 image blocks were selected as the edge pool and smoothing pool of the image, respectively, for selection by the subsequent active sampling strategy.

[0028] In the present invention, step 6 comprises:

[0029] The low-resolution images of the DIV2K mild and DIV2K bicubic domains after the first denoising are input into a GAN-based network framework for adversarial training; the network framework is based on the CinCGAN (reference: YuanY, Liu S, Zhang J, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 2223-2232.) of the two-layer cycle generative adversarial network structure (CycleGAN, reference: Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018: 701-710.) and is improved; the training process includes the following steps:

[0030] Step 6-1, actively sample each batch of images in the DIV2K mild and DIV2K bicubic domains. Randomly sample a 32×32 image block from the edge pool and smooth pool of the DIV2K mild image, and then randomly sample a 32×32 image block from the entire image. For the DIV2K bicubic image, randomly sample a 32×32 image block from the edge pool and smooth pool of the corresponding DIV2K high-resolution image according to the ×4 ratio, and then randomly sample a 32×32 image block from the entire image. That is, 3 image blocks are sampled from each low-resolution image to form an image block group.

[0031] Step 6-2, the image block group [x r , x s , x e ]After the generator G abProcessing generates new image block groups, new image block groups and image block groups sampled by DIV2K bicubic [y r ,y s ,y e ] respectively input into the discriminator D 1f To identify the domain, G ab and D 1f Conduct adversarial training, forward adversarial loss as follows:

[0032]

[0033] Among them, each corresponding image block is adversarial loss Depend on Calculated, N is the number of training samples, i is the index number of the current image sample, is an image patch sampled globally randomly from the i-th image, and They represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the i-th image, α1 and α2 are weight parameters, ||||2 is the mean square error (MSE) loss, and y r is an image patch randomly sampled globally from the entire image, y s and e Respectively represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the image; training G ab Make the images it generates difficult to be D 1f Identification, while D 1f It is trained to identify whether it belongs to the DIV2K mild domain or the DIV2K bicubic domain;

[0034] Step 6-3, use cycle consistency loss to make G ab The generated image can preserve the image content while changing the image style, that is, using the generator G ba It is expected that the generated image will be restored back, and the forward cycle consistency loss for:

[0035]

[0036] Among them, the corresponding cycle consistency loss of each image block is Depend on Calculated;

[0037] Step 6-4, use identity mapping loss (CycleGAN, reference: Zhu JY, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 2223-2232.) to make the generated image retain the original color information and prevent color jitter. Identity mapping loss As follows:

[0038]

[0039] Among them, the corresponding identity mapping loss for each image block is Depend on Calculated;

[0040] This loss is applied to images in the DIV2K bicubic domain, where i is the current image sample index. is an image patch sampled globally randomly from the i-th image, and They represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the i-th image respectively; ||||1 is the L1 loss;

[0041] Step 6-5: To strengthen the cycle consistency of step 6-3, introduce the discriminator D 1b With the generator G ba Conduct adversarial training, that is, there is a corresponding adversarial loss Cycle consistency loss and identity mapping loss

[0042]

[0043] Among them, each corresponding image block is adversarial loss Depend on Calculated;

[0044]

[0045] Among them, the corresponding cycle consistency loss of each image block is Depend on Calculated;

[0046]

[0047] Among them, the corresponding identity mapping loss for each image block is Depend on Calculated;

[0048] Step 6-6, in order to enhance the smoothness of the generated image, add the total variation loss

[0049]

[0050] Among them, the total variation loss of each corresponding image block is given by Calculated, and Represent the gradient operators in the vertical and horizontal directions respectively;

[0051] Based on this first layer of cycle generation adversarial network CycleGAN, the optimization goal is as follows: As the forward loss, As the backward loss, The total loss function is:

[0052]

[0053]

[0054]

[0055] Train the generator G in this way ab , complete the second denoising and texture generation of real-world low-resolution images, where ψ, w1, w2 and w3 are weight parameters; 10 -4 The model is trained with a learning rate until it is fitted.

[0056] In the present invention, step 7 comprises:

[0057] Fix G trained in step 6 ab , G ba , D 1b and D 1f Parameters, add the EDSR super-resolution network SR and generator G trained in step 2 Ba and discriminator D 2b and D 2f , in order to fine-tune the super-resolution network;

[0058] The training process includes the following steps:

[0059] Step 7-1, through G ab The image block group after inputting the super-resolution SR (Super-Resolution, SR) network to obtain the high-resolution image block group and the DIV2K high-resolution image block group [Yr , Y s , Y e ] forms adversarial training, where Y r represents an image patch randomly sampled globally from the DIV2K high-resolution image, Y s and Y e They represent image patches randomly sampled from the smooth pool and edge pool corresponding to the same DIV2K high-resolution image, and are input into the discriminator D 2f To identify, that is, the SR network is here with D 2f Form adversarial training, adversarial loss for:

[0060]

[0061] Among them, each corresponding image block is adversarial loss Depend on Calculated;

[0062] Step 7-2, as described in step 6, requires cycle consistency, using generator G Ba Restore the reconstructed high-resolution image to the low-resolution image, with cycle consistency loss for:

[0063]

[0064] Among them, the corresponding cycle consistency loss of each image block is Depend on Calculated;

[0065] Step 7-2, similarly, requires identity mapping loss. Here, supervised super-resolution mean square error MSE loss is used as identity mapping loss.

[0066]

[0067] Among them, the corresponding identity mapping loss for each image block is Depend on Calculated;

[0068] Here y and Y represent the pairing in DIV2K bicubic low-resolution image and DIV2K high-resolution image; i is the current image sample index, Y i r is an image patch sampled globally randomly from the i-th image, Y i s and Y i e They represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the i-th image;

[0069] Step 7-3, same as step 6-5, add D 2b Enhanced G Ba Performance of the backward adversarial loss for:

[0070]

[0071] Among them, each corresponding image block is adversarial loss Depend on Calculated; SR() represents the super-resolution model;

[0072] Step 7-4: In order to enhance the smoothness of the super-resolution reconstructed image, add the total variation loss

[0073]

[0074] Among them, the total variation loss of each corresponding image block Depend on Calculated; and Represent the gradient operators in the vertical and horizontal directions respectively;

[0075] The optimization goal of the second-layer cycle-generated adversarial network CycleGAN is as follows: As the forward loss, As the backward loss, As the total loss:

[0076]

[0077]

[0078]

[0079] In this way, the super-resolution model SR is fine-tuned to G ab The final reconstructed super-resolution image is obtained after the result is input; here λ1, λ2 and λ3 are weight parameters, with 2×10 -4 The learning rate is used to train the model until it is fitted and a trained super-resolution network model is obtained.

[0080] In the present invention, step 8 comprises:

[0081] The trained super-resolution network model is used to process the DIV2K mild images to be tested, and all the final high-resolution images are obtained to improve the resolution of real-world images.

[0082] In step 6-6 of the present invention, the weight parameters w1, w2 and w3 are 10, 5, 0.5, and ψ is 0.2; in step 7-4, the weight parameters λ1, λ2 and λ3 are 10, 150 and 2, respectively.

[0083] Beneficial effects:

[0084] By adopting a GAN-based network framework and an active sampling strategy based on edge detection, real-world images can be processed without estimating the degradation parameters of real-world images to obtain high-resolution images. The denoising difficulty of the GAN network framework is reduced by introducing an unsupervised denoising network, and the learning direction of the GAN network framework is optimized through an active sampling strategy, which can achieve better image reconstruction effects. In addition, the super-resolution network only needs to be trained once in a supervised form in advance, and only the super-resolution network model needs to be fine-tuned for different real-world image data sets, saving the cost of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0086] Figure 1 It is a schematic diagram of the workflow of the present invention.

[0087] Figure 2 It is a schematic diagram of the overall framework of the present invention;

[0088] Figure 3 It is a schematic diagram of the DIV2K high-resolution image sampled in the present invention.

[0089] Figure 4 It is a schematic diagram of a DIV2K bicubic low-resolution image sampled in the present invention.

[0090] Figure 5 It is a schematic diagram of the DIV2K mild low-resolution image sampled in the present invention.

[0091] Figure 6 This is a schematic diagram of the output image of the DIV2K mild low-resolution image after being processed by the unsupervised denoising network for the first time in the present invention.

[0092] Figure 7 It is an edge information map obtained by edge detection of the DIV2K mild low-resolution image in the present invention.

[0093] Figure 8 It is a schematic diagram of an example of an image block set of edge areas and smooth areas obtained by active sampling according to an edge information graph in the present invention.

[0094] Fig. 9 It is a schematic diagram of a high-resolution picture output by the model during testing in the present invention.

[0095] Fig.10 It is a schematic diagram of some details of the high-resolution picture output by the model during the test in the present invention. DETAILED DESCRIPTION

[0096] The present invention discloses a real-world image super-resolution method based on active sampling adversarial learning, which is applied to super-resolution reconstruction of low-resolution images in the real world. The method is applicable to real-world images with unknown noise, unknown blur and no high-resolution image pairing. The method is applicable to hardware platforms with different requirements. Through a GAN-based network framework and an active sampling strategy based on edge detection, the super-resolution reconstruction of real-world images can be performed efficiently by fine-tuning the existing super-resolution model, providing better image visual effects.

[0097] Step 1: Select the data set and super-resolution backbone network.

[0098] Step 2: Use the DIV2K high-resolution image dataset and its bicubic-degraded low-resolution image dataset (DIV2K bicubic) to train a super-resolution network in a supervised manner.

[0099] Step 3: Use an unsupervised denoising network to preprocess the real-world low-resolution image for the first time.

[0100] Step 4: Use edge detection algorithm to perform edge detection on each image to obtain edge information graphs of all images.

[0101] Step 5: For each edge information map, 400 random image blocks are sampled (32×32 for low-resolution images and 128×128 for high-resolution images), and the edge pixels of each image block are counted and sorted to obtain the “edge” block pool and the “smooth” block pool, with the size of each pool being 50.

[0102] Step 6: Input the real-world low-resolution image and DIV2K bicubic image after step 3 into the GAN-based network framework, and use global random sampling and random sampling of the "edge" block pool and "smooth" block pool in step 4 combined with the sampling data for training.

[0103] Step 7: Add the supervised super-resolution network trained in step 2, fix the trained module parameters in step 6, and add the DIV2K high-resolution image fine-tuning super-resolution network.

[0104] Step 8: Test the trained model on the real low-resolution image and reconstruct the final high-resolution image.

[0105] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0106] This paper proposes a new real-world image super-resolution method based on active sampling adversarial learning. Its overall logic and workflow are as follows: Figure 1 As shown, it is mainly divided into four stages:

[0107] In the first stage, the real-world image is subjected to the first round of denoising through the unsupervised denoising network, such as Figure 5 A low-resolution image with random blur and noise for DIV2Kmild;

[0108] In the second stage, the DIV2K high-resolution image and its bicubic interpolated low-resolution image (DIV2Kbicubic) are prepared to train the super-resolution network in a supervised manner. Figure 3 and Figure 4 They are DIV2K high-resolution images and the corresponding DIV2K bicubic low-resolution images respectively.

[0109] The third stage is the training of the first layer of CycleGAN in the GAN-based network framework, mainly training the generator G ab , G ba and the discriminator D 1f , D 1b ,like Figure 2 shown.

[0110] The fourth stage is to train the second layer of CycleGAN in the GAN-based network framework. The main focus is on fine-tuning the super-resolution model EDSR, the generator G Ba and the discriminator D 2f , D 2b ,like Figure 2 shown.

[0111] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 1 includes:

[0112] Selecting datasets and super-resolution backbone networks: Here, the real-world low-resolution image dataset uses the 800-image DIV2K mild dataset from the "New Trends in Image Enhancement and Restoration (NTIRE)" challenge held in 2018, which is degraded by ×4 times. The DIV2K high-resolution image dataset and the DIV2K bicubic low-resolution image dataset are both 800 images, of which the DIV2K bicubic low-resolution images are degraded by ×4 times. The super-resolution model uses the EDSR model, and the unsupervised denoising network uses the Neighbor2Neighbor model. The DIV2K high-resolution images and the DIV2K bicubic low-resolution images form image pairs, and the DIV2K mild images have no high-resolution images paired with them.

[0113] The EDSR model is based on the SRResNet network structure, which removes the batch normalization operation that takes up memory. With the same computing resources, more network layers can be stacked or more features can be extracted per layer, thereby achieving better performance. Neighbor2Neighbor is a self-supervised image denoising method that does not require clean images. It is an extension of Noise2Noise. Through theoretical analysis, Noise2Noise is extended to two scenarios: a single noisy image and similar noisy images. A similar noisy image is constructed from a single noisy image by designing a sampler.

[0114] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 2 includes:

[0115] The super-resolution network of EDSR structure is used as the training model, and the paired DIV2K high-resolution images and DIV2K bicubic low-resolution images are used as training datasets. The ADAM optimizer is used, and the parameters are β1=0.9, β2=0.999, ∈=10. -8 , set the batch size to 16 and the initial learning rate to 10 -4 , every 2×10 5 After a small batch, the number of pixels is reduced to half of the original value. In this way, the EDSR super-resolution network is trained in a supervised manner until it is fitted.

[0116] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 3 includes:

[0117] The unsupervised denoising method Neighbor2Neighbor based on self-supervised strategy is used as the denoising network. The DIV2K mild low-resolution image dataset is used as the training set, and data augmentation operations of random flipping and random 90° rotation are performed. The compromise parameter γ=1 is set, and the denoising network is trained to fit. The trained denoising network is processed with 800 DIV2Kmild low-resolution images to obtain the DIV2K mild low-resolution images after the first denoising process, as shown in Figure 6 shown.

[0118] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 4 includes:

[0119] The Canny edge detection operator is used as the edge detection tool, where the double thresholds of the gradient for determining the edge are 60 and 180 respectively. Perform edge detection on each low-resolution image in DIV2K mild to obtain the corresponding edge information map; perform edge detection on each DIV2K high-resolution image to obtain the corresponding edge information map, such as Figure 7 shown.

[0120] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 5 includes:

[0121] Each edge information map is randomly sampled 400 times. For low-resolution images, the size of each image block is 32×32, and for high-resolution images, the size of each image block is 128×128. The 400 image blocks are processed by the non-maximum suppression (NMS) algorithm to reduce duplication, and the intersection-over-union (IOU) threshold is set to 0.2. The number of edge pixels in all processed image blocks is counted, and then sorted from large to small. The first 50 image blocks and the last 50 image blocks are selected as the "edge" pool and "smooth" pool of the image, respectively, as shown in Figure 8 As shown, it can be selected by the subsequent active sampling strategy.

[0122] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 6 includes:

[0123] The low-resolution images of the DIV2K mild and DIV2K bicubic domains after the first denoising are input into the GAN-based network framework for adversarial training. The network framework is based on a two-layer cycle generative adversarial network (CycleGAN) structure and is optimized, such as Figure 2 shown.

[0124] The training process includes the following steps:

[0125] Step 6-1, actively sample each batch of images in the two domains of DIV2K mild and DIV2K bicubic. For the images in DIV2K mild, a 32×32 image block is randomly sampled from the "edge" pool and the "smooth" pool respectively, and then a 32×32 image block is randomly sampled from the entire image globally; for the DIV2K bicubic images, a 32×32 image block is randomly sampled from the "edge" pool and the "smooth" pool of the corresponding DIV2K high-resolution image according to the ratio of ×4, and then a 32×32 image block is randomly sampled from the entire image globally. That is, 3 image blocks are sampled from each low-resolution image to form an image block group.

[0126] Step 6-2, DIV2K mild image block group for each image ([x r , x s , x e ]) through the generator G ab Processing generates new image block groups, new image block groups and DIV2K bicubic sampled image block groups ([y r ,y s ,y e ]) Input the discriminator D respectively 1f To identify the domain, G ab and D 1f Conduct adversarial training:

[0127]

[0128] in,

[0129] N is the number of training samples, x r is an image patch randomly sampled globally from the entire image, x s and x e Respectively represent the image patches randomly sampled from the “smooth” pool and the “edge” pool corresponding to the image, α1 and α2 are weight parameters, and ||||2 is the mean square error (MSE) loss. Training G ab Make the images it generates difficult to be D 1f Identification, while D 1f It is also trained to identify which domain it belongs to (DIV2K mild or DIV2K bicubic).

[0130] Step 6-3, use cycle consistency loss to make G ab The generated image can preserve the image content while changing the image style, that is, using the generator G ba It is expected that the generated image will be restored back:

[0131]

[0132] in,

[0133] Step 6-4, use identity mapping loss to make the generated image retain the original color information and prevent color changes:

[0134]

[0135] in,

[0136] Acts on DIV2K bicubic, y r is an image patch randomly sampled globally from the entire image, y s and e Respectively represent image patches randomly sampled from the "smooth" pool and "edge" pool corresponding to the image. ||||1 is the L1 loss.

[0137] Step 6-5: To strengthen the cycle consistency of step 6-3, introduce the discriminator D 1b With Gb a Conduct adversarial training, that is, there are corresponding adversarial losses, cycle consistency losses, and identity mapping losses:

[0138]

[0139] in,

[0140]

[0141] in,

[0142]

[0143] in,

[0144] Step 6-6, in order to enhance the smoothness of the generated image, add the total variation loss (Total Variation loss):

[0145]

[0146] in,

[0147] The optimization goal of this first layer of CycleGAN is as follows:

[0148]

[0149]

[0150]

[0151] Train the generator G in this way ab , which can complete the second denoising and texture generation of low-resolution images in the real world. w1, w2 and w3 are weight parameters, which are 10, 5, 0.5 respectively, and ψ is 0.2. -4 The learning rate is used to train the model until it is fitted.

[0152] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 7 includes:

[0153] Fix G trained in step 6 ab , G ba , D 1b and D 1f Parameters, add the EDSR super-resolution network SR and generator G trained in step 2 Ba and the discriminator (D 2b and D 2f ), to fine-tune the super-resolution network.

[0154] The training process includes the following steps:

[0155] Step 7-1, through G ab The high-resolution image block group obtained by inputting the SR network and the DIV2K high-resolution image block group ([Y r , Y s , Y e ]) to form adversarial training, and input the discriminator D respectively 2f To identify, that is, the SR network is here with D 2f Forming adversarial training:

[0156]

[0157] in,

[0158] Step 7-2, as described in step 6, requires cycle consistency, using generator G Ba Restore the reconstructed high-resolution image to the low-resolution image:

[0159]

[0160] in,

[0161] Step 7-2, similarly requires identity mapping loss, here we use supervised super-resolution MSE loss as identity mapping loss:

[0162]

[0163] in,

[0164] Here y and Y represent the pairing in the DIV2K bicubic low-resolution image and the DIV2K high-resolution image.

[0165] Step 7-3, same as step 6-5, add D 2b Enhanced G Ba Performance:

[0166]

[0167] in,

[0168] Step 7-4, in order to enhance the smoothness of the super-resolution reconstructed image, the total variation loss (TotalVariation loss) is also added:

[0169]

[0170] in,

[0171] The optimization goal of this second layer CycleGAN is as follows:

[0172]

[0173]

[0174]

[0175] In this way, the super-resolution model SR is fine-tuned to G ab The final reconstructed super-resolution image is obtained after the result is input. Here, λ1, λ2 and λ3 are weight parameters, which are 10, 150 and 2 respectively. -4 The learning rate is used to train the model until it is fitted.

[0176] In the real-world image super-resolution method based on active sampling adversarial learning described in this embodiment, step 8 includes:

[0177] Use the above GAN-based network framework to process the DIV2K mild images to get all the final high-resolution images. Fig. 9 To reconstruct high-resolution images, Fig.10 For details example.

[0178] The existing technologies for improving image resolution include traditional methods and deep learning-based methods, but most of the current technologies assume that the input image and the high-resolution image are degraded by predefined degradation parameters (such as bicubic interpolation). However, images in the real world may be closely related to the environment, equipment, and the photographer's operation, which often makes the images in the real world have unknown blur and noise, and there is no corresponding high-resolution image pairing for supervised training. The models trained by existing supervised super-resolution methods cannot process real-world low-resolution images. Even some popular blind super-resolution methods in recent years can only estimate limited degradation parameters and are difficult to adapt to real-world images.

[0179] This paper proposes a real-world image super-resolution method based on active sampling adversarial learning. The main body is a GAN-based network framework, which combines an unsupervised denoising network and uses an active sampling strategy based on edge detection. Compared with the existing technology, it can better reconstruct the details of real-world images and remove blur and noise to obtain high-resolution images, making the image quality higher and the viewing experience better without the need for high training costs.

[0180] The present invention provides a method and idea for a real-world image super-resolution method based on active sampling adversarial learning. There are many methods and approaches to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented using existing technologies.

Claims

1. A real-world image super-resolution method based on active sampling adversarial learning, characterized in that: The following steps are involved: Step 1: Select the data set and super-resolution backbone network; Step 2, use the high-resolution image dataset DIV2K and its low-resolution image dataset DIV2K bicubic degraded by bicubic interpolation to train the super-resolution network in a supervised manner; Step 3, using an unsupervised denoising network to perform a first denoising preprocessing on the real-world low-resolution image in the data set described in step 1; Step 4: Use edge detection algorithm to perform edge detection on each image to obtain edge information graphs of all images; Step 5: Sample 400 random image blocks for each edge information map; the low-resolution image is 32×32 and the high-resolution image is 128×128. Count and sort the edge pixels of each image block to obtain an edge block pool and a flat slider pool, and the size of each pool is 50. Step 6, input the real-world low-resolution image after step 3 and the DIV2K bicubic image into the network framework based on the generative adversarial network GAN, and use global random sampling and random sampling of the edge block pool and the flat slider pool in step 5 combined with the sampling data for training; Step 7, adding the super-resolution network trained in a supervised manner in step 2, fixing the relevant module parameters trained in step 6, adding the DIV2K high-resolution image to adjust the super-resolution network, and obtaining a trained super-resolution network model; Step 8: Test the trained super-resolution network model on a real-world low-resolution image to reconstruct the final high-resolution image; Wherein, step 3 comprises: The denoising network is constructed using the unsupervised denoising method Neighbor2Neighbor based on a self-supervised strategy; The DIV2K mild low-resolution image dataset is used as the training set, and random flipping and random 90° rotation data augmentation operations are performed. The compromise parameter γ is set to 1, and the denoising network is trained to fit. The trained denoising network is used to process the DIV2K mild low-resolution image to obtain the DIV2K mild low-resolution image after the first denoising process; Step 4 includes: The Canny operator is used as an edge detection tool, where the double thresholds of the gradient for determining the edge are set to 60 and 180 respectively; edge detection is performed on each low-resolution image in DIV2K mild to obtain the corresponding edge information map; edge detection is performed on each DIV2K high-resolution image to obtain the corresponding edge information map; Step 5 includes: Each edge information map is randomly sampled with a number of 400. For low-resolution images, the size of each image block is 32×32, and for high-resolution images, the size of each image block is 128×128. The 400 image blocks were processed by the non-maximum suppression algorithm to reduce duplication, and the intersection-over-union ratio (IOU) threshold was set to 0.

2. The number of edge pixels of all processed image blocks was counted and sorted from large to small. The first 50 image blocks and the last 50 image blocks were selected as the edge pool and smoothing pool of the image, respectively, for selection in the subsequent active sampling process. Step 6 includes: The low-resolution images of the DIV2K mild and DIV2K bicubic domains after the first denoising are input into the GAN-based network framework for adversarial training and learning. The network framework is based on the CinCGAN with a two-layer cyclic generative adversarial network structure. The training process includes the following steps: Step 6-1, actively sample each batch of images in the two domains DIV2K mild and DIV2K bicubic. For the images in DIV2Kmild, a 32×32 image block is randomly sampled from the edge pool and the smooth pool respectively, and then a 32×32 image block is randomly sampled from the entire image globally; for the DIV2K bicubic images, a 32×32 image block is randomly sampled from the edge pool and the smooth pool of the corresponding DIV2K high-resolution image according to the ×4 ratio, and then a 32×32 image block is randomly sampled from the entire image globally, that is, 3 image blocks are sampled from each low-resolution image to form an image block group; Step 6-2, the image block group [x r ,x s ,x e ]After the generator G ab Processing generates new image block groups, new image block groups and image block groups sampled by DIV2K bicubic [y r ,y s ,y e ] respectively input the discriminator D 1f To identify the domain, G ab and D 1f Conduct adversarial training, forward adversarial loss as follows: Among them, each corresponding image block is adversarial loss Depend on Calculated, N is the number of training samples, i is the index number of the current image sample, is an image patch sampled globally randomly from the i-th image, and They represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the i-th image, α1 and α2 are weight parameters, ||||2 is the mean square error MSE loss, and y r is an image patch randomly sampled globally from the entire image, y s and e Respectively represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the image; training G ab Make the images it generates difficult to be D 1f Identification, while D 1f It is trained to identify whether it belongs to the DIV2K mild domain or the DIV2K bicubic domain; Step 6-3, use cycle consistency loss to make G ab The generated image can preserve the image content while changing the image style, that is, using the generator G ba It is expected that the generated image will be restored back, and the forward cycle consistency loss for: Among them, the corresponding cycle consistency loss of each image block is Depend on Calculated; Step 6-4, use identity mapping loss to make the generated image maintain the original color information and prevent color jitter. As follows: Among them, the corresponding identity mapping loss for each image block is Depend on Calculated; This loss is applied to the DIV2K bicubic domain image, where i is the current image sample index number. is an image patch sampled globally randomly from the i-th image, and They represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the i-th image respectively; ||||1 is the L1 loss; Step 6-5: To strengthen the cycle consistency of step 6-3, introduce the discriminator D 1b With the generator G ba Conduct adversarial training and learning, that is, there is a corresponding adversarial loss Cycle consistency loss and identity mapping loss Among them, each corresponding image block is adversarial loss Depend on Calculated; Among them, the corresponding cycle consistency loss of each image block is Depend on Calculated, here y is the image in the DIV2K bicubic domain; Among them, the corresponding identity mapping loss for each image block is Depend on Calculated; Step 6-6, in order to enhance the smoothness of the generated image, add the total variation loss Among them, the total variation loss of each corresponding image block is given by Calculated, and Represent the gradient operators in the vertical and horizontal directions respectively; Based on this first layer of cycle generation adversarial network CycleGAN, the optimization goal is as follows: As the forward loss, As the backward loss, The total loss function is: Train the generator G in this way ab , complete the second denoising and texture generation of real-world low-resolution images, where ψ, w1, w2 and w3 are weight parameters; 10 -4 The model is trained with a learning rate until it is fitted.

2. The real-world image super-resolution method based on active sampling adversarial learning according to claim 1, characterized in that: The data sets described in step 1 include: a real-world low-resolution image data set, a high-resolution image data set DIV2K, and a low-resolution image data set DIV2K bicubic; the real-world low-resolution image data set is a DIV2K mild data set that is degraded by ×4 times, and the DIV2K bicubic low-resolution image also has a degradation factor of ×4 times. The super-resolution model selects the EDSR model, and the unsupervised denoising network selects the Neighbor2Neighbor model.

3. The real-world image super-resolution method based on active sampling adversarial learning according to claim 2, characterized in that: Step 2 includes: The super-resolution network of EDSR structure is used as the super-resolution model to be trained, and the paired DIV2K high-resolution images and DIV2K bicubic low-resolution images are used as training data sets. The ADAM optimizer is used, and the batch size is set to 16, and the initial learning rate is 10. -4 , every 2×10 5 After a small batch, it decays to half of the original value; the EDSR super-resolution network is trained in this supervised way until it is fitted.

4. The real-world image super-resolution method based on active sampling adversarial learning according to claim 3, characterized in that: Step 7 includes: Fix G trained in step 6 ab , G ba , D 1b and D 1f Parameters, add the EDSR super-resolution network SR and generator G trained in step 2 Ba and discriminator D 2b and D 2f , in order to adjust the super-resolution network; The adjustment process includes the following steps: Step 7-1, through G ab The image block group after inputting the super-resolution network SR obtains the high-resolution image block group and the DIV2K high-resolution image block group [Y r ,Y s ,Y e ] forms adversarial training, where Y r represents an image patch randomly sampled globally from the DIV2K high-resolution image, Y s and Y e They represent image patches randomly sampled from the smooth pool and edge pool corresponding to the same DIV2K high-resolution image, and are input into the discriminator D 2f To identify, that is, the SR network is here with D 2f Form adversarial training, adversarial loss for: Among them, each corresponding image block is adversarial loss Depend on Calculated; Step 7-2, use generator G Ba Restore the reconstructed high-resolution image to the low-resolution image, with cycle consistency loss for: Among them, the corresponding cycle consistency loss of each image block is Depend on Calculated; Similarly, identity mapping loss is required. Here, supervised super-resolution mean square error MSE loss is used as identity mapping loss Among them, the corresponding identity mapping loss for each image block is Depend on Calculated; Here y and Y represent the pairing in DIV2K bicubic low-resolution image and DIV2K high-resolution image; i is the current image sample index number, is an image patch sampled globally randomly from the i-th image, and They represent the image blocks randomly sampled from the smooth pool and edge pool corresponding to the i-th image; Step 7-3, same as step 6-5, add D 2b Enhanced G Ba Performance of the backward adversarial loss for: Among them, each corresponding image block is adversarial loss Depend on Calculated; SR() represents the super-resolution model; Step 7-4: In order to enhance the smoothness of the super-resolution reconstructed image, add the total variation loss Among them, the total variation loss of each corresponding image block Depend on Calculated; and Represent the gradient operators in the vertical and horizontal directions respectively; The optimization goal of the second-layer cycle-generated adversarial network CycleGAN is as follows: As the forward loss, As the backward loss, As the total loss: In this way, the super-resolution model SR is fine-tuned to G ab The final reconstructed super-resolution image is obtained after the result is input; here λ1, λ2 and λ3 are weight parameters, with 2×10 -4 The learning rate is used to train the model until it is fitted and a trained super-resolution network model is obtained.

5. The real-world image super-resolution method based on active sampling adversarial learning according to claim 4, characterized in that: Step 8 includes: Use the above-trained super-resolution network model to process the DIV2K mild images to be tested, obtain all the final high-resolution images, and complete the super-resolution task of real-world images.

6. The real-world image super-resolution method based on active sampling adversarial learning according to claim 5, characterized in that: In step 6-6, the weight parameters w1, w2 and w3 are 10, 5, 0.5, and ψ is 0.2 respectively; in step 7-4, the weight parameters λ1, λ2 and λ3 are 10, 150 and 2 respectively.

Images