A single-frame image super-resolution method based on epistemic uncertainty

Abstract

The present invention discloses a single-frame image super-resolution method based on cognitive uncertainty. An easy-to-difficult super-resolution generator with cognitive uncertainty is established; a cognitive uncertainty map generator is established for the training of the easy-to-difficult super-resolution generator; the cognitive uncertainty map generator is integrated into the training process of the easy-to-difficult super-resolution generator for training; a single-frame image as a low-resolution image to be tested is input into the trained easy-to-difficult super-resolution generator for processing to obtain the final super-resolution result map. The present invention reduces the pseudo-texture generated by the generative adversarial network when generating images, generates more realistic structure and texture information, and has a natural effect, can achieve optimal or suboptimal results, and improves the reconstruction performance of the super-resolution method.

Description

Technical Field

[0001] The present invention relates to an image super-resolution processing method in the fields of image processing technology, computer vision, and deep learning, and in particular to a single-frame image super-resolution method based on a generative model in the field of image super-resolution technology. Background Art

[0002] Single-frame image super-resolution tasks have a wide range of applications in the field of computer vision, such as target recognition, medical image processing, aerial imaging, etc. The task of single-frame image super-resolution aims to restore high-resolution images from low-resolution images. This problem is ill-posed because each low-resolution input corresponds to multiple high-resolution images.

[0003] In recent years, with the development of deep learning, super-resolution methods based on convolutional neural networks have been proposed one after another. Now there are mainly two categories: PSNR-Oriented Methods and Perceptual-Driven Methods. PSNR-Oriented Methods generally adopts a uniform pixel loss function (for example, MSE), so the super-resolution results generated by this method tend to be smooth. In order to improve the visual quality of the image, Perceptual-Driven Methods proposes to use the GAN network to guide the PSNR-Oriented Methods super-resolution network, which improves the visual quality of the image, but often produces pseudo textures.

[0004] Both of these methods process low-resolution images in a unified manner, ignoring the fact that different types of networks have different processing capabilities for different areas of the image. Different areas of the low-resolution image are not processed differently, so optimal processing cannot be achieved.

[0005] In order to process different areas of the image differently, some methods have proposed to distinguish the corners and flat areas of the low-resolution image by extracting them. However, when the low-resolution image is severely degraded, it is often impossible to effectively extract the information that distinguishes different low-resolution areas, and the actual processing difficulty of these distinguishing information domains may not match that of the network. How to process low-resolution images more accurately and finely has become an important research topic at present. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the prior art, solve the problems existing in the background technology, and provide a single-frame image super-resolution method based on cognitive uncertainty, which can perform super-resolution processing on low-resolution images to generate high-resolution images with clear structure, natural texture and good visual quality.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] S1. Build an easy-to-difficult super-resolution generator with epistemic uncertainty.

[0009] S2, establish a cognitive uncertainty map generator for training super-resolution generators from easy to difficult processing;

[0010] S3, integrating the cognitive uncertainty map generator into the training process of the easy-to-difficult super-resolution generator for training to complete the network model training of the easy-to-difficult super-resolution generator;

[0011] S4. Input the single frame image as the low-resolution image to be tested into the trained easy-to-difficult super-resolution generator to obtain the final super-resolution result map. The cognitive uncertainty map of the low-resolution image can also be output, but the result does not need the cognitive uncertainty map.

[0012] The low resolution and high resolution mentioned in the present invention refer to the relative relationship that the resolution of a low resolution image is lower than that of a high resolution image.

[0013] The method includes two modules: Uncertainty Map Creater and Easy To Hard Super Resolution Net.

[0014] During the training process of S3, the low-resolution image LR and its corresponding high-resolution image HR are input into the cognitive uncertainty map generator to obtain a preliminary super-resolution result map corresponding to the low-resolution image LR, and the cognitive uncertainty map Var_img is obtained using the preliminary super-resolution result map. Then, the low-resolution image LR and its corresponding cognitive uncertainty map Var_img are input into the easy-to-difficult super-resolution generator, and the cognitive uncertainty map is used to process the low-resolution image in different regions and stages to obtain the final super-resolution result map.

[0015] During the actual test, by inputting a low-resolution image into the easy-to-difficult super-resolution generator, we can obtain the cognitive uncertainty map corresponding to the low-resolution image and the final super-resolution result map.

[0016] The specific structure of the easy-to-difficult processing super-resolution generator includes a low-resolution branch;

[0017] The low-resolution branch mainly consists of a convolutional layer, a second dense residual structure (RRDB, Residual-in-Residual Dense Block) module and a third dense residual structure (RRDB, Residual-in-Residual Dense Block) module connected in sequence, as well as a simple area upsampling module Up*4_a and a difficult area upsampling module Up*4_b; the input and output of the second dense residual structure module are added and input into the simple area upsampling module Up*4_a and the third dense residual structure module respectively, and the input and output of the third dense residual structure module are added and input into the difficult area upsampling module Up*4_b. The second dense residual structure module and the third dense residual structure module are both composed of 10 dense residual blocks connected in sequence;

[0018] The simple area upsampling module Up*4_a and the difficult area upsampling module Up*4_b have the same topological structure, both of which include two parallel convolution upsampling modules. Each convolution upsampling module is mainly composed of three convolution layers connected in sequence and an image upsampling module connected between the first convolution layer and the second convolution layer.

[0019] In the low-resolution branch,

[0020] The low-resolution image is input into the convolution layer. After adding the input and output of the second dense residual structure module itself, a simple region feature map srEasy_feature is obtained. After adding the input and output of the third dense residual structure module itself, a difficult region feature map srHard_feature is obtained. The simple region feature map srEasy_feature is processed by a convolution upsampling module to obtain a simple region super-resolution result map srEasy and a simple region distribution map srEasy_map. The difficult region feature map srHard_feature is processed by another convolution upsampling module to obtain a difficult region super-resolution result map srHard and a difficult region distribution map srHard_map.

[0021] Then, according to the simple area super-resolution result map srEasy and the simple area distribution map srEasy_map, as well as the difficult area super-resolution result map srHard and the difficult area distribution map srHard_map, the final super-resolution result map ETHSRout is obtained according to the following formula:

[0022] ETHSRout=srHard×srHard_map+srEasy×srEasy_map.

[0023] The simple area distribution map srEasy_map and the difficult area distribution map srHard_map represent the network's processing capabilities for different areas of this low-resolution image.

[0024] The easy-to-difficult super-resolution generator also includes a cognitive uncertainty branch, which is mainly composed of a binarization operation, a dilation operation and a negation operation connected in sequence.

[0025] Specifically, the binarization operation adopts Otsu binarization method.

[0026] In the cognitive uncertainty branch, the cognitive uncertainty map is first binarized by the Otsu binarization method to obtain a binarized image, and then a dilation operation is performed. The structural elements in the dilation are all-1 square structural elements, and the difficulty distribution map gt_Hardmap corresponding to the cognitive uncertainty map is obtained. The difficulty distribution map gt_Hardmap is negated to obtain a simple distribution map gt_Easymap.

[0027] In S4, for the single-frame image processing of the low-resolution image to be tested, the single-frame image is input into the low-resolution branch of the easy-to-difficult super-resolution generator for processing to obtain the final super-resolution result image ETHSRout without the need for processing by the cognitive uncertainty branch.

[0028] The present invention provides a super-resolution generator that processes from easy to difficult, which can perform super-resolution processing on low-resolution images in different regions and rationally allocate different computing powers to each region, thereby improving the objective quality and visual quality of the super-resolution results reconstructed by the super-resolution network.

[0029] The present invention also specially sets up a cognitive uncertainty branch of the easy-to-difficult super-resolution generator and a cognitive uncertainty map generator. Cognitive uncertainty can be used to quantify the difficulty of the network model in processing super-resolution tasks. Cognitive uncertainty can be used to guide the easy-to-difficult super-resolution generator to perform targeted processing on different low-resolution areas, making the processing of the easy-to-difficult super-resolution generator more reasonable, and ultimately improving the algorithm performance of the single-frame image super-resolution network.

[0030] The specific structure of the cognitive uncertainty map generator is mainly composed of a feature extraction module, an image upsampling module and two consecutive post-processing modules connected in sequence, and each post-processing module is composed of a Monte Carlo Dropout sampling module and a convolutional layer connected in sequence;

[0031] The feature extraction module is mainly composed of a first convolutional layer, a first dense residual structure (RRDB) module and a second convolutional layer. The first dense residual structure module is set between the first convolutional layer and the second convolutional layer and a residual learning method is adopted. The output of the first convolutional layer and the output of the first dense residual structure module are weighted and fused and then input into the second convolutional layer. The first dense residual structure module is composed of 20 dense residual blocks connected in sequence.

[0032] The Monte Carlo Dropout sampling module adopts a Dropout layer.

[0033] The topological structure of each dense residual block is composed of a convolutional layer and residual learning, specifically: the convolutional layer and the LRELU activation function are connected in sequence as an activated convolutional layer, and then the four activated convolutional layers are connected in sequence. The input of each activated convolutional layer is a feature set of the input of the entire dense residual block and the output of the previous activated convolutional layer. For example, the input of the second activated convolutional layer is a feature set of the input of the dense residual block and the output of the first activated convolutional layer. The feature set here means that the inputs of the activated convolutional layers are connected in the channel dimension, and the final output is the input of the dense residual block plus 0.2 times the fourth activated convolutional layer.

[0034] The image upsampling module in the cognitive uncertainty map generator / easy-to-difficult super-resolution generator is composed of two two-fold upsampling operations, and each two-fold upsampling operation is mainly composed of two bilinear interpolation upsampling operations, convolution layers, and LRELU activation functions connected in sequence.

[0035] In the cognitive uncertainty map generator, Monte Carlo Dropout sampling is turned on, a low-resolution image LR and its corresponding high-resolution image HR are input, the low-resolution image LR is passed through the first convolution layer of the feature extraction module to obtain the image initial feature I_ori, and the network deep feature I_fea is extracted from the image initial feature I_ori using the first dense residual structure module group; the two features of the image initial feature I_ori and the network deep feature I_fea are fused through the residual learning method and then processed by the second convolution layer, and then the feature result obtained by the second convolution layer is upsampled through the image upsampling module, and the feature result obtained by the upsampling of the image upsampling module is subjected to two Dropout+convolution post-processing operations through the post-processing module, and the preliminary super-resolution result map SRout is output;

[0036] The same low-resolution image is tested N times to obtain N different preliminary super-resolution result images SRout[SRout_1, SRout_2, …, SRout_N-1, SRout_N]. The variance map Var is obtained based on the N different preliminary super-resolution result images, and then the cognitive uncertainty map Var_img is obtained based on the variance map Var, which is expressed as:

[0037]

[0038]

[0039] Among them, Var represents the variance map, Var_img represents the cognitive uncertainty map, .MAX() represents taking the maximum value of the pixel value in the map; SRout_i represents the i-th preliminary super-resolution result map obtained by the i-th test.

[0040] During the training process, the original higher-resolution image in the known data set is used as the high-resolution image HR for downsampling processing to reduce the resolution and obtain a lower-resolution low-resolution image LR. A high-resolution image HR and its corresponding low-resolution image LR together constitute a pair of training images, which are then input into the cognitive uncertainty map generator.

[0041] During the training process of S3, the cognitive uncertainty graph generator uses the content loss L1, the perceptual loss Perceptual_Loss, and the GAN loss RaGAN_loss comprehensive loss for fusion processing as the total loss, and updates the generator parameters:

[0042] First, the content loss L1 is obtained according to the relationship between the preliminary super-resolution result map SRout and the high-resolution map HR obtained by the cognitive uncertainty map generator. The perceptual loss Perceptual_Loss is obtained according to the relationship between the preliminary super-resolution result map SRout and the high-resolution map HR obtained by the cognitive uncertainty map generator after being processed by the Vgg19 network. It is specifically expressed as:

[0043] L pix =E‖SRout-HR‖1

[0044] L Per =E‖Vgg(SRout)-Vgg(HR)‖1

[0045] Among them, Vgg represents the trained Vgg19 network, Vgg() represents the use of the Vgg19 network to extract the feature map, the initial super-resolution result map SRout and the high-resolution map HR are input into the Vgg19 network to convert them into the feature space to obtain the feature map in the feature space, ‖‖1 represents the absolute value, and E represents the mean;

[0046] Then, the discriminator D is established. According to the relationship between the initial super-resolution result map SRout and the high-resolution map HR obtained by the cognitive uncertainty map generator after being processed by the discriminator D, the GAN loss RaGAN_loss is obtained, which is specifically expressed as:

[0047] L GAN_D =-E[log(D(HR,SRout))]-E[log(1-D(SRout,HR)]

[0048] L GAN_G =-E[log(1-D(HR,SRout))]-E[log(D(SRout,HR)]

[0049] Among them, L GAN_D , L GAN_G They represent the GAN loss used to update the discriminator D and the GAN loss used to update the cognitive uncertainty map generator respectively; D represents the discriminator, specifically the true and false discrimination information obtained by inputting the discriminator D, D(HR,SRout) represents the true and false discrimination information of the preliminary super-resolution result map SRout obtained by inputting the discriminator D, and D(SRout,HR) represents the true and false discrimination information of the high-resolution map HR obtained by inputting the VGG128 network;

[0050] Finally, the weighted addition of content loss L1, perceptual loss Perceptual_Loss, and GAN loss RaGAN_loss is performed to obtain the following comprehensive loss:

[0051] L G =α*L pix +β*L Per +γ*L GAN_G

[0052] L D_total =L GAN_D

[0053] Among them, L G represents the total generator loss of the epistemic uncertainty graph generator, L D_total Represents the total discriminator loss of the cognitive uncertainty graph generator; α, β, and γ represent the three weights of content loss L1, perceptual loss Perceptual_Loss, and GAN loss RaGAN_loss, and their specific values ​​are 0.1, 1, and 0.05 respectively.

[0054] The discriminator D specifically adopts a Vgg128 network structure.

[0055] During the test of the cognitive uncertainty graph generator, Monte Carlo Dropout sampling needs to be enabled, and the upsampling rate should be kept consistent with that in the training phase.

[0056] In the training process of S3, the simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss are fused as the total loss for the super-resolution generator from easy to difficult, and the generator parameters are updated:

[0057] The simple area loss loss is obtained according to the relationship between the simple distribution map gt_Easymap in the easy-to-difficult super-resolution generator and the high-resolution map HR and the simple area super-resolution result map srEasy input by the cognitive uncertainty map generator:

[0058] L easy =E‖gt_Easymap*HR-gt_Easymap*srEasy‖1

[0059] The difficult area loss Loss is obtained according to the relationship between the difficult distribution map gt_Hardmap in the easy-to-difficult super-resolution generator and the high-resolution map HR and the difficult area super-resolution result map srHard input by the cognitive uncertainty map generator:

[0060] L hard =E‖gt_Hardmap*HR, gt_Hardmap*srHard‖1

[0061] In the specific implementation, any image obtained after processing by the convolution upsampling module is used as a distribution map and input into the above formula to obtain the simple area loss loss and the difficult area loss Loss.

[0062] The overall super-resolution loss Loss is obtained by processing in the same way as the total loss of the cognitive uncertainty map generator, including the content loss L1, the perceptual loss Perceptual_Loss, and the GAN loss RaGAN_loss comprehensive loss of the super-resolution generator generator from easy to difficult to process, and the total loss L of the generator generator from easy to difficult to process super-resolution generator is obtained. G ;

[0063] The frequency domain loss Loss is obtained by processing the relationship between the final super-resolution result image ETHSRout output by the easy-to-difficult super-resolution generator and the high-resolution image HR input by the cognitive uncertainty map generator:

[0064] L sr_frequency =E‖fft2(ETHSRout)-fft2(HR)‖1+BCE(PSD(ETHSRout),PSD(HR))

[0065] Among them, fft2 represents two-dimensional Fourier transform, PSD represents the spectrum curve of the two-dimensional image, and BCE is the binary cross entropy loss function, which is used to measure the similarity of two curves;

[0066] The present invention can further enhance the network's ability to process high frequencies and limit the generation of pseudo textures by the GAN method by adopting a specially added frequency domain loss Loss.

[0067] Finally, the total loss of the network generator is obtained by weighted addition of the simple area loss, the difficult area loss, the overall super-resolution loss and the frequency domain loss. The total loss is the sum of the above loss weights, expressed as:

[0068] L G_total =δ*L easy +ε*L hard +μ*L G +k*L sr_frequency

[0069] Among them, L easy , L hard , L G , L sr_frequency They represent simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss, respectively. δ, ε, μ and k represent the four weights of simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss, respectively, which are equal to 1, 4, 1 and 1. The discriminator and the Loss used by the discriminator of the easy-to-difficult super-resolution generator are consistent with the settings of the cognitive uncertainty map generator.

[0070] Therefore, the present invention realizes that the super-resolution network divides the low-resolution image into simple areas and complex areas for separate processing, and at the same time realizes the restoration of the "simple" area first and then the "complex" area.

[0071] The present invention applies cognitive uncertainty to super-resolution methods for the first time, and uses the difficulty of the network in processing images as a network prior for the first time to construct a network structure that processes images in different regions and stages.

[0072] The uncertainty map generator of the present invention is constructed by a dense residual structure module, and the upsampled features are resampled by applying a sampling module Dropout, and Dropout is added for sampling during testing to achieve approximate construction of a cognitive uncertainty map.

[0073] During the testing process, the cognitive uncertainty map generator performs N tests on the same low-resolution image input, obtains N different super-resolution results and calculates their variance values ​​to generate a cognitive uncertainty map that approximates the low-resolution image. The uncertainty map can characterize the different processing capabilities of the network for different regions of the image. The easy-to-difficult super-resolution generator uses the uncertainty map generator to construct a cognitive uncertainty map corresponding to the low-resolution image, and constructs an easy-to-difficult super-resolution generator. The input low-resolution image first processes the "easy" processing area, and then processes the "difficult" area. Specifically, it includes two processes: the training process and the testing process of the easy-to-difficult super-resolution generator. In the training process, the low-resolution image, the high-resolution original image and the corresponding cognitive uncertainty map are input, and the low-resolution image is processed by region and stage. Different loss functions are used for different regions, and the final super-resolution result is obtained by fusing the super-resolution results of different regions. In the final test process, only the low-resolution image needs to be input to obtain the final super-resolution result.

[0074] Beneficial effects of the present invention:

[0075] The innovation of the present invention lies in that, for low-resolution images, a cognitive uncertainty map can be first generated through a cognitive uncertainty map generator to clearly understand the difficulty of processing different areas of the image by the super-resolution network; secondly, a more accurate and refined processing model for low-resolution images is constructed using the easy-to-difficult processing super-resolution generator, thereby realizing image region processing.

[0076] The present invention can reduce the pseudo texture produced by the generative adversarial network (GAN) when generating images, can produce more realistic structure and texture information, has a natural effect, and has good visual quality. Objective and subjective indicators on multiple data sets can reach optimal or suboptimal values, thereby improving the reconstruction performance of the super-resolution method. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of an epistemic uncertainty graph generator of the method of the present invention;

[0078] Figure 2 A schematic diagram of a super-resolution generator from easy to difficult processing according to the method of the present invention;

[0079] Figure 3 This is the cognitive uncertainty map generated by the cognitive uncertainty map generator of the method example of the present invention. It can be seen that the super-resolution method using GAN often has good processing capabilities for large structural information and flat areas of low-resolution images, but is relatively insufficient for processing smaller details and texture information;

[0080] Figure 4srEasy_map and srHard_map obtained by the easy-to-difficult super-resolution generator in the example of the method of the present invention, and it can be seen that they have a high consistency with the epistemic uncertainty map generated by the epistemic uncertainty generator through the low-resolution image;

[0081] Figure 5 This is a comparison chart of the 4× super-resolution effect of natural images in the example of the present invention and other methods. From left to right, each column corresponds to the ESRGAN effect diagram, the NatSR effect diagram, the SPSR effect diagram, the effect diagram of the present invention and the high-resolution original image. For the low-resolution images with more structural information in the first row, the present invention is more capable of recovering image features with higher spatial rate. Other GAN-based methods cannot recover distinguishable image periodic structures, and the image structural information recovered by the present invention is obvious; for the low-resolution images with worse quality in the second row, other methods will produce obvious pseudo textures, and the textures recovered by the present invention are real and natural; for the relatively flat areas in the third row of images, compared with other methods, the results produced by the present invention are more natural and the structural features are obvious; for the facial information in the fourth row of images, compared with other methods, the results produced by the present invention are more natural and the facial structure distortion is relatively small.

[0082] Table 1 is a comparison of subjective and objective evaluation indicators of the present invention and other super-resolution methods on 4× natural images. The evaluation indicators are PSNR, SSIM, LPIPS, and NIQE. The higher the PSNR and SSIM, the better the super-resolution effect. The lower the LPIPS and NIQE values, the better the super-resolution effect. Compared with other methods, the present invention obtains the best or suboptimal indicators in almost all data sets, which shows the advancement of the present invention in the field of single-frame image super-resolution algorithms. DETAILED DESCRIPTION

[0083] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0084] The embodiments and implementation processes fully implemented according to the content of the present invention are as follows:

[0085] The first step is to build the data sets needed for the training and testing process of the super-resolution generator and cognitive uncertainty map generator from easy to difficult: first, build the training data set, download the high-resolution data set DIV2K, take the training pictures in the data set, first use MATLAB's image scaling function resize function to reduce the image to one-fourth of the original image, and the reduction method used is the bicubic interpolation method to obtain high-resolution and low-resolution image data pairs, and then cut the high-resolution and low-resolution image data pairs into image blocks with a length and width of 480 pixels, and the training data set is completed; second, build the test data set, download the Set5, Set14, BSDS100, and Urban100 test data sets as experimental test sets.

[0086] The second step is to build an epistemic uncertainty graph generator: Figure 1 The figure shows a schematic diagram of the cognitive uncertainty map generator of the present method. The construction of the cognitive uncertainty map generator is divided into two steps. The first is Train Line. The low-resolution image LR is passed through the first convolution layer of the feature extraction module to obtain the image initial feature I_ori. The image initial feature I_ori is extracted from the network deep feature I_fea using the first dense residual structure module group; the two features of the image initial feature I_ori and the network deep feature I_fea are fused through residual learning and then processed by the second convolution layer. Subsequently, the feature result obtained by the second convolution layer is upsampled through the image upsampling module. The feature result obtained by upsampling in the image upsampling module is post-processed by two Dropout+convolution operations through the post-processing module to output the preliminary super-resolution result map SRout. The output result is the same as the high-resolution original image using the total loss L of the generator of the cognitive uncertainty map generator. G Calculate the loss, use the Adam optimizer to feed the obtained loss back to the network to update the network parameters until the network training reaches the fit. Then the Test Line tests the same low-resolution image N times, and obtains N different preliminary super-resolution result images SRout[SRout_1,SRout_2,…,SRout_N-1,SRout_N]. According to the N different preliminary super-resolution result images, the variance map Var is calculated, and then the cognitive uncertainty map Var_img is obtained according to the variance map Var. That is, the cognitive uncertainty map of the network for the low-resolution image can be approximated. The area with a large variance value indicates that the network has poor processing ability for the area, and when the variance value is small, it indicates that the network has good processing ability for the area.

[0087] In the third step, the cognitive uncertainty map generated by the cognitive uncertainty map generator in the second step is combined to construct a super-resolution generator from easy to difficult processing: Figure 2The figure shows the schematic diagram of the super-resolution generator from easy to difficult processing in this method. The creation of the super-resolution generator from easy to difficult processing is divided into two steps. First, Train Line, the overall input of the network is: the corresponding low-resolution image, cognitive uncertainty map, and high-resolution original image. The cognitive uncertainty map is first binarized using the Otsu method, and then the binarized image is expanded using a 3×3 all-1 structure element to obtain gtEasy_map. The gtEasy_map is negated to obtain gtHard_map, which is used to locate the "simple" and "difficult" areas. The low-resolution image is input into the convolution layer. After adding the input and output of the second dense residual structure module itself, the simple area feature map srEasy_feature is obtained. After adding the input and output of the third dense residual structure module itself, the difficult area feature map srHard_feature is obtained. The simple area feature map srEasy_feature is processed by a convolution upsampling module to obtain the simple area super-resolution result map srEasy and the simple area distribution map srEasy_map. The difficult area feature map srHard_feature is processed by another convolution upsampling module to obtain the difficult area super-resolution result map srHard and the difficult area distribution map srHard_map;

[0088] Then, according to the simple area super-resolution result map srEasy and the simple area distribution map srEasy_map as well as the difficult area super-resolution result map srHard and the difficult area distribution map srHard_map, srEasy and srHard are fused to obtain the final super-resolution result map ETHSRout.

[0089] Network learning uses different loss functions for different regions. For the supervision of srEasy and srHard, the present invention uses content loss L1 for supervised learning. For the supervised learning of the final super-resolution result image ETHSRout, we use content loss L1, perceptual loss Perceptual_Loss, GAN loss RaGAN_loss comprehensive loss, and finally add frequency domain loss to obtain the total loss of the network generator lossL G_total , using the total loss loss L of the network generator G_total Calculate the sum of losses in different regions and use Adam optimization to optimize the parameters of the network until the total loss of the network generator is L G_total When testing the line, only low-resolution images are input, and the trained easy-to-difficult super-resolution generator can be directly used to perform super-resolution processing on the low-resolution images, obtaining super-resolution results with clear structures and real and natural textures.

[0090] Table 1

[0091]

[0092] According to the above-described implementation method, a trained easy-to-difficult super-resolution generator is obtained. In order to prove the present invention, the method of the present invention is compared with ESRGAN, NatSR, and SPSR methods in the field of single-frame image super-resolution, and NIQE, PSNR, SSIM, and LPIPS are used as evaluation indicators, wherein NIQE is a reference-free image evaluation indicator that can directly give the quality score of the image. The lower the value, the higher the image quality. LPIPS measures the visual quality similarity of the super-resolution image and the high-resolution image in the feature space. The lower the value, the higher the image quality. As can be seen from the above table, compared with other methods, the present invention can achieve the best or second-best PSNR / SSIM indicators, and the present invention achieves the best visual quality NIQE and LPIPS indicators on almost all data sets.

[0093] The experimental results prove that the present invention is advanced in both objective and visual indicators.

[0094] The present invention applies cognitive uncertainty to the field of single-frame image super-resolution for the first time, achieving more reasonable and refined processing of different areas of low-resolution images, which can produce clearer structures and more realistic texture information with natural effects and good visual quality. Compared with other mentioned super-resolution methods, the present invention can achieve optimal or suboptimal objective performance indicators and subjective performance indicators on multiple data sets, and is advanced in the field of single-frame image super-resolution.

Claims

1. A single-frame image super-resolution method based on cognitive uncertainty, characterized by: The method proceeds as follows: S1. Build an easy-to-difficult super-resolution generator with epistemic uncertainty. S2, establish a cognitive uncertainty map generator for training super-resolution generators from easy to difficult processing; S3, integrate the cognitive uncertainty map generator into the training process of the super-resolution generator from easy to difficult processing; S4, inputting the single frame image as the low-resolution image to be tested into the trained easy-to-difficult super-resolution generator to obtain the final super-resolution result image; The specific structure of the easy-to-difficult processing super-resolution generator includes a low-resolution branch; The low-resolution branch mainly consists of a convolutional layer, a second dense residual structure module, a third dense residual structure module, a simple area upsampling module Up*4_a, and a difficult area upsampling module Up*4_b connected in sequence; the input and output of the second dense residual structure module are added and input into the simple area upsampling module Up*4_a and the third dense residual structure module respectively, and the input and output of the third dense residual structure module are added and input into the difficult area upsampling module Up*4_b. The second dense residual structure module and the third dense residual structure module are both composed of 10 dense residual blocks connected in sequence; The simple area upsampling module Up*4_a and the difficult area upsampling module Up*4_b have the same topological structure, both of which include two parallel convolution upsampling modules, each of which is mainly composed of three sequentially connected convolution layers and an image upsampling module connected between the first convolution layer and the second convolution layer; The specific structure of the cognitive uncertainty map generator is mainly composed of a feature extraction module, an image upsampling module and two consecutive post-processing modules connected in sequence, and each post-processing module is composed of a Monte Carlo Dropout sampling module and a convolutional layer connected in sequence; The feature extraction module is mainly composed of a first convolutional layer, a first dense residual structure module and a second convolutional layer. The first dense residual structure module is set between the first convolutional layer and the second convolutional layer and a residual learning method is adopted. The output of the first convolutional layer and the output of the first dense residual structure module are weighted and fused and then input into the second convolutional layer. The first dense residual structure module is composed of 20 dense residual blocks connected in sequence. The Monte Carlo Dropout sampling module adopts a Dropout layer; In the cognitive uncertainty map generator, Monte Carlo Dropout sampling is turned on, a low-resolution image LR and its corresponding high-resolution image HR are input, the low-resolution image LR is passed through the first convolution layer of the feature extraction module to obtain the image initial feature I_ori, and the network deep feature I_fea is extracted from the image initial feature I_ori using the first dense residual structure module group; the two features of the image initial feature I_ori and the network deep feature I_fea are fused through the residual learning method and then processed by the second convolution layer, and then the feature result obtained by the second convolution layer is upsampled through the image upsampling module, and the feature result obtained by the upsampling of the image upsampling module is subjected to two Dropout+convolution post-processing operations through the post-processing module, and the preliminary super-resolution result map SRout is output; The same low-resolution image is tested N times to obtain N different preliminary super-resolution result images SRout[SRout_1, SRout_2, …, SRout_N-1, SRout_N]. The variance map Var is obtained based on the N different preliminary super-resolution result images, and then the cognitive uncertainty map Var_img is obtained based on the variance map Var, which is expressed as: Where Var represents the variance map, Var_img represents the cognitive uncertainty map, .MAX() represents the maximum value of the pixel value in the map; SRout_i represents the i-th preliminary super-resolution result map obtained by the i-th test; During the training process of S3, the cognitive uncertainty graph generator uses the content loss L1, the perceptual loss Perceptual_Loss, and the GAN loss RaGAN_loss comprehensive loss for fusion processing as the total loss: First, the content loss L1 is obtained according to the relationship between the preliminary super-resolution result map SRout and the high-resolution map HR obtained by the cognitive uncertainty map generator. The perceptual loss Perceptual_Loss is obtained according to the relationship between the preliminary super-resolution result map SRout and the high-resolution map HR obtained by the cognitive uncertainty map generator after being processed by the Vgg19 network. It is specifically expressed as: L pix =E||SRout-HR‖1 L Per =E||Vgg(SRout)-Vgg(HR)||1 Among them, Vgg represents the trained Vgg19 network, Vgg() represents the feature map extracted by the Vgg19 network, ‖‖1 represents the absolute value, and E represents the mean; Then, the discriminator D is established. According to the relationship between the initial super-resolution result map SRout and the high-resolution map HR obtained by the cognitive uncertainty map generator after being processed by the discriminator D, the GAN loss RaGAN_loss is obtained, which is specifically expressed as: L GAN_D =-E[log(D(HR,SRout))]-E[log(1-D(SRout,HR)] L GAN_G =-E[log(1-D(HR,SRout))]-E[log(D(SRout,HR)] Among them, L GAN_D , L GAN_G They represent the GAN loss used to update the discriminator D and the GAN loss used to update the cognitive uncertainty map generator respectively; D represents the discriminator, specifically the true and false discrimination information obtained by inputting the discriminator D, D(HR,SRout) represents the true and false discrimination information of the preliminary super-resolution result map SRout obtained by inputting the discriminator D, and D(SRout,HR) represents the true and false discrimination information of the high-resolution map HR obtained by inputting the VGG128 network; Finally, the weighted addition of content loss L1, perceptual loss Perceptual_Loss, and GAN loss RaGAN_loss is performed to obtain the following comprehensive loss: L G =α*L pix +β*L Per +γ*L GAN_G L D_total =L GAN_D Among them, L G represents the total generator loss of the epistemic uncertainty graph generator, L D_total Represents the total discriminator loss of the cognitive uncertainty graph generator; α, β, and γ represent the three weights of content loss L1, perceptual loss Perceptual_Loss, and GAN loss RaGAN_loss; In the training process of S3, the simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss are fused as the total loss for the super-resolution generator from easy to difficult: The simple area loss loss is obtained according to the relationship between the simple distribution map gt_Easymap in the easy-to-difficult super-resolution generator and the high-resolution map HR and the simple area super-resolution result map srEasy input by the cognitive uncertainty map generator: L easy =E||gt_Easymap*HR-gt_Easymap*srEasy|‖1 The difficult area loss Loss is obtained according to the relationship between the difficult distribution map gt_Hardmap in the easy-to-difficult super-resolution generator and the high-resolution map HR and the difficult area super-resolution result map srHard input by the cognitive uncertainty map generator: L hard =E||gt_Hardmap*HR,gt_Hardmap*srHard‖1 The overall super-resolution loss Loss is obtained by processing in the same way as the total loss of the cognitive uncertainty map generator, and the total loss L of the generator of the super-resolution generator generator is obtained from easy to difficult. G ; The frequency domain loss Loss is obtained by processing the relationship between the final super-resolution result image ETHSRout output by the easy-to-difficult super-resolution generator and the high-resolution image HR input by the cognitive uncertainty map generator: L sr_frequency =E||fft2(ETHSRout)-fft2(HR)||1+BCE(PSD(ETHSRout),PSD(HR)) Among them, fft2 represents two-dimensional Fourier transform, PSD represents the spectrum curve of the two-dimensional image, and BCE is the binary cross entropy loss function; Finally, the simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss are weighted and added to obtain: L G_total =δ*L easy +e*L hard +μ*L G +k*L sr_frequency Among them, L easy , L hard , L G , L sr_frequency They represent simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss respectively. δ, ε, μ and k represent the four weights of simple area loss, difficult area loss, overall super-resolution loss and frequency domain loss respectively.

2. The single-frame image super-resolution method based on cognitive uncertainty according to claim 1, characterized in that: During the training process of S3, the low-resolution image LR and its corresponding high-resolution image HR are input into the cognitive uncertainty map generator to obtain a preliminary super-resolution result map corresponding to the low-resolution image LR, and the preliminary super-resolution result map is used to obtain the cognitive uncertainty map Var_img. Then, the low-resolution image LR and its corresponding cognitive uncertainty map Var_img are input into the easy-to-difficult super-resolution generator to obtain the final super-resolution result map.

3. The single-frame image super-resolution method based on cognitive uncertainty according to claim 1, characterized in that: The easy-to-difficult super-resolution generator also includes a cognitive uncertainty branch, which is mainly composed of a binarization operation, a dilation operation and a negation operation connected in sequence.

4. A single-frame image super-resolution method based on cognitive uncertainty according to claim 1 or 3, characterized in that: In S4, for the single-frame image processing of the low-resolution image to be tested, the single-frame image is input into the low-resolution branch of the easy-to-difficult super-resolution generator for processing to obtain the final super-resolution result image ETHSRout.

5. The single-frame image super-resolution method based on cognitive uncertainty according to claim 1, characterized in that: The image upsampling module in the cognitive uncertainty map generator / easy-to-difficult super-resolution generator is composed of two two-fold upsampling operations, and each two-fold upsampling operation is mainly composed of two bilinear interpolation upsampling operations, convolution layers, and LRELU activation functions connected in sequence.

Images