Unsupervised image denoising method based on uncertainty perception

Abstract

The present invention belongs to the field of digital image processing technology, and specifically is an unsupervised image denoising method based on uncertainty perception. The method of the present invention comprises: constructing a smooth noise extraction module and a rough denoising network to simulate noise blocks and perform rough denoising on images; constructing a network for modeling accidental uncertainty, estimating pixels with high uncertainty in the noise image, generating an uncertainty map, and guiding the next stage of the network to perform denoising; constructing a fine denoising network, and guiding the network to reduce the influence of uncertainty on the denoising process through the weighted objective function of the uncertainty map, so that the network can achieve better denoising effect. Experimental results show that the present invention can effectively remove noise from images, and the denoised images have good texture structure and visual quality.

Description

Technical Field

[0001] The invention belongs to the technical field of digital image processing, and in particular relates to an image denoising method. Background Art

[0002] Image denoising is a classic and challenging image processing task that has attracted more and more attention. The noise in the image is mainly generated in the process of acquiring the image. Two commonly used types of image sensors, CCD and CMOS, will introduce various noises due to their own sensor material properties, working environment, electronic components and circuit structure, such as thermal noise caused by resistors, channel thermal noise of field effect transistors, photon noise, dark current noise, and light response non-uniformity noise. The presence of noise will not only lead to a decrease in the visual quality of the image, but may even bring more serious effects. For example, low-dose radiation CT images are noisy due to the reduced radiation intensity, and these noises will affect doctors' judgment of the lesion area and the patient's condition.

[0003] In recent years, the emergence of deep convolutional neural network (DCNN) technology has greatly improved the performance of denoising networks. In 2016, Zhang et al. [2] proposed a DnCNN model that combines convolutional neural networks and residual learning for image denoising. Because the model uses paired clean and noisy image datasets to train the network, its denoising performance depends largely on the quality and quantity of the training dataset. In 2018, Lehtinen et al. [3] A weakly supervised denoising method, Noise2Noise, is proposed. Instead of using a clean image dataset to train the network, the model uses a paired noisy image dataset. Experiments show that the network trained with a paired noisy image dataset can effectively remove additive noise with a mean of 0 and independent of the clean image. Inspired by the Noise2Noise method, Krull et al. [4] A blind convolution structure was designed and the Noise2Void method was proposed. This model can train the network to achieve denoising using only a single noisy image. Although the above methods have good performance in denoising tasks, they tend to generate the average of multiple possible denoising results and do not consider the impact of accidental uncertainty in the training data set on the denoising network, resulting in the network denoising results being too smooth and the visual effect being poor. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an unsupervised image denoising method based on uncertainty perception, which mainly reduces the impact of accidental uncertainty in the training data set on the network training process so that the denoising result can maintain a clear texture structure and achieve satisfactory visual quality.

[0005] The unsupervised image denoising method based on uncertainty perception provided by the present invention has the following specific steps:

[0006] (1) Construct a smooth noise extraction module and a rough denoising network to simulate noise blocks and perform rough denoising on images;

[0007] (2) Construct a network to model accidental uncertainty, estimate pixels with high uncertainty in the noisy image, and generate an uncertainty map to guide the next stage of the network to perform denoising;

[0008] (3) Construct a precise denoising network and guide the network to reduce the impact of uncertainty on the denoising process through the uncertainty graph weighted objective function, so that the network can achieve better denoising effect.

[0009] In step (1), the noise extraction module and the rough denoising network are extracted using GAN2GAN [1] The method in which:

[0010] The noise extraction module first randomly cuts out a 96×96 image block from the original noise image and processes the image block using discrete wavelet transform (DWT). The rules for determining whether an image block is a smooth noise block are as follows:

[0011]

[0012] σ(W(p)) represents the DWT processing process. The four images obtained after the image block p is processed by DWT contain the low-frequency information, horizontal high-frequency information, vertical high-frequency information, and diagonal high-frequency information of p. We define these four images as If the image p satisfies equation (1), then p is determined to be a smooth noise block. In equation (1), σ is the empirical standard deviation coefficient of discrete wavelet transform, and λ is a hyperparameter that satisfies λ∈(0,1).

[0013] The rough denoising network includes three generators and two discriminators. The three generators are denoted as G1, G2, and G3, and the two discriminators are denoted as C1 and C2. G1 simulates the extracted noise module and can generate independent and identically distributed noise. With the assistance of G3, G2 realizes rough denoising of the noise block. C1 and C2 judge the quality of the generation results of G1 and G2 respectively.

[0014] In step (2), the aleatoric uncertainty network adopts a binary structure; the network is constructed on the basis of DnCNN, specifically, a network module consisting of a convolutional layer, a batch normalization layer and an ELU activation layer module stacked 5 times is inserted in the middle of the DnCNN network as a branch network of the binary structure, and the second half of the original DnCNN network becomes another branch network of the binary structure; the two branch networks share the network structure of the first half of the DnCNN network, and the two branch networks respectively output the mean M and variance U in the aleatoric uncertainty in the form of a graph, where U is the uncertainty graph.

[0015] Furthermore, before training the network for predicting accidental uncertainty, we first need to use the noise simulator and the rough denoising network trained in step (1) to synthesize a pair of noise images. Assuming the noise image is Y, first pass through the rough denoising network G2 to obtain a relatively clean image X; generate two independent and identically distributed noises N1 and N2 from G1, add them to X respectively, and obtain a pair of noise images Z1 and Z2;

[0016] Furthermore, the loss function for training a binary network that models aleatoric uncertainty is as follows:

[0017]

[0018] Where K is the total number of noise samples input to the network, f(·) represents a branch network in the binary network, and f(z2 i ) is similar to z1 i The image of , corresponds to the concept of "mean" in aleatoric uncertainty. i It is the uncertainty map output by the other branch of the binary network, corresponding to the concept of "variance" in accidental uncertainty. Specifically, the uncertainty map measures f(z2 i ), that is, in u i In the , the pixel area with larger value has higher uncertainty, that is, lower confidence.

[0019] In step (3), the fine denoising network is constructed based on DnCNN; the first two layers of the network are convolutional layers and ReLU activation layers, followed by 15 modules consisting of convolutional layers, batch normalization layers and ReLU activation layers, and the last layer is a convolutional layer; the number of channels of the first convolutional layer of the network is 2.

[0020] Furthermore, the input of the fine denoising network is the result of channel-wise concatenation of the noise image and the gradient map. Specifically, firstly, the rough denoising network trained in step (1) is used to obtain the rough denoising image D of the noise image Z2, and then the Sobel operator and formula (3) are used to obtain the gradient map G corresponding to D:

[0021]

[0022]

[0023] Among them, S h and S v There are two types of convolution kernels, and * represents the convolution operation. Finally, the gradient map G and the noise image Z2 are concatenated by channel and sent to the network.

[0024] Furthermore, the fine denoising network adopts the MSE loss weighted by the uncertainty map U; first, Z2 obtains the variance map U, that is, the uncertainty map, by predicting the binary network of accidental uncertainty. Pixels with high uncertainty need to be valued by the network; then, the uncertainty map U is used to weight the loss function of the network; so the loss function of the network is:

[0025]

[0026] Furthermore, the model is trained in stages, and the parameters of all networks in each stage are fixed after each stage of training. During testing, first, the original noisy image is input into the trained rough denoising network G2 to obtain a relatively clean image, and then the gradient of the relatively clean image is calculated to obtain a gradient map. The gradient map and the original noisy image are spliced ​​by channel and sent to the trained fine denoising network to obtain the final denoising result.

[0027] The beneficial effects of the present invention are:

[0028] The present invention can predict an uncertainty map for each input noise sample, and increase the penalty for areas with higher uncertainty to make the network more focused on the accuracy of generating these pixels, thereby reducing the interference of uncertainty on the network and making the denoising result have a clear texture structure and higher confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the present invention.

[0030] Figure 2 The following are the results of denoising a noisy image using the present invention, where (a) is the original noisy image and (b) is the denoised image. DETAILED DESCRIPTION

[0031] The embodiments of the present invention are described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0032] The specific steps are:

[0033] (1) Before training, a number of smooth noise blocks of size 96×96 were extracted using the smooth noise block extraction rule (the number of noise blocks is closely related to the noise intensity). 20,000 noise image blocks of size 96×96 were randomly cropped from the original noise image as the training data set. The network model parameters were initialized using the default method.

[0034] (2) During training, randomly crop the 96×96 noise image and smooth the noise block to 64×64. The training cycle is 30, the initial learning rate is set to 0.001, the decay rate is 0.5, and it decays once every ten cycles. The loss function is minimized using the small batch stochastic gradient descent method. The batch size is set to 128. The model adopts a staged training method. After a stage of training, the parameters of all networks in that stage are fixed.

[0035] (3) During the test, the original noisy image is 512×512 in size. It is first input into the trained rough denoising model to obtain a cleaner image, and then the gradient of the cleaner image is calculated to obtain a gradient map. The gradient map and the original noise are concatenated by channel and sent into the trained fine denoising network to obtain the final denoising result.

[0036] References

[0037] [1]Cha S, Park T, Moon T.GAN2GAN: Generative Noise Learning for Iterative Blind Denoising with Single Noisy Images[J].2019.

[0038] [2] Zhang K, Zuo W, Chen Y, et al. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising[J]. IEEE Transactions on ImageProcessing, 2016, 26(7): 3142-3155.

[0039] [3]Lehtinen J, Munkberg J, Hasselgren J, et al. Noise2Noise: LearningImage Restoration without Clean Data[J].2018.

[0040] [4]Krull A,Buchholz T O,Jug F.Noise2Void-Learning Denoising FromSingle Noisy Images[C] / / 2019IEEE / CVF Conference on Computer Vision andPattern Recognition(CVPR).IEEE,2019。

Claims

1. An unsupervised image denoising method based on uncertainty perception, characterized in that: The specific steps are as follows: (1) Construct a noise extraction module and a rough denoising network to simulate noise blocks and perform rough denoising on images; (2) Construct a network to model accidental uncertainty, estimate pixels with high uncertainty in the noisy image, and generate an uncertainty map to guide the next stage of the network to perform denoising; (3) Construct a fine denoising network and guide the network to reduce the impact of uncertainty on the denoising process through the uncertainty graph weighted objective function, so that the network can achieve better denoising effect; In step (1), the rough denoising network includes three generators and two discriminators, the three generators are respectively denoted as G1, G2, and G3, and the two discriminators are respectively denoted as C1 and C2; G1 simulates the extracted noise module and can generate independent and identically distributed noise; with the assistance of G3, G2 realizes rough denoising of the noise block; C1 and C2 respectively judge the quality of the generated results of G1 and G2; In step (2), the network of accidental uncertainty adopts a binary structure; the network is constructed on the basis of DnCNN, specifically, a network module consisting of a convolutional layer, a batch normalization layer and an ELU activation layer module stacked 5 times is inserted in the middle of the DnCNN network as a branch network of the binary structure, and the second half of the original DnCNN network becomes another branch network of the binary structure; the two branch networks share the network structure of the first half of the DnCNN network, and the two branch networks respectively output the mean M and variance U in the accidental uncertainty in the form of a graph, where U is the uncertainty graph; Before training the network for predicting aleatoric uncertainty, we first use the noise module and the rough denoising network trained in step (1) to synthesize a pair of noise images. Assuming the noise image is Y, we first pass it through the rough denoising network G2 to obtain a relatively clean image X. G1 generates two independent and identically distributed noises N1 and N2, which are added to X to obtain a pair of noise images Z1 and Z2. The loss function of the network for training modeling aleatoric uncertainty is as follows: Where K is the total number of noise samples input to the network, f(·) represents a branch network in the binary network, and f(z2 i ) is similar to z1 i The image of corresponds to the concept of "mean" in accidental uncertainty; u i It is the uncertainty map output by the other branch of the binary network, corresponding to the concept of "variance" in accidental uncertainty; specifically, the uncertainty map measures f(z2 i ), that is, in u i In the , the pixel area with larger value has higher uncertainty, that is, lower confidence.

2. The method according to claim 1, characterized in that In step (1), the noise extraction module and the rough denoising network adopt the method in GAN2GAN; wherein: The noise extraction module first randomly cuts out a 96×96 image block from the original noise image, processes the image block using discrete wavelet transform DWT, and determines whether the image block is a smooth noise block. The determination rules are as follows: σ(W(p)) represents the DWT processing process. The four images obtained after the image block p is processed by DWT contain the low-frequency information, horizontal high-frequency information, vertical high-frequency information, and diagonal high-frequency information of p. These four images are uniformly defined as If the image p satisfies formula (1), then p is determined to be a smooth noise block; in formula (1), σ is the empirical standard deviation coefficient of discrete wavelet transform, and λ is a hyperparameter that satisfies λ∈(0,1).

3. The method according to claim 1, characterized in that In step (3), the fine denoising network is constructed based on DnCNN; the first two layers of the network are convolutional layers and ReLU activation layers, followed by 15 modules consisting of convolutional layers, batch normalization layers and ReLU activation layers, and the last layer is a convolutional layer; the number of channels of the first convolutional layer of the network is 2.

4. The method according to claim 3, characterized in that The input of the fine denoising network is the result of channel-wise concatenation of the noise image and the gradient map. Specifically, the rough denoising network trained in step (1) is first used to obtain the rough denoising image D of the noise image Z2, and then the Sobel operator and formula (4) are used to obtain the gradient map G corresponding to D: Among them, S h and S v There are two types of convolution kernels, * represents the convolution operation; finally, the gradient map G and the noise image Z2 are concatenated by channel and sent to the network.

5. The method according to claim 4, characterized in that The fine denoising network adopts the MSE loss weighted by the uncertainty map U. First, Z2 obtains the variance map U, that is, the uncertainty map, through the network that predicts the accidental uncertainty. Pixels with high uncertainty need to be valued by the network. Then, the uncertainty map U is used to weight the loss function of the network. Therefore, the loss function of the network is:

6. The method according to any one of claims 1 to 5, characterized in that: During training, the model adopts a staged training method. After each stage of training, the parameters of all networks in that stage are fixed. During testing, first, the original noisy image is input into the trained rough denoising network G2 to obtain a cleaner image. Then the gradient of the cleaner image is calculated to obtain the gradient map, which is concatenated with the original noisy image by channel and sent to the trained fine denoising network to obtain the final denoising result.

Images