Robust self-supervised image denoising method

Abstract

This invention belongs to the field of digital image processing technology, specifically a robust self-supervised image denoising method based on constructing paired noise samples. The method includes: obtaining a coarsely denoised image of the noisy image through a pre-denoising network; subtracting the original noisy image from the coarsely denoised image in groups to obtain approximately realistically distributed noise; performing a cyclic shift operation within each group on the noise groups twice, adding the results to the coarsely denoised image respectively to obtain two groups of paired noisy samples; training a two-branch denoising network using the constructed paired noisy samples and an uncertainty-aware loss function to improve its denoising performance and robustness. Experimental results show that this method overcomes the shortcomings of previous self-supervised image denoising methods, effectively improving the clarity of the denoised image, and the method of constructing noisy samples has strong practical value.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, specifically relating to a robust self-supervised image denoising method. Background Technology

[0002] Image denoising aims to recover clean signals from noisy observations and is one of the important tasks in image processing and low-level computer vision. Recently, with the rapid development of neural networks, learning-based supervised denoising models have achieved satisfactory performance. However, these methods largely rely on paired noise-clean or noise-noise images. In practical applications, collecting such paired images is complex and expensive. Even in tasks such as dynamic scenes and medical imaging, suitable paired images are simply unavailable due to real-world limitations. This makes supervised image denoising methods difficult to adapt to real-world denoising scenarios or achieve ideal denoising results.

[0003] Compared to supervised image denoising methods, self-supervised image denoising methods are more practically valuable because they do not require clean images or paired images as references. Currently, most self-supervised methods can train denoising models using only a single noisy image. The core idea of ​​these methods is to construct pairs of noisy samples from a single noisy image. However, this process is extremely challenging, and the quality of the constructed pairs of noisy samples is closely related to the performance of the denoising model. Existing self-supervised denoising methods widely employ two strategies for constructing pairs of samples for learning: blind convolution and subsampling, such as N2V. [1] NBR [4] and B2UB [5] These strategies achieve the goal of learning denoising results from a single noisy image, but the performance improvement of the denoising model is hindered by problems such as insufficient information utilization and pixel misalignment in the strategies.

[0004] Furthermore, most self-supervised image denoising methods focus on improving the model's denoising performance while paying little attention to its robustness. This makes these denoising models highly sensitive to unseen noise and prone to severe degradation. In practice, since the noise distribution and intensity of collected noisy images cannot be guaranteed to strictly meet requirements, denoising networks must maintain robustness when facing complex scenes. Previous research has used an uncertainty-aware loss function... [3] It has been proven to effectively improve model stability, but this idea has not yet been discussed in denoising tasks. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing self-supervised image denoising tasks and propose a robust self-supervised image denoising method that significantly improves the performance of denoising networks.

[0006] The robust self-supervised image denoising method provided by this invention is based on constructing paired noise samples. The noisy samples synthesized by this method have the same "clean" scene and independent and identically distributed noise, which can effectively alleviate the problems of low information utilization and image misalignment, and significantly improve the performance of the denoising network. At the same time, an uncertainty-aware loss function is introduced to improve the robustness of the denoising model.

[0007] The robust self-supervised image denoising method provided by this invention comprises the following steps:

[0008] (1) Pre-denoising: The N2V denoising model proposed by Krull et al. [1] is used to perform coarse denoising on the noisy image;

[0009] (2) Constructing pairs of noisy samples: Using the original noisy image and the coarsely denoised image, pairs of noise samples are constructed through simple addition, subtraction and shifting operations;

[0010] (3) Training a dual-branch denoising network: A dual-branch denoising network is trained using the paired noise samples constructed in step (2). During the training process, an uncertainty-aware loss function is used to constrain the network, thereby significantly improving the denoising performance and robustness of the network.

[0011] In step (1), the N2V denoising model, denoted as f(·), is used to pre-denoise a set of noisy images Y = {y1, y2, y3}, resulting in the corresponding coarse denoising result X′ = {x′1, x′2, x′3}, where numerical subscripts mark different noisy images; that is:

[0012] X′=f(Y)

[0013] This process does not involve updating network parameters.

[0014] In step (2), the noise group N = {n1, n2, n3}, which approximates the true distribution, is first obtained by calculating the difference between the noisy image and the coarse denoising result.

[0015] N=YX′

[0016] Note that the noise within the noise group is independent and identically distributed at this point.

[0017] Secondly, performing two cyclic shift operations on the noise group yields two noise groups, N1 and N2, differing only in their order. Specifically, the noise element at the first position in each noise group is moved to the last position, repeated twice. Thus:

[0018] N1 = {n2, n3, n1}

[0019] N2 = {n3, n1, n2}

[0020] Finally, the two sets of noise are added to the coarse denoising result to construct pairs of noisy sample groups, Y′. A and Y′ B ;Right now:

[0021] Y′ A =X′+N1={y′ A1 y′ A2 y′ A3}

[0022] Y′ B =X′+N2={y′ B1 y′ B2 y′ B3}

[0023] Furthermore, the two pairs of noisy images constructed through the above process, for example: y′ A1 and y′ B1 Having the same "clean" scene x′ * And independent and identically distributed noise, where * represents any numerical subscript.

[0024] In step (3), a dual-branch denoising network is trained using the pairs of noisy samples constructed in step (2). Specifically, one of the noisy samples in the pairs is used as the input of the dual-branch network, and the other noisy sample is used as the target to be learned by the network.

[0025] The dual-noise reduction branch network in DnCNN [2] Built upon the foundation of DnCNN, the first two layers are convolutional layers and ReLU activation layers, followed by 15 modules consisting of convolutional layers, batch normalization layers, and ReLU activation layers, with the last layer being a convolutional layer. In the dual-branch network, a network module consisting of seven stacked convolutional layers, batch normalization layers, and ReLU activation layers is inserted in the middle of the DnCNN network as a denoising branch; the latter half of the original DnCNN forms the other branch, the uncertainty branch; the two branches share the first half of DnCNN. The convolutional kernel size in the branch networks is 3×3. The denoising branch outputs the denoised image, and the uncertainty branch outputs the uncertain image.

[0026] During the training of the dual-branch network, an uncertainty-aware loss function is used. [3] :

[0027]

[0028] Where K represents the total number of samples in the noisy dataset, f1(·) represents the branch in the two-branch network that outputs the denoised result, and u i It is the branch output result of predicting uncertainty in a two-branch network.

[0029] The uncertainty map of the uncertainty branch output measures the confidence level of the denoising result of the f1(·) branch output at the pixel level, i.e., in u i In the process, pixel regions with larger values ​​have higher uncertainty, and the corresponding regions in the denoising results of f1(·) output have lower confidence. In other words, pixel values ​​with high uncertainty deviate more from the pixel values ​​of the real clean image.

[0030] Compared to the MSE loss function commonly used in denoising tasks:

[0031]

[0032] The dual-branch network trained with the uncertainty loss function has the ability to perceive the difficulty of denoising in different regions of an image. When denoising a noisy image, it does not treat each pixel region equally, but makes a more intelligent choice for each pixel. This enables the dual-branch network to distinguish the impact of unseen noise on the difficulty of denoising when dealing with unseen noise, thus achieving better denoising performance and robustness. Attached Figure Description

[0033] Figure 1 This is a flowchart of the present invention.

[0034] Figure 2 This invention constructs a flowchart for pairing noisy samples.

[0035] Figure 3 A schematic diagram of the dual-branch denoising network structure of this invention.

[0036] Figure 4 The denoising results of a noisy image obtained using this invention are shown in Figure (a), where (b) is the noisy image and (a) is the denoised image. Detailed Implementation

[0037] The embodiments of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0038] The specific steps are as follows:

[0039] (1) When obtaining the coarse denoising result, the noisy image input to the network is randomly cropped into 128×128 noisy image blocks, and the image size remains unchanged when constructing pairs of noisy samples. A total of 40,000 pairs of noisy samples are constructed as the training set;

[0040] (2) During training, one pair of noisy samples is used as input, and the other is used as the target for the two-branch network to learn. The training cycle of the two-branch network is 100, the initial learning rate is set to 0.0003, the decay rate is 0.5, and it decays once every 20 cycles. The training process uses the mini-batch stochastic gradient descent method to minimize the loss function, and the batch size is set to 4;

[0041] (3) During testing, the entire test image is input into the network at its original size, and only the result of the branch of the output denoised image is used as the final denoising result.

[0042] References:

[0043] [1]Krull A,Buchholz TO,Jug F.Noise2Void-Learning Denoising FromSingle Noisy Images[C] / / 2019IEEE / CVF Conference on Computer Vision andPattern Recognition(CVPR).IEEE,2019.Mount J.The equivalence of logisticregression and maximum entropymodels[J].2011.

[0044] [2] Zhang, Kai, et al. "Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising." IEEE transactions on image processing 26.7 (2017): 3142-3155.

[0045] [3]Kendall, Alex, and Yarin Gal. "What uncertainties do we need inbayesian deep learning for computer vision?." Advances in neural informationprocessing systems 30 (2017).

[0046] [4]Huang T,Li S,Jia X,et al.Neighbor2neighbor:Self-superviseddenoising from single noisy images[C] / / Proceedings of the IEEE / CVF conferenceon computer vision and pattern recognition.2021:14781-14790.

[0047] [5]Wang Z,Liu J,Li G,et al.Blind2Unblind:Self-Supervised ImageDenoising with Visible Blind Spots[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2022:2027-2036。

Claims

1. A robust self-supervised image denoising method based on constructing pairs of noisy samples, characterized in that, The specific steps are as follows: (1) Pre-denoising: The N2V denoising model is used to perform coarse denoising on the noisy image; (2) Constructing pairs of noisy samples: Using the original noisy image and the coarsely denoised image, pairs of noise samples are constructed through addition, subtraction and shifting operations; (3) Training a dual-branch denoising network: A dual-branch denoising network is trained using the paired noise samples constructed in step (2). During the training process, an uncertainty-aware loss function is used to constrain the network, thereby significantly improving the denoising performance and robustness of the network. In step (1), the N2V denoising model is used, denoted as For a set of noisy images: Pre-denoising is performed to obtain the corresponding coarse denoising result: The numerical subscripts mark the different noisy images; that is: ； The process of constructing pairs of noisy samples in step (2) is as follows: First, a noise group that approximates the true distribution is obtained by calculating the difference between the noisy image and the coarse denoising result: ,Right now: ； At this point, the noise within the noise group is independent and identically distributed; Secondly, perform two cyclic shift operations on the noise group to obtain two noise groups that differ only in order, denoted as . N 1 and N 2; The specific operation is to move the noise in the first position of the noise group to the last position, and repeat this process twice, resulting in: ； Finally, the two sets of noise are added to the coarse denoising result to construct pairs of noisy sample groups: ; ； The two noisy images constructed through the above process have the same "clean" scene. And independent and identically distributed noise, where * represents any numerical subscript; The step (3) of training a dual-branch denoising network using the paired noisy samples constructed in step (2) specifically involves: using one noisy sample from the paired noisy samples as the input to the dual-branch network, and the other noisy sample as the target to be learned by the network; wherein, the dual-branch denoising network is constructed based on DnCNN, the first two layers of DnCNN are convolutional layers and ReLU activation layers, followed by 15 modules consisting of convolutional layers, batch normalization layers and ReLU activation layers, and the last layer is a convolutional layer; a network module consisting of convolutional layers, batch normalization layers and ReLU activation layers stacked 7 times is connected in the middle of the DnCNN network as a denoising branch of the dual-branch network; the latter half of the original DnCNN is the other branch, which is the uncertainty branch; the two branch networks share the first half of DnCNN; the convolutional kernel size of the convolutional layers in the branch networks is 3×3; the output of the denoising branch is the denoised image, and the output of the uncertainty branch is the uncertainty map; During the training of the dual-branch network, an uncertainty-aware loss function is used: ； in, K This represents the total number of samples in the noisy dataset. This represents the branch that outputs the denoised result in a two-branch network. It is the branch output result of predicting uncertainty in a two-branch network; The uncertainty graph output by the uncertainty branch is measured at the pixel level. The confidence level of the denoising result of the branch output, i.e., in In the middle, pixel regions with larger values ​​have higher uncertainty, and correspondingly The regions in the output denoising result have low confidence, meaning that the pixel values ​​with high uncertainty deviate more from the pixel values ​​of the real clean image.

Images