Self-supervised real denoising method based on asymmetric mask

Abstract

The invention discloses a self-supervised real denoising method based on an asymmetric mask. The method mainly comprises the following steps: constructing a single mask training framework of a self-supervision real denoising method based on asymmetric masks; constructing a multi-mask reasoning framework of a self-supervision real denoising method based on asymmetric masks; designing a solution of the checkerboard effect; and four self-supervised real denoising schemes based on asymmetric masks are constructed. According to the method, a noiseless clean image can be reconstructed from a real noise image, and the method is an effective real image denoising method. The method has wide application prospects in the fields of remote sensing monitoring, medical imaging, display technologies and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to image denoising technology, in particular to a mask-based self-supervised denoising method, belonging to the field of digital image processing. BACKGROUND

[0002] Obtaining high-quality images is a key goal in the field of computer vision. The denoising task can significantly improve the image quality, thus providing better visual effects. At present, deep learning has shown excellent results in the field of image denoising. However, most of these methods rely on paired datasets, and these datasets are synthesized from a large number of clean images. Therefore, due to the lack of paired data, these methods are difficult to apply in real tasks. In order to overcome this challenge, a series of real paired datasets are proposed, and the supervised denoising methods trained by applying these real paired datasets perform better in handling real tasks. However, the noise in the real world is extremely complex, with factors such as sensor noise, environmental conditions, and electromagnetic interference. These changes often make it difficult for real datasets to comprehensively cover all real noise types. In response to these challenges, self-supervised denoising methods without paired samples have emerged as a promising solution. SUMMARY

[0003] The purpose of the present application is to solve the inherent limitations in the design of Blind Spot Network (BSN) denoiser, and to propose a self-supervised real denoising method based on asymmetric mask, to overcome the limitations of denoiser design, and to achieve better real noise removal effect through asymmetric schemes in training and inference.

[0004] The self-supervised real denoising method based on asymmetric mask (Asymmetric Mask Based Self-Supervised Real Denoising Network, AMSNet) proposed by the present application mainly includes the following operation steps:

[0005] (1) Construct a single mask training framework for the self-supervised real denoising method based on asymmetric mask;

[0006] (2) Construct a multi-mask inference framework for the self-supervised real denoising method based on asymmetric mask;

[0007] (3) Design a solution to the checkerboard effect;

[0008] (4) Construct four self-supervised real denoising schemes based on asymmetric mask. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1is the training and inference framework of the asymmetric mask based self-supervised real denoising method AMSNet of the present application. Wherein, (a) is the single mask training framework of the AMSNet of the present application, and (b) is the multi-mask inference framework of the AMSNet of the present application.

[0010] Figure 2 is the pixel down-sampling (PD) execution process.

[0011] Figure 3 is the comparison chart of the denoising results of the real noise pictures of the present application and another five methods. Wherein, (a) is a real noise picture, (b) to (e) are the experimental results of methods 1 to 4, and (f) is the result of the present application.

[0012] Figure 4 is the comparison chart of the denoising results of the real noise pictures of the present application and another five methods. Wherein, (a) is a real noise picture, (b) to (e) are the experimental results of methods 1 to 4, and (f) is the result of the present application. DETAILED DESCRIPTION

[0013] The present application will be further described below in conjunction with the accompanying drawings:

[0014] Figure 1 Among them, the asymmetric mask based self-supervised real denoising method can be specifically divided into the following steps:

[0015] (1) constructing a single mask training framework of the asymmetric mask based self-supervised real denoising method;

[0016] (2) constructing a multi-mask inference framework of the asymmetric mask based self-supervised real denoising method;

[0017] (3) designing a solution to the checkerboard effect;

[0018] (4) constructing four asymmetric mask based self-supervised real denoising schemes.

[0019] Specifically, in the step (1), the single mask training framework of the asymmetric mask based self-supervised real denoising method is as shown in Figure 1 (a). Let I N ∈R c×h×w be a single noise image, where c represents the number of channels, and h and w represent the height and width of the image, respectively. Some pixels are randomly masked by a 0, 1 mask matrix M, where M c×h×w , which means that these pixels are replaced by zero. According to experience, about 50% of the noise image pixels are masked. Therefore, the masked image is M N , where represents element-wise multiplication. Then, the masked image M Nis input to the denoiser D E , a restored image D E is generated. N , θ), where θ denotes the parameters of the denoiser D E .

[0020] Based on the above analysis, the denoised pixels at the corresponding positions can be extracted from the output of the denoiser D E using the following formula:

[0021]

[0022] where D denotes the entire denoising process, including masking with M and using the inverse matrix to extract the final non-identity mapping restoration result. is the complement of M, denoting the restored pixels. Based on the BSN, the optimization process can be represented as follows:

[0023]

[0024] ||·||1 denotes the denoiser optimization using the L1 norm. The aforementioned formula also follows the assumption that the noise is zero-mean and pixel-wise independent.

[0025] However, in real-world noise scenarios, the noise often deviates from the assumption of spatial independence, so a pixel down-sampling PD strategy is introduced, denoted as P s with a step factor s. The P s operation destroys the spatial correlation between the noise, ensuring that the resulting sub-samples are consistent with the assumption, as shown in Figure 2 .

[0026] The training scheme for real-world noise scenarios is shown. After applying the P s operation to the real-world noisy image I N , a set of sub-samples can be obtained, denoted as where and I s = P s (I N ). Then, an independent mask is applied to each sub-sample using a consistent probability distribution, denoted as M s ⊙ I s , where M s is the set of binary mask matrices for all sub-samples, i.e. Each can then be processed by the denoiser D E on these masked sub-sample sets.

[0027] The overall optimization process can be represented as the minimization of the following masked self-supervised loss:

[0028]

[0029] By M s real-world noise in the mask pixels, and the BSN inherent denoiser limitations are lifted.

[0030] In step (2), the multi-mask inference framework of the asymmetric mask-based self-supervised real denoising method is as shown in Figure 1 (b) shown, in order to realize the comprehensive denoising of the whole image, an asymmetric mask-based self-supervised real denoising method is designed, and a multi-branch mask complementary denoising block (MMDB) is proposed.

[0031] The MMDB method uses k denoising branches, where k≥2, to recover the entire image. First, a P s from the original noisy image I N A sub-sample set I s is created, which helps to break the spatial correlation of noise. For each denoising branch, MMDB generates a series of mask matrix sets M s , denoted as Ensure that the coverage area of each branch is unique and non-overlapping, while maintaining roughly the same coverage ratio. The total coverage of these masks spans all pixels, as shown in the following formula:

[0032]

[0033] Each branch uses the same denoiser D E , and the output of all branches is denoted as For a given branch i, where 1≤i≤k and the corresponding is combined to independently generate the output O i :

[0034]

[0035] where D i is the i-th denoising branch, and θ is the parameter of the denoiser D E . The output of MMDB can be represented as the sum of all branch outputs:

[0036]

[0037] where D M (I s ) is the output of MMDB. Finally, the overall framework is as follows:

[0038] IDN =P -1 s(D M (P s (I N ))) (7)

[0039] in, P represents the pixel downsampling with a step size of s. s The inverse operation.

[0040] In step (3), the solution to the checkerboard effect is detailed as follows: Compared to the real image, the staggered pixel arrangement in the denoised image significantly reduces smoothness. To address this issue and promote the generation of higher quality images, a priori smoothing loss is introduced. Make fine adjustments:

[0041]

[0042] Where I represents the image, and i,j represent the coordinates of pixels. During fine-tuning, a loss function incorporating the original mask is used. and prior smoothing loss Total loss

[0043]

[0044] Where λ is the weighting coefficient of the prior smoothing loss, with an empirical value set to 0.01, and k represents the number of denoising branches.

[0045] In step (4), as summarized above, four self-supervised real denoising schemes based on asymmetric masks are constructed as follows: (The rest of the text appears to be a list of schemes and their implementations, which are not directly related to the preceding sentences and can be omitted.) The base model for loss training is denoted as AMSNet-B (represented as this invention-B), and through... The finely tuned version is denoted as AMSNet-P (represented as the present invention-P), while the versions that further refine the process by using random replacement to suppress the checkerboard effect are denoted as AMSNet-BE (represented as the present invention-BE) and AMSNet-PE (represented as the present invention-PE), respectively.

[0046] To better illustrate the effectiveness of this invention, real-world image denoising experiments were conducted on the commonly used test sets "SIDD" and "DND". Four typical real-world image denoising algorithms were selected as comparison methods in the experiments.

[0047] The real image denoising algorithm for comparison is:

[0048] Method 1: Method proposed by Neshatavar et al., reference "Neshatavar R, Yavartanoo M, Son S, et al. Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image [C] / / Proceedings of the ieee / cvf Conference on Computer Vision and Pattern Recognition. 2022: 17583-17591."

[0049] Method 2: Method proposed by Lin et al., reference "Lin X, Ren C, Liu X, et al. Unsupervised Image Denoising in Real-World Scenarios via Self-Collaboration Parallel Generative Adversarial Branches [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023: 12642-12652."

[0050] Method 3: Method proposed by Lee et al., reference "Lee W, Son S, Lee K M. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 17725-17734."

[0051] Method 4: The method proposed by Li et al., see reference "Li J, Zhang Z, Liu X, et al. Spatially adaptive self-supervised learning for real-world image denoising[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2023:9914-9924."

[0052] The comparative experiment is as follows:

[0053] Experiment 1 involved performing real-image denoising experiments using methods 1 through 4, as well as this invention, on the commonly used test sets "SIDD" and "DND". The denoising results are as follows: Figure 3 and Figure 4 As shown in Table 1, the objective evaluation parameters for the reconstruction results are used. The objective evaluation parameters are PSNR (Peak Signal to Noise Ratio) and SSIM (Structure Similarity Index). Higher values ​​for both parameters indicate better reconstruction results. It is worth noting that only through... The basic model for loss training is denoted as Invention-B, and it is obtained through... The finely tuned version is denoted as Invention-P, while the versions that further refine the reasoning process by using random replacement to suppress the checkerboard effect are denoted as Invention-BE and Invention-PE, respectively.

[0054] Table 1

[0055]

[0056] Clearly, this invention achieves better PSNR / SSIM results compared to recent methods.

[0057] Compared to other methods, this invention achieves better noise reduction with minimal blurring and artifacts. This observation confirms the practicality and effectiveness of this invention. Figure 3 For example, the denoising results of this invention show minimal noise and blur, while other methods exhibit significant noise and blur over a wide range.

[0058] In summary, compared to comparative methods, the reconstruction results of this invention have significant advantages in both subjective and objective evaluation. Therefore, this invention is an effective method for denoising realistically noisy images.

Claims

1. A self-supervised real denoising network (AMSNet) based on asymmetric masks, characterized by: Includes the following steps: Step 1: Construct a single-mask training framework for a self-supervised real denoising method based on asymmetric masks; Step 2: Construct a multi-mask inference framework for a self-supervised real denoising method based on asymmetric masks; Step 3: Design a solution for the checkerboard effect; Step 4: Construct four self-supervised real denoising schemes based on asymmetric masks.

2. The self-supervised real denoising method based on asymmetric mask according to claim 1, characterized in that... The single-mask training framework described in Step 1; this model uses a single-mask training mechanism during the training phase and a multi-mask inference mechanism during the inference phase; the details of the single-mask training mechanism are as follows: Let I N ∈R c×h×w Let P be a single noisy image, where c represents the number of channels, and h and w represent the height and width of the image, respectively. Since real noise usually deviates from the assumption of spatial independence, a pixel down-sampling (PD) strategy is introduced, denoted as P. s The step size factor is s, P s The operation disrupts the spatial correlation between noises, ensuring that the resulting subsamples are consistent with the hypothesis; in the case of real-world noisy images I N Application P s After the operation, a set of subsamples can be obtained, denoted as in And I s =P s (I N Then, for each subsample, an independent mask is applied using a consistent probability distribution, denoted as M. s ⊙I s M s It is the set of binary mask matrices for all subsamples, i.e. Each Then it can be denoised by D. E Process these masked subsample sets; The overall optimization process can be represented as minimizing the self-supervised loss of the mask as follows: pass M can be successfully removed s It represents the real-world noise in the mask pixels and removes the inherent denoising limitations of BSN.

3. The self-supervised real denoising method based on asymmetric masks according to claim 1, characterized in that... The multi-mask inference framework described in step two; this model uses a single-mask training mechanism during the training phase and a multi-mask inference mechanism during the inference phase; the details of the multi-mask inference mechanism are as follows: the multi-branch mask complementary denoising block (MMDB) uses k denoising branches, where k≥2, to recover the entire image; firstly, through P s From the original noisy image I N Create a subset I s This helps to break the spatial correlation of noise; For each denoising branch, MMDB generates a series of mask matrix sets M s , recorded as Ensure that the coverage area of ​​each branch is unique and non-overlapping, while maintaining approximately the same coverage ratio; use the same noise denoiser D for each branch. E The output of all branches is denoted as For a given branch i, where 1≤i≤k and the corresponding Combining to independently generate output O i The output of MMDB can be represented as the sum of the outputs of all branches: Finally, the overall process is as follows: I DN =P -1 s(D M (P s (I N ))), where P s -1 P represents the pixel downsampling with a step size of s. s The inverse operation.

4. The self-supervised real denoising method based on asymmetric mask according to claim 1, characterized in that... The solution to the checkerboard effect described in step three; Regarding the checkerboard effect present in the denoising results, the solution proposed in this invention is as follows: To address this problem and promote the generation of higher quality images, a priori smoothing loss is introduced. Make minor adjustments: Where I represents the image, and i,j represent the coordinates of the pixels; during fine-tuning, a loss including the original mask is used. and prior smoothing loss Total loss Where λ is the weighting coefficient of the prior smoothing loss, with an empirical value set to 0.01, and k represents the number of denoising branches.

5. The self-supervised real denoising method based on asymmetric mask according to claim 1, characterized in that... The four self-supervised real denoising schemes based on asymmetric masks described in step four; only through The base model for loss training is denoted as AMSNet-B (represented as this invention-B), and through... The finely tuned version is denoted as AMSNet-P (represented as the present invention-P), while the versions that further refine the process by using random replacement to suppress the checkerboard effect are denoted as AMSNet-BE (represented as the present invention-BE) and AMSNet-PE (represented as the present invention-PE), respectively.

Images