Unsupervised real image denoising method based on self-cooperation parallel generative adversarial network

Abstract

The invention discloses an unsupervised real image denoising method based on a self-cooperation parallel generative adversarial network. The method mainly comprises the following steps: constructing a noise extraction module based on an initial denoiser by using the superposition of noise, and extracting noise components of a real image; a parallel generation confrontation branch framework is adopted, a branch 1 synthesizes a pseudo real noise image through a non-pairing synthesis mode and then synthesizes a noise image through a self-synthesis mode, namely, non-pairing synthesis-self-synthesis; the branch 2 is used for synthesizing a noise image in a self-synthesis mode and then synthesizing a pseudo real noise image in a non-pairing synthesis mode, namely self-synthesis-non-pairing synthesis; and adopting a self-cooperation strategy, replacing the previous weaker de-noising device with the current more convergent de-noising device after each iteration until the de-noising device is completely convergent, completing network training and obtaining a final model. According to the method, noise on a real image can be well removed, and the method is an effective image denoising method.

Description

Technical Field

[0001] The present invention relates to an image denoising technology, and in particular to an unsupervised real image denoising method based on a self-collaborative parallel generative adversarial network, and belongs to the field of image processing. Background Art

[0002] Images are an important means for humans to acquire, express and transmit information. Digital images are widely used in modern society, including digital photography, video communication, medical imaging, security monitoring, artificial intelligence and other fields. With the popularization of digital image acquisition and storage technology, processing these large amounts of digital image data has become an important task. Image denoising technology is one of the classic tasks in the field of image processing. Its purpose is to restore noise-free images from noisy observations by reducing potential noise to improve the quality and visualization of images. At present, the additive white Gaussian noise (AWGN) hypothesis is widely used in image denoising. However, the noise in the real world is more complex and very different from the ideal AWGN hypothesis. Directly using the AWGN model for real scenes will result in poor performance. In addition, a variety of methods have been proposed to capture paired clean noise image datasets from real scenes to facilitate the training of deep networks. However, these image pair-based methods focus on enhancing performance by improving the network structure, and obtaining well-aligned clean noise image pairs is time-consuming and laborious. Summary of the invention

[0003] In order to solve the above problems, the present invention provides an unsupervised real image denoising method based on a self-collaborative parallel generative adversarial network. Specifically, the present invention proposes a self-collaborative strategy applicable to multiple denoisers and an effective parallel generative adversarial branch framework, through which better "synthetic noise-clean" image pairs can be generated, thereby providing a more powerful denoiser for the next iteration, so that the noise in the real image can be effectively removed without increasing the complexity of network reasoning.

[0004] The unsupervised real image denoising method based on a self-collaborative parallel generative adversarial network proposed in the present invention mainly includes the following steps:

[0005] (1) Utilizing the superposition of noise, a noise extraction module based on the initial denoiser is constructed to extract the noise components of the real image;

[0006] (2) Constructing an “Unpaired Synthetic-Self Synthetic” (US) adversarial branch, in which the unpaired synthesis operation uses the clean image and the unpaired noise as the input of the generator to generate a pseudo-real noise image. Then, in the self-synthesis operation, the pseudo-real noise image is passed through the denoiser and the noise extraction module to obtain the estimated clean image and the paired noise, which are then input into the next generator to generate a new pseudo-real noise image. Finally, the complementary constraints of the unpaired synthesis operation and the self-synthesis operation are realized.

[0007] (3) Constructing a “Self Synthetic-Unpaired Synthetic” (SU) adversarial branch, which first performs a self-synthesis operation and then an unpaired synthesis operation, forming a complementary branch constraint with the “unpaired synthesis-self-synthesis” adversarial branch, thus promoting better training results for the network;

[0008] (4) Construct the loss function of the entire network and train the network model by minimizing the loss function;

[0009] (5) Using a self-collaborative strategy, after each iteration, the more convergent denoiser replaces the weaker denoiser to generate better “synthetic noise-clean” image pairs, while promoting the training of the denoiser and further improving its performance;

[0010] (6) After the current network is fully converged, the network training is completed and the final model is obtained, and the performance of the denoiser in the trained network model is tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a principle block diagram of the unsupervised real image denoising method based on the self-collaborative parallel generative adversarial network of the present invention.

[0012] Figure 2 It is the denoiser structure used in the network of the present invention.

[0013] Figure 3 It is a block diagram of the noise extraction module in the network of the present invention.

[0014] Figure 4 : is a comparison chart of the denoising effect of the present invention and different methods on the SIDD verification data set, where (a) is the original clean image, (b) is the noisy image, (c) to (f) are the reconstruction results of other methods, and (g) is the denoising result of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with the accompanying drawings:

[0016] Figure 1In the paper, the unsupervised real image denoising method based on self-collaborative parallel generative adversarial network can be divided into the following seven steps:

[0017] (1) Utilizing the superposition of noise, a noise extraction module based on the initial denoiser is constructed to extract the noise components of the real image;

[0018] (2) Constructing an “Unpaired Synthetic-Self Synthetic” (US) adversarial branch, in which the unpaired synthesis operation uses the clean image and the unpaired noise as the input of the generator to generate a pseudo-real noise image. Then, in the self-synthesis operation, the pseudo-real noise image is passed through the denoiser and the noise extraction module to obtain the estimated clean image and the paired noise, which are then input into the next generator to generate a new pseudo-real noise image. Finally, the complementary constraints of the unpaired synthesis operation and the self-synthesis operation are realized.

[0019] (3) Constructing a “Self Synthetic-Unpaired Synthetic” (SU) adversarial branch, which first performs a self-synthesis operation and then an unpaired synthesis operation, forming a complementary branch constraint with the “unpaired synthesis-self-synthesis” adversarial branch, thus promoting better training results for the network;

[0020] (4) Construct the loss function of the entire network and train the network model by minimizing the loss function;

[0021] (5) Using a self-collaborative strategy, after each iteration, the more convergent denoiser replaces the weaker denoiser to generate better “synthetic noise-clean” image pairs, while promoting the training of the denoiser and further improving its performance;

[0022] (6) After the current network is fully converged, the network training is completed and the final model is obtained, and the performance of the denoiser in the trained network model is tested.

[0023] Specifically, in the step (1), the superposition of noise is utilized to construct a noise extraction module based on the initial denoiser to extract the noise component of the real image:

[0024] n y =NE(y)=y-DN(y)

[0025] Among them, y represents the real noise image, n y represents noise, DN represents denoiser, and NE represents noise extraction module.

[0026] In the step (2), an "Unpaired Synthetic-SelfSynthetic" (US) adversarial branch is constructed, wherein the unpaired synthesis operation uses the clean image and the unpaired noise as the input of the generator to generate a pseudo-real noise image, and then in the self-synthesis operation, the pseudo-real noise image is respectively passed through the denoiser and the noise extraction module to obtain an estimated clean image and paired noise, and are input together to the next generator to generate a new pseudo-real noise image, and finally the complementary constraints of the unpaired synthesis operation and the self-synthesis operation are realized. The whole process is expressed by the formula as follows:

[0027] x u-syn =G(x,NE(y))

[0028] x rec =DN(x u-syn )

[0029] x s-syn =G(x rec ,NE(x u-syn ))

[0030] Among them, x represents the clean image, G represents the generator, and x u-syn represents the pseudo-real noise image generated by the generator in an unpaired manner, x rec represents the clean image estimated by the denoiser, x s-syn Represents a synthetically generated noise image.

[0031] In the step (3), a "Self Synthetic-Unpaired Synthetic" (SU) adversarial branch is constructed. In the self-synthesis operation, the real noise image is passed through the denoiser and the noise extraction module to obtain an estimated clean image and paired noise, and the two are input together to the generator to generate a pseudo-real noise image. In the unpaired synthesis operation, the generated pseudo-real noise image and the unpaired noise are input together to the next generator to generate a new pseudo-real noise image. The whole process is expressed by the following formula:

[0032] y rec =DN(y)

[0033] y s-syn =G(y rec ,NE(y))

[0034] y u-syn =G(y rec ,NE(x u-syn ))

[0035] Among them, y rec represents the clean image estimated by the denoiser, ys-syn represents the noise image generated by the generator in a self-synthetic way, y u-syn Represents a noisy image generated by the generator in an unpaired manner.

[0036] In step (4), the loss function of the entire network is constructed to prevent model degradation during training and improve the representation ability of the network, such as Figure 1 As shown, there are four adversarial losses in the network structure of the present invention. We use the least squares loss as L adv1 Adversarial loss:

[0037]

[0038] Among them, D represents the discriminator, and the adversarial loss L adv1 is limited to y and y s-syn The other three adversarial losses can also be constructed similarly by constraining the current generated noise image and y.

[0039] We further apply a background guidance module (BGM) to provide additional reliable constraints. BGM maintains the consistency of the background between the synthetic noisy image and the clean image, constraining their low-frequency contents to be similar. The low-frequency contents are extracted by using multiple low-pass filters and constrained to be similar to each other through the L1 norm loss to synthesize a pseudo-real noisy image x u-syn BGM loss between the clean image x:

[0040]

[0041] Among them, x represents the clean image, x u-syn represents the noise image generated by the generator in an unpaired manner, B σ (·) represents the Gaussian filter operator with blur kernel size σ, λ σ represents the weight of level σ. We empirically set σ-s to 3, 9, and 15, and λ-s to 0.01, 0.1, and 1.

[0042] Finally, we train the denoiser by optimizing the following function:

[0043]

[0044] Among them, m represents the total number of sample pairs, Θ represents the parameters in the network, (x rec ) i is the denoised image output by the denoiser, i represents the i-th sample image in 1-m, L SSIM SSIM loss is used to constrain the structural information of the image, λ SSIM YesL SSIM The weight of .

[0045] In the step (5), a self-collaborative strategy is used. In NE, k represents the number of iterations. When k=0, it is a learnable convolution block. When k>0, the weaker denoiser DN0 is replaced by the current more convergent denoiser DN, which enables the synthesis of better "synthetic noise-clean" image pairs and promotes the training of the denoiser to further improve its performance. This iterative process produces a series of increasingly powerful denoisers.

[0046] In the step (6), after the current network is fully converged, the network training is completed and the final model is obtained, and the performance of the denoiser in the trained network model is tested.

[0047] In order to better illustrate the effectiveness of the present invention, we selected three commonly used test data sets for comparative experiments, including the SIDD validation set, the SIDD benchmark set, and the DND benchmark set. Four representative denoising methods were selected for comparison, including two supervised methods and two unsupervised methods.

[0048] The chosen algorithm is:

[0049] Method 1: The method proposed by Guo et al., reference "Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 1712-1722, 2019.".

[0050] Method 2: The method proposed by Jang et al., reference “C2n: Practical generative noise modeling for real-world denoising. In Proceedings of the IEEE International Conference on Computer Vision, 2350-2359, 2021.”

[0051] Method 3: The method proposed by Neshatavar et al., reference “Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from mage. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 17583-17591, 2022.”.

[0052] Method 4: The method proposed by Lee et al., reference "Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 17725-17734, 2022.".

[0053] The comparative experiment contents are as follows:

[0054] Experiment 1, using the real noise dataset SIDD validation set to conduct experiments, using method 1, method 2, method 3, method 4 and the method of the present invention to denoise the noisy image, the results are shown in Figure 4 The objective evaluation results of the reconstruction results are shown in Table 1. PSNR (Peak Signal to Noise Ratio, unit: dB) and SSIM (Structure Similarity Index) are used to evaluate the reconstruction effect. The higher the PSNR / SSIM value, the better the reconstruction effect.

[0055] Experiment 2, using the SIDD benchmark set of a real noise dataset, denoising the noisy image using Method 1, Method 2, Method 3, Method 4 and the method of the present invention. The objective evaluation results of the reconstruction results are shown in Table 1. PSNR (PeakSignal to Noise Ratio, unit dB) and SSIM (Structure Similarity Index) are used to evaluate the reconstruction effect, and the higher the PSNR / SSIM value, the better the reconstruction effect.

[0056] Experiment 3, using the real noise dataset DND benchmark set, denoising the noisy image using Method 1, Method 2, Method 3, Method 4 and the method of the present invention. The objective evaluation results of the reconstruction results are shown in Table 1. PSNR (PeakSignal to Noise Ratio, unit dB) and SSIM (Structure Similarity Index) are used to evaluate the reconstruction effect, and the higher the PSNR / SSIM value, the better the reconstruction effect.

[0057] As can be seen from Table 1, the present invention achieves higher PSNR and SSIM. Figure 4 It can be seen that the reconstruction result of the present invention has clear and natural edges and shows more details, while the reconstruction result of the contrast algorithm has certain artifacts and blurred edges in subjective visual effects. In summary, compared with the contrast method, the reconstruction result of the present invention has a greater advantage in both subjective and objective evaluation. Therefore, the present invention is an effective image denoising method.

[0058] Table 1

[0059]

Claims

1. An unsupervised real image denoising method based on a self-collaborative parallel generative adversarial network, characterized by The following steps are involved: Step 1: Using the superposition of noise, a noise extraction module based on the initial denoiser is constructed to extract the noise component of the real image; Step 2: Construct an "unpaired synthesis-self-synthesis" adversarial branch, in which the unpaired synthesis operation uses the clean image and the unpaired noise as the input of the generator to generate a pseudo-real noise image. Then, in the self-synthesis operation, the pseudo-real noise image is passed through the denoiser and the noise extraction module to obtain the estimated clean image and the paired noise, which are then input into the next generator to generate a new pseudo-real noise image. Finally, the complementary constraints of the unpaired synthesis operation and the self-synthesis operation are realized. Step 3: Construct a "Self Synthetic-Unpaired Synthetic" (SU) adversarial branch, which first performs a self-synthesis operation and then an unpaired synthesis operation, forming a complementary branch constraint with the "unpaired synthesis-self-synthesis" adversarial branch to promote better training results of the network; Step 4: Construct the loss function of the entire network and train the network model by minimizing the loss function; Step 5: Use a self-collaborative strategy to replace the weaker denoiser with the more convergent denoiser after each iteration to generate better "synthetic noise-clean" image pairs, while promoting the training of the denoiser and further improving its performance. Step 6: After the current network is fully converged, complete the network training and obtain the final model, and test the performance of the denoiser in the trained network model.

2. The unsupervised real image denoising method based on self-collaborative parallel generative adversarial network according to claim 1 is characterized in that In step 1, the superposition of noise is used to construct a noise extraction module, and a noise extraction module based on the initial denoiser is constructed to extract the noise component of the real image. The specific formula is as follows: n y =NE(y)=y-DN(y) Where y represents the input noise image, n y represents the noise corresponding to the image y, DN represents the denoiser, and NE represents the noise extraction module.

3. The unsupervised real image denoising method based on self-collaborative parallel generative adversarial network according to claim 1 is characterized in that The "unpaired synthesis-self-synthesis" adversarial branch constructed in step 2, the unpaired synthesis operation uses the clean image and unpaired noise as the input of the generator to generate a pseudo-real noise image. The pseudo-real noise image is passed through the denoiser to obtain an estimated clean image, and the paired noise is obtained through the noise extraction module. The self-synthesis operation inputs the estimated clean image and paired noise together into the next generator to generate a noise image. The specific formula is as follows: x u-syn =G(x,NE(y)) x s-syn =G(DN(x u-syn ),NE(x u-syn )) Among them, x represents the clean image, y represents the input noisy image, and x u-syn represents the pseudo-real noise image generated by the generator in an unpaired manner, DN represents the denoiser, NE represents the noise extraction module, G represents the generator, and x s-syn Represents a synthetically generated noisy image.

4. The unsupervised real image denoising method based on self-collaborative parallel generative adversarial network according to claim 1 is characterized in that The "self-synthesis-unpaired synthesis" adversarial branch constructed in step 3 passes the real noise image through the denoiser to obtain the estimated clean image, and obtains the paired noise through the noise extraction module. The self-synthesis operation inputs the estimated clean image and the paired noise into the generator to generate a noisy image. The unpaired synthesis operation uses the generated noisy image and the unpaired noise as the input of the next generator to generate a pseudo-real noise image. The specific formula is as follows: and s-syn =G(DN(y),NE(y)) y u-syn =G(DN(y),NE(x u-syn )) Among them, y represents the input noise image, y s-syn represents the noise image generated by the generator in a self-synthetic way, DN represents the denoiser, NE represents the noise extraction module, G represents the generator, and y u-syn Represents a pseudo-realistic noise image generated by the generator in an unpaired manner.

5. The unsupervised real image denoising method based on self-collaborative parallel generative adversarial network according to claim 1 is characterized in that In step five, a self-collaborative strategy is used. In NE, k represents the number of iterations. When k = 0, it is a learnable convolutional block. When k>0, the weaker denoiser DN0 is replaced by the current more convergent denoiser DN, which enables the synthesis of better "synthetic noise-clean" image pairs and promotes the training of the denoiser, further improving its performance. This iterative process produces a series of increasingly powerful denoisers.

Images