Unsupervised Real Image Denoising Method Based on Self-Cooperative Parallel Generative Adversarial Networks

Abstract

The present invention discloses an unsupervised real image denoising method based on a self-collaborative parallel generative adversarial network. The method mainly includes the following steps: utilizing the superposition of noise, constructing a noise extraction module based on an initial denoiser to extract the noise component of the real image; using a parallel generative adversarial branch framework, branch 1 first synthesizes a pseudo-real noise image through unpaired synthesis and then synthesizes a noise image through self-synthesis, i.e., "unpaired synthesis-self-synthesis"; branch 2 first synthesizes a noise image through self-synthesis and then synthesizes a pseudo-real noise image through unpaired synthesis, i.e., "self-synthesis-unpaired synthesis"; using a self-collaborative strategy, after each iteration, the more converged denoiser replaces the previous weaker denoiser until the denoiser fully converges, completing network training and obtaining the final model. The method described in the present invention can effectively remove noise from real images and is an effective image denoising method.

Description

Technical Field

[0001] This invention relates to image denoising technology, specifically to an unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks, belonging to the field of image processing. Background Technology

[0002] Images are a crucial means for humans to acquire, express, and transmit information. Digital images are widely used in modern society, including in fields such as digital photography, video communication, medical imaging, security monitoring, and artificial intelligence. With the widespread adoption of digital image acquisition and storage technologies, processing these massive amounts of digital image data has become a critical task. Image denoising, a classic task in image processing, aims to recover noise-free images from noisy observations by reducing potential noise, thereby improving image quality and visualization. Currently, the additive white Gaussian noise (AWGN) assumption is widely used in image denoising. However, real-world noise is far more complex and differs significantly from the ideal AWGN assumption. Directly applying the AWGN model to real-world scenes leads to poor performance. Furthermore, various methods have been proposed to capture paired clean noisy image datasets from real-world scenes to facilitate the training of deep networks. However, these image pair-based methods focus on enhancing performance through improving network structure, and obtaining well-aligned clean noisy image pairs is both time-consuming and laborious. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides an unsupervised real-image denoising method based on a self-cooperative parallel generative adversarial network. Specifically, this invention proposes a self-cooperative strategy applicable to multiple denoisers and an effective parallel generative adversarial branching framework. This strategy can generate better "synthetic noise-clean" image pairs, thereby providing a more powerful denoiser for the next iteration, enabling effective removal of noise from real images without increasing network inference complexity.

[0004] The unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks proposed in this invention mainly includes the following steps:

[0005] (1) Utilizing the superposition property of noise, a noise extraction module based on the initial denoising unit is constructed to extract noise components from real images;

[0006] (2) Construct an adversarial branch of “Unpaired Synthetic-Self Synthetic (US)”, in which the unpaired synthesis operation takes the clean image and unpaired noise as input to the generator to generate a pseudo-real noise image. Then, in the self-synthetic operation, the pseudo-real noise image is passed through the denoiser and the noise extraction module to obtain the estimated clean image and the paired noise, and then input together into the next generator to generate a new pseudo-real noise image. Finally, the complementary constraints between the unpaired synthesis operation and the self-synthetic operation are realized.

[0007] (3) Construct a “Self Synthetic-Unpaired Synthetic” (SU) adversarial branch, which first performs self-synthetic operation and then performs unpaired synthesis operation, forming a complementary branch constraint with the “Unpaired Synthetic-Self Synthetic” adversarial branch, promoting better training results for the network.

[0008] (4) Construct the loss function for the entire network and train the network model by minimizing the loss function;

[0009] (5) Using a self-cooperation strategy, after each iteration, the weaker denoiser is replaced with the more convergent denoiser to generate better “synthetic noise-clean” image pairs, while promoting the training of the denoiser and further improving its performance.

[0010] (6) After the current network has fully converged, complete the network training and obtain the final model, and test the performance of the denoiser in the trained network model. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating the principle of the unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks of this invention.

[0012] Figure 2 This is the noise denoiser structure used in the network of this invention.

[0013] Figure 3 This is a block diagram of the noise extraction module in the network of this invention.

[0014] Figure 4 This is a comparison of the denoising effects of the present invention and different methods on the SIDD validation dataset, where (a) is the original clean image, (b) is the noisy image, (c) to (f) are the reconstruction results of other methods, and (g) is the denoising result of the present invention. Detailed Implementation

[0015] The present invention will be further described below in conjunction with the accompanying drawings:

[0016] Figure 1In this paper, the unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks can be divided into the following seven steps:

[0017] (1) Utilizing the superposition property of noise, a noise extraction module based on the initial denoising unit is constructed to extract noise components from real images;

[0018] (2) Construct an adversarial branch of “Unpaired Synthetic-Self Synthetic (US)”, in which the unpaired synthesis operation takes the clean image and unpaired noise as input to the generator to generate a pseudo-real noise image. Then, in the self-synthetic operation, the pseudo-real noise image is passed through the denoiser and the noise extraction module to obtain the estimated clean image and the paired noise, and then input together into the next generator to generate a new pseudo-real noise image. Finally, the complementary constraints between the unpaired synthesis operation and the self-synthetic operation are realized.

[0019] (3) Construct a “Self Synthetic-Unpaired Synthetic” (SU) adversarial branch, which first performs self-synthetic operation and then performs unpaired synthesis operation, forming a complementary branch constraint with the “Unpaired Synthetic-Self Synthetic” adversarial branch, promoting better training results for the network.

[0020] (4) Construct the loss function for the entire network and train the network model by minimizing the loss function;

[0021] (5) Using a self-cooperation strategy, after each iteration, the weaker denoiser is replaced with the more convergent denoiser to generate better “synthetic noise-clean” image pairs, while promoting the training of the denoiser and further improving its performance.

[0022] (6) After the current network has fully converged, complete the network training and obtain the final model, and test the performance of the denoiser in the trained network model.

[0023] Specifically, in step (1), a noise extraction module based on the initial denoising unit is constructed by utilizing the superposition property of noise to extract noise components from the real image:

[0024] n y =NE(y) = y - DN(y)

[0025] Where y represents the real noise image, and n y The symbols represent noise, DN represents noise denoiser, and NE represents noise extraction module.

[0026] In step (2), an adversarial branch of "Unpaired Synthetic-Self Synthetic (US)" is constructed. In the unpaired synthesis operation, the clean image and unpaired noise are used as inputs to the generator to generate a pseudo-realistic noise image. Then, in the self-synthetic operation, the pseudo-realistic noise image is passed through a denoiser and a noise extraction module to obtain the estimated clean image and paired noise, which are then input together into the next generator to generate a new pseudo-realistic noise image. Finally, the complementary constraints between the unpaired synthesis operation and the self-synthetic operation are realized. The whole process is expressed by the following formula:

[0027] x u-syn =G(x,NE(y))

[0028] x rec =DN(x u-syn )

[0029] x s-syn =G(x) rec ,NE(x u-syn ))

[0030] Where x represents a clean image, G represents the generator, and x u-syn x represents a pseudo-realistic noise image generated by the generator in an unpaired manner. rec x represents the clean image estimated by the denoiser. s-syn This represents a noisy image generated by a self-synthesis method.

[0031] In step (3), a "Self Synthetic-Unpaired Synthetic (SU)" adversarial branch is constructed. In the self-synthetic operation, the real noise image is passed through a denoiser and a noise extraction module to obtain an estimated clean image and paired noise, which are then input together into the generator to generate a pseudo-real noise image. In the unpaired synthesis operation, the generated pseudo-real noise image and unpaired noise are input together into the next generator to generate a new pseudo-real noise image. The whole process is expressed by the following formula:

[0032] y rec =DN(y)

[0033] y s-syn =G(y rec ,NE(y))

[0034] y u-syn =G(y rec ,NE(x u-syn ))

[0035] Among them, y rec y represents the clean image estimated by the denoiser.s-syn This represents a noisy image generated by the generator's self-synthesis method, y u-syn This represents a noisy image generated by the generator in an unpaired manner.

[0036] In step (4), a loss function is constructed for the entire network. This is done to prevent model degradation during training and to improve the network's representational power. Figure 1 As shown, the network structure of this invention contains four adversarial losses, and we use the least squares loss as L. adv1 The losses incurred in the confrontation:

[0037]

[0038] Where D represents the discriminator, and L represents the adversarial loss. adv1 Limited to y and y s-syn In between, the other three adversarial losses can also be constructed similarly by constraining the currently generated noisy image and y.

[0039] We further apply a Background Guidance Module (BGM) to provide additional reliable constraints. The BGM maintains the consistency of the background between the synthesized noisy image and the clean image, constraining their low-frequency content to be similar. This is achieved by extracting low-frequency content using multiple low-pass filters and limiting the low-frequency content to be similar to each other through L1 norm loss, thus synthesizing a pseudo-realistic noisy image x. u-syn BGM loss between the clean image x and the clean image x:

[0040]

[0041] Where x represents a clean image, x u-syn B represents a noisy image generated by the generator in an unpaired manner. σ (·) represents a Gaussian filter operator with a fuzzy kernel size of σ, where λ σ This represents the weight of level σ. Based on experience, we set σ-s to 3, 9, and 15 respectively, and λ-s to 0.01, 0.1, and 1 respectively.

[0042] Finally, we train the denoiser by optimizing the following function:

[0043]

[0044] Where m represents the total number of sample pairs, Θ represents the parameters in the network, (x rec ) i This is the denoised image output by the denoiser, where i represents the i-th sample image from 1 to m, and L... SSIM The SSIM loss is used to constrain the structural information of the image, λ SSIM It is L SSIM The weight.

[0045] In step (5), a self-cooperative strategy is used. In NE, k represents the number of iterations. When k = 0, it is a learnable convolutional block. When k > 0, the weaker denoiser DN0 is replaced by the current more convergent denoiser DN. This enables the synthesis of better “synthetic noise-clean” image pairs, while promoting the training of the denoiser and further improving its performance. This iterative process produces a series of increasingly powerful denoisers.

[0046] In step (6), after the current network has fully converged, the network training is completed and the final model is obtained. The performance of the denoiser in the trained network model is then tested.

[0047] To better illustrate the effectiveness of this invention, we selected three commonly used test datasets for comparative experiments: the SIDD validation set, the SIDD benchmark set, and the DND benchmark set. Four representative denoising methods were used for comparison, including two supervised methods and two unsupervised methods.

[0048] The selected algorithm is:

[0049] Method 1: The method proposed by Guo et al., see reference "Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, 1712-1722, 2019."

[0050] Method 2: The method proposed by Jang et al., see reference "C2n: Practical generative noise modeling for real-world denoising. In Proceedings of the IEEE International Conference on Computer Vision, 2350-2359, 2021."

[0051] Method 3: The method proposed by Neshatavar et al., see reference "Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 17583-17591, 2022."

[0052] Method 4: The method proposed by Lee et al., see reference "Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 17725-17734, 2022."

[0053] The comparative experiment is as follows:

[0054] Experiment 1 was conducted using the SIDD validation set of a real noise dataset. Methods 1, 2, 3, 4, and the method of this invention were used to denoise the noisy images. The results were obtained by... Figure 4 As shown in Table 1, the objective evaluation results of the reconstruction are presented. PSNR (Peak Signal to Noise Ratio, in dB) and SSIM (Structure Similarity Index) are used to evaluate the reconstruction effect, with a higher PSNR / SSIM value indicating a better reconstruction effect.

[0055] Experiment 2 was conducted using the SIDD benchmark dataset, a real noise dataset. Noise was removed from the images using methods 1, 2, 3, 4, and the method of this invention. The objective evaluation results of the reconstruction are shown in Table 1. PSNR (PeakSignal to Noise Ratio, dB) and SSIM (Structure Similarity Index) were used to evaluate the reconstruction effect; a higher PSNR / SSIM value indicates a better reconstruction effect.

[0056] Experiment 3 was conducted using the real noise dataset DND benchmark set. Noise was removed from the noisy images using methods 1, 2, 3, 4, and the method of this invention. The objective evaluation results of the reconstruction are shown in Table 1. PSNR (PeakSignal to Noise Ratio, in dB) and SSIM (Structure Similarity Index) were used to evaluate the reconstruction effect; a higher PSNR / SSIM value indicates a better reconstruction effect.

[0057] As can be seen from Table 1, this invention achieves high PSNR and SSIM. Figure 4 As can be seen, the reconstruction result of this invention has clear and natural edges, revealing more details, while the reconstruction result of the contrast algorithm exhibits certain artifacts and blurred edges in terms of subjective visual effect. In summary, compared with the contrast method, the reconstruction result of this invention achieves significant advantages in both subjective and objective evaluation. Therefore, this invention is an effective image denoising method.

[0058] Table 1

[0059]

Claims

1. An unsupervised real-image denoising method based on self-cooperative parallel generative adversarial networks, characterized in that... Includes the following steps: Step 1: Utilizing the superposition property of noise, a noise extraction module based on the initial denoiser is constructed to extract noise components from the real image. An adversarial branch, "unpaired synthesis-unpaired synthesis," is built. In this branch, the unpaired synthesis operation uses the clean image and unpaired noise as input to the generator to produce a pseudo-realistic noise image. Then, in the self-synthesis operation, the pseudo-realistic noise image passes through the denoiser and noise extraction module to obtain the estimated clean image and paired noise, which are then input together into the next generator to produce a new pseudo-realistic noise image. This achieves complementary constraints between the unpaired synthesis and self-synthesis operations. A "self-synthetic-unpaired synthesis" adversarial branch is also constructed. This branch first performs self-synthesis and then unpaired synthesis, forming a complementary branch constraint with the "unpaired synthesis-self-synthesis" adversarial branch, promoting better training results. A loss function is constructed for the entire network, and the network model is trained by minimizing this loss function. The network structure of this invention has four adversarial losses, using the least squares loss as L... adv1 The losses incurred in the confrontation: Where D represents the discriminator, y represents the real noise image, and y s-syn This represents a noisy image generated by the generator's self-synthesis method, with adversarial loss L. adv1 Limited to y and y s-syn In addition, the other three adversarial losses can also be constructed similarly by constraining the currently generated noisy image and y; by further applying the Background Guidance Module (BGM) to provide additional reliable constraints, the BGM maintains the consistency of the background between the synthesized noisy image and the clean image, constraining their low-frequency content to be similar. This is achieved by extracting the low-frequency content using multiple low-pass filters and restricting the low-frequency content to be similar to each other through L1 norm loss, thus synthesizing a pseudo-realistic noisy image x. u-syn BGM loss between the clean image x and the clean image x: Where x represents a clean image, x u-syn B represents a noisy image generated by the generator in an unpaired manner. σ (·) represents a Gaussian filter operator with a fuzzy kernel size of σ, where λ σ The weights of level σ are represented by σ-s; empirically, σ-s are set to 3, 9, and 15 respectively, and λ-s are set to 0.01, 0.1, and 1 respectively; finally, the denoiser is trained by optimizing the following function: Where m represents the total number of sample pairs, Θ represents the parameters in the network, (x rec ) i This is the denoised image output by the denoiser, where i represents the i-th sample image from 1 to m, and L... SSIM The SSIM loss is used to constrain the structural information of the image, λ SSIM It's L SSIM The weights; Step 2: Use a self-cooperative strategy to replace the weaker denoiser with the more convergent denoiser after each iteration to generate better "synthetic noise-clean" image pairs, while promoting the training of the denoiser and further improving its performance. Step 3: After the current network has fully converged, complete the network training and obtain the final model, and test the performance of the denoiser in the trained network model.

2. The unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks according to claim 1, characterized in that... Utilizing the superposition property of noise from step one, a noise extraction module is constructed. This module, based on the initial denoising unit, is used to extract noise components from the real image. The specific formula is as follows: n y =NE(y)=y-DN(y) Where y represents the input noisy image, and n y DN represents the noise corresponding to image y, DN represents the denoiser, and NE represents the noise extraction module.

3. The unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks according to claim 1, characterized in that... The "unpaired synthesis - self-synthesis" adversarial branch in step one is constructed. The unpaired synthesis operation uses the clean image and unpaired noise as input to the generator to generate a pseudo-realistic noise image. The pseudo-realistic noise image is passed through a denoiser to obtain an estimated clean image, and then through a noise extraction module to obtain paired noise. The self-synthesis operation inputs the estimated clean image and paired noise together into the next generator to generate a noise image. The specific formula is as follows: x u-syn =G(x,NE(y)) x s-syn =G(DN(x u-syn ),NE(x u-syn )) Where x represents the clean image and y represents the input noisy image, x u-syn This represents a pseudo-realistic noisy image generated by the generator in an unpaired manner. DN represents the denoiser, NE represents the noise extraction module, G represents the generator, and x represents the noise generator. s-syn This represents a noisy image generated by a self-synthesis method.

4. The unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks according to claim 1, characterized in that... The "self-synthesis-unpaired synthesis" adversarial branch in step one is constructed. The real noisy image is processed by a denoiser to obtain an estimated clean image. Paired noise is obtained through a noise extraction module. The self-synthesis operation inputs the estimated clean image and paired noise together into the generator to generate a noisy image. The unpaired synthesis operation uses the generated noisy image and unpaired noise as input to the next generator to generate a pseudo-realistic noise image. The specific formula is as follows: and s-syn =G(DN(y),NE(y)) y u-syn <G(DN(y),NE(x)> u-syn )) Where y represents the input noisy image, y s-syn This represents a noisy image generated by the generator's self-synthesis method. DN represents the denoiser, NE represents the noise extraction module, G represents the generator, and y represents the noise generator. u-syn This represents a pseudo-realistic noise image generated by the generator in an unpaired manner.

5. The unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks according to claim 1, characterized in that... Using the self-cooperative strategy in step one, in NE, k represents the number of iterations. When k = 0, it is a learnable convolutional block. When k > 0, the weaker denoiser DN0 is replaced by the current more convergent denoiser DN. This enables the synthesis of better "synthetic noise-clean" image pairs, while promoting the training of the denoiser and further improving its performance. This iterative process produces a series of increasingly powerful denoisers.

Images