Noise Reconstruction for Image Denoising

Abstract

This paper describes an apparatus (901) for image denoising, comprising a processor (904) for: receiving (701) an input image captured by an image sensor (902); executing a trained artificial intelligence model to: form (702) an estimate of a noise pattern in the input image; form (703) an estimate of at least one noise statistic of the image sensor that captured the input image; refine (704) the noise pattern estimate based on the estimate of the at least one noise statistic; and form (705) an output image by subtracting the refined noise pattern estimate from the input image. A method (800) for training the model is also disclosed herein. Taking into account the noise statistics of the sensor that captured the input image can improve the quality of the denoised image.

Description

Invention Field

[0001] This invention relates to computer vision, and more particularly to image analysis using deep neural networks, such as convolutional neural networks (CNNs). Background Technology

[0002] Image denoising aims to estimate the underlying clean image from noisy observations of the underlying clean image, and is an important step in many digital imaging and computer vision systems.

[0003] Figure 1 (a) shows how the presence of noise affects image quality. Figure 1 (b) shows how image quality can be improved by applying image denoising techniques to noisy images (from "Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising" by Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng and Lei Zhang, IEEE Transactions on Image Processing, 2017).

[0004] The camera sensor outputs RAW data in a linear color space, where pixel measurements are proportional to the number of photoelectrons collected. The main sources of noise are shot noise and readout noise. Shot noise is a Poisson process with variance equal to the signal level, while readout noise is an approximate Gaussian process caused by various sensor readout effects. These effects are well modeled using a signal-dependent Gaussian distribution.

[0005]

[0006] Where x p It is the true intensity y at pixel p. p The noise measurement value. Noise parameter σ r and σ s It is fixed for each image, but can vary from image to image as the sensor gain (ISO) changes.

[0007] However, noise in real images comes from a variety of sources (such as dark current noise and thermal noise) and is much more complex. While noise in RAW sensor data is relatively well understood, post-processing performed between capture and display in the RGB domain (such as demosaicing, sharpening, tone mapping, and compression) makes noise models more complex, making image denoising tasks more challenging.

[0008] For example, by considering the in-camera image processing pipeline, the channel-independent noise assumption may not hold. Generally, realistic noise models and in-camera image processing pipelines are important aspects of training CNN-based methods for real-world image denoising.

[0009] Traditional single-image denoising algorithms typically use mathematical tools and models (including partial differential equations, sparse coding, and low-rank approximations) to analyze and model the characteristics of the image and the noise these algorithms aim to remove. Most of these methods rely on very limited human knowledge or prior assumptions about the image, which limits their ability to recover complex image structures. In contrast, modern denoising methods often use neural networks to learn a mapping from noisy to noise-free images. Deep learning is capable of representing the complex properties of both images and noise, but training these models requires large paired datasets. Therefore, most learning-based denoising techniques rely on synthetic training data. Recent benchmarks have shown that some deep learning models often outperform traditional hand-picked extraction algorithms when evaluated on real-world noisy images.

[0010] By employing stacked convolutions, batch normalization, ReLU layers, and residual learning, the DnCNN method, published in IEEE Transactions on Image Processing (2017) by Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, achieves a significantly higher PSNR exponent than traditional state-of-the-art methods. This is achieved through stacked convolutions, batch normalization, ReLU layers, and residual learning. To further enhance denoising accuracy, several complex networks have been proposed, such as Memnet (CVPR 2017), presented by Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu.

[0011] Generative adversarial network (GAN) denoising methods consist of a generator module and a discriminator module, which are typically optimized using alternating gradient descent. The generator is derived from a prior distribution p. z Sample z, for example, a uniform distribution, and try to apply it to the target distribution p. d Modeling. The purpose of discriminator D is to distinguish between the sample distribution generated from the model and the target (ground truth) distribution.

[0012] Conditional GANs (cGANs), as described by Mehdi Mirza and Simon Osindero in their 2014 arXiv preprint "Conditional generative adversarial networks" (arXiv:1411.1784), extend the formula by providing additional labels to the generator. The generator G typically takes the form of an encoder-decoder network, where the encoder projects labels into a low-dimensional latent subspace, and the decoder performs the opposite mapping, from the low-dimensional subspace to a high-dimensional subspace. If s represents the conditional label and y represents a sample in the target distribution, then the adversarial loss is expressed as:

[0013]

[0014] By solving the following min-max problem:

[0015]

[0016] Among them, w G w D Let represent the parameters of the generator and discriminator, respectively. For the sake of simplicity, the dependencies on the parameters and the noise z are omitted in the following description.

[0017] One approach to extended conditional GANs is the Robust Conditional GAN ​​(RoCGAN), as described by G. Chrysos et al. in their paper "Robust Conditional Generative Adversarial Networks" (ICLR, 2019) presented at the International Conference on Learning Representations in 2019. In RoCGAN, the GAN has a generator module and a discriminator module, which are typically optimized using alternating gradient descent methods. The generator samples from a prior distribution (e.g., a uniform distribution) and attempts to model the target distribution. The discriminator attempts to distinguish between samples generated from the model and the target (ground truth) distribution.

[0018] In this approach, the generator is enhanced via unsupervised paths to encourage its output to traverse the target manifold, even in the presence of significant noise. The first path, called the reg path, performs a regression (denoising) similar to its counterpart in cGAN. This path takes samples from the source domain (noisy image) and maps them to the target domain (clean image). An additional AE path acts as an autoencoder in the target domain.

[0019] In RoCGAN, the AE path results in the following loss term:

[0020]

[0021] Where f d This indicates the divergence metric (L1 loss). The superscript "AE" is an abbreviation for the AE path module, and "G" is an abbreviation for the reg path module. (AE) (y (n) )=d (AE) (e (AE) (y (n) )) is the output of the AE path.

[0022] Both paths share the same encoder-decoder network. By sharing decoder weights, RoCGAN facilitates regression output across the target manifold without inducing arbitrarily large errors. Skip connections can be used in RoCGAN, enabling deeper layers to capture more abstract representations without needing to memorize all information. Lower-level representations are propagated directly to the decoder via shortcuts.

[0023] Despite sharing encoder weights, RoCGAN forces the latent representations of both paths to traverse the same space. To further reduce the distance between the two representations in the latent space, a latent loss term is used. This minimizes the distance between encoder outputs, meaning that the two representations are spatially close (within the subspace spanned by the encoder).

[0024] The potential loss term is given by the following formula:

[0025]

[0026] Feature matching loss enables the network to match the distributions of data and models more quickly. Intuitively, in order to match the high-dimensional distribution of data with the reg path, their projections in the low-dimensional space are encouraged to be similar.

[0027] The feature matching loss is given by the following formula:

[0028]

[0029] In this process, π() extracts features from the penultimate layer of the discriminator.

[0030] Skip connections allow deeper layers to capture more abstract representations without needing to memorize all the information. Lower-level representations are propagated directly to the decoder via shortcuts, making it more difficult to train longer paths—i.e., networks that don't include skip connections. This challenge is implicitly addressed by maximizing the variance captured by longer path representations. Using a Deov loss term, which penalizes the relevance in the (layer's) representations, implicitly encouraging representations to capture diverse and useful information. This loss can be applied to a single or multiple layers in the network, and for j... th Layer, this loss is defined as:

[0031]

[0032] The `diag()` function calculates the diagonal elements of a matrix. j It is j th The covariance matrix of the layer representation. When the covariance matrix is ​​a diagonal matrix, the loss is minimized, i.e., minimizing the covariance of the hidden units without restricting the diagonal elements that include the variance of the hidden representation, incurs a cost.

[0033] By defining G(s) (n) )=d (G) (e (G) (s (n) For the output of the reg path, RoCGAN's final loss function combines the original cGAN loss term with three additional terms from the AE path:

[0034]

[0035] Where λ c ,λ π , λ ae , λ l , and λ d It is a hyperparameter for balancing the loss term.

[0036] This method can be used for image denoising, but it is designed for object-related image denoising.

[0037] The Autoencoder (AE) path is an unsupervised learning method whose hidden layers contain representations of the input data, used to compress (and decompress) the data while losing as little information as possible. However, even with skipped connections, the AE path cannot reconstruct all natural scenes and patterns. In other words, it is unrealistic to define a nonlinear manifold that can accurately reconstruct image patterns from a wide variety of real-world complex objects / scenes using an autoencoder. Therefore, previous methods, such as RoCGAN, often produce the illusion of complex image structures by introducing severe blurring effects or unnatural image patterns / artifacts.

[0038] The computational and memory requirements of these methods also hinder their application on hardware-constrained devices such as smartphones or consumer electronics. Furthermore, these methods attempt to leverage image priors to better model clean images. Given the diversity of all natural image patterns, this is a highly complex problem.

[0039] The method described by I. Marras et al., presented at the 2020 European Conference on Computer Vision (ECCV2020) in "Reconstructing the Noise Variance Manifold for Image Denoising," performs image denoising using reconstructed noise across the signal-correlated noise manifold of the target image. It employs an adversarial neural network based on an encoder-decoder generator with two path modules and shared decoder parameters. The first path performs regression as a generator based on residual learning, while the generator is enhanced by a second path, which facilitates the removal of residual noise across the signal-correlated noise manifold of the target image from the noisy input. A schematic diagram of the generator is shown below. Figure 2As shown. The reg path is shown at 201, and the AE path is shown at 202.

[0040] With annual smartphone sales exceeding 1.5 billion units, it's no surprise that smartphones now take far more photos than DSLRs and point-and-shoot cameras. While the popularity of smartphones makes them a convenient photography device, their images are often degraded by higher noise levels due to the smaller sensors and lenses in their cameras. This issue urgently requires optimization of image noise reduction, especially for smartphone images.

[0041] There is a need to develop an image denoising method to overcome these problems. Summary of the Invention

[0042] According to a first aspect, an apparatus for image noise reduction is provided, the apparatus including a processor for: receiving an input image captured by an image sensor; executing a trained artificial intelligence model to: form an estimate of a noise pattern in the input image; form an estimate of at least one noise statistic of the image sensor that captured the input image; refine the noise pattern estimate based on the estimate of the at least one noise statistic; and form an output image by subtracting the refined noise pattern estimate from the input image.

[0043] Taking into account the noise statistics of the sensor that captures the input image can improve the quality of the denoised image.

[0044] This model can be used to refine the estimation of noise patterns by projecting onto a trained noise manifold. Using this method, meaningful image structure can be better preserved through the denoising process, thus improving image quality.

[0045] This model can be used to project noise pattern estimates and estimates of at least one noise statistic onto the same trained noise manifold. This allows noise statistics, such as noise standard deviation information, to be used to correct the noise statistics predicted by the model's initial estimates.

[0046] At least one noise statistic may include the noise variance of the image sensor. The noise variance projection layer of the network can explicitly utilize the noise standard deviation information to correct for the statistical information of the noise predicted by the generator's initial estimate.

[0047] The noise pattern can be estimated using a spatially adaptive Gaussian distribution. This allows for convenient noise estimation.

[0048] This device can be used to receive an indication of the specific image sensor type of the image sensor that captured the input image, and the device is used to provide this indication as input to the model. Providing the noise model with an indication of the sensor type that captured the image can improve image quality.

[0049] The apparatus may include an imaging device with an image sensor that captures an input image. This apparatus can be used to generate an input image using the imaging device, and to provide an indication of the specific image sensor type as input to the model. Providing the noisy model with an indication of the type of sensor that captured the image can improve image quality.

[0050] This model can be used to estimate at least one noise statistic based on the noise parameters of an image sensor. At least one of the noise parameters can be learnable.

[0051] The trained artificial intelligence model can be a neural network. This can be a convenient implementation method.

[0052] Neural networks can include encoder-decoder generator architectures with a first path and a second path. Enhancing the generator with the second path can help facilitate the generator's removal of residual noise from noisy inputs that crosses the noise manifold related to the target image signal.

[0053] The weights of the decoders for the first and second paths can be shared. By sharing the decoder weights, RoCGAN facilitates the regression output across the target manifold without producing arbitrarily large errors.

[0054] The first path can be used as a generator to perform regression and is enhanced by a second path, which facilitates a refined estimate of the generator by subtracting noisy patterns from the input image.

[0055] The input image can be an RGB image. The input image can also be a RAW image. This allows the device to be used in devices such as smartphones to remove noise from images captured by the cameras of such devices.

[0056] According to a second aspect, a method for image denoising is provided, the method comprising: receiving an input image captured by an image sensor; executing a trained artificial intelligence model to: form an estimate of a noise pattern in the input image; form an estimate of at least one noise statistic of the image sensor that captured the input image; refine the noise pattern estimate based on the estimate of the at least one noise statistic; and form an output image by subtracting the refined noise pattern estimate from the input image.

[0057] Taking into account the noise statistics of the sensor that captures the input image can improve the quality of the denoised image.

[0058] According to another aspect, a method for training a model to perform noise reduction on images is provided, the method comprising: receiving a plurality of input images, each input image being captured by an image sensor; receiving a plurality of noise signatures; receiving at least one noise statistic from the image sensor that captured the plurality of input images; for each of the plurality of input images:

[0059] (i) Select one of a plurality of noise signatures and apply the noise signature to the input image to form a noisy input image;

[0060] (ii) By implementing a candidate version of the model on a noisy input image, a first noise estimate in the input image and a first estimate of at least one noise statistic of the image sensor that captured the input image are formed;

[0061] (iii) Refine the first noise estimate based on a first estimate of at least one noise statistic;

[0062] (iv) An estimate of the corresponding input image is formed by subtracting the refined first noise estimate from the noisy input image;

[0063] (v) By implementing a candidate version of the model on the corresponding input image, the selected noise signature, and at least one noise statistic of the image sensor that captured the corresponding input image, a second estimate and a second noise estimate of at least one noise statistic are formed;

[0064] (vi) Adjust the candidate version of the model based on (a) the difference between the corresponding input image and the estimate of the corresponding input image, (b) the difference between the second noise estimate and the selected noise signature, and (c) the difference between the second estimate of at least one noise statistic and at least one noise statistic of the image sensor that captured the corresponding input image.

[0065] Formation step (ii) can be performed in the first path. Formation step (v) can be performed in the second path. Enhancing the image processor's generator via the second path facilitates the generator's removal of residual noise from the noisy input that crosses the noise manifold related to the target image signal.

[0066] Each of the first and second paths can include an encoder-decoder network. The weights of the decoders in the first and second paths can be shared. By sharing the decoder weights, RoCGAN facilitates the regression output across the target manifold without producing arbitrarily large errors.

[0067] The first and second paths can each be based on a fully convolutional network. This can be a convenient implementation approach.

[0068] The second approach is to implement unsupervised learning methods. The hidden layers of unsupervised learning methods can contain sufficiently powerful representations of the input data to compress (and decompress) the data while minimizing information loss.

[0069] The first path can include one or more skip connections. Skip connections, such as connecting intermediate layers of the encoder to corresponding intermediate layers of the decoder, can force the network to learn the residuals between the predicted image noise and the features corresponding to the actual image noise. This can lead to faster convergence.

[0070] Each of the multiple input images can be a RAW image or an RGB image. This allows the method to be used in devices such as smartphones to remove noise from images captured by the cameras of such devices.

[0071] The model could be a convolutional neural network. This could be a convenient implementation.

[0072] According to another aspect, an apparatus for training a model to perform noise reduction on an image is provided, the apparatus having a processor for performing the steps of the above-described method. Attached Figure Description

[0073] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0074] Figure 1 (a) shows a noisy image.

[0075] Figure 1 (b) illustrates the application of known image noise reduction techniques. Figure 1 (a) Improves image quality by reducing noise in the image.

[0076] Figure 2 This diagram illustrates the generator used in the method described by I. Marras et al. in their paper "Reconstructing the Noise Variance Manifold for Image Denoising" (ECCV 2020), presented at the European Conference on Computer Vision (ECCV 2020).

[0077] Figure 3(a) illustrates a schematic implementation of the generator described in this paper. For simplicity, the ResNet-type backbone at the beginning of the reg path is omitted.

[0078] Figure 3(b) shows a schematic diagram of network training.

[0079] Figure 4 The projection operation in the noise variance projection layer of the network is illustrated schematically.

[0080] Figure 5The diagram illustrates the building blocks used in the backbone residual network (ResNet) (Deep residual learning for image recognition, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016, K. He, X. Zhang, S. Ren and J. Sun).

[0081] Figure 6 An example of the Unet architecture used in the methods described herein is illustrated schematically.

[0082] Figure 7 A summary of examples of image denoising methods is shown.

[0083] Figure 8 A summary of examples of methods for training models to perform noise reduction on images is shown.

[0084] Figure 9 An example of a camera used to implement the apparatus and methods described herein is shown.

[0085] Figure 10 (a)- Figure 10 (d) shows the noise reduction results and a comparison with state-of-the-art methods. Figure 10 (a) shows a ground truth image of a clean ground surface. Figure 10 (b) shows the noisy RGB input image. Figure 10 (c) shows an image denoised using the method described by I. Marras et al. in “Reconstructing the Noise Variance Manifold for Image Denoising” (ECCV 2020). Figure 10 (d) shows an example of a denoised image produced using this method. Detailed Implementation

[0086] This invention relates to a method for image denoising based on an explicit understanding and utilization of the structure of noise variance associated with an image sensor. In one implementation, the predicted residual image (predicted noise) to be subtracted from the noisy input image is first corrected to match noise statistics of a specific image sensor noise characteristic. Therefore, meaningful image structure is better preserved through the denoising process, and image quality is improved.

[0087] The goal is to perform image denoising by removing generated noise distribution samples from a noisy input image using an explicitly constrained generator, where the reconstructed image noise and noise variance span the target image signal-correlated noise variance manifold.

[0088] In one embodiment, the noise variance projection layer explicitly uses noise standard deviation information to correct for statistical information of the predicted noise initially estimated by the generator.

[0089] Figure 3(a) schematically illustrates an embodiment of the CNN design for the generator. An encoder-decoder generator with two path modules and shared decoder parameters utilizes an adversarial neural network. As schematically shown in Figure 3(a), the generator typically takes the form of an encoder-decoder network, where the encoder projects labels into a low-dimensional latent subspace, and the decoder performs the reverse mapping, i.e., from the low-dimensional subspace to the high-dimensional subspace.

[0090] The generator includes a first path 301 (referred to here as the reg path) and a second path 302 (referred to here as the AE path). Both reg path 301 and AE path 302 are based on deep learning and can, for example, apply a CNN to process the input image. The CNN learns a set of filters that are applied to the image via convolutions. The convolutions are designed to be spatially invariant, meaning that they have the same effect when applied anywhere in the image.

[0091] The first path 301 acts as a generator, performing regression based on residual learning. This path (through projection) forces the noise statistics of the predicted residual image to conform to the statistics of the camera sensor noise.

[0092] The generator is enhanced by a second path 302, which facilitates the generator in removing residual noise from the noise input that crosses the noise manifold associated with the target image signal, while the noise statistics are close to the estimated noise statistics of each camera sensor that crosses the same manifold.

[0093] In other words, the network can learn how to transform a suboptimal image into an ideal one by simply looking at the structure of the predicted residual image (the predicted noise) that is forced to match the noise statistics of the camera sensor. Figure 3(a) and 3(b)As shown. This makes the task of image denoising much easier. Furthermore, by explicitly learning a low-dimensional manifold of the noise sources and their standard deviations, it is possible not only to remove noise from images more effectively, but also to combine all noise sources in the same pipeline when there are many different types of noise sources.

[0094] During training, the AE path not only learns how to reconstruct image noise (residual image), but also how to reconstruct the noise standard deviation.

[0095] The manifold of the image signal-related noise and its standard deviation is estimated to constrain the generator to remove these generated noise distribution samples from the noisy input, where the variance spans the target image signal-related noise variance manifold. Therefore, the reg path implicitly removes potentially clean images through operations in the hidden layers, while explicitly correcting the estimated camera sensor-based noise statistics. In other words, the task of the AE path is not to learn the underlying structure of a large number of complex image patterns, but rather to understand how image structure is affected by structured noise and how to define noise statistics for each camera sensor.

[0096] During training, as shown in Figure 3(b), the network learns convolutional filters, specifically the filter weights. This is accomplished using training pairs, each consisting of a noisy RGB input and a reference clean RGB image used as the ground truth (GT). Initially, the convolutional filters are set to random values. As shown in Figure 3(b), the noisy input image is preferably formed by adding a noisy signature (which can be generated from the noisy model) to the clean input image.

[0097] As described above, the training pair includes a clean RGB input image y (reference image 304) and a noisy RGB input image s = y + v (reference image 305) in Figure 3(b), where y is used as the ground truth (GT) image and v is the ground truth (GT) residual image, i.e., the noise signature 306 of the image. As described above, the noise signature v 306 can be generated by the noise model and applied to the corresponding ground truth input image y 304 to give the noisy input image s 305. Initially, the convolutional filter can be set to random values. The noisy RGB input image s 305 is input into the network, and the network regresses and denoises the output image 307.

[0098] During training, AE path 302 learns how to reconstruct image noise 309 (residual image) and noise variance 310.

[0099] The second path 302 receives the ground truth residual image (noise signature) v, 306 and the ground truth residual noise variance 308, and outputs the reconstructed ground truth residual image 309 and noise variance 310.

[0100] Therefore, the network receives multiple training input images 304 captured by an image sensor, multiple noise signatures 306, and noise statistics from the image sensor, which captures multiple input images, such as 310.

[0101] For each training image, a noise signature is applied to the image to form a noisy input image 305. On the reg path, a first noise estimate in the input image and a first estimate of the noise statistics of the image sensor capturing the input image are formed by implementing a candidate version of the model on the noisy input image 305. The first noise estimate is refined based on the first estimate of the noise statistics. Then, the estimate of the corresponding input image is obtained by subtracting the refined first noise estimate from the noisy input image.

[0102] On the AE path, a candidate version of the model is formed by implementing the model on the corresponding input image, the selected noise signature, and at least one noise statistic of the image sensor that captured the corresponding input image, forming a second noise estimate 309 and a second estimate of the noise statistic 310.

[0103] The candidate versions of the model depend on the differences between the estimates of the corresponding input image 304 and the corresponding input image 307, the differences between the second noise estimate 309 and the noise signature 306, and the differences between the second estimate and the noise statistics 308 of the image sensor that captured the corresponding input image.

[0104] For example, the difference between the regression output image 307 and the ground truth image 304 forms an error, which is then backpropagated from the output to the input via gradients through the network. The network weights are then updated to reduce the error.

[0105] The training process preferably uses a large set of images for iteration until the network weights converge.

[0106] Once the network is trained, during inference, only reg path 301 is applied to the noisy RGB input image to produce its denoised version.

[0107] As described above, path 301 takes the noisy RGB image s as input, which is then preferably processed by the backbone ResNet model, followed by U-Net, as described by Ronneberger, Olaf et al. in their paper "U-Net: Convolutional Networks for Biomedical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention (MICCAI)," Springer, LNCS, Vol. 9351: 234-241, 2015. The output is the predicted residual image (noise) G(s), which is then removed from the noisy input image to obtain the denoised image.

[0108] Therefore, the first path predicts the residual noise in the input image and subtracts the predicted residual noise, refined according to noise statistics, from the input image to obtain a denoised image. Thus, the first path 301 acts as a generator, performing regression based on residual learning. The generator is enhanced by a second path 302, which facilitates the generator's removal of residual noise from the noisy input across the noise manifold related to the target image signal during training.

[0109] Therefore, the first path does not directly output a denoised image, but is designed to predict the ground truth residual image, that is, the difference between the noisy observation and the clean (ground truth) image.

[0110] The method described above introduces the explicit use of camera noise statistics to predict overshoot or undershoot residual images to match these statistics. For this purpose, a noise standard deviation projection layer 303 is placed after the generator. The input to the noise standard deviation projection layer 303 is the predicted residual image (predicted image noise) and the predicted noise standard deviation.

[0111] The first path does not directly output a denoised image; instead, it is designed to predict the ground truth residual image G(s), which is the difference between the noisy observation s and the clean (ground truth) image y. Compared to RoCGAN, the first path also predicts the image standard deviation K(s). G(s) and K(s) are inputs to projection layer 303, which (through projection) forces the noise statistics of G(s) to conform to the statistics of camera sensor noise. The final residual image estimate is the output of projection layer G'(s). This can then be removed from the noisy input image.

[0112] In one implementation, the model is used to estimate noise statistics based on the noise parameters of the image sensor. Figure 4 The projection operation of the noise variance projection layer 303 is illustrated. The threshold ε used in the projection layer is parameterized, such as... Figure 4 As shown. The threshold ε is the predicted pixel-based noise standard deviation σ, and the number of image pixels N. t A function with a learnable parameter α.

[0113] In some implementations, multiple ε values ​​can be learned for different camera sensors and different image ISO group values. The reg path implicitly estimates the potential clean image through operations in the hidden layers.

[0114] Unsupervised path 302 acts as an autoencoder in the common domain of the real residual image v = s – y and the real image noise standard deviation b. The input v of this path is concatenated with y and b, v ⊙ y ⊙ b. Thus, the task of AE path 302 is not to understand how the image structure is affected by the presence of structured noise, but rather how the noise statistics relate to the actual noise. By sharing the decoder weights, the generator employs a residual learning strategy to remove image information from the noisy observations that crosses the common manifold of image noise and image noise standard deviation.

[0115] Adding noise is a challenging process that can be reversed by a few convolutional layers in the encoder of the recursive path, especially in object-independent scenes. To address this, a backbone network can be used before the recursive path to extract complex feature representations that help preserve both low and high image frequencies later. As mentioned above, the backbone network can be a residual network (ResNet), such as the paper "Deep residual learning for image recognition" by K. He, X. Zhang, S. Ren, and J. Sun, published in the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. Figure 5 As shown. Therefore, the input to the reg path can be features extracted from the input image (e.g., in the form of a tensor), rather than the image itself.

[0116] Therefore, the reg path preferably takes an RGB image as input, which is then processed by the backbone ResNet model and subsequently by the Unet model. The output is the predicted residual image (noise), which is then removed from the noisy input image to obtain the denoised image. Unet uses an encoder-decoder architecture with two transverse Unet-style skip connections (more details are shown in...). Figure 6 (As described below). These skip connections connect the intermediate layers of the encoder to the corresponding intermediate layers of the decoder, forcing the network to learn the residuals between the features corresponding to the predicted image noise and the actual image noise. This has an impact on speeding up convergence, as empirically detected. The AE path takes the true (ground truth) residual image and the true (ground truth) noise standard deviation as input, and is then processed by a Unet similar to the Unet in the reg path. The output is a reconstruction of the true residual image and the true noise standard deviation.

[0117] In this implementation, the identifier remains the same as in RoCGAN. It accepts the predicted clean image s–G'(s) and y as input.

[0118] In this implementation, the input is RGB image data. The network can also accept RAW images as input.

[0119] Both the reg path and the AE path are based on deep learning (e.g., using CNNs). Both paths apply convolutional neural networks to process RGB images. CNNs learn a set of filters that are applied to the image via convolutions. Convolutions are designed to be spatially invariant, meaning they have the same effect when applied anywhere in the image.

[0120] Figure 6 An example of the Unet architecture that can be used for the method described in this paper is shown. Unet uses an encoder-decoder architecture with two transverse Unet-style skip connections. The encoder portion of the network is typically shown at 601, the decoder at 602, and the skip connections at 603. These skip connections connect intermediate layers of the encoder to corresponding intermediate layers of the decoder, forcing the network to learn the residuals between the features corresponding to the predicted image noise and the actual image noise. This has an impact on speeding up convergence, as empirically detected, the AE path takes the true (ground truth) residual image as input and is then processed by Unet, similar to Unet in the reg path. The output is a reconstruction of the true residual image.

[0121] In this example, the encoder portion of the network, typically shown at 601, processes a noisy RGB input with six consecutive layers. Each layer applies a strided convolution with a 3x3 convolutional filter (along with ReLU activation and batch normalization) to its input. The strided convolution doubles the number of filters (i.e., channels) while halving the spatial image resolution (i.e., from H, W, C to H / 2, W / 2, C). The image undergoes multi-scale processing, and the network adapts to different frequency contents. This produces output channels that capture the inherent features of the data and are relevant to the RGB image denoising task.

[0122] The decoder portion of the network, typically shown at 602, processes the encoder's output using five consecutive layers of transposed convolutional operations with 3x3 convolutional filters (along with ReLU activation functions and batch normalization). The transposed convolution is an upsampling layer that doubles the spatial resolution in each dimension (width and height) and halves the number of filters.

[0123] Figure 6 The skip connections shown at point 603 allow deeper layers to capture more abstract representations without having to memorize all the information. Lower-level representations are propagated directly to the decoder via a shortcut. In the case of layers with Unet-style skip connections, the input to each of these decoder layers is a concatenation of i) high-resolution features from the encoded portion associated with the same spatial resolution and ii) the output of the previous decoder layer (i.e., spatially upsampled features). Subsequent convolutional learning assembles a more accurate output based on the concatenated input. The input to each of the remaining decoder layers is simply the output of the previous decoder layer.

[0124] By sharing the weights of the decoder, the generator employs a residual learning strategy to remove image information that crosses the image noise manifold from noisy observations.

[0125] In this implementation, the reg path and AE path both include fully convolutional networks.

[0126] Figure 7 An image denoising method 700 is summarized, which can be implemented using the networks and apparatus described herein. In step 701, the method includes receiving an input image captured by an image sensor. The method includes executing a trained artificial intelligence model to perform steps 702-704. In step 702, the method includes forming an estimate of a noise pattern in the input image. In step 703, the method includes forming an estimate of at least one noise statistic of the image sensor that captured the input image. In step 704, the method includes refining the noise pattern estimate based on the estimate of the at least one noise statistic. In step 705, the method includes forming an output image by subtracting the refined noise pattern estimate from the input image.

[0127] Figure 8 A method 800 for training a model to perform noise reduction on images is summarized. In step 801, the method includes: receiving a plurality of input images, each input image captured by an image sensor; receiving a plurality of noise signatures; receiving at least one noise statistic from the image sensor that captured the plurality of input images; and for each of the plurality of input images:

[0128] (i) Select one of a plurality of noise signatures and apply the noise signature to the input image to form a noisy input image;

[0129] (ii) By implementing a candidate version of the model on the noisy input image, a first noise estimate in the input image and a first estimate of at least one noise statistic of the image sensor that captured the input image are formed.

[0130] (iii) Refine the first noise estimate based on a first estimate of at least one noise statistic;

[0131] (iv) An estimate of the corresponding input image is formed by subtracting the refined first noise estimate from the noisy input image;

[0132] (v) By implementing a candidate version of the model on the corresponding input image, the selected noise signature, and at least one noise statistic of the image sensor that captured the corresponding input image, a second estimate and a second noise estimate of at least one noise statistic are formed;

[0133] (vi) Adjust the candidate version of the model based on (a) the difference between the corresponding input image and the estimate of the corresponding input image, (b) the difference between the second noise estimate and the selected noise signature, and (c) the difference between the second estimate of at least one noise statistic and at least one noise statistic of the image sensor that captured the corresponding input image.

[0134] Figure 9 An example architecture of a camera that can implement the methods described above is shown. Camera 901 is connected to a communication network. Camera 901 includes an image sensor 902. The camera also includes a memory 903, a processor 904, and a transceiver 905. The memory stores code that can be executed by the processor 904 in a non-transitory form. In some implementations, this code may include a trained artificial intelligence model as described above. The model may include code and / or parameters that can be directly executed by the processor, such as neural network weights, which are not directly executable instructions but are used to configure other executable code stored in memory 903. Transceiver 905 may be able to send and receive data through one or both wired and wireless communication channels. For example, it may support cellular protocols such as Ethernet, IEEE 802.11b, and / or 4G or 5G.

[0135] Such a camera 901 typically includes some onboard processing capabilities. This can be provided by a processor 904. The processor 904 can also be used for the basic functions of the device.

[0136] Transceiver 905 is capable of communicating with other entities 910, 911 via a network. These entities may be physically located away from camera 901. The network may be a publicly accessible network, such as the Internet. Entities 910, 911 may be cloud-based. Entity 910 is a computing entity. Entity 911 is a command and control entity. These entities are logical entities. In practice, they may be provided by one or more physical devices (such as servers and data storage), and the functionality of two or more entities may be provided by a single physical device. The physical device implementing each entity includes a processor and memory. These devices may also include transceivers for sending data to and receiving data from transceiver 905 of camera 901. The memory stores, in a non-transitory manner, code executable by the processor, thereby implementing the respective entities in the manner described herein.

[0137] Command and control entity 911 can train the model. This is typically a computationally intensive task, even if the resulting model can be efficiently described; therefore, developing the model in the cloud is likely to be efficient, given the abundance of energy and computing resources available there. Predictably, this is more efficient than forming such a model on a typical camera.

[0138] In one implementation, once the model is developed in the cloud, the command and control entities can automatically generate the corresponding model and send it to the relevant camera device. In this example, noise reduction is performed by the processor 904 at the camera 901.

[0139] In another possible implementation, the image can be captured by camera sensor 902, and the image data can be sent to the cloud for processing by transceiver 905. The resulting denoised image can then be sent back to camera 901, as shown below. Figure 9 As shown at position 912 in the diagram.

[0140] Therefore, this method can be deployed in various ways; for example, in the cloud, on a device, or in dedicated hardware. As mentioned above, cloud facilities can perform training to develop new models or improve existing ones. Depending on the computing power near the data corpus, training can be performed near the source data or in the cloud, for example, using an inference engine.

[0141] Camera 901 can execute a trained model on an input image to form a noise estimate in the image, and subtract the noise estimate from the input image to form a denoised output image. The camera can perform a method including the following steps: receiving an input image captured by an image sensor; executing a trained artificial intelligence model to: form an estimate of a noise pattern in the input image; form an estimate of at least one noise statistic of the image sensor that captured the input image; refine the noise pattern estimate based on the estimate of the at least one noise statistic; and form an output image by subtracting the refined noise pattern estimate from the input image.

[0142] The embodiments of the methods described herein have many advantages.

[0143] For the AE path, reconstructing the real image noise and noise standard deviation based on the same low-dimensional manifold allows for a better understanding of camera sensor noise statistics. In other words, it's possible to learn how to transform an undesirable image into an ideal one by simply examining the structure of the predicted residual image, which is forced to match the camera sensor's noise statistics. This property enables the method to adapt to different camera sensors and different image ISO values, and helps the denoiser largely avoid over-smoothing the image.

[0144] The noise variance projection layer explicitly gives any noise reducer the ability to adjust the noise reduction level based on the ISO value of the image and the camera sensor used to capture that image.

[0145] In the case of multi-source image noise, multiple AE paths can be used. In this case, each path is responsible for removing noise information from a specific noise source with specific noise statistics.

[0146] The method described in this paper simplifies adapting existing trained models to new camera sensors (inter-domain switching). While this requires retraining the AE path, it simplifies fine-tuning the reg path using only a small number of paired training samples obtained with the new sensor.

[0147] By learning to constrain the residual image noise variance, the denoising task becomes simpler compared to the methods described by RoCGAN or I. Marras et al. in "Reconstructing the NoiseVariance Manifold for Image Denoising" (ECCV 2020).

[0148] For RoCGAN, there is no explicit method to ensure that the noise statistics predicted by the generator match the expected noise statistics of the camera sensor. Thus, the network cannot explicitly understand that the estimated noise standard deviation should be significantly smaller in images with lower ISO values ​​than in images with higher ISO values. Consequently, in many cases, the denoiser destroys high-frequency details in the image, especially for low-ISO images. The use of different models for different ISO groups also makes it unsuitable for resource-constrained electronic devices such as smartphones, where runtime performance is critical.

[0149] In RoCGAN, the AE path is an unsupervised learning method whose (hidden) representations of the input data are strong enough to compress (and decompress) the data while minimizing information loss. The AE path directly reconstructs image noise but not noise statistics. In other words, using an autoencoder to define a nonlinear manifold capable of accurately reconstructing image noise does not guarantee that the predicted noise standard deviation is (close to) the correct noise standard deviation. For example, given a specific camera sensor, there exists a maximum standard deviation for a given image. Therefore, it is difficult to ensure that the RoCGAN network can adapt to different camera sensors and different ISO values. Consequently, RoCGAN often produces the illusion of complex image structures by introducing severe blurring effects or strange image patterns / artifacts. Compared to denoised images produced using RoCGAN and other existing methods, the method described in this paper can improve image quality.

[0150] The applicant hereby discloses each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, regardless of whether such features or combinations of features solve any problem disclosed herein; and without limiting the scope of the claims. The applicant notes that aspects of the invention may include any such individual feature or combination of features. In view of the foregoing description, those skilled in the art will appreciate that various modifications can be made within the scope of the invention.

Claims

1. An apparatus (901) for image denoising, characterized in that, The device includes a processor (904) for: Receive (701) the input image captured by the image sensor (902); Execute the trained artificial intelligence model to: Forming (702) an estimate of the noise pattern in the input image; (703) Form an estimate of at least one noise statistic of the image sensor that captures the input image; Based on the estimation of the at least one noise statistic, the estimation of the noise pattern is refined (704); wherein the estimation of the noise pattern is refined by projection onto a trained noise manifold; (705) An output image is formed by subtracting the refined noise pattern estimate from the input image.

2. The apparatus (901) according to claim 1, characterized by The model is used to project the estimate of the noise pattern and the estimate of the at least one noise statistic onto the same trained noise manifold.

3. The apparatus (901 ) according to claim 1 or 2, characterized by The at least one noise statistic includes the noise variance of the image sensor.

4. The apparatus (901 ) according to claim 1 or 2, characterized by The estimation of the noise pattern is a spatially adaptive Gaussian distribution.

5. The apparatus (901 ) according to claim 1 or 2, characterized by The device is used to receive an indication of the specific image sensor type of the image sensor that captures the input image, wherein the device is used to provide the indication as input to the model.

6. The apparatus (901) according to claim 1 or 2, characterized in that, The apparatus includes an imaging device having the image sensor (902) that captures the input image, and the apparatus is configured to: generate the input image using the imaging device, and provide an indication of a specific image sensor type as input to the model.

7. The apparatus (901) according to claim 1 or 2, characterized in that, The model is used to estimate at least one noise statistic based on the noise parameters of the image sensor (902).

8. The apparatus (901) according to claim 7, characterized in that, At least one of the noise parameters is learnable.

9. The apparatus (901) according to claim 1 or 2, characterized in that, The trained artificial intelligence model is a neural network.

10. The apparatus (901) according to claim 9, characterized in that, The neural network includes an encoder-decoder generator architecture with a first path (301) and a second path (302).

11. The apparatus (901) according to claim 10, characterized in that, The decoders of the first path (301) and the second path (302) share weights.

12. The apparatus (901) according to claim 10 or 11, characterized in that, The first path (301) is used to act as a generator to perform regression and is enhanced by the second path (302), which facilitates the generator to subtract the refinement estimate of the noise pattern from the input image.

13. The apparatus (901) according to claim 1 or 2, characterized in that, The input image is an RGB image or a RAW image.

14. A method (700) for image denoising, characterized in that, The method includes: Receive (701) an input image captured by an image sensor; Execute the trained artificial intelligence model to: Forming (702) an estimate of the noise pattern in the input image; (703) Form an estimate of at least one noise statistic of the image sensor that captures the input image; Based on the estimation of the at least one noise statistic, the estimation of the noise pattern is refined (704); wherein the estimation of the noise pattern is refined by projection onto a trained noise manifold; (705) An output image is formed by subtracting the refined noise pattern estimate from the input image.

15. A method (800) for training a model to perform noise reduction on an image, characterized in that, The method includes: Receive (801) multiple input images (304), each input image being captured by an image sensor (902); Receive (802) multiple noise signatures (306); Receive (803) at least one noise statistic (308) of the image sensor that captures the plurality of input images; For each of the plurality of input images: (i) Select one of the plurality of noise signatures (306) (804) and apply the noise signature to the input image (304) to form a noisy input image (305). (ii) By implementing a candidate version of the model on the noisy input image, a first noise estimate in the input image and a first estimate of at least one noise statistic of the image sensor that captured the input image are formed (805); (iii) Refine (806) the first noise estimate based on the first estimate of the at least one noise statistic; (iv) An estimate of the corresponding input image (307) is formed (807) by subtracting the refined first noise estimate from the noisy input image (305); (v) By implementing the candidate version of the model on the corresponding input image, the selected noise signature and the at least one noise statistic of the image sensor that captured the corresponding input image, a second noise estimate (309) and a second estimate (310) of the at least one noise statistic are formed (808). (vi) Adjust (809) the candidate version of the model based on (a) the difference between the estimates of the input image (304) and the corresponding input image (307), (b) the difference between the second noise estimate (309) and the selected noise signature (306), and (c) the difference between the second estimate of the at least one noise statistic (310) and the at least one noise statistic (306) of the image sensor that captured the input image (304).

16. The method according to claim 15, characterized in that, Step (ii) is performed in the first path (301), and step (v) is performed in the second path (302).

17. The method according to claim 16, characterized in that, Each of the first path (301) and the second path (302) includes an encoder-decoder network.

18. The method according to claim 17, characterized in that, The decoders of the first path (301) and the second path (302) share weights.

19. The method according to any one of claims 16 to 18, characterized in that, The first path (301) and the second path (302) are based on fully convolutional networks, respectively.

20. The method according to any one of claims 16 to 18, characterized in that, The second path (302) implements an unsupervised learning method.

21. The method according to any one of claims 16 to 18, characterized in that, The first path (301) includes one or more skip connections (603).

22. The method according to any one of claims 15 to 18, characterized in that, Each of the plurality of input images (304) is a RAW image or an RGB image.

23. The method according to any one of claims 15 to 18, characterized in that, The model is a convolutional neural network.

24. An apparatus (901) for training a model to perform noise reduction on an image, characterized in that, The device has a processor (904) for performing the steps of the method according to any one of claims 15 to 23.

Images