Noise reconstruction for image denoising

Abstract

An apparatus (901) for denoising an image, the apparatus (901) having a processor to receive an input image, implement a trained artificial intelligence model to form an estimate of a noise pattern in the input image and form an output image by subtracting the estimate of the noise pattern from the input image, the model being to form the estimate of the noise pattern such that the estimate of the noise pattern is representative of a noise pattern characteristic of a particular image sensor type.

Description

TECHNICAL FIELD

[0001] The present invention relates to computer vision, in particular to image analysis using deep neural networks such as convolutional neural networks. BACKGROUND

[0002] Image denoising aims to estimate the underlying clean image from noisy observations. Denoising is an important step in many digital image and computer vision systems. Figure 1 (a) illustrates how the presence of noise affects image quality. Figure 1 (b) illustrates the improvement in image quality obtained after applying an image denoising technique to the noisy image of (a) (according to Kai Zhang, Kai Zhang, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond Gaussian denoising: Residual learning of deep CNN for image denoising”, IEEE Transactions on Image Processing, 2017). Figure 1 (a) illustrates how the presence of noise affects image quality. Figure 1 (b) illustrates the improvement in image quality obtained after applying an image denoising technique to the noisy image of (a) (according to Kai Zhang, Kai Zhang, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond Gaussian denoising: Residual learning of deep CNN for image denoising”, IEEE Transactions on Image Processing, 2017).

[0003] Camera sensors output RAW data in a linear color space, where pixel measurements are proportional to the number of photoelectrons collected. The main sources of noise are shot noise (a Poisson process with variance equal to the signal level) and readout noise (an approximately Gaussian process caused by various sensor readout effects). These effects can be well modeled by a signal-dependent Gaussian distribution:

[0004]

[0005] where x p is the noisy measurement of the true intensity y p at pixel p. The noise parameters σ r and σ s are fixed for each image, but the noise parameters can vary between images as the sensor gain (ISO) changes.

[0006] However, the noise in real images originates from different sources (e.g. dark current noise and thermal noise) and is more complex. Although the noise in RAW sensor data is relatively easy to understand, in the RGB domain, the post-processing performed between capture and display (e.g. demosaicing, sharpening, tone mapping, and compression) makes the noise model more complex, making the image denoising task more challenging.

[0007] For example, considering the in-camera image processing pipeline, the noise assumption of being channel-independent can not hold. In general, realistic noise models and in-camera image processing pipelines are important aspects for training CNN-based denoising methods for real photos.

[0008] Traditional single-image denoising algorithms usually analyze and model the properties of the image and the noise they are designed to remove. Many methods have been developed using different mathematical tools and models, including partial differential equations, sparse coding, and low-rank approximations. These methods mostly rely on very limited human knowledge or assumptions about the image, limiting their ability to recover complex image structures.

[0009] Modern denoising methods usually employ neural networks to learn a mapping from noisy images to noise-free images. Deep learning is able to represent complex properties of images and noise, but training these models requires large paired datasets. Therefore, most learning-based denoising techniques rely on synthetic training data.

[0010] By stacking convolutional, batch normalization, ReLU layers, and employing the idea of residual learning, the DnCNN method, as described in Kai Zhang, Kai Zhang, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Transactions on Image Processing, 2017, achieves higher PSNR indices than traditional state-of-the-art methods. Some complex networks have also been proposed, such as in Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu, “MemNet: A multi-memorized deep network for image restoration,” CVPR, 2017.

[0011] The method of denoising with a generative adversarial network (GAN) includes a generator and a discriminator module, which are usually optimized with an alternating gradient descent method. The generator samples z from a prior distribution p z (such as a uniform distribution) and attempts to model the target distribution p d . The discriminator D aims to distinguish between the samples generated by the model and the target (ground truth) distribution.

[0012] A conditional GAN (cGAN), as described in Mehdi Mirza and Simon Osindero, “Conditional generative adversarial nets,” arXiv preprint arXiv:1411.1784, 2014, extends the formulation by providing the generator with an additional label s. The generator G usually takes the form of an encoder-decoder network, where the encoder projects the label s into a low-dimensional latent subspace, and the decoder performs the inverse mapping, i.e., from the low-dimensional subspace to the high-dimensional one. If s represents the conditioning label and y represents a sample in the target distribution, then the adversarial loss is represented as:

[0013]

[0014] by solving the following min-max problem:

[0015]

[0016] where w G and w D denote the parameters of the generator and discriminator, respectively. To simplify notation, the dependence on the parameters and the noise z is omitted in the following description.

[0017] A recent method that extends the conditional GAN approach is the Robustly conditioned GAN (RoCGAN), as described in Gregorios Kerasos et al., “Robustly conditioned GAN for improving the robustness to label noise of image classification networks”, International Conference on Learning Representations (ICLR), 2019. Figure 2 A conventional generator according to the RoCGAN approach is schematically illustrated. In this method, the generator is augmented using an unsupervised pathway to encourage the output of the generator to span the target manifold, even in the presence of a large amount of noise. A first pathway 201, referred to as the reg pathway, performs a similar regression (denoising) as its counterpart in cGAN. It takes a sample from the source domain (noisy image) and maps it to the target domain (clean image). An additional AE pathway 202 is used as an autoencoder in the target domain.

[0018] In RoCGAN, the AE pathway results in the following loss term:

[0019]

[0020] where f d denotes the difference measure ( loss), the superscript “AE” abbreviates the modules of the AE pathway, “G” abbreviates the modules of the reg pathway, and G (AE) (y (n) ) = d (AE) (e (AE) (y (n) )) is the output of the AE pathway.

[0021] Although the weights of the encoder are shared, RoCGAN forces the latent representations of the two pathways to span the same space. To further reduce the distance of the two representations in the latent space, a latent loss term This term minimizes the distance between the outputs of the encoders, i.e., the two representations are close in space (in the subspace spanned by the encoder).

[0022] The latent loss term is given by:

[0023]

[0024] The feature matching loss enables the network to match the distribution of data and model faster. Intuitively, to match the high-dimensional distribution of data with the reg path, one encourages their projections in the low-dimensional space to be similar.

[0025] The feature matching loss is given by:

[0026]

[0027] where π() extracts features from the second-to-last layer of the discriminator.

[0028] Skip connections enable deep layers to capture more abstract representations without having to remember all the information. Low-level representations

[0029] By propagating directly to the decoder through a shortcut, it makes training longer paths (i.e., networks that do not include skip connections) more difficult. This challenge can be addressed implicitly by maximizing the variance captured by longer paths representations. A Decov loss term is used, which penalizes the correlation in the (layer’s) representation and thus implicitly encourages the representation to capture diverse and useful information. This loss can be applied to a single layer or multiple layers in the network, while for j th layer, the loss is defined as:

[0030]

[0031] where diag() computes the diagonal elements of a matrix, C j is the covariance matrix of the j th layer representation. When the covariance matrix is diagonal, the loss is minimized, i.e., it penalizes to minimize the covariance of hidden units without restricting the diagonal elements that include the variance of the hidden representation.

[0032] By defining G(s (n) ) = d (G) (e (G) (s (n) )) as the output of the reg path, the final loss function of RoCGAN combines the loss terms of the original cGAN with the additional three terms of the AE path:

[0033]

[0034] where λ c λ π , λ ae , λ l , and λ d are hyperparameters used to balance the loss terms.

[0035] An AE pathway is an unsupervised learning method whose hidden layers contain representations of the input data that are used to compress (and decompress) the data while losing as little information as possible. However, even with skip connections, an AE pathway cannot reconstruct all natural scenes and patterns. In other words, it is unrealistic to use one autoencoder to define a non-linear manifold from which image patterns can be accurately reconstructed from a variety of real complex objects / scenes. Therefore, previous methods like RoCGAN often create an illusion of complex image structures by introducing severe blurring effects or non-natural image patterns / artifacts.

[0036] The large amount of computation and memory usage of these methods also hinders their application on hardware-constrained devices, such as smartphones or consumer electronics. Furthermore, these methods attempt to leverage image priors to better model clean images; giving various natural image patterns is a very complex problem.

[0037] While the popularity of smartphones makes them a convenient device for photography, their image quality is often reduced due to higher levels of noise because of the smaller sensors and lenses in the cameras. This problem increases the need for image denoising, especially in the case of smartphone images.

[0038] Therefore, it is desirable to develop an image denoising method that can overcome these problems. SUMMARY

[0039] According to a first aspect, there is provided an apparatus for denoising an image, the apparatus having a processor for receiving an input image, implementing a trained artificial intelligence model to form an estimate of a noise pattern in the input image and forming an output image by subtracting the estimate of the noise pattern from the input image, the model being for forming the estimate of the noise pattern such that the estimate of the noise pattern is representative of a noise pattern specific to a particular image sensor type.

[0040] The model can be for forming an estimate of a noise pattern specific to each of a plurality of types of image sensor, the apparatus can be for receiving an indication of the particular image sensor type of an image sensor that captured the input image, the apparatus can be for providing the indication as input to the model, and the model can be for generating the estimate of the noise pattern in dependence on the indication. The apparatus can comprise an imaging device having the image sensor that captured the input image, and the apparatus can be for generating the input image using the imaging device and providing the indication of the particular image sensor type as input to the model. Providing an indication of the type of sensor that captured an image to the noise model can result in improved image quality.

[0041] The apparatus can comprise an imaging device in which the image sensor is comprised, and the apparatus can be used to generate the input image using the imaging device and provide information indicative of the noise behavior of the image sensor as input to the model. Providing further information as input to the model, such as parameters of the noise model of the sensor, can lead to improved image quality.

[0042] The noise pattern is non-Gaussian. This can lead to improved quality of the denoised image in processes such as demosaicing, sharpening, tone mapping, and compression, and to accommodate more complex noise models required due to post-processing performed between capture and display.

[0043] The model is used to form the estimate of the noise pattern by means of a projection on a noise manifold trained. Using this approach, meaningful image structure can be better preserved by the denoising process, leading to improved image quality.

[0044] According to a second aspect, there is provided a method for training a model to perform denoising of images, the method comprising: receiving a plurality of training images; receiving a plurality of noise features; for each of the plurality of training images: (i) selecting one of the plurality of noise features and applying the noise feature to the training image to form a noisy input image; (ii) forming a first noise estimate in the noisy input image by implementing a candidate version of the model on the noisy input image, and forming an estimate of the respective training image by subtracting the first noise estimate from the noisy input image; (iii) forming a second noise estimate by implementing the candidate version of the model on the respective training image and the selected noise feature; and (iv) adapting the candidate version of the model according to (a) a difference between the respective training image and the estimate of the respective training image and (b) a difference between the second noise estimate and the selected noise feature. The plurality of training images comprises a plurality of clean ground truth input images, and the plurality of noise features comprises a plurality of real residual images.

[0045] The forming step (ii) is performed in a first pass, and the forming step (iii) is performed in a second pass. The generator of the image processing unit is prompted by the second pass to remove residual noise from the noisy input that spans the target image signal related noise manifold.

[0046] Each of the first pass and the second pass can comprise an encoder-decoder network. The weights of the decoders of the first pass and the second pass can be shared. By sharing the weights of their decoders, this facilitates the regression output to span the target manifold without causing arbitrarily large errors.

[0047] The first pathway and the second pathway can each be based on a fully convolutional network. This can be a convenient implementation.

[0048] The second pathway implements an unsupervised learning method. The hidden layers of an unsupervised learning method can contain a sufficiently powerful representation of the input data that can be used to compress (and decompress) the data while losing as little information as possible.

[0049] The first pathway can include one or more skip connections. A skip connection that connects an intermediate layer of an encoder to a corresponding intermediate layer of a decoder can force the network to learn the residual between the predicted image noise and the corresponding features of the actual image noise. This can result in faster convergence.

[0050] Each of the plurality of training images and / or input images can be a RAW image or an RGB image. This can allow the method to be used for devices such as smartphones to remove noise from images captured by the cameras of such devices.

[0051] The model can be a convolutional neural network. This can be a convenient implementation.

[0052] According to a third aspect, there is provided a device for training a model to perform denoising of images, the device having a processor for performing the steps of: receiving a plurality of training images; receiving a plurality of noise features; for each of the plurality of training images: (i) selecting one of the plurality of noise features and applying the noise feature to the training image to form a noisy input image; (ii) forming a first noise estimate in the noisy input image by implementing a candidate version of the model on the noisy input image and forming an estimate of the respective training image by subtracting the first noise estimate from the noisy input image; (iii) forming a second noise estimate by implementing the candidate version of the model on the respective training image and the selected noise feature; and (iv) adapting the candidate version of the model in dependence on (a) a difference between the respective training image and the estimate of the respective training image and (b) a difference between the second noise estimate and the selected noise feature. The plurality of training images comprises a plurality of clean ground truth input images and the plurality of noise features comprises a plurality of true residual images. BRIEF DESCRIPTION OF DRAWINGS

[0053] The application will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0054] Figure 1 (a) shows a noisy image.

[0055] Figure 1 (b) shows the result of applying a candidate version of the model to the noisy image. Figure 1(a) shows the improvement in image quality achieved after applying denoising techniques to the noisy image.

[0056] Figure 2 A conventional generator based on the RoCGAN method described in "Robust Conditional Generative Adversarial Networks" (ICLR, 2019) by G. Chrysos et al. is illustrated.

[0057] Figure 3 A schematic diagram of a generator for image denoising in the RGB domain is shown.

[0058] Figure 4 The building blocks used in the backbone residual network (ResNet) are illustrated schematically. (K. He, X. Zhang, S. Ren and J. Sun, “Deep Residual Learning for Image Recognition” (IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016)).

[0059] Figure 5 This schematically illustrates the use of more than one AE path in the case of multi-source image noise from a camera.

[0060] Figure 6 An example of a discriminator is illustrated schematically.

[0061] Figure 7 An example of the Unet architecture used in the methods described herein is illustrated schematically.

[0062] Figure 8 This describes the workflow of the method used to train the model to perform noise reduction on the image.

[0063] Figure 9 An example of an imaging device for implementing the image processor described herein is illustrated schematically.

[0064] Figure 10 The denoising results are shown compared to those obtained using DnCNN (Zhang et al., 2017). Figure 10 (a) shows a noisy RGB input image. Figure 10 (b) shows the denoised image obtained using the DnCNN method. Figure 10 (c) shows a denoised image processed by the method described herein. Figure 10 (d) shows a clean ground truth image. Detailed Implementation

[0065] This paper describes an image denoising method based on an explicit understanding of the noise structure added to the image captured by the image sensor. This method directly reconstructs the image noise. Using this method, meaningful image structure can be better preserved during the denoising process, thereby improving image quality.

[0066] The goal of this method is to perform image denoising using reconstructed image noise across a target image signal-related noise manifold. The input to the image processor can include RAW image data or RGB image data. The image processor includes generator and discriminator modules, each comprising a convolutional neural network (CNN) optimized using an alternating gradient descent method. The generator samples from a prior distribution (e.g., a uniform distribution) and is designed to model the target distribution. The discriminator aims to distinguish the model-generated samples from the target (ground truth) distribution.

[0067] A preferred embodiment of the CNN design for the generator is illustrated as follows: Figure 3 As shown. Figure 3 The inputs and outputs of the generator's pathways during model training are schematically illustrated. An encoder-decoder generator is used, employing an adversarial neural network with two pathway modules and shared decoder parameters. Figure 3 As illustrated, the generator typically takes the form of an encoder-decoder network, where the encoder projects labels into a low-dimensional latent subspace, while the decoder performs the opposite mapping, i.e., from the low-dimensional subspace to the high-dimensional subspace.

[0068] The generator comprises a first path 301 (referred to herein as reg path 301) and a second path 302 (referred to herein as AE path 302). Both reg path 301 and AE path 302 are based on deep learning and can, for example, apply a CNN to process images. The CNN learns a set of filters that are applied to the image via convolution. The convolutions are designed to be spatially invariant, meaning that they have the same effect when applied to any location in the image.

[0069] In the reg path, especially in object-independent scenes, noise is a challenging process that can be reversed by several convolutional layers in the encoder. To address this, a backbone network, as shown at 303, can be used before the reg path to extract complex feature representations, which help preserve them later on low and high image frequencies. The backbone network 303 can be a Residual Network (ResNet), as described in "Deep Residual Learning for Image Recognition" by K. He, X. Zhang, S. Ren, and J. Sun (IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016), which is created by stacking building blocks, such as... Figure 4The input to the reg pathway 301 can thus be features extracted from the image to be denoised (e.g. in the form of a tensor), rather than the image itself.

[0070] During training, the network learns the convolutional filter weights. This can be done using a number of training pairs comprising a reference clean RGB input image y, 307 (which is used as a ground truth (GT) image (i.e. the training image)) and a noisy RGB input image s = y + v, 304, where v is a true (GT) residual image 306, which is the noise feature of the image. The noise feature v, 306, can be generated by a noise model and applied to each ground truth input image y, 307 to get the noisy input image s, 304. Initially, the convolutional filter is set to random values. The noisy RGB input image s, 304 is input into the network and the network regresses a denoised output image 305 (i.e. the predicted clean image). The difference between the regressed denoised output image 305 and the clean GT image 307 forms an error, which is then backpropagated through the gradient from the output to the input of the network. The weights of the network are then updated to reduce the error. The training process is iterated using a large number of training images until the network weights converge.

[0071] During training, the AE pathway 302 learns how to reconstruct the image noise (residual image). The second pathway 302 receives the ground truth residual image (noise feature) v, 306 and the clean ground truth image y, 307 and outputs a reconstructed ground truth residual image G'(s), 308. The input to the AE pathway 302 is the GT residual image (i.e. the noise feature) v, 306, concatenated with the clean ground truth input image y, 307 (i.e. the input is v © y). In this way, the task of the AE pathway 302 is not to learn the underlying structure of a large number of diverse complex image patterns, but to learn how the presence of an image structure affects the image structure. By sharing the weights of its decoder, the generator adopts a residual learning strategy to remove the image information spanning the image noise manifold from the noisy observation.

[0072] As described above, the reg pathway 301 takes as input a noisy RGB image s, 304, which is then processed by a backbone ResNet model 303 and Unet, as described in Olaf Ronneberger et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” MICCAI, Springer, Lecture Notes in Computer Science (LNCS), Vol. 9351: 234-241, 2015. The output of the pathway 301 is a predicted residual image (noisy) G'(s) (i.e., an estimate of the noise in the image), which is then removed from the noisy input image 304 to obtain the denoised output image 305, i.e., the predicted clean RGB image.

[0073] The reg pathway 301 implicitly removes the latent clean image with operations in the hidden layers. Thus, the unsupervised AE pathway 302 acts as an autoencoder in the domain of the true residual image 306 (the noise features of the training images), v = s - y.

[0074] Once the network is trained, during inference, only the reg pathway 301 is applied to a noisy RGB input image to produce its denoised clean version. At inference, the first pathway can predict the residual noise in an input image captured by an imaging device and subtract the predicted residual noise from the input image to obtain a denoised output image.

[0075] Thus, the first pathway 301 acts as a generator based on residual learning and performs regression. During training, the generator is augmented by the second pathway 302, prompting the generator to remove the residual noise from the noisy input image that spans the target image signal-related noise manifold.

[0076] The present approach directly reconstructs the image noise. The first pathway 301 in the approach described herein does not directly output a denoised image, but is designed to predict a ground truth residual image G'(s), i.e., a first noise estimate, which is the difference between the noisy observation s and the clean (ground truth) image y.

[0077] In further embodiments, this residual learning can benefit from any conditional information related to the noise characteristics of a particular image sensor type and / or sensor c, as in Figure 3The proposed conditional reg path 301 implicitly removes the latent clean image with operations in the hidden layers. In other words, the unsupervised AE path 302 acts as a conditional autoencoder in the domain of v. The input to this path is v concatenated with y and c. In this way, by explicitly giving y as an additional input, the task of the AE path 302 is not to learn the underlying structure of a large variety of complex image patterns, but to learn how the presence of structural noise affects the image structure. By sharing the weights of its decoder, the generator employs a residual learning strategy to remove the image information across the image noise manifold from the noisy observations.

[0078] In the case of image denoising in the RGB domain, the image is based on a 3- channel tensor. On the other hand, in the RAW domain, each pixel in a conventional image sensor (linear Bayer sensor reading) is covered by a single red, green or blue color filter arranged in a Bayer array, like R-G-G-B. Again, the information represented by c is different. In the case where the noise model of the image sensor of the imaging device is known, c can contain two noise parameters σ r and σ s (same for each pixel). In the case of more than one imaging device sensor, c can also contain one heat vector per pixel to define the imaging device ID (e.g. camera ID) used to take each picture.

[0079] Thus, the model can be used to form an estimate of the noise pattern specific to each of a plurality of types of image sensors. An apparatus implementing the model can be configured to receive an indication of the particular image sensor type of the image sensor that captured the input image, and the apparatus can provide the indication as input to the model. The model can then generate an estimate of the noise pattern in accordance with the indication. The model can also be provided with an indication of the particular sensor type used to generate the input image, or information indicative of the noise behavior of the image sensor as input. This can help the model to form an estimate of the noise pattern specific to the particular type of image sensor.

[0080] Figure 5 An example of the use of more than one AE path 501, 502 in the generator is schematically illustrated, the generator can be used in the case of multi-source image noise. The Reg paths are illustrated at 503 and 504. Thus, the generator of the image processor can comprise at least one further path having the same structure as the main AE path. In this case, the AE paths 501, 502 can each be responsible for removing noise information from a particular source of noise.

[0081] Figure 6One example of a discriminator 600 of an image processor is schematically illustrated. The discriminator 600 accepts as input the predicted clean image s-G'(s) (or s-G'(s c)) 601, and y 602.

[0082] In the methods described herein, the content loss comprises two terms that compute the pixel-wise difference between the predicted clean image and the clean (ground truth) image. These two terms are i) the ground truth versus the generator output loss and ii) the loss of its gradient.

[0083]

[0084] The loss term for the unsupervised (AE) module is:

[0085]

[0086] where G'(s) = d(e(v y c)) is the AE pass output, f (AE) (v (n) y c) represents the (n) y c) = d(e(v y c)) is the AE pass output, f (n) ) = d (AE) (e (AE) (v (n) y c) represents the (n) y c) = d(e(v y c)) is the AE pass output, f (n) ) = d d loss due to auto-encoding in the domain of the residual image.

[0087] The latent loss term is given by:

[0088]

[0089] The feature matching loss is given by:

[0090]

[0091] The final loss function is given by:

[0092]

[0093] where l c , l cg , l π , l ae , l l , and l d are hyperparameters used to balance the loss terms.

[0094] In the examples described herein, the reg pass 301, 503, 504 and the AE pass 302, 501, 502 comprise a “fully convolutional” network.​

[0095] Figure 7 An example of a Unet architecture that can be used in the methods described herein is schematically illustrated. The Unet uses an encoder-decoder architecture with two lateral Unet-style skip connections. The encoder part of the network is generally illustrated at 701, the decoder at 702, and the skip connections at 703. These skip connections 703, which connect intermediate layers of the encoder with corresponding intermediate layers of the decoder 702, force the network to learn the residual between predicted and actual image noise corresponding features. This has the effect of faster convergence according to empirical detection. The AE pathway takes as input the real (ground truth) residual image, which is then processed by a Unet similar to the one in the reg pathway. The output is a reconstruction of the real residual image.

[0096] The encoder part of the network (generally illustrated at 701) processes the noisy RGB input with six consecutive layers. Each layer applies a strided convolution to its input using a 3x3 convolutional filter (along with a ReLU activation function and batch normalization). The strided convolution increases the number of filters (i.e. channels) by a factor of two, while also reducing the spatial image resolution by a factor of two (i.e. from H, W, C to H / 2, W / 2, C). The image is processed at multiple scales, and the network adapts to different frequency content. This produces output channels that capture features inherent in the data and relevant to the RGB image denoising task.

[0097] The decoder part of the network (generally illustrated at 702) processes the output of the encoder with five consecutive layers of transposed convolution operations using 3x3 convolutional filters (along with a ReLU activation function and batch normalization). Transposed convolution is an upsampling layer that increases the spatial resolution by a factor of 2 in each dimension (width and height), and reduces the number of filters by a factor of 2.

[0098] In Figure 7 The skip connections, schematically illustrated at 703, enable deep layers to capture more abstract representations without needing to remember all the information. Low-level representations are directly propagated to the decoder by short cuts. In the case of layers with Unet-style skip connections, the input to each of these decoder layers is a concatenation of: i) high-resolution features from the encoding part relating to the same spatial resolution; and ii) the output of the previous decoding layer (i.e. spatially un-sampled features). Subsequent convolutions learn to assemble more accurate outputs based on the concatenated input. The input to each of the remaining decoder layers is simply the output of the previous decoding layer.

[0099] Figure 8A workflow representing a method 800 for training a model to perform denoising on images is shown. At step 801, the method includes receiving a plurality of training images 307. At step 802, the method includes receiving a plurality of noise features 306. Then, for each of the plurality of training images 307, the method includes performing the following steps. At step 803, the method includes selecting one of the plurality of noise features 306 and applying that noise feature 306 to the training image 307 to form a noisy input image 304. At step 804, the method includes forming a first noise estimate in the noisy input image 304 by implementing a candidate version of the model on the noisy input image 304, and forming an estimate 305 of the respective training image 307 by subtracting the first noise estimate from the noisy input image 304. At step 805, the method includes forming a second noise estimate 308 by implementing the candidate version of the model on the respective training image 307 and the selected noise feature 306. At step 806, the method includes adapting the candidate version of the model according to (a) a difference between the respective training image 307 and the estimate 305 of the respective training image, and (b) a difference between the second noise estimate 308 and the selected noise feature 306.

[0100] Figure 9 An example of an apparatus 901 comprising an imaging device 901a that can implement the above-described method 800 is shown schematically. According to some embodiments, the imaging device 901a can be a camera. The imaging device 901a is connected to a communication network. The imaging device 901a comprises an image sensor 902. The imaging device 901a further comprises a memory 903, a processor 904, and a transceiver 905. The memory stores code in a non-transitory form that is executable by the processor 904. In some implementations, the code can comprise a data-driven model as described above. The model comprises code that is directly executable by the processor and / or parameters such as neural network weights that are not directly executable instructions but are used to configure other executable code stored in the memory 903. The transceiver 705 is capable of transmitting and receiving data over either or both of wired and wireless communication channels. For example, it can support Ethernet, IEEE 802.11B, and / or cellular protocols such as 4G or 5G.

[0101] Such an imaging device 901a typically comprises some on-board processing capability. This can be provided by the processor 904. The processor 904 can also be used for the main functionality of the imaging device 901a.

[0102] The transceiver 905 is capable of communicating with other entities 910, 911 over a network. These entities can be physically remote from the imaging device 901a. The network can be a publicly accessible network, such as the Internet. The entities 910, 911 can be based in the cloud 913. The entity 910 is a computing entity. The entity 911 is a command and control entity. These entities are logical entities. In practice, each of them can be provided by one or more physical devices, such as servers and data stores, the functionality of two or more entities can be provided by a single physical device. Each physical device implementing an entity includes a processor 904 and a memory. The device can also include a transceiver for transmitting and receiving data with the transceiver 905 of the imaging device 901a. The memory stores, in a non-transitory manner, code that is executable by the processor 904 to implement the respective entity in the manner described herein.

[0103] The command and control entity 911 can train a model. Even though the resulting model can be effectively described, this is often a computationally intensive task, so it can be efficient to perform the development of the model in the cloud 913, where a large amount of energy and computing resources can be expected to be available. It can be expected that this is more efficient than forming such a model on a typical imaging device.

[0104] In one implementation, once a model has been developed in the cloud 913, the command and control entity can automatically form a corresponding model and transmit it to the relevant imaging device 901a. In this example, denoising is performed at the imaging device 901a by the processor 904.

[0105] In another possible implementation, the image sensor 902 can capture an image, and the transceiver 905 can transmit the image data to the cloud 913 for processing. The resulting target image can then be transmitted back to the imaging device 901a, as shown in 912 in Figure 9

[0106] Thus, the method can be deployed in a variety of ways; for example, in the cloud 913, on the device, or alternatively on dedicated hardware. As mentioned above, the cloud facility can perform the training to develop new models or refine existing models. Depending on the computing power close to the data corpus, the training can be performed close to the source data or in the cloud 913, for example, using an inference engine.

[0107] ​At inference, the imaging device 901a can implement the trained model on an input image in order to form a noise estimate in the image and subtract the noise estimate from the input image to form a denoised output image. At inference, the imaging device 901a can perform a method comprising the steps of: receiving an input image (i.e. a noisy image); implementing a trained artificial intelligence model to form an estimate of a noise pattern in the input image; and forming an output image by subtracting the estimate of the noise pattern from the input image, wherein the model is used to form the estimate of the noise pattern such that the estimate of the noise pattern is representative of a noise pattern specific to a particular image sensor type.

[0108] The methods described herein have a number of advantages.

[0109] For the AE pathway 302, the reconstruction of the real image noise is an easier task compared to the reconstruction of clean images and only the structure of the residual image needs to be considered. This property makes the method an object-independent image denoiser and helps the denoiser to avoid image over-smoothing, which is very important for any image denoiser.

[0110] The unsupervised AE pathway 302 enables the utilization of all samples within the residual image domain even in the absence of corresponding noisy input samples. For example, in the case of a well-defined image noise source, a large number of different residual image realizations can be generated and used to train the pathway.

[0111] With the present method, it is easier to adapt existing trained models to new imaging device sensors (domain transfer). For this, the AE pathway 302 can be retrained while the reg pathway 301 is fine-tuned using only a small number of paired training samples obtained with the new sensor.

[0112] The drawbacks of the previous RoCGAN method are related to the use (or not) of skip connections. Without skip connections, RoCGAN performs well only in the case of object-dependent image denoising (i.e. face denoising). This limits its use as different models need to be provided for different objects, making it unsuitable for resource-limited digital devices like smartphones where run-time performance is important.

[0113] Thus, a device implementing the method can learn how to transform a noisy image by only looking at the structure of the residual image. This way, the task of image denoising is simplified. Moreover, explicitly learning a low-dimensional manifold of noise sources not only enables a better removal of the noise source from the image but also enables the combination of all noise sources in the same pipeline in the case of many different types of noise sources. The device can form an estimate of a noise pattern representative of a particular image sensor type and can effectively remove the noise even in the case of a non-Gaussian noise pattern.

[0114] The residual mapping can be optimized more easily when the original mapping is more like an identity mapping. Note that the ground truth clean image is more like a noisy image compared to the output of the reg pathway (especially when the noise level is low). Therefore, the original mapping is closer to an identity mapping than the output of the reg pathway, and the residual learning formula is more suitable for image denoising.

[0115] Figure 10 (a)-(d) show examples of denoising results and comparisons with DnCNN results, as described in Kai Zhang, Kai Zhang, Yunjin Chen, Deyu Meng, and Lei Zhang, “Beyond Gaussian denoising: Residual learning of deep CNN for image denoising,” IEEE Transactions on Image Processing, 26(7):3142-3155, 2017. Figure 10 (a) shows a noisy RGB input image, Figure 10 (b) shows a denoised output image obtained by the DnCNN method, Figure 10 (c) shows a denoised output image obtained by the method described herein, Figure 10 (d) shows a clean ground truth image. It can be seen that in this embodiment, the method described herein preserves high frequency image details.

[0116] Applicant hereby discloses all individually-indicated features described herein and any combination of two or more such features to the general knowledge of those of ordinary skill in the art, which can practice such features or combinations based on the present specification, whether or not such features or combinations address any of the problems disclosed herein, and without limitation to the scope of the claims. Applicant indicates that aspects of the present application can consist of any of such individual features or combinations of features. In light of the description provided herein, various modifications will be apparent to those of ordinary skill in the art.

Claims

1. An apparatus (901) for denoising an image, characterized in that, The apparatus (901) has a processor (904) for receiving an input image, implementing a trained artificial intelligence model to form an estimate of a noise pattern in the input image and forming an output image by subtracting the estimate of the noise pattern from the input image, the model being for forming the estimate of the noise pattern by means of a projection onto a trained noise manifold, such that the estimate of the noise pattern is representative of a noise pattern specific to a particular image sensor type; The processor (904) has a generator module, the generator comprising a first pass (301) and a second pass (302), during training of the model, the first pass (301) taking as input a noise input image (304), outputting a first noise estimate and forming an estimate (305) of a training image (307) by subtracting the first noise estimate from the noise input image (304); the second pass (302) taking as input a concatenation of the training image (307) and a noise feature (306), outputting a second noise estimate; the noise feature (306) being applied to the training image (307) to form the noise input image (304).

2. The apparatus (901) according to claim 1, characterized by The model is for forming an estimate of a noise pattern specific to each of a plurality of types of image sensor (902), and wherein the apparatus (901) comprises an imaging device (901a) having the image sensor (902) that captures the input image, and wherein the apparatus (901) is for generating the input image using the imaging device (901a) and providing an indication of the particular image sensor type as input to the model, and the model is for generating the estimate of the noise pattern in dependence on the indication; the indication of the particular image sensor type comprising an imaging device (901a) ID, the indication of the particular image sensor type being concatenated with the input image as input to the model.

3. The apparatus (901 ) according to claim 1 or 2, characterized by The apparatus (901) is configured to provide information indicative of a noise behavior of the image sensor (902) as input to the model, the information indicative of the noise behavior of the image sensor (902) comprising a noise parameter σ r and σ s , the information indicative of the noise behavior of the image sensor (902) being input to the model in series with the input image.

4. The apparatus (901 ) according to claim 1 or 2, characterized by The noise pattern is non-Gaussian.

5. A method (800) for training a model to perform noise reduction on an image, characterized in that, The method comprises: receiving (801) a plurality of training images (307); receiving (802) a plurality of noise features (306); for each of the plurality of training images (307): (i) selecting (803) one of the plurality of noise features (306) and applying the noise feature (306) to the training image (307) to form a noise input image (304); (ii) inputting the noise input image (304) into a first pass (301), processing by a candidate version of the model, forming (804) a first noise estimate in the noise input image (304) and forming an estimate (305) of the respective training image (307) by subtracting the first noise estimate from the noise input image (304); (iii) inputting the respective training image (307) and the selected noise feature (306) in series into a second pass (302) for processing by the candidate version of the model to form (805) a second noise estimate (308); each of the first pass (301) and the second pass (302) comprises an encoder-decoder network and shares weights of the decoder; and (iv) adapting (806) the candidate version of the model according to (a) a difference between the respective training image (307) and the estimate (305) of the respective training image (307) and (b) a difference between the second noise estimate (308) and the selected noise feature (306).

6. The method of claim 5, wherein, The first pass (301) and the second pass (302) are each based on a fully convolutional network.

7. The method according to claim 5 or 6, characterized in that, The second pass (302) implements an unsupervised learning method.

8. The method according to claim 5 or 6, characterized in that, The first pass (301) comprises one or more skip connections (703).

9. The method according to claim 5 or 6, characterized in that, Each of the plurality of training images (307) is a RAW image or an RGB image.

10. The method of claim 5 or 6, wherein, The model is a convolutional neural network.

11. A device (901a) for training a model to perform denoising on images, characterized in that, The device has a processor for performing the steps of: receiving (801) a plurality of training images (307); receiving (802) a plurality of noise features (306); for each of the plurality of training images (307): (i) selecting (803) one of the plurality of noise features (306) and applying the noise feature (306) to the training image (307) to form a noisy input image (304); (ii) inputting the noisy input image (304) into a first pass (301) for processing by a candidate version of the model to form (804) a first noise estimate in the noisy input image (304) and an estimate (305) of the respective training image (307) by subtracting the first noise estimate from the noisy input image (304); (iii) inputting the respective training image (307) and the selected noise feature (306) in series into a second pass (302) for processing by the candidate version of the model to form (805) a second noise estimate (308); each of the first pass (301) and the second pass (302) comprises an encoder-decoder network and shares weights of the decoder; and (iv) adapting (806) the candidate version of the model according to (a) a difference between the respective training image (307) and the estimate (305) of the respective training image (307) and (b) a difference between the second noise estimate (308) and the selected noise feature (306).

Images