Deep residual network for color filter array image denoising

Abstract

Embodiments of deep residual networks dedicated to color filter array mosaic patterns are described herein. Mosaic strided convolutional layers are introduced to match the mosaic pattern of a multispectral filter array (MSFA) or a color filter array raw image. Embodiments of data augmentation using MSFA shift and dynamic noise are applied to make the model robust to different noise levels. L is normalized by using the noise standard deviation 1 The loss function can create an implementation of the network optimization criterion. Comprehensive experiments show that the disclosed implementation of the deep residual network outperforms the prior art denoising algorithms in the field of MSFA.

Description

Background Art

[0001] A. Technical field

[0002] The present disclosure generally relates to systems and methods for image processing. More specifically, the present disclosure relates to systems and methods for multispectral image denoising.

[0003] B. Background Technology

[0004] Multispectral filter arrays (MSFA) are increasingly used in multispectral imaging. Although many previous works have investigated denoising algorithms for color filter array (CFA) based cameras, denoising of MSFA raw images has rarely been discussed.

[0005] Compared with CFA, the main challenges of denoising MSFA data include: 1) MSFA data contains more channels than CFA and no channel is dominant; and 2) designing the denoising process to be compatible with the subsequent demosaicing process is nontrivial.

[0006] Therefore, there is a need for systems and methods for MSFA or CFA image denoising to improve performance. Summary of the invention

[0007] In a first aspect, the present disclosure provides a computer-implemented method for denoising a noisy image using a deep residual network, the method comprising: using a first block including at least a mosaic strided convolution layer to generate a convolution output image from the noisy image, the mosaic strided convolution layer including multiple strided convolution channels to sample the noisy image into multiple sampled images, wherein each sampled image is filtered into a filtered image through a strided convolution, and the filtered images from multiple strided convolution channels are merged to generate a convolution output image; using one or more residual blocks in a deep residual network to generate an intermediate output image from a rectified output, each residual block including one or more convolution layers and a rectified linear unit (ReLU), the ReLU having a shortcut connection across one or more convolution layers and the ReLU; using the convolution layers in the deep residual network to transform the intermediate output image into an estimated noisy image; and subtracting the estimated noisy image from the noisy image to obtain a denoised image.

[0008] In a second aspect, the present disclosure provides a method for training a deep residual network for image denoising, the method comprising: inputting a set of training image pairs into the deep residual network, wherein the training image pairs include a noise-free image and a noise image corresponding to the noise-free image, and both the noise-free image and the noisy image have a mosaic pattern, the deep residual network includes a first block, one or more residual blocks and a final block, the first block includes a mosaic strided convolution layer; using the deep residual network, generating a denoised image for each noise image, the steps comprising: using the mosaic strided convolution layer to generate a convolution output image from the noise image, the mosaic strided convolution layer including multiple A strided convolution channel is used to sample a noisy image into multiple sampled images, wherein each sampled image is filtered into a filtered image through a strided convolution, and the filtered images from multiple strided convolution channels are merged into a convolution output image; an intermediate output image is generated from the convolution output image using one or more residual blocks in a deep residual network; the intermediate output image is transformed into an estimated noisy image using a convolution layer in the deep residual network; and the estimated noisy image is subtracted from the noisy image to obtain a denoised image; and a loss function is used to train the deep residual network, the loss function involving a loss between multiple ground truth noise-free original images and corresponding denoised images.

[0009] In a third aspect, the present disclosure provides a non-temporary computer-readable medium or medium comprising one or more instruction sequences, which, when executed by one or more processors, cause the execution of steps for denoising a noisy image, the steps comprising: sampling the noisy image into multiple sampled images in a mosaic strided convolution layer comprising multiple strided convolution filters; filtering each of the multiple sampled images into a filtered image using a corresponding strided convolution filter having a strided convolution kernel; merging all filtered images into a convolution output image; using one or more residual blocks in a deep residual network to generate an intermediate output image from the convolution output image, each residual block comprising one or more convolution layers and a rectified linear unit (ReLU), the ReLU having a shortcut connection across one or more convolution layers and a PReLU; transforming the intermediate output image into an estimated noisy image using a convolution layer; and subtracting the estimated noisy image from the noisy image to obtain a denoised image.

[0010] In a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium or medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute a method for training a deep residual network for image denoising, the method comprising: inputting a set of training image pairs into a deep residual network, wherein the training image pairs include a noise-free image and a noise image corresponding to the noise-free image, and both the noise-free image and the noisy image have a mosaic pattern, the deep residual network includes a first block, one or more residual blocks and a final block, the first block includes a mosaic strided convolution layer; using the deep residual network, generating a denoised image for each noise image, the steps of which include: using the mosaic strided convolution layer to generate a denoised image from the noise image into a convolution output image, the mosaic strided convolution layer includes multiple strided convolution channels to sample the noisy image into multiple sampled images, wherein each sampled image is filtered into a filtered image through a strided convolution, and the filtered images from the multiple strided convolution channels are merged into a convolution output image; using one or more residual blocks in a deep residual network, an intermediate output image is generated from the convolution output image; using a convolution layer in the deep residual network, the intermediate output image is transformed into an estimated noise image; and the estimated noise image is subtracted from the noise image to obtain a denoised image; and a loss function is used to train the deep residual network, the loss function involving the loss between multiple ground truth noise-free original images and the corresponding denoised images.

[0011] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements a computer-implemented method for denoising a noisy image using a deep residual network, the method comprising: using at least a first block including a mosaic strided convolution layer to generate a convolution output image from a noisy image, the mosaic strided convolution layer including multiple strided convolution channels to sample the noisy image into multiple sampled images, wherein each sampled image is filtered into a filtered image through a strided convolution, and the filtered images from multiple strided convolution channels are merged to generate a convolution output image; using one or more residual blocks in a deep residual network to generate an intermediate output image from a rectified output, each residual block including one or more convolution layers and a rectified linear unit (ReLU), the ReLU having a shortcut connection across one or more convolution layers and the ReLU; using the convolution layers in the deep residual network to transform the intermediate output image into an estimated noisy image; and subtracting the estimated noisy image from the noisy image to obtain a denoised image.

[0012] In a sixth aspect, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements a method for training a deep residual network for image denoising, the method comprising: inputting a set of training image pairs into a deep residual network, wherein the training image pairs include a noise-free image and a noise image corresponding to the noise-free image, and both the noise-free image and the noisy image have a mosaic pattern, the deep residual network includes a first block, one or more residual blocks and a final block, the first block includes a mosaic strided convolution layer; using the deep residual network, generating a denoised image for each noise image, the steps of which include: using the mosaic strided convolution layer to generate a convolution output from the noise image An image, a mosaic strided convolution layer includes multiple strided convolution channels to sample a noisy image into multiple sampled images, wherein each sampled image is filtered into a filtered image through a strided convolution, and the filtered images from the multiple strided convolution channels are merged into a convolution output image; using one or more residual blocks in a deep residual network, an intermediate output image is generated from the convolution output image; using a convolution layer in the deep residual network, the intermediate output image is transformed into an estimated noise image; and the estimated noise image is subtracted from the noise image to obtain a denoised image; and a loss function is used to train the deep residual network, the loss function involving a loss between multiple ground truth noise-free original images and corresponding denoised images. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Reference will be made to embodiments of the present invention, examples of which may be shown in the accompanying drawings. These drawings are intended to be illustrative and not restrictive. Although the present invention has been generally described in the context of these embodiments, it should be understood that the present invention is not intended to limit the scope of the present invention to these specific embodiments. The items in the drawings are not drawn to scale.

[0014] Figure 1 The process of mosaic strided convolution according to an embodiment of the present document is described.

[0015] Figure 2 An example of a mosaic strided convolution according to an embodiment of the present document is illustrated.

[0016] Figure 3 A network architecture for MSFA denoising according to an embodiment of the present document is described.

[0017] Figure 4 The process of MSFA denoising according to an embodiment of the present document is described.

[0018] Figure 5 The diagram illustrates a pipeline for MSFA separate channel denoising according to an embodiment of the present document.

[0019] Figure 6A process for MSFA separation channel denoising according to an embodiment of the present document is shown.

[0020] Figure 7 A simplified block diagram of a computing device / information processing system is depicted in accordance with an implementation of the present document. DETAILED DESCRIPTION

[0021] In the following description, for the purpose of explanation, specific details are set forth in order to provide an understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without these details. In addition, those skilled in the art will recognize that the embodiments of the present invention described below can be implemented in various ways (e.g., processes, devices, systems, equipment, or methods) on a tangible computer-readable medium.

[0022] The components or modules shown in the accompanying drawings are exemplary illustrations of embodiments of the present invention and are intended to avoid obscuring the present invention. It should also be understood that throughout this discussion, components may be described as separate functional units (which may include subunits), but those skilled in the art will recognize that various components or portions thereof may be divided into separate components, or may be integrated together (including integration into a single system or component). It should be noted that the functions or operations discussed herein may be implemented as components. Components may be implemented in software, hardware, or a combination thereof.

[0023] In addition, the connections between components or systems in the drawings are not intended to be limited to direct connections. Instead, the data between these components may be modified, reformatted, or otherwise changed by intermediate components. In addition, additional or fewer connections may be used. It should also be noted that the terms "coupled," "connected," or "communicatively coupled" are understood to include direct connections, indirect connections through one or more intermediate devices, and wireless connections.

[0024] References in this specification to "one embodiment", "preferred embodiment", "embodiment" or "multiple embodiments" indicate that the specific features, structures, characteristics or functions described in conjunction with the embodiment are included in at least one embodiment of the present invention, and may be included in more than one embodiment. In addition, the appearance of the above-mentioned phrases in various places in this specification does not necessarily all refer to the same embodiment or multiple identical embodiments.

[0025] The use of certain terms in various places in this specification is intended to be illustrative and should not be construed as limiting. A service, function, or resource is not limited to a single service, a single function, or a single resource; use of these terms may refer to a distributed or aggregated grouping of related services, functions, or resources. A "layer" may include one or more operations. The words "best," "optimized," "optimized," and the like refer to improvements to a result or process, and do not require that a specified result or process has reached an "optimal" or peak state.

[0026] The terms "include," "comprising," "including," and "comprising" should be understood as open-ended terms, and any listings thereafter are examples and are not intended to be limited to the listed items. Any headings used herein are for organizational purposes only and should not be used to limit the scope of the specification or claims. Each reference mentioned in this patent document is incorporated herein by reference in its entirety.

[0027] In addition, those skilled in the art should recognize that: (1) certain steps may be performed optionally; (2) the steps may not be limited to the specific order described herein; (3) certain steps may be performed in a different order; and (4) certain steps may be performed simultaneously.

[0028] A. Introduction

[0029] Hyperspectral and multispectral imaging have been widely used in many applications. Although conventional push-broom scanners have been the most commonly used hyperspectral imaging system, snapshot multispectral cameras have recently attracted significant popularity due to their low cost and high acquisition speed. They use multispectral filter array (MSFA) technology to capture multispectral images (MSI) in a single exposure. MSFA data benefits from a large number of computer vision applications by providing higher spectral resolution than RGB (red-green-blue) only data, such as spectral-spatial SIFT and human recognition using hand skin. Low-level imaging processing methods such as denoising and demosaicing have been studied for CFA-based imaging. However, previous work rarely considers denoising and demosaicing of MSFA raw images. When CFA with Bayer pattern was first disclosed, most demosaicing methods were proposed as a preprocessing step for denoising. Some people noticed that the demosaicing process changes the Gaussian characteristics of sensor noise and proposed switching processing commands before demosaicing to denoise the CFA raw image. Alternatively, various methods have been introduced to jointly solve demosaicing and denoising. In recent years, with the rapid development of convolutional neural networks (CNNs), some deep learning models have been proposed for denoising, demosaicing, or joint demosaicing and denoising.

[0030] However, as further detailed in this paper, most of these CFA-oriented methods are not suitable for MSFA-based multispectral cameras. To bridge this gap, certain demosaicing methods for CFA have been recently proposed. For example, in the case of a 4×4 MSFA pattern, the dominant green band is used as a guide for residual interpolation to achieve MSI demosaicing. For MSFA patterns without a dominant band, the pseudo-panchromatic image difference (PPID) method first estimates a pseudo-panchromatic image, which is used as a guide to smooth its difference with each sampling band. Recently, deep learning methods proposed by Shinoda et al. ("Deep demosaicking for multispectral filter arrays", arXiv preprint arXiv:1808.08021, 2018) and Habtegebrial et al. ("Deep convolutional networks for snapshot hyperspectral demosaking", 2019 IEEE 10th Workshop on Hyperspectral Image and Signal Processing: Evolution in Recessing) showed superior results over PPID. The inventors of the present invention previously recovered the deep panchromatic image (DPI) from the MSFA raw image using a novel DPI-Net and performed demosaicing using the DPI as a guide for residual interpolation. The results showed further improvements over the Shinoda model with higher computational efficiency. However, in the case of denoising, there are few methods designed for MSFA raw images.

[0031] In this paper, an implementation of a deep residual network is disclosed to solve the MSFA raw image denoising problem. Given the uniqueness of the MSFA mosaic pattern, an implementation of the network uses a mosaic strided convolutional layer before the layer of the residual block. In addition, an implementation using L normalized by the noise standard deviation is introduced. 1The embodiment of the loss function of the loss is disclosed to optimize the network convergence. In addition, MSFA shift and dynamic noise for noise tolerant data enhancement are disclosed. For comparison purposes, two baseline methods are implemented: a separate channel denoising process for MSFA raw images based on the existing CFA method, and a deep learning based method (DPI-Net). All three methods (the embodiment of the present disclosure and the two baselines) are evaluated using the peak signal-to-noise ratio (PSNR) metric. The comparison results show that the embodiment of the present disclosure is superior at all noise levels with significant tolerance.

[0032] B. Implementation of MSFA denoising before demosaicing

[0033] In one or more embodiments, the MSFA raw image Y may be an observed form of the multispectral image X without the potential noise as shown below:

[0034] y=Mx+n (1)

[0035] in, are vectorized versions of Y and X. C is the number of spectral channels, N is the number of pixels per channel, M is a matrix representing the sampling process used for the MSFA mosaic, and n is the noise that accounts for all errors introduced in image acquisition, assumed to be signal-independent additive white Gaussian noise (AWGN) with zero mean.

[0036] In one or more embodiments, the mosaic sampling process can also be expressed by equation (2), where z is the noise-free MSFA original image, is an image averaged from all spectral channels of X (eg, in the form of a full-color image), and m is the additive signal introduced by mosaic sampling.

[0037]

[0038] To denoise the denoised observations before demosaicing the MSFA raw data as shown in Equation (3), removing n while retaining m may be challenging because both are grayscale images (or full-color images) that are altered in a similar manner. The additive term of the image signal in .

[0039]

[0040] In one or more embodiments, the optimal denoiser can be found by solving the following optimization problem:

[0041]

[0042] Among them, ||·|| l Indicates Ll -norm, S(n), S(M), and S(x) are the parameter space sets for all noise, mosaic sampling processes, and potential multispectral images, respectively, and Mx refers to the ground truth noise-free image. When given a MSFA mosaic pattern of C channels, the set of possible M sampling processes is restricted to unique options, as shown in Equation (5), where each M i The sampling of x starts from a spectrum channel i.

[0043] S(M)={M i |i∈{1,2,…,C}} (5)

[0044] C. Implementation of Color Filter Array Denoising Method

[0045] 1. Mosaic strided convolution implementation

[0046] A unique challenge in MSFA raw image denoising is the separation of m and n in equation (4). The difference between m and n is that the variance of n is spatially uniform, while m has a mosaic pattern caused by signal variations in different spectral channels. In one or more embodiments, a mosaic strided convolution scheme is first introduced to address the differences between spectral channels. This can also partially overcome the first challenge mentioned above. For a w×h mosaic pattern, when a convolution filter is applied with a stride of w×h, the center pixels of all receptive fields are from a specific spectral channel to obtain the channel weight based on the noise.

[0047] Figure 1 A process for mosaic strided convolution according to an embodiment of the present document is described. Given a noisy color filter array image (e.g., a noisy MSFA image) having a mosaic pattern of w×h, the image is sampled (105) into C=w×h sampled images using C=w×h strided convolution channels. A mosaic strided convolution comprising C=w×h strided convolution filters is applied (110) to the sampled images, where each sampled image is centered on one of the C spectral channels. In one or more embodiments, the strided convolution channels correspond to spectral channels in the color filter array. Each sampled image is filtered by a convolution kernel with a stride of w×g. In one or more embodiments, the size of the convolution kernel can vary according to different designs. The filtered images output from all C filters are then merged (115) back into an output image of the same size as the input, when appropriate padding is included.

[0048] Figure 2 An example of a 4×4 mosaic strided convolution process is shown in FIG. It should be noted that the convolution kernel size is set to 3×3 and the MSFA image is shown in pseudo grayscale, where each grayscale level represents a pixel of a spectral channel. Figure 2As shown, an MSFA image 210 having a 4×4 mosaic pattern is sampled using 16 channels or filters to obtain 16 sampled images 220. In this embodiment, each sampled image is filtered into a filtered image 230 by a 3×3 convolution with a 4×4 stride, although the filters and / or the starting positions of the filters may be different among the 16 filters. All filtered images are then merged into a convolution output image 240 having the same size as the MSFA image 210. In one or more embodiments, the convolution output image 240 may then be processed at one or more rectified linear unit (Relu) filters to output a rectified output for further denoising processing. In one or more embodiments, the one or more ReLU filters are parameterized rectified linear unit (PReLU) filters. Although in Figure 2 In the illustrated embodiment, the convolution kernel of the filter has a size of 3×3, and thus, image 220 is a sampled image from the mosaic original input, but those skilled in the art will appreciate that the convolution kernel size may have different sizes. In one or more embodiments, the convolution kernel may be equal to or even larger than the mosaic pattern, and thus, all pixels of the original image are required for each mosaic strided convolution even if the filtered image from each channel consists of only a subset.

[0049] Although the MSFA images are used for illustration purposes, Figure 1 and Figure 2 However, those skilled in the art will appreciate that the disclosed methods and processes may also be applicable to CFA image denoising.

[0050] 2. Implementation of Color Filter Array Denoising Network

[0051] In one or more embodiments, a deep residual network is disclosed for estimating noise from a noisy color filter array raw mosaic image. The raw mosaic image can be obtained with a color filter array (e.g., RGGB filter) or MSFA, which can include more channels than traditional RGGB filters and can exceed the visible band. According to one or more embodiments of the present document, in Figure 3 The architecture of the denoising network is described in . Figure 4 References to one or more embodiments of the present invention are described. Figure 3 The denoising network shown in FIG. 1 performs the process of MSFA denoising. The network 300 has a total depth D (or the number of blocks, e.g., 7 or 9). The first block 305 includes a set of mosaic strided convolutions. The first block 305 is coupled to receive (405) an input of a noisy MSFA raw image 302 and output a convolution output 306. In one or more embodiments, the above-described mosaic strided convolutions in Section C.1 are also applicable to Figure 3For the intermediate D-2 residual blocks 310, each includes two convolutional layers 314 and 316 and a shortcut connection 313 across these two layers, where the input 312 of each residual block is added to the output 317 of the second layer 316 to obtain the residual block output 318 of each residual block. Figure 3 PReLU is shown in the residual block 310 in , but it will be appreciated by those skilled in the art that other activation functions, such as general ReLU or leaky ReLU, may also be employed in one or more embodiments of the present disclosure. Such variations would be within the scope of the present invention. One or more intermediate blocks 310 generate (410) an intermediate output 319. The last or final block 320 is a convolutional layer for transforming or reorganizing (415) all channels from the intermediate output 319 into an estimated noise image 322, which is subtracted (420) from the input noisy MSFA original image 302 to obtain a denoised output image 324. In one or more embodiments, the first block 305 may also include one or more activation functions, such as Relu or PReLU filters, which are used for further processing after the mosaic strided convolution, and thus the convolution output 306 from the first block 305 may be a rectified convolution output.

[0052] In one or more embodiments, the first block 305 includes multiple mosaic strided convolutional layers, which can be connected in series for multiple implementations. In one or more embodiments, the mosaic strided convolutional layer can be followed by one or more parameterized rectified linear unit (PReLU) filters, so that the first block has a structure of mosaic strided convolutional layer, PReLU filter, mosaic strided convolutional layer, PReLU filter, ..., mosaic strided convolutional layer, PReLU filter. The convolution output from one mosaic strided convolutional layer is rectified by a subsequent PReLU filter and then processed by the next mosaic strided convolutional layer for another implementation of sampling, strided convolution and merging, etc.

[0053] Apart from Figure 3 In addition to the novel structure shown above, according to one or more embodiments of the present document, two new enhancement methods, MSFA shift and dynamically varying noise, can be applied. When x is known, MSFA shift uses different M in equation (5). i For mosaic sampling, because each M iMove the sampling window by one pixel vertically or horizontally. In addition, in order to minimize the prediction error at all noise levels, a noise-free mosaic original image can be enhanced to include various noise levels as input. In order to improve the enhancement efficiency, the training data can be dynamically enhanced by changing the added noise in different iterations. This dynamic enhancement can be achieved to save storage space for training, thereby greatly improving the training efficiency. The MSFA shift increases the sampling in the S(M) space of equation (4), and the dynamic noise enhancement method also enhances the sampling in the S(n) space. In this way, they can alleviate the two challenges mentioned above for at least the following two reasons. First, it provides more training samples, which can improve D n In addition, the above two enhancements are independent, so it helps to separate m and n in the denoising process.

[0054] To further improve the robustness of the model to various noise levels, in one or more embodiments, a new form of loss function is used, which normalizes L by the noise standard deviation as shown below: 1 Loss (e.g., minimum absolute deviation).

[0055]

[0056] where N is the number of pixels, σ is the noise level, z is the noise-free MSFA original image as the ground truth, and is the denoised MSFA image. Using the new data augmentation and normalization loss, embodiments of the present disclosure do not require explicit input of the noise standard deviation, as in a previous CFA raw image denoising work (H. Akiyama et al., "Pseudofour-channel image denoising for noisy CFA raw data", 2015 IEEE International Conference on Image Processing, pp. 4778-4782).

[0057] 3. Separate channel denoising implementation

[0058] In the prior art CFA raw image denoising work of Akiyama (described above), a four-channel image denoising method with block artifact removal is shown to outperform the prior art methods. For comparison, in one or more embodiments of the present document, this four-channel image denoising method is generalized to MSFA applications. Figure 5 and Figure 6 The pipeline and process of MSFA separation channel denoising according to the implementation of this document are described respectively. For the MSFA pattern of C channels, as Figure 5As shown, the noisy MSFA original image 510 can be segmented into multi-channel images, each corresponding to one of the C spectral channels. Figure 5 The noisy MSFA raw image 510 is segmented (605) into C different subsets of multi-channel images 520, using a 4×4 square 512 at the head image in FIG. 1 . The sampling window 512 describes a sampling window for each subset, and the 4×4 pixels in the window correspond to pixels at fixed positions of the subset of the 16-channel image. To obtain all pixels of the subset, the sampling window scans the entire original image without overlap and without gaps between adjacent windows. For each subset, multi-channel denoising is processed (610) into a multi-channel denoised image 530, and then the denoised image 530 is merged (615) back into an instance of the denoised MSFA raw image 540 of the subset. All instances of the denoised MSFA raw image (one for each multi-channel subset) are combined (620), for example by averaging, to reduce blocking artifacts and generate a final denoised MSFA raw image 550.

[0059] D. Some experimental results

[0060] It should be noted that these experiments and results are provided by way of illustration and were performed under specific conditions using one or more specific embodiments; therefore, these experiments and their results are not intended to limit the scope of the disclosure of this patent document.

[0061] Four different multispectral datasets (including three indoor and one outdoor dataset) were used for testing in one or more experimental settings. All datasets were converted to reflectance values ​​between 0 and 1 and resampled to have λ i 4{469, 480, 489, 499, 513, 524, 537, 551, 552, 566, 580, 590, 602, 613, 621, 633} (nm). In one or more experimental settings, these pre-processed MSI images are sampled as MSFA raw images using a 4×4 mosaic pattern. In one or more experimental settings, two indoor datasets and one outdoor dataset are used to train and validate the deep learning network implementation of the present disclosure. The remaining indoor dataset is used in the test of all three sets of experiments to compare the average PSNR of all 31 images.

[0062] The first set of experiments, denoted as 1*, used the separate channel denoising method as in Section C.3. In one or more experimental settings, the non-local averaging method (A. Buades et al., "A non-local algorithm for image denoising", 2015 IEEE Computer Science Conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 60-65) was selected instead of block matching and 3D filtering (BM3D) for the multi-channel denoising step to obtain higher PSNR results.

[0063] The second set of experiments, denoted as 2*, utilized the above DPI-Net for denoising, keeping the network architecture and configurations such as kernel size and number of filters. It should be noted that in the second set of experiments, the network depth D was increased to 11. All training images were segmented into 128×128 patches without data augmentation. Static AWGN noise was added to each patch, and the σ of this noise was randomly selected from a uniform distribution between 0 and 0.2 before being used as training input. The network was trained by minimizing the mean squared error (MSE) loss between the predicted and ground truth MSFA original images using the Adam optimizer, all of which were implemented in the PyTorch platform.

[0064] For the last set of experiments, embodiments of the present disclosure were trained in three steps. First, starting from the DPI-Net model trained in Experiment 2*, the network was further trained with data augmentation, including flipping, rotation, and MSFA shift. The training data was also dynamically augmented by changing the added noise of each input image after one iteration. Second, the normalized L 1 The model from the previous step is further trained using the loss. Finally, the weights from the DPI-Net model trained in the second step are passed to the deep residual network implementation, where each group of strided convolutional filters in the mosaic strided convolutional layer within the deep residual network implementation is initialized using the weights from the first convolutional layer of the DPI-Net.

[0065] The average PSNR of the images for all three sets of experiments is shown in Table 1. The added noise level varies from 0.01 to 0.2, and the PSNR from the noisy original image is included as a reference. It shows that the deep learning baseline method implementation 2* produces significantly higher PSNR compared to 1* at all noise levels except σ=0.01. It is also demonstrated that the results of the proposed method are the best at all noise levels. It exceeds methods 1* and 2* by 9.88 (σ=0.2) and 3.79 (σ=0.01) in PSNR, respectively.

[0066] Table 1: Average PSNR of images with the best results at each noise level, indicated in bold. * Means MSFA denoising applied to the original method.

[0067]

[0068]

[0069] The PSNR results of the intermediate results from the last experiment are compared in Table 2 to show the effect of each step. As shown in the proposed implementation (1), the suggested data augmentation technique significantly improves the PSNR performance for all noise levels. In the proposed implementation (2), for lower noise levels, the normalized loss function significantly increases the PSNR while maintaining the performance for all levels, except for a drop of 0.2 for σ=0.2. Finally, the addition of mosaic strided convolutions further improves the well-trained model at all noise levels. σ=0.2

[0070] Table 2: Comparison of results from three training stages of the proposed model implementations: (1) adding data augmentation; (2) changing to normalized loss; (3) adding mosaic strided convolutions.

[0071]

[0072] E. Some conclusions

[0073] In this patent disclosure, an implementation of a deep residual network is disclosed to address the challenge of denoising MSFA raw images. Several novel steps are introduced in the proposed network, including mosaic strided convolution, data augmentation techniques such as MSFA shift and dynamic noise, and L-normalization by noise standard deviation. 1 loss. All three steps have been shown to effectively improve the performance of the network. When they are combined, embodiments of the disclosed method have shown significant improvements over other prior art methods based on CFA raw denoising or DPI-Net used in MSFA demosaicing.

[0074] Those skilled in the art will recognize that the embodiments herein can be easily modified or adapted for use in CFA raw image denoising, and for joint denoising and demosaicing of MSFA raw images.

[0075] System Implementation

[0076] In embodiments, aspects of the present patent document may relate to, may include, or may be implemented on one or more information processing systems / computing systems. A computing system may include any means or combination of means operable to calculate, compute, determine, classify, process, transmit, receive, retrieve, initiate, route, exchange, store, display, communicate, visualize, detect, record, reproduce, process, or utilize any form of information, intelligence, or data. For example, a computing system may be or may include a personal computer (e.g., a laptop), a tablet, a tablet phone, a personal digital assistant (PDA), a smart phone, a smart watch, a smart package, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A computing system may include random access memory (RAM), one or more processing resources (e.g., a central processing unit (CPU) or hardware or software control logic), ROM, and / or other types of memory. Additional components of a computing system may include one or more disk drives, one or more network ports for communicating with external devices, and various input and output (I / O) devices (e.g., a keyboard, a mouse, a touch screen, and / or a video display). The computing system may also include one or more buses operable to transmit communications between the various hardware components.

[0077] Figure 7 A simplified block diagram of a computing device / information processing system (or computing system) according to an embodiment of the present disclosure is depicted. It should be understood that the computing system may be configured differently and include different components, including, for example, Figure 7 Although fewer or more components may be shown in , it should be understood that the functionality shown for system 700 is operable to support various implementations of computing systems.

[0078] like Figure 7 As shown, the computing system 700 includes one or more central processing units (CPUs) 701, which provide computing resources and control the computer. The CPU 701 may be implemented with a microprocessor, etc., and may also include one or more graphics processing units (GPUs) 719 and / or floating point coprocessors for mathematical calculations. The system 700 may also include a system memory 702, which may be in the form of a random access memory (RAM), a read-only memory (ROM), or both.

[0079] like Figure 7As shown, multiple controllers and peripherals may also be provided. Input controller 703 represents an interface to various input devices 704, such as a keyboard, mouse, touch screen, and / or stylus. Computing system 700 may also include a storage controller 707 for docking with one or more storage devices 708, each of which includes a storage medium (such as a tape or disk) or an optical medium (which can be used to record a program of instructions for operating systems, utilities, and applications, which may include implementations of programs that implement various aspects of the present invention). Storage device 708 may also be used to store processed data or data to be processed according to the present invention. System 700 may also include a display controller 709 for providing an interface for display device 711, which may be a cathode ray tube (CRT), a thin film transistor (TFT) display, an organic light emitting diode, an electroluminescent panel, a plasma panel, or other types of displays. Computing system 700 may also include one or more peripheral controllers or interfaces 705 for one or more peripheral devices 706. Examples of peripheral devices may include one or more printers, scanners, input devices, output devices, sensors, and the like. The communication controller 714 can be interfaced with one or more communication devices 715, which enables the system 700 to connect to remote devices via any of a variety of networks (including the Internet, cloud resources (such as Ethernet cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) cloud, etc.), local area network (LAN), wide area network (WAN), storage area network (SAN)), or via any suitable electromagnetic carrier signal (including infrared signals).

[0080] In the system shown, all major system components may be connected to bus 716, which may represent more than one physical bus. However, various system components may or may not be physically close to each other. For example, input data and / or output data may be remotely transmitted from one physical location to another physical location. In addition, programs that implement various aspects of the present invention may be accessed from a remote location (e.g., a server) via a network. Such data and / or programs may be transmitted via any of a variety of machine-readable media, including but not limited to: magnetic media such as hard disks, floppy disks, and tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices that are specifically configured to store or store and execute program code, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices.

[0081] Aspects of the present invention may utilize instructions for one or more processors or processing units to perform steps encoded on one or more non-transient computer-readable media. It should be noted that one or more non-transient computer-readable media should include volatile memory and non-volatile memory. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. The functions of hardware implementations can be implemented using ASICs, programmable arrays, digital signal processing circuits, etc. Therefore, the term "means" in any claim is intended to cover both software implementations and hardware implementations. Similarly, the term "computer-readable medium or medium" as used herein includes software and / or hardware or a combination thereof having an instruction program implemented thereon. With these conceived alternative implementations, it should be understood that the accompanying drawings and the accompanying description provide those skilled in the art with functional information required to write program code (i.e., software) and / or manufacture circuits (i.e., hardware) to perform the desired processing.

[0082] It should be noted that embodiments of the present invention may also relate to computer products having non-transitory tangible computer-readable media having computer codes thereon for performing various computer-implemented operations. The media and computer codes may be specially designed and constructed for the purposes of the present invention, or they may be known or available to those skilled in the relevant art. Examples of tangible computer-readable media include, but are not limited to: magnetic media such as hard disks, floppy disks, and tapes; optical media such as CD-ROMs and holographic devices; magneto-optical media; and hardware devices specially configured to store or store and execute program codes, such as application specific integrated circuits (ASICs), programmable logic devices (PLDs), flash memory devices, and ROM and RAM devices. Examples of computer codes include machine code (e.g., code generated by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. Embodiments of the present invention may be implemented in whole or in part as machine executable instructions in program modules that may be executed by a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules may be physically located in local, remote, or both settings.

[0083] Those skilled in the art will recognize that neither the computing system nor the programming language is important to the practice of the present invention.Those skilled in the art will also recognize that a number of the above elements may be physically and / or functionally divided into submodules or combined together.

[0084] Those skilled in the art will appreciate that the foregoing examples and implementations are exemplary and do not limit the scope of the present disclosure. It is intended to illustrate that all, permutations, enhancements, equivalents, combinations or improvements of the present invention that will be apparent to those skilled in the art after reading this specification and studying the drawings are included within the true spirit and scope of the present disclosure. It should also be noted that the elements of any claim may be arranged differently, including having multiple dependencies, configurations and combinations.

Claims

1. A computer-implemented method for denoising a noisy image using a deep residual network, the method comprising: generating a convolution output image from the noisy image using a first block comprising at least a mosaic strided convolution layer, the mosaic strided convolution layer comprising a plurality of strided convolution channels to sample the noisy image into a plurality of sampled images, wherein each sampled image is filtered into a filtered image by a strided convolution, and the filtered images from the plurality of strided convolution channels are merged to generate the convolution output image; Generate an intermediate output image from the convolution output image using one or more residual blocks in the deep residual network, each residual block comprising one or more convolution layers and a rectified linear unit (ReLU), wherein the ReLU has a shortcut connection across the one or more convolution layers and the ReLU; transforming the intermediate output image into an estimated noise image using a convolutional layer in the deep residual network; and The estimated noise image is subtracted from the noise image to obtain a denoised image.

2. The computer-implemented method of claim 1, wherein: The deep residual network is pre-trained using a loss function that converts the L between multiple ground truth noise-free original images and the corresponding denoised images by the noise standard deviation. 1 Loss normalization.

3. The computer-implemented method of claim 1, wherein: The deep residual network is pre-trained using training data that is dynamically augmented by adding random noise at different training iterations.

4. The computer-implemented method of claim 1, wherein: The deep residual network is pre-trained by shifting the mosaic sampling window for training images by one pixel vertically or horizontally to increase the mosaic sampling space.

5. The computer-implemented method of claim 1, wherein: The noise image has a w×h mosaic pattern with C=w×h spectral channels, and the mosaic strided convolution layer has C=w×h strided convolution channels, where w and h are integers.

6. The computer-implemented method of claim 5, wherein: Each strided convolution channel has a stride of size w×h and a convolution centered on one of the C spectral channels.

7. The computer-implemented method of claim 1, wherein: The plurality of sample images have the same size as the noise image.

8. The computer-implemented method of claim 1, wherein: The first block also includes one or more parameterized rectified linear unit (PReLU) filters after the mosaic strided convolution layer, and the filtered images from the multiple strided convolution channels are merged and then processed by the one or more PReLU filters, so that the convolution output from the first block is a rectified convolution output.

9. The computer-implemented method of claim 1, wherein: The first block includes multiple mosaic strided convolutional layers connected in series, multiple implementations of upsampling, strided convolution and merging, each mosaic strided convolutional layer is followed by one or more parameterized rectified linear unit (PReLU) filters.

10. A method for training a deep residual network for image denoising, the method comprising: Inputting a set of training image pairs into a deep residual network, wherein the training image pairs include a noise-free image and a noise image corresponding to the noise-free image, and both the noise-free image and the noise image have a mosaic pattern, the deep residual network including a first block, one or more residual blocks, and a final block, the first block including a mosaic strided convolutional layer; Using the deep residual network, generating a denoised image for each noisy image, the steps include: Using the mosaic strided convolution layer, generating a convolution output image from the noisy image, the mosaic strided convolution layer comprising a plurality of strided convolution channels to sample the noisy image into a plurality of sampled images, wherein each sampled image is filtered into a filtered image by strided convolution, and the filtered images from the plurality of strided convolution channels are merged into the convolution output image; generating an intermediate output image from the convolution output image using one or more residual blocks in the deep residual network; transforming the intermediate output image into an estimated noise image using a convolutional layer in the deep residual network; and subtracting the estimated noise image from the noise image to obtain a denoised image; and The deep residual network is trained using a loss function involving a loss between a plurality of ground truth noise-free original images and corresponding denoised images.

11. The method for training a deep residual network for image denoising according to claim 10, wherein: The loss between multiple ground truth noise-free original images and the corresponding denoised images is L 1 loss, the loss function converts the L 1 Loss normalization.

12. The method for training a deep residual network for image denoising according to claim 10, wherein: The noisy images of the training image pair set are dynamically enhanced from the corresponding noise-free images by adding noise to the corresponding noise-free images in different training iterations.

13. The method for training a deep residual network for image denoising according to claim 10, wherein: The noisy image of the training image pair set is obtained by vertically or horizontally shifting a mosaic sampling window used for mosaic sampling by one pixel to increase the mosaic sampling space.

14. The method for training a deep residual network for image denoising according to claim 10, wherein: The noise image has a w×h mosaic pattern, and the mosaic strided convolution layer has C=w×h strided convolution channels, where w and h are integers, and each strided convolution channel has a convolution and a stride of size w×h.

15. A non-transitory computer readable medium comprising one or more sequences of instructions, which, when executed by one or more processors, cause steps for denoising a noisy image to be performed, the steps comprising: In a mosaic strided convolution layer including a plurality of strided convolution filters, the noisy image is sampled into a plurality of sampled images using a plurality of strided convolution channels; filtering each of the plurality of sampled images into a filtered image using a corresponding strided convolution filter having a strided convolution kernel; Combine all filtered images into the convolution output image; Generate an intermediate output image from the convolution output image using one or more residual blocks in a deep residual network, each residual block comprising one or more convolutional layers and a rectified linear unit (ReLU), wherein the ReLU has a shortcut connection across the one or more convolutional layers and the PReLU; transforming the intermediate output image into an estimated noise image using a convolutional layer; as well as The estimated noise image is subtracted from the noise image to obtain a denoised image.

16. The non-transitory computer readable medium of claim 15, wherein: The noise image has a w×h mosaic pattern, and the mosaic strided convolution layer has C=w×h strided convolution channels, where w and h are integers.

17. The non-transitory computer readable medium of claim 16, wherein: The strided convolution filter has a stride of size w×h.

18. The non-transitory computer readable medium of claim 15, wherein: The plurality of sample images have the same size as the noise image.

19. The non-transitory computer readable medium of claim 15, wherein: The noisy image is captured using a multi-spectral filter array (MSFA) camera or a color filter array (CFA) camera.

20. The non-transitory computer readable medium of claim 19, wherein: Each of the strided convolution filters is centered on one of the spectral channels of the noise image.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 10-14.

22. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 10 to 14.

Images