Method and device for denoising an extremely low-light raw image

Abstract

The method and device for denoising of extremely low light original image can improve the generalization ability of the joint denoising and demosaicing method for extremely weak light image, greatly reduce the number of parameters without losing performance, and have better robustness. The method comprises the following steps: (1) inputting an extremely low light original image into a system; (2) image reconstruction, reconstructing a single-channel original image into an RGBG four-channel image with a resolution of 1 / 4 of the original image; (3) black level normalization; (4) enhancing the signal at a specified magnification ratio; (5) inputting an SUnet++ neural network; (6) outputting an RGB*4 twelve-channel image with the same resolution as the image input into the neural network; (7) reconstructing the output image into an RGB three-channel image with the same resolution as the original system input image; and (8) outputting an RGB image with normal brightness.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more particularly to a method and apparatus for denoising raw images in extremely low light conditions. Background Technology

[0002] With the rapid development of shooting, storage, and display devices, people have increasingly higher demands for image quality. Denoising research based on RGB images is gradually failing to meet the needs of the real world, making RAW (raw) images, as the original data, clearly more valuable for research. As a universal format in the real world, the convenience of RGB images in storage and display is self-evident. Therefore, studying the mapping relationship between noisy RAW images and clean RGB images is a scientific and effective method to break through this bottleneck.

[0003] Traditionally, denoising and de-mosaicing processes are often performed separately and sequentially within the image processing unit (ISP). After acquiring raw data from the imaging sensor, the conventional image processing pipeline employs a series of modules, such as white balance, de-mosaicing, denoising, sharpening, color space conversion, and gamma correction. However, the de-mosaicing process itself introduces errors, complicating the denoising process. Furthermore, because the noise in RAW images better conforms to commonly used statistical models, significant progress has been made in joint denoising and de-mosaicing based on RAW images using various techniques.

[0004] Under certain extreme conditions, these effects are limited. Recently, some methods have made groundbreaking progress in the joint denoising and desacrifice of RAW images in extremely low-light environments. These methods are primarily data-driven learning approaches, where deep convolutional neural networks are trained on paired datasets consisting of noisy RAW images and corresponding clean RGB images. However, these studies still have significant limitations in terms of color accuracy, detail, and generalization ability.

[0005] Most of the network models used in these studies are relatively simple and commonly used, with UNet currently showing the best performance. UNet has wide applications in image segmentation, natural language processing, and image processing. However, it also has some of the aforementioned drawbacks, leading many researchers to attempt improvements. Oktay O et al. proposed a plug-and-play Attention Gate module and introduced it into UNet. By selectively learning related regions in the input image and suppressing the salience of irrelevant regions, they obtained more accurate image segmentation results. Zhou Z et al. redesigned the Skip Connection to aggregate features at different semantic scales on the decoder subnetwork. By effectively integrating U-Nets of different depths, they significantly improved the generalization ability of UNet (UNet++).

[0006] While these improvements enhanced network performance, they also made the network more complex, introducing a large number of additional parameters.

[0007] Despite rapid advancements in photographic equipment, capturing high-resolution RAW images in extremely low-light environments remains a challenging problem. Existing methods utilize neural networks to automatically learn the mapping from extremely low-light-noise RAW images to long-exposure RGB images, performing joint denoising and de-mosaicing on extremely low-light images, but their performance on other datasets is very limited. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, the technical problem to be solved by the present invention is to provide a denoising method for extremely low-light raw images, which can improve the generalization ability of the joint denoising and demosaic methods for extremely low-light images, greatly reduce the number of parameters without sacrificing performance, and has better robustness.

[0009] The technical solution of this invention is: a method for denoising extremely low-light raw images, comprising the following steps:

[0010] (1) Input a raw image with extremely low light into the system;

[0011] (2) Image reconstruction: The single-channel original image is reconstructed into an RGBG four-channel image with a resolution of 1 / 4 of the original image.

[0012] (3) Black level normalization;

[0013] (4) Amplify the signal by a specified amplification ratio;

[0014] (5) Input the SUnet++ neural network;

[0015] (6) Output a twelve-channel RGB*4 image with the same resolution as the input neural network image;

[0016] (7) Reconstruct the output image into an RGB three-channel image with the same resolution as the original system input image;

[0017] (8) Output an RGB image with normal brightness.

[0018] This invention introduces separable convolution operations on the basis of Unet++, and proposes a new Unet++ network model. By replacing some conventional convolution operations with multiple separable convolution operations, it not only alleviates the problem of too many parameters in Unet++, but also enhances performance. Therefore, it can improve the generalization ability of the joint denoising and demosaic methods for extremely low light images, and greatly reduces the number of parameters without sacrificing performance, resulting in better robustness.

[0019] A noise reduction device for extremely low-light raw images is also provided, which includes:

[0020] The input module is configured to input a raw image with extremely low light into the system.

[0021] The first reconstruction module is configured to reconstruct the single-channel original image into an RGBG four-channel image with a resolution of 1 / 4 of the original image.

[0022] The normalization module is configured for black level normalization.

[0023] An amplification module, configured to enhance a signal by a specified amplification ratio;

[0024] The network input module is configured to input the SUnet++ neural network.

[0025] The network output module is configured to output a twelve-channel RGB*4 image with the same resolution as the image input to the neural network.

[0026] The second reconstruction module is configured to reconstruct the output image into an RGB three-channel image with the same resolution as the original system input image.

[0027] The output module is configured to output a normally lit RGB image. Attached Figure Description

[0028] Figure 1 This is a flowchart of the method for denoising extremely low-light raw images according to the present invention.

[0029] Figure 2 This is a topology diagram of the SUnet++ network structure of the present invention.

[0030] Figure 3 This is a flowchart of the γ-α combination operation of the present invention.

[0031] Figure 4 This is a flowchart of the α-β combination operation of the present invention.

[0032] Figure 5 The flowchart of the γ-α' combination operation of the SID algorithm is shown below.

[0033] Figure 6 The variation in the number of parameters during the α operation is shown using input channel 4 as an example.

[0034] Figure 7 Introduction to the dataset content.

[0035] Figure 8 Subjective comparison of joint denoising and de-mosaic results for extremely low-light images. Detailed Implementation

[0036] like Figure 1 As shown, this method for denoising extremely low-light original images includes the following steps:

[0037] (1) Input a raw image with extremely low light into the system;

[0038] (2) Image reconstruction: The single-channel original image is reconstructed into an RGBG four-channel image with a resolution of 1 / 4 of the original image.

[0039] (3) Black level normalization;

[0040] (4) Amplify the signal by a specified amplification ratio;

[0041] (5) Input the SUnet++ neural network;

[0042] (6) Output a twelve-channel RGB*4 image with the same resolution as the input neural network image;

[0043] (7) Reconstruct the output image into an RGB three-channel image with the same resolution as the original system input image;

[0044] (8) Output an RGB image with normal brightness.

[0045] This invention introduces separable convolution operations on the basis of Unet++, and proposes a new Unet++ network model. By replacing some conventional convolution operations with multiple separable convolution operations, it not only alleviates the problem of too many parameters in Unet++, but also enhances performance. Therefore, it can improve the generalization ability of the joint denoising and demosaic methods for extremely low light images, and greatly reduces the number of parameters without sacrificing performance, resulting in better robustness.

[0046] Preferably, in step (1), the input image is a Bayer mode RAW image of arbitrary size with 1 channel, denoted as H*W*1.

[0047] Preferably, in step (2), it is split into a 4-channel tensor with half the spatial resolution according to the RGBG format.

[0048] Preferably, in step (4), the multiplier is specified as 100 or 300 in order to eliminate black pixels.

[0049] Preferably, in step (5), the processed data is used as the input to the SUnet++ model; in step (6), a 12-channel tensor with the same resolution as the network model input is output.

[0050] Preferably, in step (7), the results of step (6) are packaged and organized into an RGB image H*W*3 with the same input and output resolution as the original system.

[0051] Preferably, in step (5), N is located in the core triangle region in the middle of the SUnet++ model. i,j F represents the j-th node in the i-th layer. i,j N represents the result of other nodes. i,j The set of operations, where positive integers i and j satisfy i+j≤6 and take values ​​within the range (1≤i≤5, 1≤j≤5); when j=1, N i,j By N i-1,j After operations on α and γ, we obtain N; when j>1, i,j By N i-1,j and N i+1,j-1 It is obtained by operating on α and β;

[0052] Operation F i,j Represented as formula (1):

[0053]

[0054] By F i,j The obtained N i,j Represented as formula (2):

[0055]

[0056] α represents a standard unit operation, consisting of a set of ordinary convolutions and m sets of separable convolutions;

[0057] γ represents the max pooling operation; when j=1, N i-1,1 After operation F i,j Get Ni,j β represents a set of operations including deconvolution, upsampling, and fusion; node N i+1,j-1 After deconvolution and downsampling at half the original channel count, and fusion of nodes N i,j-1 Then, after a basic unit operation α, N is obtained. i,j When j>1, N i,j By N i-1,j and N i+1,j-1 It is obtained by manipulating α and β.

[0058] Preferably, in step (5), N 1,5 It is a size of The tensor, after being subjected to a regular convolution with a kernel of size 12, yields a tensor of size [missing value]. The tensor is used as the output of the network.

[0059] Preferably, in step (5), m sets of separable convolutions are used instead of one set of regular convolutions, assuming the number of input channels for the convolution operation is... Number of output channels The convolution kernel size is k*k, and after a series of operations... The number of parameters obtained for a set of regular convolution operations is The number of parameters in m separable convolution operations that replace a set of regular convolutions is: Its parameters are calculated according to formulas (3) and (4):

[0060]

[0061]

[0062] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a noise reduction device for extremely low-light raw images. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0063] The input module is configured to input a raw image with extremely low light into the system.

[0064] The first reconstruction module is configured to reconstruct the single-channel original image into an RGBG four-channel image with a resolution of 1 / 4 of the original image.

[0065] The normalization module is configured for black level normalization.

[0066] An amplification module, configured to enhance a signal by a specified amplification ratio;

[0067] The network input module is configured to input the SUnet++ neural network.

[0068] The network output module is configured to output a twelve-channel RGB*4 image with the same resolution as the image input to the neural network.

[0069] The second reconstruction module is configured to reconstruct the output image into an RGB three-channel image with the same resolution as the original system input image.

[0070] The output module is configured to output a normally lit RGB image.

[0071] The present invention will now be described in more detail.

[0072] This invention proposes a novel end-to-end learning method, the process of which is as follows: Figure 1 As shown, this method directly addresses the joint denoising and de-mosaicing problem of extremely low-noise RAW images by learning a mapping from extremely low-noise RAW images to long-exposure clean RGB images. Inputting a low-brightness, noisy real camera RAW image into the trained model outputs a high-resolution RGB image with normal brightness and no noise.

[0073] The system's input data is a Bayer RAW image of arbitrary size with 1 channel (denoted as H*W*1); it is then split into a 4-channel tensor with half the spatial resolution according to the RGBG format. Next, the data is scaled according to a specified magnification ratio (e.g., x100 or x300) to eliminate black pixels; then, the processed data is used as input to the SUnet++ model, and a 12-channel tensor with the same resolution as the network model's input is output. Finally, the result is packaged and organized into an RGB image (H*W*3) with the same input and output resolution as the original system.

[0074] Main steps:

[0075] S1 inputs an extremely low-light RAW image into the system.

[0076] S2 image reconstruction reconstructs a single-channel RAW image into a RGBG four-channel image with a resolution one-quarter that of the original image.

[0077] S3 Black Level Normalization

[0078] S4 amplifies the signal by a specified amplification factor.

[0079] S5 input SUnet++ neural network

[0080] S6 outputs a twelve-channel RGB*4 image with the same resolution as the input neural network image.

[0081] S7 reconstructs the output image into an RGB three-channel image with the same resolution as the original system input image. S8 outputs a normal-brightness RGB image.

[0082] 5.2 SUnet++ Network Structure

[0083] In step S5 above, the SUnet++ neural network method is a significant achievement of this invention, and its main steps are described below:

[0084] This invention introduces separable convolution operations on the basis of Unet++, and proposes a new Unet++ network model. By replacing some conventional convolution operations with multiple separable convolution operations, it not only alleviates the problem of too many parameters in Unet++, but also enhances performance.

[0085] The overall structure of the model is as follows Figure 2 As shown. In the central core triangle region, N i,j F represents the j-th node in the i-th layer. i,j N represents the result of other nodes. i,j The set of operations, where positive integers i and j satisfy i + j ≤ 6 and take values ​​within the range (1 ≤ i ≤ 5, 1 ≤ j ≤ 5). When j = 1, N i,j By N i-1,j After operations on α and γ, we obtain N; when j>1, i,j By N i-1,j and N i+1,j-1 It is obtained by manipulating α and β.

[0086] That is, operation F i,j Represented as:

[0087]

[0088] By F i,j The obtained N i,j Represented as:

[0089]

[0090] α represents a standard unit operation, consisting of a set of ordinary convolutions and m sets of separable convolutions; γ represents a max pooling operation. When j = 1, N i-1,1 After operation F i,jGet N i,j The situation is as follows Figure 3 As shown. β represents a set of operations including deconvolution, upsampling, and fusion. Node N i+1,j-1 After deconvolution and downsampling at half the original channel count, and fusion of nodes N i,j-1 Then, after a basic unit operation α, N is obtained. i,j When j>1, N i,j By N i-1,j and N i+1,j-1 The result obtained after operating on α and β is as follows: Figure 4 As shown.

[0091] The network input is a variable of size . Tensors, after a series of operations The obtained node N i,j It is a size of The tensor of N. Therefore, N 1,5 It is a size of The tensor, after being subjected to a regular convolution with a kernel of size 12, yields a tensor of size [missing value]. The tensor is used as the output of the network.

[0092] 5.3 Parameters of Depthwise Separable Convolution

[0093] like Figure 3 and Figure 5 As shown in the comparison, this invention uses m sets of separable convolutions instead of one set of conventional convolutions. For simplicity, this paper only describes the parameter situation in the variable region, taking a tensor with 4 input channels as an example. Figure 6 This demonstrates the change in the number of parameters after the α operation.

[0094] Assume the number of input channels for the convolution operation is Number of output channels The convolution kernel size is k*k, and after a series of operations... The number of parameters obtained for a set of regular convolution operations is In the improvement of this invention, the number of parameters replacing one set of conventional convolutions in m sets of separable convolution operations is: The formula for calculating its parameters is:

[0095]

[0096]

[0097] To verify the effectiveness of the proposed network structure, the method proposed in this invention was tested on the SID and ELD datasets, respectively. Table 1 shows the quantitative results (PSNR / SSIM) of all comparative algorithms on the SID and ELD datasets. It can be seen that the state-of-the-art methods, represented by SID, show good performance on the SID dataset, but poor performance on the ELD dataset. After introducing Unet++ into the improved SID denoising architecture, its performance on the SID dataset slightly decreased, but its performance on the ELD dataset was significantly improved. This indicates that the application of Unet++ in this environment does indeed improve the model's generalization ability. Introducing the proposed Unet++ method into the improved SID denoising architecture shows a certain improvement over the original SID method, and a significant performance improvement on the ELD dataset.

[0098] from Figure 6 The thumbnail shows that the SID method achieves good results. However, it still exhibits significant noise when focusing on extremely small details. Detailed comparison charts demonstrate that this method represents a substantial improvement.

[0099] This invention uses m sets of depthwise separable convolutions in SUnet++ to replace a single set of traditional convolutions. For simplicity, only the parameters in the variable region are considered, i.e., only the parameter variations of operations α and α' are considered. Substituting the described number of channels into the formula, the parameters using a single set of conventional convolution operations are calculated to be 4.2M, while the parameters using two sets of depthwise separable convolution operations are 0.96M, reducing the parameters to 22.9% of the original. If only one set of depthwise separable convolution operations is used, this number can even be reduced to 11.5%, while the result is similar to that of a single set of conventional convolution operations. To obtain better results, double the number of separable convolution operations is used (see Table 2).

[0100] The main steps are as follows:

[0101] S1 model training. A large amount of real-world paired training data (consisting of short-exposure extremely low-light images and long-exposure normal-brightness images) as described in 7.1 was fed into the training system. 6000 training iterations were performed according to the configuration method in 7.3.

[0102] S2 Model Testing. The real-world test data (extremely low-light RAW images) described in 7.1 was fed into the test system. Testing was conducted according to the comparison method in 7.2.

[0103] 7.1 Dataset

[0104] The proposed SUnet++ method was evaluated on the SID and ELD datasets captured by a Sony A7S2 camera. The SID dataset contains 2697 short-exposure RAW images and 231 corresponding long-exposure reference RAW images. Images with different shutter speeds (i.e., different brightness) were grouped into multiple sets of paired data, such as... Figure 5 As shown, this data was divided into a training set (containing 1865 pairs of data), a test set (containing 598 pairs of data), and a validation set (containing 234 pairs of data). The ELD dataset contains 49 short-exposure RAW images and 10 corresponding long-exposure RAW images. The ELD dataset is used only as the test set. In each pair of data, the ratio of long to short exposure times is between 100 and 300.

[0105] 7.2 Comparison Algorithm

[0106] The proposed method is compared with traditional processing methods, BM3D methods, and the SID method on the SID and ELD datasets. The joint demosaicing and denoising method based on SUnet++ in this invention is an improvement on the SID method, improving its efficiency by modifying the traditional convolutional network. This invention replaces the traditional CNN with Unet++ in the SID method and further utilizes SUnet++ as the underlying scheme by using depthwise separable convolutions. These methods are denoted as SID+Unet++ and Our SUnet++.

[0107] 7.3 Experimental Setup

[0108] This invention uses L1 loss and the Adam optimizer to train the network from scratch on the SID dataset training set. The parameter m before the separable convolution operation is set to 2, meaning two separable convolutions are used instead of one regular convolution. During training, the network input is the original short-exposure RAW image; the ground truth is the corresponding long-exposure image (a reference image processed by the RAW image processing library libraw) in sRGB space; the difference in exposure time between the original and reference images is used as a magnification factor (e.g., x100, x250, or x300). In each training iteration, a 512×512 patch is randomly cropped for training, and operations such as flipping and rotation are used to randomly augment the data. The initial learning rate is set to 0.0001, and after 2000 iterations, the learning rate is reduced to 0.00001. Training is performed for a total of 6000 iterations.

[0109] Table 1

[0110]

[0111] Table 2

[0112]

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method of denoising an extremely low light raw image, characterized in that: It comprises the following steps: (1) inputting a very low light original image into the system; (2) image reconstruction, reconstructing the single-channel original image into an RGBG four-channel image with a resolution of 1 / 4 of the original image; (3) black level normalization; (4) enhancing the signal at a specified magnification ratio; (5) inputting the SUnet++ neural network; (6) outputting an RGB*4 twelve-channel image with the same resolution as the image input into the neural network; (7) reconstructing the output image into an RGB three-channel image with the same resolution as the original system input image; (8) outputting an RGB image with normal brightness; In step (5), N is located in the core triangle region in the middle of the SUnet++ model. i,j F represents the j-th node in the i-th layer. i,j N represents the result of other nodes. i,j The set of operations, where positive integers i and j satisfy i+j≤6 and take values ​​within the range (1≤i≤5, 1≤j≤5); when j=1, N i,j By N i-1,j After operations on α and γ, we obtain N; when j>1, i,j By N i-1,j and N i+1,j-1 It is obtained by operating on α and β; Operation F i,j is expressed as Equation (1): By F i,j The resulting N i,j is expressed as formula (2): Alpha represents a standard unit operation composed of a group of ordinary convolution and m groups of separable convolution; Gamma represents the maximum pooling operation; When j = 1, N i-1,1 After operation F i,j Get N i,j ; β represents a set of operations including deconvolution, upsampling and fusion operations; node N i+1,j-1 After deconvolution with a scale of half of the original channel number, downsampling, and fusion of node N i,j-1 , and then after a basic unit operation α, N i,j ; when j > 1, N i,j is obtained from N i-1,j and N i+1,j-1 after operations α and β.

2. The method of claim 1, wherein: In the step (1), the input image is a Bayer mode RAW image with an arbitrary size and a channel number of 1, denoted as H*W*1.

3. The method of claim 2, wherein: In step (2), it is split into a 4-channel tensor of halved spatial resolution in the RGBG format 4. The method of claim 3, wherein: In the step (4), the specified multiple is 100 or 300, so as to eliminate black pixels.

5. The method of claim 4, wherein: In the step (5), the processed data is input into the SUnet++ model; in the step (6), a 12-channel tensor with the same resolution as the network model input is output 6. The method of claim 5, wherein: In the step (7), the result of step (6) is packaged and arranged into an RGB image H*W*3 with the same resolution as the original system input and output.

7. The method of claim 6, wherein: In the step (5), N 1,5 is a tensor with a size of , which is subjected to a regular convolution with a convolution kernel with a size of 12 to obtain a tensor with a size of as the output of the network.

8. The method of claim 7, wherein: In the step (5), m groups of separable convolution are used instead of one group of regular convolution, assuming that the number of input channels of the convolution operation is The number of output channels is The size of the convolution kernel is k*k, and a series of operations are performed The number of parameters of one group of regular convolution operations obtained is The number of parameters of m groups of separable convolution operations replacing one group of regular convolution is The parameters are calculated according to formulas (3) and (4):

9. Apparatus for denoising very low light raw images, which performs the method according to claim 1, characterized in that: It comprises: an input module configured to input a very low light original image into the system; a first reconstruction module configured to reconstruct a single-channel original image into an RGBG four-channel image with a resolution of 1 / 4 of the original image; a normalization module configured to perform black level normalization; an amplification module configured to enhance the signal at a specified magnification ratio; a network input module configured to input the SUnet++ neural network; a network output module configured to output an RGB*4 twelve-channel image with the same resolution as the image input into the neural network; a second reconstruction module configured to reconstruct the output image into an RGB three-channel image with the same resolution as the original system input image; an output module configured to output an RGB image with normal brightness.

Images