Joint denoising and demosaicing method for low-exposure original image under low light

Abstract

The invention discloses a combined denoising and demosaicing method for a low-exposure original image under low light, and the method is divided into two stages: in the first stage, introducing a high and low frequency fusion module based on wavelet transform, and generating relatively clean pseudo long exposure data to reduce noise; and in the second stage, through a color space attention subnet, image reconstruction is carried out by using the pseudo long exposure data generated in the first stage, the color is recovered, and texture details are enhanced. In addition, the invention also provides a wavelet loss function to balance detail recovery and noise suppression, and discusses the influence of three different training strategies on the performance of the model. Experimental results show that the method provided by the invention is superior to most advanced algorithms in the aspects of quantitative indexes, color accuracy, detail retention and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for joint denoising and demosaicing of low-exposure original images under low light conditions. Background Art

[0002] With the prevalence of smart devices and digital life, the demand for image and video quality has increased significantly, especially in scenarios such as surveillance, autonomous driving, and medical imaging that rely on precise analysis and decision-making. RGB images have become the most common image format due to their wide compatibility and ease of use. However, after processing by the image signal processor (ISP), their dynamic range and color depth are often limited. In contrast, RAW images retain the original data captured by the sensor without any compression or processing, resulting in higher dynamic range and color depth. Especially in low-light environments, RAW images can capture more detail and provide greater scope for post-processing, allowing for customized adjustments such as noise reduction, white balance, and color correction. RAW images capture original photosensitive information of red, green, and blue light through a color filter array (CFA), the most common of which is a Bayer filter array, offering significant advantages. However, because each pixel only records a single color channel, unprocessed RAW images are visually difficult to interpret and must undergo ISP processing to convert them into RGB images. Even increasing gain or ISO to extract more signal can amplify noise and pixelation, further degrading image quality.

[0003] Existing image demosaicing techniques, such as fixed interpolation algorithms like bilinear, bicubic, and cubic spline interpolation, while simple to implement, suffer from two major issues: first, they ignore the correlation between color channels and fail to fully utilize the sampled color information; second, interpolation can easily lead to color distortion or image blur in high-frequency regions. Colorimetry and color difference methods use adaptive interpolation strategies to restore color images, further improving image quality. Another example is a demosaicing algorithm based on sparse representation, which can restore images with rich detail. However, it suffers from low computational efficiency and poor real-time performance. While these traditional methods have shown some improvements, they still suffer from color distortion and detail loss. Furthermore, these traditional methods typically assume a noise-free environment. However, RAW data collected in low-light conditions already contains sensor noise, necessitating a joint denoising and demosaicing process. In recent years, deep learning-based image processing methods have achieved breakthroughs due to their flexibility and superior performance. Some deep learning methods have achieved outstanding results by combining denoising and demosaicing for RAW images. These methods rely on data-driven techniques, training deep convolutional neural networks using paired RAW noisy images and their corresponding RGB image datasets. However, denoising, color information restoration, and detail preservation remain challenging in low-light conditions.

[0004] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention

[0005] The technical problem addressed by this invention is to overcome the above technical deficiencies by providing a novel two-stage joint denoising and demosaicing image processing architecture capable of generating RAW images and corresponding RGB images that approximate long-exposure effects. This method combines wavelet transforms with a color space attention mechanism to achieve high-quality image reconstruction in low-light conditions. Furthermore, a discrete wavelet loss function is proposed. By combining the pixel-level reconstruction loss Lrec with the discrete wavelet loss Lwave, the network is trained using dual supervision in both the spatial and frequency domains, effectively improving the visual quality and perceptual quality of the output image. The motivation for this work is twofold. First, denoising is key to restoring high-quality images, especially removing noise from RAW images. Image restoration typically relies on the effective processing of both high-frequency and low-frequency information, where high-frequency information represents detail and low-frequency information is closely related to the abstract semantics and brightness of the image. The discrete wavelet transform (DWT) decomposes an image into high-frequency and low-frequency subbands. The high-frequency subbands contain directional details and are where noise is primarily concentrated, while the low-frequency subbands retain approximate image information. The DWT has directional properties and perfect reconstruction capabilities. By separating and fusing information at different frequencies, the model can better focus on low-frequency semantics and high-frequency details, thereby generating natural-looking images and avoiding image quality degradation caused by missing details or semantic information. Research has shown that the skip connections in the UNet architecture primarily introduce high-frequency features, which may cause the network to ignore the core semantic information. Our method, however, uses skip connections to transmit low-frequency semantic information to the decoder, ensuring that the core semantic information is not neglected. However, the generated images still face problems such as color distortion, especially in low-light environments, where the mutual influence between color channels significantly affects image quality. In RGB images, the colors of adjacent pixels easily interfere with each other, resulting in color aliasing, which is particularly severe in low light conditions. To address this issue, this paper introduces a mutual guidance mechanism for color channels. The generated red, green, and blue spatial attention maps are used to correct the deep feature maps generated from the RG residual group, thereby reducing color aliasing. In particular, in the Bayer array, the green channel has twice the amount of light-sensitive information as the red and blue channels. This not only reflects the human eye's high sensitivity to green, but also provides more detail and information for image reconstruction, effectively improving image accuracy and reducing noise. Therefore, by reasonably rearranging the single-channel data, the details and tones of the RGB image can be restored to the maximum extent, thereby significantly improving the overall quality of the image.

[0006] In order to solve the above problems, the technical solution of the present invention is a method for joint denoising and demosaicing of low-exposure original images under low light conditions, comprising the following steps:

[0007] Step 1: Introduce Discrete Wavelet Transform (DWT) into the Unet encoder to meet the denoising requirements of RAW images. The DWT can decompose the input image into low-frequency and high-frequency subbands: the low-frequency subbands retain the overall structural information, and the high-frequency subbands contain details in the horizontal, vertical and diagonal directions.

[0008] Step 2: After the image is decomposed, two independent encoders are constructed to process the low-frequency sub-image Flow and the high-frequency sub-image Fhigh respectively. The low-frequency feature encoder extracts structural information, and the high-frequency feature encoder extracts details and removes noise. Each encoder consists of multiple double-layer convolution blocks. Each convolution block contains a convolution layer and a maximum pooling layer to gradually capture and enhance the high-level features of the image. The process of encoding the low-frequency sub-image Flow and the high-frequency sub-image Fhigh is as follows:

[0009]

[0010] Among them E low and E high Represent low-frequency feature encoder and high-frequency feature encoder respectively;

[0011] Step 3: After the encoding stage, the low-frequency feature map F low_e and high-frequency feature map F high_e are spliced together to generate a fusion feature map F fuse In order to effectively combine high-frequency and low-frequency features, the feature fusion module uses multiple convolutional layers to fuse Processing to form a comprehensive feature representation;

[0012] Step 4: Deconvolution operation (Deconv) is used to transform the fused feature map F fuse Upsampling is performed to restore it to a higher resolution. The upsampled feature map undergoes further convolution processing to enhance the detail representation. At the same time, in order to retain more context information, skip connections are used in the decoder to directly pass the high-frequency and low-frequency features of the encoder layer to the corresponding layers of the decoder;

[0013] Step 5: The processed feature map is restored to a four-channel pseudo long exposure image through inverse discrete wavelet transform (IDWT). The entire decoding process can be expressed by the following formula:

[0014]

[0015] is the initial fused feature map Ffuse ,

[0016] Deconv i Represents the deconvolution operation of the i-th layer, Skip ii Represents the skip connection feature map from the encoder at layer i

[0017] Furthermore, it also includes a Guided Color Attention Network (GCAN), which combines the mutual guidance between color channels with the RGB space color attention mechanism. The Guided Color Attention Network includes the following processes:

[0018] Assume that R, G1 G2, and B are the feature maps of the red, green, and blue channels generated by pseudo-long exposure, respectively. The feature map of the green channel is twice that of the red and blue channels. Based on the CGB (Color Guide Block) module, a mutual guidance strategy is used to enhance the detail recovery capability of each channel. After processing by the CGB module, the generated feature maps are R′, G′, and B′. The feature maps R′, B′G′ of the red, green, and blue channels generated by the mutual guidance mechanism are used to calculate the final attention maps XR, XG, and XB through the spatial attention module. The formula is as follows:

[0019] XR,XG,XB=SA(R′+R,G′+G,B′+B).

[0020] Among them, SA represents the spatial attention module. These spatial attention maps highlight important areas of the image and suppress irrelevant areas. Then, the attention weight of each color channel is calculated in the RGB spatial dimension to generate an RGB spatial color attention map, ensuring that key color information is enhanced.

[0021] Furthermore, the red channel feature map R is generated by guiding the green channel G and the blue channel B; the blue channel feature map B is generated by guiding the red channel R and the green channel G; and the green channel feature map G is generated by guiding the red channel R′ and the blue channel B′. Through this mutual guidance strategy between colors, the detail recovery capability of each color channel is enhanced during the initial feature map generation process.

[0022] Furthermore, in the feature map calibration stage, deep feature extraction is first performed by stacking residual groups (RG). RG consists of multiple residual channel attention modules (RCAB). RCAB contains two convolutional layers for feature extraction and enhances nonlinear expression capabilities through the LeakyReLU activation function. The introduction of residual connections captures high-level image features while alleviating the gradient vanishing problem. To further improve feature expression capabilities, multiple RG modules are stacked to form deeper stacked residual groups (StackedRG). After feature extraction is completed, the color attention map in the RGB space is applied to the output processed by StackedRG through the following formula:

[0023] Feature map calibration:

[0024] FR′,FG′,FB′=Fout⊙(X R ,X G ,X B ),

[0025] Among them, X R ,X G and X B They are the red, green and blue attention maps respectively, ⊙ represents the element-wise product operation, Fout represents the feature map output by the StackedRG block, and fuses all R, G, B information in the pseudo long exposure input image;

[0026] Finally, the calibrated red, green, and blue channel feature maps FR′, FG′, and FB′ are concatenated together, and a 1×1 convolutional layer is used to reduce the feature map dimension from 3C to C. The convolutional layer and PixelShuffle upsampling operation are used to reconstruct the image to obtain the final output RGB image Frgb, which is calculated as follows:

[0027] Frgb=PixelShuffle(Conv(FR′⊕FG′⊕FB′))

[0028] Among them, ⊕ represents the concatenation operation of the feature map.

[0029] Furthermore, two loss functions are included:

[0030] The L1 loss function (L1 Loss), also known as the Mean Absolute Error (MAE), is used to measure the absolute difference between the predicted value and the true value. The L1 loss is implemented as follows:

[0031]

[0032] in, is the predicted image, x is the corresponding long-exposure true image, and N is the number of pixels in the image. By calculating the absolute difference between the predicted value and the true value, the L1 loss function can directly reflect the magnitude of the prediction error, helping to better approximate the true image during model training;

[0033] Furthermore, the loss function focuses on emphasizing the spatial domain constraints and may ignore the information from the frequency domain. Based on this limitation, the loss function based on discrete wavelet transform (DWT) is introduced;

[0034] DWT decomposition: Perform DWT decomposition on the generated image and the target image, and divide them into four sub-bands: low-frequency sub-band (LL), horizontal high-frequency sub-band (LH), vertical high-frequency sub-band (HL) and diagonal high-frequency sub-band (HH). Let the generated image be G and the target image be T. Perform wavelet transform on these two images and get their respective four sub-bands: (G LL ,G LH ,G HL ,G HH ) and (T LL ,T LH ,T HL ,T HH ),

[0035] The L1 loss is calculated on each subband as follows:

[0036]

[0037] Among them, X and Y represent the corresponding subbands of the generated image and the target image respectively,

[0038] The total wavelet loss is defined as the weighted sum of the losses of each subband:

[0039]

[0040] The final loss function is defined as the weighted sum of the above loss functions:

[0041]

[0042] Among them, α∈(0,1), is used to adjust L1Loss and

[0043] Furthermore, the sub-bands respectively contain different frequency information of the image:

[0044] LL: low-frequency components, including the general outline and structure of the image;

[0045] LH: horizontal high-frequency component, containing the horizontal edge information of the image;

[0046] HL: vertical high-frequency component, containing the vertical edge information of the image;

[0047] HH: diagonal high-frequency component, containing diagonal detail information of the image.

[0048] The advantages of the present invention compared with the existing technology are:

[0049] 1. This paper proposes an image restoration method that combines wavelet transform and color space attention network. The wavelet transform module effectively denoises the RAW image by fusing different frequency subbands of the image while preserving key semantic features. The color space attention network generates color attention maps in the red, green, and blue spaces to correct and optimize the pseudo-long exposure feature maps obtained through deep learning, thereby improving the reconstruction of color space information.

[0050] 2. In order to improve the low-light image quality, the present invention designs a joint objective function consisting of two loss functions that compensate each other in the spatial domain and the frequency domain.

[0051] 3. Experimental results demonstrate the effectiveness of our approach: the proposed network is able to generate high-quality images with high sharpness, rich colors, good contrast, and low noise in extremely low-light conditions, with clear details and natural colors. The significant improvement in relevant metrics further demonstrates the superiority of our approach.

[0052] 4. This paper analyzes the impact of different training strategies on experimental results: through comparative experiments, it is verified that the two-stage end-to-end training method is the best. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a system structure framework diagram of the present invention.

[0054] Figure 2 : is a graph of the average values of the indicators on the Sony dataset of the present invention (the best score is highlighted in red and the second best score is highlighted in blue).

[0055] Figure 3 It is a data graph of the average values of the indicators on the MCR dataset of the present invention (the best score is highlighted in red and the second best score is highlighted in blue).

[0056] Figure 4 This is a visual effect diagram of the present invention on the low-light RAW image of the Sony dataset.

[0057] Figure 5 This is a graph of the ablation study data of different sub-network combinations performed on the Sony dataset by the present invention (the best scores are highlighted in bold).

[0058] Figure 61 is a table showing the average values of indicators of different training methods of the present invention on the Sony dataset (the best scores are highlighted in bold).

[0059] Figure 7 It is a visualization result diagram of the ablation study of the present invention.

[0060] Figure 8 It is a performance evaluation diagram of different loss function weight combinations of the present invention. DETAILED DESCRIPTION

[0061] In order to make the contents of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0062] like Figure 1-8 As shown in FIG, the method for jointly denoising and demosaicing a low-exposure original image under low light conditions includes the following steps:

[0063] Step 1: Introduce the discrete wavelet transform (DWT) into the Unet encoder to meet the denoising requirements of RAW images. The DWT can decompose the input image into low-frequency and high-frequency subbands: the low-frequency subband retains the overall structural information, and the high-frequency subband contains details in the horizontal, vertical, and diagonal directions. The above multi-scale decomposition not only helps to reduce computational complexity, but also reduces the loss of low-frequency information during the denoising process, allowing high-frequency noise to be processed more finely, thereby improving the image detail recovery effect.

[0064] Step 2: After image decomposition, two independent encoders are constructed, one for processing the low-frequency sub-image Flow and the other for processing the high-frequency sub-image Fhigh. The low-frequency feature encoder extracts structural information, while the high-frequency feature encoder extracts details and removes noise. Each encoder consists of multiple two-layer convolutional blocks, each containing a convolutional layer and a maximum pooling layer to gradually capture and enhance high-level features of the image. The encoding process for the low-frequency sub-image Flow and the high-frequency sub-image Fhigh is as follows:

[0065]

[0066] Among them E low and E high Represent the low-frequency feature encoder and high-frequency feature encoder respectively.

[0067] Step 3: After the encoding stage, the low-frequency feature map F low_e and high-frequency feature map F high_e are spliced together to generate a fusion feature map F fuse In order to effectively combine high-frequency and low-frequency features, the feature fusion module uses multiple convolutional layers to fuse Processing is performed to form a comprehensive feature representation.

[0068] Step 4: Next, the fused feature map F is processed by deconvolution operation (Deconv) fuse The decoder uses skip connections to preserve more context, transferring high- and low-frequency features from the encoder layer directly to the corresponding layers in the decoder. This design helps preserve high-resolution details and ensures efficient information propagation.

[0069] Step 5: The processed feature map is restored to a four-channel pseudo long exposure image through inverse discrete wavelet transform (IDWT). The entire decoding process can be expressed by the following formula:

[0070]

[0071] is the initial fused feature map F fuse ,

[0072] Deconv i Represents the deconvolution operation of the i-th layer, Skip i Represents the skip connection feature map from the encoder at layer i

[0073] A high- and low-frequency fusion network based on discrete wavelet transforms can effectively achieve contrast enhancement and noise reduction, but after the inverse wavelet transform, detail loss and color shift may still occur. In a Bayer array, each 2x2 pixel block contains two green pixels, one red pixel, and one blue pixel. Due to the larger number of green pixels, it can capture more detailed information. However, relying solely on the green channel for guidance can lead to a decline in the reconstruction quality of the red and blue channels, thus affecting the overall image reconstruction. Furthermore, in certain image regions, the green channel does not always contain more information than other channels. Therefore, relying solely on the green channel does not fully utilize the interrelationships between color information. Previous studies have explored the effectiveness of guidance mechanisms in low-light and high-noise conditions, but still face challenges with detail loss and color inconsistency. For example, color inconsistency and blurred details may occur under extreme conditions. While progress has been made in detail enhancement and resolution improvement, there is still room for improvement in complex color shift and detail loss issues, especially in recovering color information in extremely low-light conditions. To address these issues, this application proposes a Guided Color Attention Network (GCAN), which combines mutual guidance between color channels with a color attention mechanism in the RGB space. GCAN not only captures information of different color channels more effectively, but also shows good results in significantly reducing color shift and detail loss, thereby enhancing the overall quality of image restoration.

[0074] Assume that R, G1 G2, B are the feature maps of the red, green and blue channels generated by pseudo long exposure, respectively, where the feature map of the green channel is twice that of red and blue. Based on the CGB (Color Guide Block) module, a mutual guidance strategy is used to enhance the detail recovery ability of each channel. After processing by the CGB module, the generated feature maps are R', G', B'. Among them, the red channel feature map R is guided by the green channel G and the blue channel B; the blue channel feature map B is guided by the red channel R and the green channel G; the green channel feature map G is guided by the red channel R' and the blue channel B'. Through this mutual guidance strategy between colors, the detail recovery ability of each color channel is enhanced during the preliminary feature map generation process:

[0075] R′=CGB r (R,G)+CGB r (CGB r (R,G),B)

[0076] B′=CGB b (B,G)+CGB b (CGB b (B,G),R)

[0077] G′=CGB g (G,R′)+CGB g (G,B′)

[0078] The feature maps R′, B′G′ of the red, green and blue channels generated by the mutual guidance mechanism are used to calculate the final attention map X through the spatial attention module. R ,X G ,X B The formula is as follows:

[0079] X R ,X G ,X B =SA(R′+R,G′+G,B′+B).

[0080] SA stands for spatial attention module. These spatial attention maps highlight important areas of the image and suppress irrelevant areas. Then, attention weights for each color channel are calculated in the RGB spatial dimension to generate RGB spatial color attention maps, ensuring that key color information is enhanced.

[0081] In the feature map calibration stage, deep feature extraction is first performed by stacking residual groups (RG). RG consists of multiple residual channel attention blocks (RCAB). RCAB contains two convolutional layers for feature extraction and enhances nonlinear expression capabilities through the LeakyReLU activation function. RCAB also combines the channel attention layer (CALayer) to adaptively weight features and highlight key features, thereby effectively reducing detail loss. The introduction of residual connections not only captures high-level image features but also alleviates the gradient vanishing problem. In order to further improve the feature expression capability, multiple RG modules are stacked to form deeper stacked residual groups (Stacked Residual Groups, StackedRG). This deep structure not only strengthens feature extraction, but also maintains a stable flow of gradients, reducing the risk of gradient vanishing or explosion. After feature extraction is completed, the color attention map in the RGB space is applied to the output processed by StackedRG using the following formula

[0082] Feature map calibration:

[0083] FR′,FG′,FB′=Fout⊙(X R ,X G ,X B ).

[0084] Among them, X R ,X G and XB These are the red, green, and blue attention maps, respectively. ⊙ represents the element-wise product operation, and Fout represents the feature map output by the StackedRG block, which incorporates all the R, G, and B information from the pseudo-long exposure input image. This process effectively calibrates the output feature map, optimizes key color information, and improves image reconstruction quality.

[0085] Finally, the calibrated red, green, and blue channel feature maps FR′, FG′, and FB′ are concatenated together and passed through a 1×1 convolutional layer to reduce the feature map dimension from 3C to C. Then, the image is reconstructed using the convolutional layer and PixelShuffle upsampling operation to obtain the final output RGB image Frgb, which is calculated as:

[0086] Frgb=PixelShuffle(Conv(FR′⊕FG′⊕FB′))

[0087] Where ⊕ represents the concatenation of feature maps. Through this fine-grained feature calibration and reconstruction process, GCAN effectively solves the problems of color shift and detail loss, significantly improving the effects of image denoising and demosaicing.

[0088] By combining the high- and low-frequency fusion network with the spatial color attention guidance module proposed in this paper, image contrast is significantly enhanced and noise is significantly reduced, while effectively maintaining the accuracy of image details and color. This method achieves higher-quality image reconstruction while ensuring information integrity and visual consistency across different frequency and color channels.

[0089] When performing denoising and demosaicing in low-light environments, the design of the loss function is crucial because it directly impacts addressing multiple image degradation issues, such as brightness enhancement, detail restoration, color correction, and denoising. To address these complex requirements, this application employs two loss functions during training, one in the spatial domain and one in the frequency domain, to more comprehensively supervise network optimization.

[0090] The L1 loss function (L1 Loss), also known as the Mean Absolute Error (MAE), is used to measure the absolute difference between the predicted value and the true value. The L1 loss is implemented as follows:

[0091]

[0092] in, is the predicted image, x is the corresponding long-exposure true image, and N is the number of pixels in the image. By calculating the absolute difference between the predicted value and the true value, the L1 loss function directly reflects the magnitude of the prediction error, helping to better approximate the true image during model training.

[0093] Although the loss is effective, it focuses on emphasizing spatial domain constraints and may ignore information from the frequency domain. Based on this limitation, a loss function based on discrete wavelet transform (DWT) is introduced. Through DWT, the image can be decomposed into sub-bands of different frequencies and spatial resolutions, so as to better capture the differences in frequency components between the generated image and the target image. DWT decomposition: DWT decomposition is performed on the generated image and the target image, dividing them into four sub-bands: low-frequency sub-band (LL), horizontal high-frequency sub-band (LH), vertical high-frequency sub-band (HL) and diagonal high-frequency sub-band (HH). These sub-bands contain different frequency information of the image:

[0094] LL: Low-frequency components, including the general outline and structure of the image.

[0095] LH: Horizontal high-frequency component, containing the horizontal edge information of the image.

[0096] HL: vertical high-frequency component, containing the vertical edge information of the image.

[0097] HH: diagonal high-frequency component, containing diagonal detail information of the image.

[0098] Assume that the generated image is G and the target image is T. Perform wavelet transform on these two images to obtain their respective four sub-bands: (G LL ,G LH ,G HL ,G HH ) and (T LL ,T LH ,T HL ,T HH ).

[0099] The L1 loss is calculated on each subband as follows:

[0100]

[0101] Where X and Y represent the corresponding subbands of the generated image and the target image, respectively.

[0102] The total wavelet loss is defined as the weighted sum of the losses of each subband:

[0103]

[0104] The final loss function is defined as the weighted sum of the above loss functions:

[0105]

[0106] Among them, α∈(0,1), is used to adjust L1Loss and

[0107] The impact of these two losses on the total loss. By adjusting the weight parameters of these two losses, the model's performance on different performance indicators can be flexibly balanced. In subsequent ablation experiments, the α parameter was deeply analyzed to explore its impact on the model training results.

[0108] Example 1

[0109] The Sony dataset, captured with a Sony α7S II camera in low-light conditions, contains 2,697 short-exposure images and 231 long-exposure images with varying exposure times (1 / 10, 1 / 25, and 1 / 30 seconds). The image resolution is 4280×2832, with approximately 20% of the images used for the test set. The network inputs are short-exposure raw images, and the output is sRGB images. Long-exposure images generated using the libraw library serve as ground truth (GT). During training, the input size is 512x512, and data augmentation includes random cropping, flipping, and rotation. The output is a 1024x1024 sRGB image. Misaligned images in the test set (IDs: 10034, 10045, 10172) are removed.

[0110] The MCR dataset contains 4,980 images with a resolution of 1280×1024, including 3,984 low-light RAW images, 498 monochrome images (not used), and 498 sRGB images. The images were collected from indoor and outdoor scenes with varying exposure times. Specifically, the exposure time for indoor scenes ranges from 1 / 256 to 3 / 8 seconds, while the exposure time for outdoor scenes ranges from 1 / 4096 to 1 / 32 seconds. The preprocessing procedure is similar to that of the SID dataset.

[0111] During training, the batch size was 1, and LeakyReLU (negative slope 0.2) with joint loss and Adam optimizer were used for 2000 epochs. The initial learning rate was 10−4 and then reduced to 10−5. Both training and testing were performed on an NVIDIA V100. PSNR, SSIM, and LPIPS were used as quantitative evaluation metrics.

[0112] Experiments were conducted on the Sony and MCR datasets, and the experimental results were compared with a variety of existing (SOTA) methods, including CAN, SID, DID, SGN, LLPackNet, DCE, EEMEFN, REDIIRT, MCR, JDDNet, DNF, RetinexRAWMamba, etc.

[0113] The experimental results are as follows Figure 2 and Figure 3As shown in the figure, the PSNR, SSIM, and LPIPS metrics are used to evaluate the quantitative performance of various methods in RGB image reconstruction. The proposed method outperforms other methods in both PSNR and SSIM on the Sony dataset, but underperforms some methods in LPIPS. This is because the model may oversmooth the image during training, resulting in a loss of image detail. Traditional evaluation metrics, PSNR and SSIM, primarily focus on the overall structure and pixel-level similarity of the image, while LPIPS focuses more on local image structure and differences in visual perception, more sensitively capturing missing image details. This results in LPIPS not achieving the desired results. On the MCR dataset, our proposed method ranks second in both PSNR and SSIM. The exposure time on the MCR dataset ranges from 1 / 4096 second to 1 / 32 second, and the maximum exposure time can be magnified approximately 85 to 128 times compared to the minimum exposure time. On the Sony dataset, the exposure time can reach up to 300 times. Our method performs better on the Sony dataset than on the MCR dataset, demonstrating the effectiveness of our method for image restoration in extremely dark environments.

[0114] Figure 4 Some representative results from the Sony dataset are shown to facilitate intuitive comparison of the effects of different methods. The results of SID, SUnet++, and MCR are presented side by side. The original short-exposure images come from camera sensors with limited light sensitivity. Although short-exposure images can be processed by traditional ISP processes, the images are often close to black due to the low pixel values. Only after the amplification factor is applied can visible results be obtained under adverse conditions, but the amplification factor will cause a sharp increase in noise and the image quality will be affected, such as Figure 4 As shown in the first column of . The first row of the figure shows the results with a magnification factor of 100 times. It can be seen from the figure that the images generated by each method are similar to the real images. As the magnification factor increases, noise begins to significantly affect the image quality. The second and third rows are for a magnification factor of 250 times. Compared with other methods, our method can truly restore the colors of objects in the image. For example, the color of the electric stew pot in the second row and the red area on the ground in the third row are more accurately restored. In the case of a magnification factor of 300 times in the second and third rows, our method can effectively avoid color distortion compared with other methods. Experimental results show that the proposed method performs well in restoring low-light RAW images, especially in maintaining the accuracy of image details and colors at high magnifications. Compared with the most advanced existing methods, our method can better handle noise suppression and color restoration problems in low-light environments, avoiding excessive color distortion and detail loss.

[0115] In order to evaluate the effectiveness of each part of the method proposed in this application, an ablation experiment was conducted on the Sony dataset. Figure 3As shown, a Unet network is first used to replace the wavelet-based high- and low-frequency segmentation network. The result is (PSNR 29.51 / SSIM 0.799), but it is still lower than the entire pipeline. This proves the effectiveness of using wavelet-based high- and low-frequency segmentation to process separately. Next, the mutual guided spatial color attention (GCAN) is removed to verify its effectiveness. Experimental results show that by learning from long-exposure RAW images and leveraging the mutual guidance between colors, high-quality RGB images can be generated. Without this sub-network, low-quality RAW images would interfere with the training process of subsequent images, resulting in overall performance degradation.

[0116] To better verify the effectiveness of each subnet, we Figure 7 The ablation experiment results of the method are shown in Figure 3, including GT (ground truth) images and visual RGB images of each stage.

[0117] In the proposed method, two different loss functions are combined to optimize the network model parameters. To evaluate the effectiveness of these two loss functions in the joint denoising and demosaicing tasks, detailed ablation experiments were designed. Specifically, the influence of each loss function was removed separately, while keeping the network structure unchanged, to explore the independent contribution of each loss function to the model performance. Figure 5 The necessity of using these two loss functions together is demonstrated. Quantitative evaluation results show that the model combining the two loss functions shows significant improvements in color restoration, detail preservation, and brightness enhancement, fully leveraging the advantages of each loss function.

[0118] In order to further determine the optimal weight ratio of the two loss functions, the weights are tested by adjusting the α value in the formula. The experimental results are as follows Figure 8 As shown in the figure, when α is set to 0.5, the model performance is the best, indicating that the balanced weight can effectively combine the advantages of the two loss functions, especially achieving a good balance between detail recovery and noise suppression. Therefore, α = 0.5 was finally selected as the optimal weight.

[0119] In experiments, we thoroughly compared the effects of three different training methods on the proposed network, aiming to comprehensively evaluate the potential of various strategies to improve network performance. The proposed network is divided into two closely connected stages, each with a specific task. The experiments analyzed the impact of the three training methods on network performance.

[0120] Training Method 1: A phased training strategy is employed, with different constraints introduced between the two phases. In the first phase, the network focuses on generating pseudo-long-exposure data and optimizing network performance through direct comparison with real-world long-exposure data. After completing the first phase of training, all parameters are frozen to ensure they remain unchanged during subsequent training. The second phase of training then proceeds, performing detailed comparisons of the generated images with real-world RGB images to further optimize image quality.

[0121] Training Method 2: Similar to Training Method 1, this method also uses a phased training strategy. However, the key difference is that the parameters of the first phase are not frozen. This means that the first phase remains active throughout the training process and can be dynamically adjusted according to the needs of the second phase.

[0122] Training method three: Combine the two stages into one and perform end-to-end training. By directly comparing the generated images with the real RGB images, the network can more efficiently learn the complete mapping relationship from the original input to the final output.

[0123] according to Figure 6 It can be seen that although training method 1 optimizes the training process by freezing the parameters of the first stage, this method fails to achieve optimal performance and may limit the network's comprehensive exploration of the parameter space. Although training method 2 does not freeze the parameters of the first stage, allowing it to be adjusted dynamically, it still fails to surpass the effect of end-to-end training. On the contrary, training method 3 can more effectively capture global information through end-to-end overall optimization and ultimately achieve the best performance. This shows that in the task of joint denoising and demosaicing, the importance of global consistency may exceed the optimization of local stages, especially in low-light image processing, where a single global optimization objective can more effectively balance detail recovery and noise suppression. This finding provides an important direction for subsequent research and emphasizes the potential advantages of global optimization in complex scenes.

[0124] The above description of the present invention and its embodiments is non-limiting. In short, if a person skilled in the art is inspired by the above description and designs a similar structure and embodiment to the technical solution without departing from the purpose of the present invention, they should fall within the scope of protection of the present invention.

Claims

1. A method for joint denoising and demosaicing of low-exposure raw images in low light conditions, characterized by: The following steps are involved: Step 1: Introduce Discrete Wavelet Transform (DWT) into the Unet encoder to meet the denoising requirements of RAW images. The DWT can decompose the input image into low-frequency and high-frequency subbands: the low-frequency subbands retain the overall structural information, and the high-frequency subbands contain details in the horizontal, vertical and diagonal directions. Step 2: After the image is decomposed, two independent encoders are constructed to process the low-frequency sub-image Flow and the high-frequency sub-image Fhigh respectively. The low-frequency feature encoder extracts structural information, and the high-frequency feature encoder extracts details and removes noise. Each encoder consists of multiple double-layer convolution blocks. Each convolution block contains a convolution layer and a maximum pooling layer to gradually capture and enhance the high-level features of the image. The process of encoding the low-frequency sub-image Flow and the high-frequency sub-image Fhigh is as follows: Among them E low and E high Represent low-frequency feature encoder and high-frequency feature encoder respectively; Step 3: After the encoding stage, the low-frequency feature map F low_e and high-frequency feature map F high_e are spliced together to generate a fusion feature map F fuse In order to effectively combine high-frequency and low-frequency features, the feature fusion module uses multiple convolutional layers to fuse Processing to form a comprehensive feature representation; Step 4: Deconvolution operation (Deconv) is used to transform the fused feature map F fuse Upsampling is performed to restore it to a higher resolution. The upsampled feature map undergoes further convolution processing to enhance the detail representation. At the same time, in order to retain more context information, skip connections are used in the decoder to directly pass the high-frequency and low-frequency features of the encoder layer to the corresponding layers of the decoder; Step 5: The processed feature map is restored to a four-channel pseudo long exposure image through inverse discrete wavelet transform (IDWT). The entire decoding process can be expressed by the following formula: is the initial fused feature map F fuse , Deconv i Represents the deconvolution operation of layer i, Skip i Represents the skip connection feature map from the encoder at layer i 2. The method for joint denoising and demosaicing of low-light, low-exposure original images according to claim 1, characterized in that: It also includes a Guided Color Attention Network (GCAN), which combines the mutual guidance between color channels with the RGB spatial color attention mechanism. The Guided Color Attention Network includes the following processes: Assume that R, G1G2, and B are the feature maps of the red, green, and blue channels generated by pseudo-long exposure, respectively. The feature map of the green channel is twice that of the red and blue channels. Based on the CGB (Color Guide Block) module, a mutual guidance strategy is used to enhance the detail recovery capability of each channel. After processing by the CGB module, the generated feature maps are R′, G′, and B′. The feature maps R′, B′G′ of the red, green, and blue channels generated by the mutual guidance mechanism are used to calculate the final attention maps XR, XG, and XB through the spatial attention module. The formula is as follows: XR,XG,XB=SA(R′+R,G′+G,B′+B). Among them, SA represents the spatial attention module. These spatial attention maps highlight important areas of the image and suppress irrelevant areas. Then, the attention weight of each color channel is calculated in the RGB spatial dimension to generate an RGB spatial color attention map, ensuring that key color information is enhanced.

3. The method for joint denoising and demosaicing of low-light, low-exposure original images according to claim 2, characterized in that: The red channel feature map R is generated by guiding the green channel G and the blue channel B; the blue channel feature map B is generated by guiding the red channel R and the green channel G; and the green channel feature map G is generated by guiding the red channel R′ and the blue channel B′. Through this mutual guidance strategy between colors, the detail recovery capability of each color channel is enhanced during the initial feature map generation process.

4. The method for joint denoising and demosaicing of low-light, low-exposure original images according to claim 1, characterized in that: In the feature map calibration stage, deep feature extraction is first performed by stacking residual groups (RG). RG consists of multiple residual channel attention blocks (RCAB). RCAB contains two convolutional layers for feature extraction and enhances nonlinear expression capabilities through the LeakyReLU activation function. The introduction of residual connections captures high-level image features while alleviating the gradient vanishing problem. To further improve feature expression capabilities, multiple RG modules are stacked to form deeper stacked residual groups (StackedRG). After feature extraction is completed, the color attention map in the RGB space is applied to the output processed by StackedRG through the following formula Feature map calibration: FR′,FG′,FB′=Fout⊙(X R ,X G ,X B ), Among them, X R ,X G and X B They are the red, green and blue attention maps respectively, ⊙ represents the element-wise product operation, Fout represents the feature map output by the StackedRG block, and fuses all R, G, B information in the pseudo long exposure input image; Finally, the calibrated red, green, and blue channel feature maps FR′, FG′, and FB′ are concatenated together, and a 1×1 convolutional layer is used to reduce the feature map dimension from 3C to C. The convolutional layer and PixelShuffle upsampling operation are used to reconstruct the image to obtain the final output RGB image Frgb, which is calculated as follows: Frgb=PixelShuffle(Conv(FR′⊕FG′⊕FB′)) Among them, ⊕ represents the concatenation operation of the feature map.

5. The method for joint denoising and demosaicing of low-light, low-exposure original images according to claim 1, characterized in that: Two loss functions are also included: The L1 loss function (L1Loss), also known as the Mean Absolute Error (MAE), is used to measure the absolute difference between the predicted value and the true value. The L1 loss is implemented as follows: in, is the predicted image, x is the corresponding long-exposure true image, and N is the number of pixels in the image. By calculating the absolute difference between the predicted value and the true value, the L1 loss function directly reflects the magnitude of the prediction error, helping to better approximate the true image during model training.

6. The method for joint denoising and demosaicing of low-exposure original images in low light conditions according to claim 5, characterized in that: The loss function focuses on emphasizing the spatial domain constraints and may ignore the information from the frequency domain. Based on this limitation, the loss function based on discrete wavelet transform (DWT) is introduced; DWT decomposition: Perform DWT decomposition on the generated image and the target image, and divide them into four sub-bands: low-frequency sub-band (LL), horizontal high-frequency sub-band (LH), vertical high-frequency sub-band (HL) and diagonal high-frequency sub-band (HH). Let the generated image be G and the target image be T. Perform wavelet transform on these two images and get their respective four sub-bands: (G LL ,G LH ,G HL ,G HH ) and (T LL ,T LH ,T HL ,T HH ), The L1 loss is calculated on each subband as follows: Among them, X and Y represent the corresponding subbands of the generated image and the target image respectively, The total wavelet loss is defined as the weighted sum of the losses of each subband: The final loss function is defined as the weighted sum of the above loss functions: Among them, α∈(0,1), is used to adjust L1Loss and 7. The method for joint denoising and demosaicing of low-exposure original images in low light conditions according to claim 6, characterized in that: The sub-bands contain different frequency information of the image: LL: low-frequency components, including the general outline and structure of the image; LH: horizontal high-frequency component, containing the horizontal edge information of the image; HL: vertical high-frequency component, containing the vertical edge information of the image; HH: diagonal high-frequency component, containing diagonal detail information of the image.

Images