A low-light image enhancement method and device based on RAW image

Abstract

The present invention relates to a method and device for low-light and weak-light image enhancement based on RAW images, and belongs to the field of image processing technology. This method proposes a multi-scale parallel deep neural network for low-light and weak-light image enhancement, and realizes the cascade of network branches of different depths through convolution modular design, so as to fully extract complementary features and reasonably allocate computing resources. At the same time, it deeply analyzes the data advantages of low-light and weak-light images based on RAW format, and makes full use of the spatial information and channel characteristics of high-resolution RAW images, especially the green channel, so as to achieve better detail feature extraction and hidden information mining, and realizes efficient low-light and weak-light image enhancement based on RAW images, so as to achieve a balance between image enhancement quality and model processing efficiency. The present invention effectively improves the robustness recovery capability of the low-light and weak-light image enhancement network for low-light and weak-light images with different information content, improves the quality of low-light and weak-light enhancement of real images, and ensures the authenticity of real low-light and weak-light enhanced images.

Description

Technical Field

[0001] The present invention relates to a low-light image enhancement method and device based on a RAW image, belonging to the technical field of image processing. Background Art

[0002] Low-light environment refers to an environment with low light intensity, such as the environment under city lights, moonlight, starlight at night. Images collected under low-light conditions are called low-light images. At this time, the light is insufficient, the image sensor collects fewer photons, and the image signal is weak, resulting in low-light images with strong noise and low signal-to-noise ratio, and severe loss of scene information that is difficult to recover. RAW image (RAW Image Format) is the raw data that the image sensor converts the captured light source signal into a digital signal. It is an unprocessed and uncompressed format. Compared with the commonly used JPEG format sRGB image, RAW image retains more information and has a better dynamic range. The enhancement technology for low-light images can solve a series of image quality degradation problems such as insufficient brightness, low contrast and color distortion in low-light areas, so that the image can display more visual information. It is a technology that has a significant impact on image processing and downstream vision-related tasks and industries.

[0003] Currently, mainstream image processing methods are primarily designed and processed for sRGB (standard Red Green Blue) visualization images in formats such as 8-bit quantized JPEG. These images are processed from RAW domain images using the image acquisition device's built-in ISP (Image Signal Processor) algorithm. This process further loses the limited semantic information in low-light images and sometimes even leads to permanent loss of scene information. If RAW format images are directly used as input for image processing algorithms, on the one hand, RAW images acquired from the camera sensor contain more information and higher bit values ​​than sRGB images. Extracting information directly from RAW images, which are typically 14 bits, can effectively avoid the information loss caused by quantization of sRGB images. On the other hand, RAW images almost maintain a linear correlation between image exposure and scene brightness, that is, image pixel values ​​are directly related to real-world signal strength, further avoiding the loss of limited information in low-light scenes.

[0004] Existing low-light image enhancement technologies primarily utilize deep learning architectures to transform low-light images into normal-light images. These technologies can effectively remove noise and improve the image signal-to-noise ratio (SNR). Their performance far exceeds that of traditional methods based on artificial prior models. However, due to their large parameter space, these methods inevitably lead to high memory usage and computational costs. Therefore, they are difficult to run on embedded platforms with limited computing and storage resources. Furthermore, their long inference times prevent efficient low-light image enhancement and make them unsuitable for deployment in real-world scenarios.

[0005] In practical applications, data-driven low-light image enhancement methods must, on the one hand, address efficiency requirements, such as small memory usage, fewer model parameters, and low computational cost, to meet real-time and lightweight requirements. On the other hand, performance must also address challenges related to image enhancement, such as brightness restoration, color deviation, and noise suppression. Because performance and efficiency are often conflicting in actual algorithm design, existing methods often sacrifice model speed and computational efficiency for improved performance, even forgoing processing complex, computationally intensive scenarios like 4K resolution images. This significantly reduces the practicality of these methods.

[0006] In summary, in order to achieve rapid acquisition of high-resolution true low-light image enhancement results on mobile devices and overcome the problem that existing technologies find it difficult to balance image enhancement quality and model computational efficiency, there is an urgent need for an efficient and effective low-light image enhancement method that can use a lightweight deep learning framework to solve the problems of noise, color deviation and detail restoration in 4K low-light images. Summary of the Invention

[0007] The purpose of the present invention is to address the defects of the existing technology, especially the technical problems and shortcomings such as the difficulty in balancing algorithm performance and model efficiency, the inability to be effectively deployed and applied to embedded mobile platforms with limited computing and storage resources, and the loss of information after compression processing of sRGB format images commonly used as input. The present invention creatively proposes a low-light image enhancement method and device based on RAW images.

[0008] The innovative features of the present invention include:

[0009] For the first time, this paper proposes an efficient, high-performance multi-scale parallel deep neural network for low-light image enhancement. Furthermore, through a convolutional modular design, it achieves a cascade of different deep network branches to fully extract complementary features and rationally allocate computing resources. Furthermore, by deeply analyzing the data advantages of low-light images in RAW format, it fully utilizes the spatial information and channel characteristics of high-resolution RAW images, especially the illumination-sensitive properties and channel correlation of the green channel in RAW images, to achieve better detail feature extraction and hidden information mining. This achieves efficient low-light image enhancement based on RAW images, achieving a balance between image enhancement quality and model processing efficiency.

[0010] The present invention is implemented by adopting the following technical solutions.

[0011] A low-light image enhancement method based on a RAW image comprises the following steps:

[0012] Step 1: Analyze the data characteristics of low-light RAW images and design a green channel-guided convolution branch.

[0013] Because the human eye is more sensitive to green, visible light camera sensors are designed to perceive color based on this. RAW images stored in a Bayer pattern have twice as many green pixels as red or blue pixels.

[0014] Therefore, based on the data characteristics of the green channel in low-light RAW images, which has higher brightness and perceptual intensity and more information, a green channel-guided convolution branch is designed to deeply mine useful information from the green channel and guide the feature extraction of the backbone network.

[0015] Step 2: Based on the RAW image channel characteristics and residual dense structure, a channel attention-guided residual dense module is designed.

[0016] Specifically, the attention mechanism is combined to optimize the existing residual dense structure and realize modular design, thereby achieving good hidden information mining and detail feature extraction under low computational cost and storage space constraints.

[0017] Step 3: Use the green channel to guide the convolution branch and the channel attention to guide the residual dense module to build a multi-scale parallel feature extraction backbone network.

[0018] Traditional deep network models mostly adopt a sequential serial structure, and small-scale sequential data processing can lead to significant inference latency and computational overhead. Therefore, from the perspective of RAW image utilization, a multi-scale parallel feature extraction architecture is designed, combined with a channel-attention-guided residual dense module, to downsample the input Bayer-patterned RAW image to different scales, while simultaneously extracting and learning features. This framework effectively parallelizes the green channel-guided convolutional branches and shares the richer hidden information and detailed features in the green channel, effectively achieving network acceleration and feature enhancement.

[0019] Step 4: Establish a training objective function for real image overexposure enhancement, train the parameters of the convolutional neural network, and obtain the mapping relationship between the low-light RAW image and the reference image.

[0020] Step 5: Input the low-light RAW image to be tested and the mapping relationship between the low-light RAW image and the reference image obtained in Step 4. By establishing a mapping relationship between the low-light RAW image and the normal-light reference image, the low-light RAW image is mapped into the image format required for normal-light conditions. This enables efficient, high-quality low-light image enhancement, which can be deployed on embedded mobile platforms with limited computing and storage resources for practical scenarios.

[0021] On the other hand, in order to achieve the objectives stated in the present invention, the present invention proposes a low-light image enhancement device based on RAW images, including a green channel guidance submodule, a channel attention guided residual dense submodule and a multi-scale parallel backbone network submodule.

[0022] Among them, the green channel guidance submodule is used to design the green channel guidance convolution branch, so that it can share the retained information based on the data characteristics of the real low-light images in the low-light RAW format, take advantage of the information advantages of the green channel with higher brightness and perceptual intensity, guide the backbone network to recover the hidden or lost detail features and color deviations in the low-light area.

[0023] The channel attention-guided residual dense submodule is used to design a dense residual structure based on the RAW image channel characteristics. Combined with the attention mechanism, it performs in-depth feature mining on feature maps of different scales to fully extract hidden detail information with limited computational cost.

[0024] The multi-scale parallel backbone network submodule is used to achieve cross-scale collaborative feature extraction and aggregate image context. Its multi-branch structure and progressive fusion approach effectively alleviate the learning difficulty of the model and leverage the complementarity and potential guidance between features at different scales. This submodule is trained using real paired RAW-format low-light enhancement data, resulting in a network designed for real-world low-light image enhancement on end devices.

[0025] The connection relationship between the above modules is as follows:

[0026] The output of the green channel guidance submodule and the output of the channel attention guided residual dense submodule are both connected to the input of the multi-scale parallel backbone network submodule.

[0027] Beneficial effects

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. This paper proposes an efficient multi-scale parallel network architecture for low-light RAW images, which can achieve a balance between model efficiency and performance through a multi-branch structure and progressive fusion approach, and can be directly applied and deployed on embedded mobile platforms with limited computing and storage resources.

[0030] 2. Based on the illumination sensitivity and channel correlation of the green channel of the RAW domain low-light image, the present invention designs a green channel guided convolution calculation branch, and combines the beneficial effect 1 to construct a multi-scale parallel network, which can additionally extract and share richer texture information and detail features in the green channel, help guide the low-light image to restore missing pixels, and effectively improve the low-light image enhancement network's robustness in restoring low-light images with different information contents.

[0031] 3. The present invention uses a convolutional neural network to learn the mapping relationship between low-light RAW images and normal-light reference images. Combined with beneficial effects 1 and 2, it can improve the generalization and robustness of the convolutional neural network, and can improve the quality of low-light enhancement of real images, ensuring the authenticity of real low-light enhanced images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 Schematic diagram of the high-efficiency low-light image enhancement network based on RAW images in the present invention;

[0034] Figure 3 It is a schematic diagram of the structure of the device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings.

[0036] like Figure 1 As shown, a low-light image enhancement method based on a RAW image includes the following steps:

[0037] Step 1: Analyze the data characteristics of low-light RAW images and design a green channel-guided convolution branch.

[0038] Because the human eye has varying visual sensitivities to different colors, visible light camera sensors are designed for color perception based on this. In RAW images stored in a Bayer pattern, the number of green pixels is twice that of red or blue pixels. Therefore, due to the green channel's higher brightness and perceived intensity in low-light conditions, the green channel in low-light images typically contains more information than the red or blue channels. To leverage this characteristic of RAW images, a green channel-guided convolutional branch is designed to extract illumination-sensitive features, thereby guiding the backbone network to recover lost pixel information in low-light areas.

[0039] Specifically, if Figure 2 As shown, first, in each 2×2 block of the Bayer pattern RAW image, the pixels of the two green channels at the corresponding positions are extracted to obtain the green channel, which is then sent to the green channel guided convolution branch.

[0040] Furthermore, the channel-guided convolution branch is implemented through spatially adaptive normalization (SAN). Specifically, to extract illumination information and channel hidden features, a 3×3 convolution is used to process the two green channels of the input. Then, two separate 3×3 convolutions are performed to generate element-wise parameters γ and β. Finally, they are added to the batch-normalized activation elements through multiplication and summation operations.

[0041] At this point, the construction of the green channel guided convolution branch is completed, thereby realizing the mining and extraction of hidden features of the green channel at a limited computational cost.

[0042] Step 2: Based on the RAW image channel characteristics and residual dense structure, a channel attention-guided residual dense module is designed.

[0043] Specifically, if Figure 2 As shown in Figure 1, the channel attention-guided residual dense module starts with a 3×3 convolution operation. To balance the performance and computational cost of this module, its depth is flexibly set to n, that is, each channel attention-guided residual dense module can be set to a module with n layers of ReLU activation functions and 3×3 convolution components according to actual computational requirements.

[0044] In addition, after the convolution and feature fusion operations, adding an efficient channel attention sub-module to the module can further improve the feature extraction capability with almost negligible additional computational cost.

[0045] At this point, the design of the channel-guided convolution branch is completed. The subsequent network framework uses this low-cost plug-and-play module to efficiently improve the network feature extraction capability.

[0046] Step 3: Use the green channel to guide the convolution branch and the channel attention to guide the residual dense module to build a multi-scale parallel feature extraction backbone network.

[0047] Optimally, from the perspective of RAW image utilization, a multi-scale parallel feature extraction architecture is designed, combined with the channel-attention-guided residual dense module constructed in step 2, to downsample the input Bayer pattern RAW image to different scales for simultaneous feature extraction and learning. This framework effectively parallels the green channel-guided convolution branch constructed in step 1, sharing the richer hidden information and detailed features in the green channel, effectively achieving network acceleration and feature enhancement.

[0048] Specifically, if Figure 2 As shown in the figure, the overall network first decomposes the input Bayer pattern RAW image into a four-channel image, then downsamples it (e.g., 4x, 8x, and 16x) for multi-scale feature learning in three different branches and extracts the corresponding feature maps. Since the branch that extracts low-resolution image features requires less computational cost than the branch that extracts high-resolution image features, more computation is loaded in the low-level branches in order to mine more detailed features.

[0049] Finally, in order to utilize the intrinsic correlation of features at different scales, the feature maps obtained at different scales are upsampled and then spliced ​​and fused to achieve collaborative feature extraction and contextual feature information aggregation across different image scales.

[0050] At this point, the construction of a multi-scale parallel feature extraction backbone network using RAW data has been completed. The multi-branch structure and progressive fusion scheme effectively alleviate the learning difficulty of the model, taking into account the computational efficiency and enhanced performance of the model, and making full use of the complementarity and potential guidance between features of different scales to effectively restore the detailed features and color deviations of low-light images or areas.

[0051] Step 4: Establish a training objective function for real image overexposure enhancement, train the parameters θ of the convolutional neural network, and obtain the mapping relationship F between the low-light RAW image and the reference sRGB image.

[0052] Specifically, the overall training objective function of the overexposure correction convolutional neural network is for:

[0053]

[0054] Among them, θ represents the network parameters; I out Represents the low-light image after input network enhancement, Igt represents the reference sRGB image; 0.5 is the loss function weight; represents the mean absolute error loss; represents the wavelet structural similarity loss; represents the wavelet mean square error loss.

[0055] In formula 1 Further expressed as:

[0056]

[0057] is defined in formula 1 First, for image I out and I gt Perform two-dimensional discrete wavelet transform DWT (Discrete Wavelet Transform).

[0058] Take I out For example, it is expressed as:

[0059]

[0060] in, is the downsampled image, For horizontal edge detection image, For vertical edge detection image, Detect corners in an image.

[0061] Furthermore, in order to adopt the iterative DWT algorithm, we combine Equation 2 and let Express it as:

[0062]

[0063] Here, i is an integer ranging from 0 to 2.

[0064] Similarly, I gt The iterative DWT operation is defined and executed as shown in Equations 3 and 4.

[0065] In summary, in formula 1 Expressed as:

[0066]

[0067] Where r represents the ratio of multi-frequency SSIM (structural similarity index, abbreviated as SSIM, used to measure the structural similarity between two images) loss, preferably, r = 0.4; SSIM (·) is the original SSIM loss; w represents the output of different filters, and w∈{LL,HL,LH,HH}, and the values ​​of w are substituted into Equation 5. is the downsampled image, For horizontal edge detection image, For vertical edge detection image, Detect corners in an image.

[0068] The network uses two-dimensional Haar wavelet to implement cubic wavelet transform. Expressed as:

[0069]

[0070] Among them, WT t (·) indicates that the out and I gt The t-th order scale of the wavelet coefficients computed in the image.

[0071] By optimizing the objective function formula 1 of the training, the optimized network parameters θ are obtained.

[0072] At this point, the training of the efficient low-light image enhancement network is completed, and the mapping relationship F between the low-light RAW image and the reference sRGB image is obtained.

[0073] Step 5: Input the low-light RAW image to be tested, and the mapping relationship F between the low-light RAW image and the reference sRGB image obtained in Step 4. Using the mapping relationship F between the low-light RAW image and the reference sRGB image, the low-light RAW image is mapped into a high-quality normal-light sRGB image, achieving efficient and high-quality low-light image enhancement and improving the algorithm's practicality on embedded mobile platforms with limited computing and storage resources.

[0074] Specifically, low-light RAW images are Y. They are low-light RAW images and normal-light sRGB images respectively.

[0075] Preferably, a GPU is used to complete the network training process in step 4 and the low-light image enhancement process in step 5, and the cuDNN library is used to speed up the operation of the convolutional neural network.

[0076] On the other hand, in order to achieve the purpose of the present invention, the present invention further proposes a low-light image enhancement device based on RAW images, such as Figure 3 As shown, it includes a green channel guidance submodule 10, a channel attention guidance residual dense submodule 20, and a multi-scale parallel backbone network submodule 30.

[0077] Among them, the green channel guidance submodule 10 is used to design a green channel guidance convolution branch so that it can share retained information based on the data characteristics of real low-light images in low-light RAW format, utilize the information advantages of the green channel with higher brightness and perceptual intensity, guide the backbone network to recover hidden or lost detail features and color deviations in low-light areas, etc.

[0078] The channel attention-guided residual dense submodule 20 is used to design a channel-based residual dense structure based on the RAW image, combined with the attention mechanism, to perform in-depth feature mining on feature maps of different scales, so as to fully extract hidden detail information under limited computational cost.

[0079] The multi-scale parallel backbone network submodule 30 is used to achieve cross-scale collaborative feature extraction, thereby aggregating image context information. Its multi-branch structure and progressive fusion approach effectively alleviate the learning difficulty of the model, leveraging the complementarity and potential guidance between features at different scales. This submodule is trained using real paired RAW format low-light enhancement data, and the resulting network can be used for real-world low-light image enhancement on mobile devices.

[0080] The connection relationship between the above modules is as follows:

[0081] The output end of the green channel guidance submodule 10 is connected to the input end of the multi-scale parallel backbone network submodule 30 .

[0082] The output of the channel attention-guided residual dense submodule 20 is also connected to the input of the multi-scale parallel backbone network submodule 30.

[0083] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A low-light image enhancement method based on RAW images, characterized in that: The following steps are involved: Step 1: Analyze the data characteristics of low-light RAW images and design a green channel-guided convolution branch; In each 2×2 block of the Bayer pattern RAW image, the pixels belonging to the two green channels at the corresponding position are extracted to obtain the green channel, and they are sent together to the green channel guided convolution branch; The channel-guided convolution branch is implemented by spatially adaptive normalization. For the extraction of illumination information and channel hidden features, the two green channels of the input are processed using a 3×3 convolution, and then two separate 3×3 convolutions are performed to generate element-level parameters γ and β. Finally, through multiplication and addition operations, it is added to the activation elements after batch normalization; Step 2: Based on the RAW image channel characteristics and residual dense structure, design a channel attention guided residual dense module; Combined with the attention mechanism, the existing residual dense structure is optimized to achieve modular design; The channel attention-guided residual dense module starts with a 3×3 convolution operation, and its depth is flexibly set to n, which means that each channel attention-guided residual dense module can be set as a module with n layers of ReLU activation function and 3×3 convolution combination according to actual computational requirements; Step 3: Use the green channel to guide the convolution branch and the channel attention to guide the residual dense module to build a multi-scale parallel feature extraction backbone network; From the perspective of RAW image utilization, by designing a multi-scale parallel feature extraction architecture and combining it with a channel attention-guided residual dense module, the input Bayer pattern RAW image is downsampled to different scales, and feature extraction and learning are performed simultaneously; First, the input Bayer pattern RAW image is decomposed into a four-channel image, which is then downsampled for multi-scale feature learning in three different branches and the corresponding feature maps are extracted. Finally, the feature maps obtained at different scales are upsampled and then spliced ​​and fused to achieve collaborative feature extraction and context feature information aggregation across different image scales; Step 4: Establish a training objective function for real image overexposure enhancement, train the parameters of the convolutional neural network, and obtain the mapping relationship between the low-light RAW image and the reference image; Step 5: Input the low-light RAW image to be tested and the mapping relationship between the low-light RAW image obtained in step 4 and the reference image; By constructing a mapping relationship between low-light RAW images and normal-light reference images, the low-light RAW images are mapped into images in the format required for normal-light conditions.

2. The low-light image enhancement method based on RAW image according to claim 1, characterized in that: In step 4, the overall training objective function of the overexposure correction convolutional neural network is for: Among them, θ represents the convolutional neural network parameters; I out Represents the low-light image after input network enhancement, I gt represents the reference sRGB image; 0.5 is the loss function weight; represents the mean absolute error loss; represents the wavelet structural similarity loss; represents the wavelet mean square error loss; In formula 1 Further expressed as: is defined in formula 1 First, for image I out and I gt Perform two-dimensional discrete wavelet transform DWT; For I out , expressed as: in, is the downsampled image, For horizontal edge detection image, For vertical edge detection image, Detect images for corners; To use the iterative DWT algorithm, combine Equation 2 and let Express it as: Where i is an integer ranging from 0 to 2; I gt The iterative DWT operation is defined and executed as shown in Equations 3 and 4. Then in Equation 1 Expressed as: Where r represents the ratio of multi-frequency SSIM loss; SSIM(·) is the original SSIM loss; w represents the output of different filters, and w∈{LL,HL,LH,HH}, and the values ​​of w are substituted into Equation 5 respectively; for is the downsampled image, For horizontal edge detection image, For vertical edge detection image, Detect images for corners; The network uses two-dimensional Haar wavelet to implement cubic wavelet transform; Expressed as: Among them, WT t (·) indicates that the out and I gt The t-order scale of the wavelet coefficients calculated in the image; By optimizing the objective function formula 1 of the training, the optimized network parameters θ are obtained; At this point, the mapping relationship F between the low-light RAW image and the reference sRGB image is obtained.

3. The low-light image enhancement method based on RAW image according to claim 2, characterized in that: In formula 5, r = 0.

4.

4. The low-light image enhancement method based on RAW image according to claim 1, characterized in that: In step 5, the low-light RAW image is Y. are low-light RAW images and normal-light sRGB images respectively; F represents the mapping relationship between the low-light RAW image and the reference sRGB image.

5. The low-light image enhancement method based on RAW image according to claim 1, characterized in that: Use the GPU to complete the network training process in step 4 and the low-light image enhancement process in step 5, and use the cuDNN library to speed up the operation of the convolutional neural network.

6. A low-light image enhancement device based on a RAW image that implements the method of claim 1, characterized in that: It includes a green channel guidance submodule, a channel attention guided residual dense submodule, and a multi-scale parallel backbone network submodule; The green channel guidance submodule is used to design a green channel guidance convolution branch. Based on the data characteristics of real low-light RAW format low-light images, it takes advantage of the information of the green channel with higher brightness and perceptual intensity, shares retained information, and guides the backbone network to recover hidden / lost detail features and color deviations in low-light areas. The channel attention-guided residual dense submodule is used to design a dense residual structure based on the RAW image channel characteristics. It combines the attention mechanism to conduct in-depth feature mining on feature maps of different scales, thereby fully extracting hidden detail information with limited computational cost. The multi-scale parallel backbone network submodule is used to achieve cross-scale collaborative feature extraction and aggregate image context information. This submodule is trained using real paired RAW format low-light enhancement data, and the trained network is used for real low-light image enhancement of terminal devices. The connection relationship between the above modules is as follows: The output of the green channel guidance submodule and the output of the channel attention guided residual dense submodule are both connected to the input of the multi-scale parallel backbone network submodule.

Images