A real-time low-illumination image enhancement method based on convolutional neural network
By using the lightweight convolutional neural network ADU-Net to preprocess and enhance low-light images, the problem of high computational overhead in existing technologies is solved, achieving efficient and real-time low-light image enhancement and improving image quality.
Patent Information
- Application Number
- CN202310482075.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-29
- Publication Date
- 2026-02-17
- Estimated Expiration
- 2043-04-29
AI Technical Summary
Existing low-light image enhancement methods face challenges in terms of computational overhead and device deployment, especially the computational burden brought by large networks, and traditional methods often lead to image distortion or noise amplification.
We employ a lightweight convolutional neural network, ADU-Net, to achieve efficient enhancement of low-light images through RAW image preprocessing and enhancement network processing, including image rearrangement, normalization, pre-enlargement, and feature extraction and adaptive feature fusion of the U-Net structure.
Without increasing computing resources, high-quality low-light image enhancement was achieved to meet the needs of real-time image processing, especially on edge devices such as mobile phones and surveillance cameras, improving image brightness and color restoration.
Smart Images
Figure CN116579940B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a method for enhancing low-light images. Background Technology
[0002] Photography in low-light conditions is an extremely challenging task. Due to insufficient light, the captured images often suffer from problems such as low brightness and color distortion; these images are generally referred to as low-light images. Low-light images not only have a poorer aesthetic appeal but also significantly reduce the performance of a range of downstream tasks, such as object detection and facial recognition, thus negatively impacting related applications such as road monitoring and outdoor security. Therefore, enhancing low-light images has significant practical importance.
[0003] Currently, low-light image enhancement can be divided into two categories: physical methods and algorithmic methods. The core of physical methods lies in increasing the amount of light entering the camera, but this is often difficult to achieve. For example, using a larger aperture can increase the amount of light entering the camera, but it reduces the depth of field, leading to image blur, and it cannot be implemented on devices with size limitations such as mobile phones and cameras; extending the exposure time is only suitable for shooting static scenes, and motion blur will occur in dynamic scenes; using a flash will lead to uneven exposure; increasing the camera's ISO will amplify noise. Therefore, academia and industry pay more attention to algorithm-based enhancement methods. Currently, representative low-light image enhancement algorithms mainly include methods based on histogram equalization, Retinex theory, and deep learning. Among them, histogram equalization methods globally adjust the image to make the image brightness conform to the prior distribution, which can effectively improve image contrast, but ignores the local information of the image, often leading to distorted results. Retinex theory-based methods can effectively preserve the structural information of the image; however, this method involves solving non-convex optimization problems, which is computationally intensive. In addition, Retinex theory has an inherent flaw: it ignores the noise component in the image, which often leads to distortion in the enhancement results on noisy images.
[0004] Deep learning-based methods are currently the mainstream in academia and industry. Deep methods use neural networks (such as convolutional neural networks) to directly learn the mapping relationship between low-light and normal-light images, demonstrating better performance than the aforementioned methods. Currently, research on deep learning-based augmentation methods has achieved some results both domestically and internationally; however, most of these methods are limited to using large-scale networks for augmentation, ignoring the computational overhead of large networks and the deployment requirements of practical devices. Summary of the Invention
[0005] The purpose of this invention is to provide a real-time low-light image enhancement method based on convolutional neural networks to solve the problem of image quality degradation caused by insufficient light intake and significant imaging noise in low-light environments.
[0006] The real-time low-light image enhancement method based on convolutional neural networks provided by the present invention includes (i) RAW image preprocessing and (ii) using the ADU-Net enhancement network to enhance the preprocessed image.
[0007] (i) The RAW image preprocessing includes three steps: image rearrangement, normalization, and pre-enlargement; the preprocessing process is used to provide translation invariance for subsequent convolution operations and to reduce image resolution to accelerate computation.
[0008] (ii) The ADU-Net enhancement network receives the preprocessed RAW image, performs low-light enhancement, and obtains the enhanced sRGB image.
[0009] In this invention, the RAW image preprocessing method specifically includes the following steps:
[0010] (1) Rearrange the RAW image according to the color filter array, and arrange pixels at the same position on the array to the same channel; taking a Bayer RAW image as an example, the image will be rearranged according to the RGBG color channel order to obtain a four-channel image with the original width and height as the original width and height. Figure 1 The process of creating a half-image can be represented as:
[0011]
[0012] Where I represents the input image, I′ represents the rearranged image, the subscript c represents the channel number, and x, y are spatial coordinates; the rearrangement operation is used to provide translation invariance for subsequent convolution operations and accelerate computation.
[0013] (2) Normalize the rearranged image, which can be represented as follows:
[0014] I″ c =(I′) c -BL c ) / (WL c -BL c ),
[0015] Where BL represents the black level and WL represents the saturation level, the subscript 'c' indicates the corresponding channel number. The black level and saturation level are directly read from the metadata of the RAW image;
[0016] (3) Amplify the normalized image using a pre-amplifier:
[0017] Iinput =I″*γ,
[0018] Wherein, γ is the amplification factor, which can be determined by the user.
[0019] In this invention, the ADU-Net enhancement network enhances the preprocessed image; wherein:
[0020] The enhanced network model, ADU-Net, is a U-Net structure comprising an encoder and a decoder, and incorporates a feature extraction module based on dilated convolution and residual connections, as well as an attention-based adaptive feature fusion module. The network accepts a preprocessed low-light RAW image as input. First, the encoder extracts multi-scale feature representations, which are then fed into the decoder for bottom-up feature reconstruction. The decoder uses the feature extraction module based on dilated convolution and residual connections to capture the semantic information of the image, and employs the attention-based adaptive feature fusion module to fuse the semantic information with shallow texture information from the encoder. After feature reconstruction by the decoder, the enhanced sRGB image is finally obtained.
[0021] Furthermore, the ADU-Net enhancement network performs enhancement processing on the preprocessed image, specifically through the following steps:
[0022] (a) For the input image I input First, a 3×3 convolution is projected onto the feature space.
[0023] (b) The feature representations obtained in the previous step are fed into the encoder to extract multi-scale feature representations; each encoder contains convolution and pixel-unshuffle downsampling. [1] , is represented as:
[0024]
[0025] in, f represents the output of the i-th convolutional layer; conv (.) indicates a convolution operation, and ↓ indicates a pixel-unshuffle downsampling operation. [1] .
[0026] (c) The features obtained in the previous step are fed into the decoder network for feature reconstruction. The features from the decoder are bilinearly interpolated and then reconstructed through the feature extraction module. Simultaneously, the shallow features from the encoder are passed through skip connections and undergo an adaptive feature fusion process to obtain the enhanced feature representation. This process can be represented as:
[0027]
[0028] in, This indicates the operation of the feature extraction module. This indicates the adaptive feature fusion module operation, where the superscript i represents the module number, and ↑ represents the bilinear interpolation operation; specifically:
[0029] (cI) The feature extraction module contains N residual modules and a feature aggregation operation for efficient feature extraction; the residual modules use dilated convolutions to increase the receptive field and use 1×1 convolutions as feature connection paths; the feature aggregation operation is implemented by 1×1 convolutions.
[0030] In the feature extraction module, different hole rates are set for the residual modules at different locations, specifically:
[0031]
[0032] i is the index of the residual module, i = 0, 1, ..., N-1;
[0033] The mechanism of the feature extraction module can be represented as follows:
[0034] Y = f conv ([X1,…,X N ]),
[0035] Among them, X i Y represents the intermediate feature, and Y represents the output feature. [...] indicates concatenation along the channel dimension. Residual module [2] The superscript i indicates the module number;
[0036] (c-II) Adaptive feature fusion process is used to fuse shallow texture features X from the encoder. t and deep semantic features X from the decoder s The process first uses convolution to extract complementary feature representations Y from the two features. comp Then, it is injected into the deep semantic features from the decoder through residual connections; subsequently, the weights of each channel are adjusted through a residual channel attention mechanism; the mechanism of the adaptive feature fusion process can be represented as:
[0037] Y comp =f conv ([X t ,X s ]),
[0038]
[0039] A = f conv (AvgPool([X t ,X s ])),
[0040]
[0041] Where A is the learned attention weight, and Y... fused For output features, AvgPool(.) represents global average pooling, [...] represents concatenation along the channel dimension.
[0042] (d) The features output by the decoder are upsampled by pixel-shuffle. [1] After that, the enhanced sRGB image is obtained.
[0043] The specific operation process of this invention is as follows:
[0044] (1) Prepare the dataset for training the augmented network;
[0045] (2) Use the RAW image preprocessing method described above to preprocess the low-light images in the dataset;
[0046] (3) Train the augmented network model ADU-Net;
[0047] (4) Use the trained augmentation network to process the pre-processed low-light image.
[0048] The process of training and enhancing the network model includes setting a training strategy; starting training; and adjusting the training strategy multiple times to obtain the model with the best performance.
[0049] The training strategy includes the image patch size, batch size, learning rate adjustment strategy, and gradient descent strategy used for training.
[0050] This invention utilizes an efficient encoder-decoder network architecture, along with efficient feature extraction and adaptive feature fusion processes, to obtain a lightweight and efficient image enhancement network, overcoming the problem of excessive computational overhead in previous low-light image enhancement methods. Experiments on a GTX 1080Ti GPU show that, without special optimization, this invention can enhance 4K high-resolution extremely dark images (brightness less than 5 lux) at 25fps, while peak memory usage does not exceed 1GB. Therefore, this invention can meet the real-time image processing needs of edge devices such as mobile phones and surveillance cameras, and has good application value. Attached Figure Description
[0051] Figure 1 This is a diagram of the ADU-Net network structure used in this invention.
[0052] Figure 2 This is a structural diagram of the feature extraction module based on dilated convolution and residual connections used in this invention.
[0053] Figure 3 This is a structural diagram of the adaptive feature fusion module used in this invention.
[0054] Figure 4 This is a comparison of the experimental results on real low-light images. Detailed Implementation
[0055] The network structure used in this invention for real-time low-light image enhancement is as follows: Figure 1 As shown, the low-light RAW images are fed into the network after preprocessing, and then processed as follows: Figure 2 Feature extraction module and Figure 3 After the adaptive feature fusion module, the enhanced sRGB image is finally obtained.
[0056] The embodiments of the present invention include the following steps:
[0057] (1) Preparation of dataset
[0058] This article selects the SID dataset. [3] The network was trained and tested. The SID dataset contains pairs of low-light and normal-light images taken under extremely dark conditions, with ambient light levels between 0.03 lux and 5 lux. The exposure time for low-light images ranged from 0.033 s to 0.1 s, while the exposure time for normal-light images ranged from 10 s to 30 s. The dataset contains 5094 image pairs, taken with a Sony α7S II and a Fujifilm X-T2 camera, with image resolutions of 4240×3842 and 6000×4000 respectively, stored in RAW format. The SID dataset provides a standard training / validation / test set partition, and this invention maintains this partitioning method. Without loss of generality, this embodiment selects Bayer RAW images taken with a Sony α7S II for training and evaluation.
[0059] (2) Data preprocessing
[0060] The original low-light RAW image needs to be rearranged, normalized, and pre-enlarged. The specific steps are as follows:
[0061] (2a) Rearrange the Bayer RAW image according to the RGBG color channel order to obtain four channels, with the length and width as originally... Figure 1 The process of creating a half-image can be represented as: Where I represents the input image, I′ represents the rearranged image, and c is the channel number. The rearrangement operation is used to provide translation invariance for subsequent convolution operations and to speed up computation.
[0062] (2b) Normalize the rearranged image, which can be represented as I″ c =(I′)c -BL c ) / (WL c -BL c ), where BL represents the black level and WL represents the saturation level. The black level and saturation level are directly read from the metadata of the RAW image;
[0063] (2c) Preamplify the normalized image: I input =I″*γ, where γ is the magnification factor, set as the exposure duration ratio of long / short exposure images in the SID dataset.
[0064] (3) Model Training
[0065] The network in this invention is trained using a single GTX 1080Ti GPU with 512×512 image patches and a batch size of 1. The network is trained for 4000 epochs with an initial learning rate of 1×10⁻⁶. -4 It decreased to 1×10 after 2000 epochs. -5 The network was trained using the Adam optimizer.
[0066] (4) Model Testing
[0067] After training, the network is evaluated on the SID test set images. The image preprocessing method is the same as in step (2). Figure 4 The first line shows the experimental results on the SID test set images. It observes that the original low-light images have extremely low brightness, while after magnification, the images exhibit significant noise and color distortion; compared to the method proposed by Lamba et al. [4] Compared to the previous version, the present invention has achieved a significant improvement in image restoration, with the image presenting more accurate colors and better detail information.
[0068] Figure 4 The second line shows the experimental results of the generalization of this invention, where low-light images were captured by a Canon 6D camera and augmented using a model trained on the SID dataset. Compared to the method proposed by Lamba et al., this invention achieves more accurate color restoration.
[0069] Table 1 presents the performance data of this invention. The relevant data on the GPU are test results on a single GTX 1080Ti card, and the relevant data on the CPU are test results on a quad-core Intel i7-6700K. The algorithm runs in single-linear mode. It can be observed that this invention has a fast running speed and low computational resource overhead, which can meet the needs of real-time image processing.
[0070] Table 1
[0071] MACs(G) Number of parameters (M) Inference time (s) on CPU / GPU Peak GPU memory usage (GB) 0.18 0.635 1.1791 / 0.0367 0.79
[0072] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Any modifications or substitutions made by those skilled in the art within the scope of the technology disclosed in this invention should be included within the scope of this invention.
[0073] References
[0074] [1]Shi W,Caballero J,Huszár F,et al.Real-time single image and videosuper-resolution using an efficient sub-pixel convolutional neural network[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2016:1874-1883.
[0075] [2]He K, Zhang X, Ren S, et al.Deep residual learning for imagerecognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.
[0076] [3]Chen C, Chen Q, Xu J, et al. Learning to see in the dark[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2018:3291-3300.
[0077] [4]Lamba M,Mitra K.Restoring extremely dark images in real time[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2021:3487-3497。
Claims
1. A real-time low-light image enhancement method based on convolutional neural networks, characterized in that, The specific steps are as follows: (a) RAW image preprocessing, including image rearrangement, normalization and pre-enlargement; the preprocessing process is used to provide translation invariance for subsequent convolution operations and reduce image resolution to accelerate computation. (II) The ADU-Net enhancement network is used to enhance the low-light performance of the preprocessed RAW image, resulting in an enhanced sRGB image; where: The enhanced network model ADU-Net is a U-Net structure, which includes an encoder and a decoder, and adds a feature extraction module based on dilated convolution and residual connections, as well as an attention-based adaptive feature fusion module. The augmented network model takes a preprocessed low-light RAW image as input. First, the encoder extracts multi-scale feature representations, which are then fed into the decoder for bottom-up feature reconstruction. The decoder uses a feature extraction module based on dilated convolution and residual connections to capture the semantic information of the image, and uses an attention-based adaptive feature fusion module to fuse the semantic information with the shallow texture information from the encoder. After feature reconstruction by the decoder, the augmented sRGB image is finally obtained.
2. The real-time low-light image enhancement method according to claim 1, characterized in that, The specific steps of the RAW image preprocessing are as follows: (1) Rearrange the RAW image according to the color filter array, and arrange the pixels at the same position on the array to the same channel; for the Bayer RAW image, rearrange according to the RGBG color channel order to obtain a four-channel image with half the length and width of the original image. This process is represented as: Where I represents the input image, I′ represents the rearranged image, the subscript c represents the channel number, and x, y are the spatial coordinates; (2) Normalize the rearranged image, specifically as follows: to c =(I′ c -BL c ) / (WL c -BL c ), Where BL represents the black level and WL represents the saturation level, and the subscript c is the corresponding channel number; the black level and saturation level are directly read from the metadata of the RAW image; (3) Amplify the normalized image using a pre-amplifier: I input =I″*γ, Wherein, γ is the amplification factor, which can be determined by the user.
3. The real-time low-light image enhancement method according to claim 1, characterized in that, The ADU-Net enhancement network enhances the preprocessed image through the following steps: (a) For the input image I input First, a 3×3 convolution is projected onto the feature space; (b) Feed the feature representation obtained in the previous step into the encoder to extract multi-scale feature representations; each encoder contains convolution and pixel-unshuffle downsampling; (c) The features obtained in the previous step are fed into the decoder network for feature reconstruction. The features from the decoder are reconstructed by the feature extraction module after bilinear interpolation. Meanwhile, the shallow features from the encoder are passed through skip connections and then through the adaptive feature fusion module to obtain the enhanced feature representation. (d) The features output by the decoder are upsampled by Pixel-Shuffle to obtain the enhanced sRGB image.
4. The real-time low-light image enhancement method according to claim 3, characterized in that, In step (c), the feature extraction module contains N residual modules and a feature aggregation module for efficiently extracting feature representations; the residual modules use dilated convolutions to increase the receptive field and use 1×1 convolutions as feature connection paths. The feature aggregation module is implemented using 1×1 convolutions; In the feature extraction module, different hole rates are set for the residuals at different locations, specifically: i is the index of the residual module, i = 0, 1, ..., N-1; The mechanism of the feature extraction module is represented as follows: Among them, X i As an intermediate feature, This indicates residual module operation, with the superscript i indicating the module number; Y is the output feature, [...] indicating concatenation along the channel dimension.
5. The real-time low-light image enhancement method according to claim 4, characterized in that, In step (c), the adaptive feature fusion module is used to fuse the shallow texture features X from the encoder. t and deep semantic features X from the decoder s The process first uses convolution to extract complementary feature representations Y from the two features. comp Then, it is injected into the deep semantic features from the decoder through residual connections; then, the weights of each channel are adjusted through the residual channel attention mechanism. The mechanism of adaptive feature fusion process is represented as follows: Y comp =f conv ([X t ,X s ]), A=f conv (AvgPool([X t ,X s ])), Where A is the learned attention weight, and Y... fused For output features, AvgPool(.) represents the global average pooling operation, [...] represents concatenation along the channel dimension.
6. The real-time low-light image enhancement method according to any one of claims 1-5, characterized in that, The specific operating procedure is as follows: (1) Prepare the dataset for training the augmented network; (2) Use the RAW image preprocessing method described above to preprocess the low-light images in the dataset; (3) Train the augmented network model ADU-Net; (4) Use the trained augmentation network to process the pre-processed low-light image; The training of the augmentation network model includes setting a training strategy; starting training; and adjusting the training strategy multiple times to obtain the model with the best performance. The training strategy includes the image patch size used for training, the batch size, the learning rate adjustment strategy, and the gradient descent strategy.