Low-illumination image enhancement method based on deep learning

Abstract

The invention discloses a low-illumination image enhancement method based on deep learning, and the method comprises the following steps: obtaining an image data set, and dividing the image data set into a training set, a verification set and a test set; preprocessing the image data set; carrying out denoising processing on the preprocessed image data set; inputting the de-noised training set into a low-illumination image enhancement model for training; inputting the de-noised verification set into a low-illumination image enhancement model, and adjusting hyper-parameters; inputting the de-noised test set into the trained low-illumination image enhancement model, and finally outputting enhanced image data; the image is decomposed into a low-frequency component reflecting the overall brightness and a high-frequency component containing detail features by adopting wavelet transform, and the low-frequency component is enhanced by utilizing a deep network, so that the overall brightness of the image is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to an image enhancement method, in particular to an image enhancement method for low-light environment. BACKGROUND

[0002] With the rapid development of computer vision technology, image acquisition and analysis have been widely used in industrial detection, security monitoring, intelligent transportation, robot navigation and many other fields. However, in low-light environment, such as night work, underground space or bad weather conditions, the visual system often faces problems such as insufficient image brightness, decreased signal-to-noise ratio, blurred details, etc., which seriously affects the accuracy and stability of subsequent image recognition, target detection and scene understanding tasks. Therefore, how to effectively improve the visual quality of low-light images and restore the structure and texture details in the images has become a key problem in current computer vision research.

[0003] At present, the methods for low-light image enhancement mainly adopt traditional image processing methods such as histogram equalization, Retinex algorithm and gamma correction. However, the above image processing methods have certain limitations, for example, histogram equalization may cause detail loss or over-enhancement when the contrast of local image region is too high; when processing images with more noise, it may amplify noise and intensify noise interference; Retinex algorithm is prone to detail blur and color distortion when processing extreme low-illumination or complex illumination scenes; gamma correction is sensitive to parameters and needs to manually adjust the gamma value, which is prone to overexposure or underexposure and lacks adaptive ability; and excessive correction may cause detail loss in over-bright regions of the image. SUMMARY

[0004] In view of the problem that the current visual system cannot effectively obtain clear images in low-light environment, the present application provides a low-light image enhancement method based on wavelet transform and deep network.

[0005] The technical solution adopted by the present application to solve the above technical problems is: A low-light image enhancement method based on deep learning, comprising the following steps: Step 1: acquiring video data in low-light environment through a camera device, and extracting image data frame by frame from the video data to form an image data set, dividing the image data set into a training set, a validation set and a test set; Step 2: preprocessing the image dataset, converting the image format in the image dataset to PNG format and converting the data type to 8-bit unsigned integer data, then scaling the image dataset and performing color balance check to remove color offset of the image dataset; converting the image dataset from RGB color mode to LAB color mode and adjusting the brightness channel of the image dataset to enhance the contrast of the image data; Step 3: denoising the image dataset after step 2 preprocessing, Gaussian filter and median filter are respectively used to denoise the image dataset for Gaussian noise and salt and pepper noise existing in the image dataset; in order to protect the edge and detail information of the image dataset, guided filter and anisotropic diffusion filter are also used to preserve the edge features in the image dataset; Step 4: inputting the training set after step 3 denoising into the low-light image enhancement model for training, optimizing the weight value of the low-light image enhancement model; then inputting the validation set after step 3 denoising into the low-light image enhancement model, adjusting the hyperparameters to improve the prediction accuracy of the low-light image enhancement model; Step 5: inputting the test set after step 3 denoising into the low-light image enhancement model trained in step 4, and finally outputting the enhanced image data.

[0006] The data processing flow of the low-light image enhancement model is as follows: receiving the image dataset after step 3 denoising, using wavelet transform to decompose the image dataset in frequency domain, extracting low-frequency components reflecting overall brightness and high-frequency components containing detail features, obtaining low-frequency approximation coefficients corresponding to low-frequency part of image signal and high-frequency detail coefficients corresponding to high-frequency part of image signal, then inputting the low-frequency approximation coefficients and high-frequency detail coefficients obtained by wavelet transform into deep network for image enhancement, weighting and fusing the enhanced low-frequency approximation coefficients and high-frequency detail coefficients processed by the deep network with the low-frequency approximation coefficients and high-frequency detail coefficients, then reconstructing the weighted and fused low-frequency approximation coefficients and high-frequency detail coefficients into enhanced image data by inverse wavelet transform, and outputting the enhanced image data; using wavelet transform to decompose the image into low-frequency components reflecting overall brightness and high-frequency components containing detail features, and using deep network to enhance the low-frequency components, thereby effectively improving the overall brightness of the image.

[0007] The deep network comprises an encoder, a decoder and a bottleneck layer, the encoder extracts abstract features of the low-frequency approximation coefficients and the high-frequency detail coefficients, then inputs the abstract features into the bottleneck layer for high-dimensional mapping, and restores details by upsampling on the decoder, and convolves and fuses the abstract features of the same level of the decoder and the encoder to make up for the image details lost in the upsampling process; the deep network based on U-Net enhances the high-frequency detail coefficients, can enhance image edges, textures and detail information, and can further improve the clarity of image data.

[0008] The deep network further comprises a convolutional attention module, the convolutional attention module comprises a channel attention module and a spatial attention module, the channel attention module and the spatial attention module respectively perform weighted enhancement on channel weights and spatial positions of the abstract features to highlight key frequency domain information.

[0009] The image data set is divided into a training set, a validation set and a test set according to a 7:1:2 image quantity ratio.

[0010] The specific calculation formula of the Gaussian filter used to remove Gaussian noise is: ; wherein, and is a coordinate offset relative to a center point, denotes a standard deviation; The specific calculation formula of the median filter used to remove salt and pepper noise is: ; wherein denotes a pixel value of an image data set, denotes a filtered pixel value, is a neighborhood window, and represent offset amounts of the neighborhood window relative to a current pixel.

[0011] The specific formula of the guided filter is: ; wherein denotes output image data, is image data of a guide image, and are linear parameters; The specific formula of the anisotropic diffusion filter is: ; wherein is an original image, is a divergence, is a gradient operator, is a conduction function of diffusion, is a scale parameter.

[0012] The beneficial effects of the present application are: the present application uses wavelet transform to perform multi-scale frequency domain decomposition on the image, extracts low-frequency components reflecting overall brightness and high-frequency components containing detail features respectively; then a U-Net deep network with a fusion attention mechanism is used to perform targeted feature enhancement on each frequency domain component; finally, image reconstruction is realized through inverse wavelet transform to obtain an enhanced image in a low-light environment; the image is decomposed into low-frequency components reflecting overall brightness and high-frequency components containing detail features by using wavelet transform; the high-frequency detail coefficients are enhanced by using a U-Net-based deep network, which can enhance image edges, textures and detail information, and further improve the clarity of image data. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present application will be further described below in conjunction with the drawings and embodiments.

[0014] Figure 1 is a schematic diagram of the principle of the present application.

[0015] Figure 2 is a schematic diagram of the deep network. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the present application clearer, further detailed description will be made below in conjunction with specific embodiments and reference to the drawings. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0017] It is understood that these descriptions are only exemplary and are not intended to limit the scope of the present application.

[0018] Referring to Figure 1 and Figure 2 , a low-light image enhancement method based on deep learning includes the following steps: Step 1: acquiring video data in a low-light environment through a camera device, and extracting image data from the video data frame by frame to form an image data set, dividing the image data set into a training set, a validation set and a test set; Step 2: preprocessing the image data set, converting the image format in the image data set to PNG format and converting the data type to 8-bit unsigned integer data, then performing adaptive scaling on the image, scaling the image size to 256x256, and performing color balance check to remove color offset of the image data set; converting the image data set from RGB color mode to LAB color mode, and adjusting the brightness channel of the image data set to enhance the contrast of the image data; Step 3: The image data set pre-processed in step 2 is subjected to denoising processing. Gaussian noise and salt and pepper noise existing in the image data set are subjected to denoising processing by using Gaussian filtering and median filtering respectively. In order to protect the edges and detail information of the image data set, edge features in the image data set are also reserved by guided filtering and anisotropic diffusion filtering. Before training the low-light image enhancement model, the training configuration is determined. First, the initial weights of the model are randomly generated. Second, the batch size is set to 32, the number of iterations is 400 times, the pixel loss includes L1 loss and SSIM, the weights are 1 and 0.5 respectively, the frequency domain loss weight is 0.3, and the Dropout probability is 0.4. Finally, all experiments are completed in an NVIDIA GPU environment, the optimizer is selected as the Adam optimizer, and the parameter settings are learning rate lr=0.0001, betas=(0.9, 0.999), and eps=1.

[0019] Step 4: The training set subjected to denoising processing in step 3 is input into the low-light image enhancement model for training, and the weight values of the low-light image enhancement model are optimized. Then the verification set subjected to denoising processing in step 3 is input into the low-light image enhancement model, and the hyperparameters are adjusted to improve the prediction accuracy of the low-light image enhancement model. Due to the influence of factors such as sensor characteristics and environmental interference during image acquisition, different types and degrees of noise are inevitably introduced into the image. In order to effectively improve the image quality, the noise in the image needs to be detected first. By calculating the gray variance and histogram distribution of the local region of the image, the noise type is identified. In this embodiment, the main noise sources are Gaussian noise and salt and pepper noise, therefore, in this embodiment, Gaussian filtering and median filtering are used to denoise the above two main noises.

[0020] Step 5: The test set subjected to denoising processing in step 3 is input into the low-light image enhancement model completed in step 4, and the enhanced image data is finally output.

[0021] In recent years, with the development of deep learning technology, the image enhancement method based on convolutional neural network has shown significant advantages in illumination correction and detail recovery. However, most of the methods are still limited to spatial domain processing, which is difficult to enhance brightness while considering detail preservation and noise suppression. The present application proposes a low-light image enhancement method based on wavelet transform combined with deep network, which fully utilizes the multi-scale frequency domain analysis capability of wavelet transform and the nonlinear feature learning advantage of deep neural network, effectively improving the overall brightness, contrast and detail definition of low-light images. Through the whole process of mapping the image from the wavelet domain to the spatial domain, end-to-end mapping from low-light images to enhanced images is realized, solving the contradiction between brightness enhancement and detail preservation in traditional enhancement methods.

[0022] The data processing flow of the low-light image enhancement model is as follows: receiving the image data set after denoising processing in step 3, using wavelet transform to decompose the image data set in frequency domain, extracting the low-frequency component reflecting the overall brightness and the high-frequency component containing the detail features, obtaining the low-frequency approximation coefficient corresponding to the low-frequency part of the image signal and the high-frequency detail coefficient corresponding to the high-frequency part of the image signal, then inputting the low-frequency approximation coefficient and high-frequency detail coefficient obtained by wavelet transform into the deep network for image enhancement, and weighting and fusing the enhanced low-frequency approximation coefficient and high-frequency detail coefficient processed by the deep network with the low-frequency approximation coefficient and high-frequency detail coefficient; To further balance the enhancement effect and the fidelity of the original structure, the low-frequency coefficient is weighted and fused in this embodiment, that is, the original low-frequency and the enhanced low-frequency are combined linearly to introduce the enhancement effect while preserving the image structure information; the high-frequency coefficient is completely used as the enhanced result, then the weighted and fused low-frequency approximation coefficient and high-frequency detail coefficient are reconstructed into enhanced image data by inverse wavelet transform, and the enhanced image data is output; the image is decomposed into low-frequency component reflecting overall brightness and high-frequency component containing detail features by wavelet transform, and the low-frequency component is enhanced by using deep network, so as to effectively improve the overall brightness of the image; common wavelet functions include Haar wavelet, Daubechies, Symlets, Coiflets, etc., different wavelets are suitable for different types of image or signal feature extraction; in specific implementation, different wavelet functions need to be selected according to different types of images, and Haar wavelet is preferred in this embodiment for multi-level decomposition and representation of images.

[0023] The wavelet transform in this embodiment adopts tower decomposition structure; the first layer decomposition: the original image is divided into one low-frequency subband (LL1) and three high-frequency subbands (LH1, HL1, HH1), which correspond to horizontal, vertical and diagonal direction detail information respectively; subsequent layer decomposition: the low-frequency subband of the last layer is further decomposed to form a multi-level structure, and the low-frequency information of finer granularity is extracted layer by layer.

[0024] To adapt the input format of the deep network, the low-frequency approximation coefficients and the high-frequency detail coefficients need to be reorganized. Each subband corresponding to the RGB three channels is arranged in order by layer and direction, and finally spliced into a tensor with a shape of [12, H, W]. The 12 channels correspond to the combination of 4 subbands (LL, LH, HL, HH) and 3 color channels respectively.

[0025] As shown in Figure 2 The deep network includes an encoder, a decoder and a bottleneck layer. The encoder extracts abstract features of the low-frequency approximation coefficients and the high-frequency detail coefficients, then inputs the abstract features into the bottleneck layer for high-dimensional mapping, and restores details through upsampling on the decoder, and convolves and fuses the abstract features of the same level of the decoder and the encoder to make up for the image details lost in the upsampling process. The deep network also includes a convolutional attention module, which includes a channel attention module and a spatial attention module. The channel attention module and the spatial attention module respectively weight and enhance the channel weight and the spatial position of the abstract features to highlight key frequency domain information. The deep network based on U-Net enhances the high-frequency detail coefficients, which can enhance image edges, textures and detail information, and further improve the clarity of image data. Figure 2 In the formula, Conv is convolution, BN is normalization layer, and ReLU is activation function.

[0026] The deep network takes the 12-channel frequency domain coefficients generated after wavelet transform as input, extracts multi-level abstract features through multiple encoders, and each layer uses a convolutional block to gradually extract higher-level semantic information. Then, the abstract features enter the bottleneck layer for high-dimensional mapping, and the spatial details are gradually restored through upsampling and skip connection in the decoder. To further improve the enhancement effect, the deep network of the embodiment also introduces a convolutional attention mechanism (CBAM) to weight and enhance the channel weight and spatial position of the feature map, thereby more effectively highlighting key frequency domain information. Finally, the deep network outputs a 12-channel tensor with the same size as the input but with enhanced features, corresponding to the enhanced wavelet coefficients, which is used for subsequent image reconstruction. The deep network combines the feature reuse advantage of the U-shaped network and the feature selection ability of the attention mechanism, and realizes efficient enhancement of frequency domain information.

[0027] The image dataset is divided into training set, validation set and test set according to the image number ratio of 7:1:2.

[0028] The specific calculation formula of the Gaussian filter used to remove Gaussian noise is: ; wherein, and is a coordinate offset relative to the center point, represents a standard deviation; The specific calculation formula of the median filter used to remove salt and pepper noise is: ; wherein represents a pixel value of the image data set, represents a filtered pixel value, is a neighborhood window, and represents an offset of the neighborhood window relative to the current pixel.

[0029] The specific formula of the guided filter is: ; wherein represents output image data, is image data of a guide image, and is a linear parameter; The specific formula of the anisotropic diffusion filter is: ; wherein is an original image, is a divergence, is a gradient operator, is a conduction function of diffusion, is a scale parameter.

[0030] To sum up, the application constructs an efficient and robust low-light image enhancement framework through the organic combination of wavelet transform, deep network and feature fusion strategy. The method not only is superior to traditional enhancement algorithms in subjective visual effect, but also has significant advantages in objective evaluation indexes PSNR and SSIM, has good generalization ability and engineering application prospect, and can be widely applied to night monitoring, industrial detection, intelligent transportation, unmanned aerial vehicle visual navigation and the like, thereby providing a feasible technical scheme for image quality improvement under a complex illumination environment.

[0031] In the present application, the term "a plurality of" refers to two or more, unless otherwise explicitly limited. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. The terms "mounting", "connecting", "connection", "fixing" and the like should be interpreted broadly, for example, "connection" can be fixed connection, detachable connection or integral connection; "connection" can be direct connection or indirect connection through an intermediate medium. For those skilled in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0032] It is to be understood that the terms "mounted", "attached", "connected" or "set up" as used herein, unless otherwise specified, are to be construed as referring to an arrangement in which one element is directly connected to another element, or indirectly connected through intervening elements. As used herein, the terms "vertical", "horizontal", "upper", "lower", "left", "right", and similar terms are intended to describe the orientation of the elements as shown in the drawings and are not meant to be limiting.

[0033] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "certain embodiments", and the like, means that the particular feature, structure, material or characteristic being described is included in at least one embodiment or example of the present application. In the present specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the description of the particular feature, structure, material or characteristic can be combined in any one or more embodiments or examples in a suitable manner.

[0034] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.

Claims

1. A low-light image enhancement method based on deep learning, characterized in that... Includes the following steps: Step 1: Acquire video data in low-light environment using camera equipment, extract image data frame by frame from the video data to form an image dataset, and divide the image dataset into training set, validation set and test set; Step 2: Preprocess the image dataset by converting the image format in the image dataset to PNG format and the data type to 8-bit unsigned integer data. Then, scale the image dataset and perform a color balance check to remove color shifts. Convert the image dataset from RGB color mode to LAB color mode and adjust the brightness channel of the image dataset to enhance the contrast of the image data. Step 3: Denoise the image dataset after preprocessing in Step 2. Gaussian filtering and median filtering are used to denoise the Gaussian noise and salt-and-pepper noise in the image dataset, respectively. In order to preserve the edge and detail information of the image dataset, guided filtering and anisotropic diffusion filtering are also used to preserve the edge features in the image dataset. Step 4: Input the training set after the denoising process in Step 3 into the low-light image enhancement model for training and optimize the weight values ​​of the low-light image enhancement model; then input the validation set after the denoising process in Step 3 into the low-light image enhancement model and adjust the hyperparameters to improve the prediction accuracy of the low-light image enhancement model. Step 5: Input the test set after denoising in Step 3 into the low-light image enhancement model trained in Step 4, and finally output the enhanced image data.

2. The low-light image enhancement method based on deep learning according to claim 1, characterized in that... The data processing flow of the low-light image enhancement model is as follows: The image dataset after denoising in step 3 is received. Wavelet transform is used to perform frequency domain decomposition on the image dataset, extracting low-frequency components reflecting overall brightness and high-frequency components containing detail features. Low-frequency approximation coefficients of the low-frequency part of the corresponding image signal and high-frequency detail coefficients of the high-frequency part of the corresponding image signal are obtained. Then, the low-frequency approximation coefficients and high-frequency detail coefficients obtained by wavelet transform are input into a deep network for image enhancement. The enhanced low-frequency approximation coefficients and enhanced high-frequency detail coefficients processed by the deep network are weighted and fused with the low-frequency approximation coefficients and high-frequency detail coefficients. Then, the weighted and fused low-frequency approximation coefficients and high-frequency detail coefficients are reconstructed into enhanced image data through inverse wavelet transform, and the enhanced image data is output.

3. The low-light image enhancement method based on deep learning according to claim 2, characterized in that... The deep network includes an encoder, a decoder, and a bottleneck layer. The encoder extracts abstract features from the low-frequency approximation coefficients and high-frequency detail coefficients. Then, the abstract features are input into the bottleneck layer for high-dimensional mapping. The decoder performs upsampling to restore details and convolutionally fuses the abstract features at the same level as the encoder to compensate for the image details lost during upsampling.

4. The low-light image enhancement method based on deep learning according to claim 3, characterized in that... The deep network further includes a convolutional attention module, which comprises a channel attention module and a spatial attention module. The channel attention module and the spatial attention module respectively weight and enhance the channel weights and spatial positions of the abstract features to highlight key frequency domain information.

5. The low-light image enhancement method based on deep learning according to claim 4, characterized in that... The image dataset is divided into a training set, a validation set, and a test set according to an image quantity ratio of 7:1:

2.

6. The low-light image enhancement method based on deep learning according to claim 5, characterized in that... The specific calculation formula for Gaussian filtering used to remove Gaussian noise is as follows: ;in, and It is the coordinate offset relative to the center point. Indicates standard deviation; The specific calculation formula for median filtering used to remove salt-and-pepper noise is as follows: ;in Represents the pixel values ​​of an image dataset. This represents the filtered pixel value. For neighborhood windows, and This represents the offset of the neighboring window relative to the current pixel.

7. The low-light image enhancement method based on deep learning according to claim 6, characterized in that... The specific formula for the guided filtering is as follows: ;in This represents the output image data. It is the image data of the guide map. and It is a linear parameter; The specific formula for the anisotropic diffusion filter is as follows: ;in It is the original image. It is divergence. It is the gradient operator. It is the diffusion transfer function. It is a scale parameter.

Images