A low-light image enhancement method based on deep learning

Abstract

The embodiment of the present invention discloses a method for low-light image enhancement based on deep learning, including: step 1, constructing a multi-exposure and multi-scale reflectance image extraction model; step 2, constructing a brightness weight matrix estimation model, the brightness weight matrix estimation model is used to estimate the input image brightness weight matrix and the reflectance image brightness weight matrix; step 3, constructing a lighting enhancement model, combining the outputs of the multi-exposure and multi-scale reflectance image extraction model and the brightness weight matrix estimation model, and establishing a lighting enhancement model for predicting the input low-light image to obtain a normal-light image; step 4, constructing a loss function, and training based on the gradient descent method to solve the optimal parameters of the overall low-light image enhancement model. The embodiment of the present invention solves the problem that it is difficult to achieve both image processing effects of restoring details and maintaining colors in the existing low-light image enhancement technology, so as to improve the visual perception effect.

Description

Technical Field

[0001] The present invention relates to but is not limited to the field of image processing, and specifically to a low-light image enhancement method based on deep learning. Background Art

[0002] In recent years, the research on low-light image enhancement technology has been one of the hot topics in image processing. The implementation methods are divided into traditional image enhancement methods based on Retinex theory and new image enhancement methods based on neural networks.

[0003] Traditional image enhancement methods based on Retinex theory include MSR, MSRCR and MSRCP methods, etc. The above methods have obvious effects in low-light image enhancement, but they all have problems such as long processing time and color distortion, resulting in poor visual perception.

[0004] New image enhancement methods based on neural networks include RetinexNet, LLNet, MBLLEN, and KinDNet. Most of the above methods have good processing speed and color retention, but they are still insufficient in detail processing and more accurate correction of low-light image color performance.

[0005] It can be seen that the above two types of image enhancement methods are difficult to achieve both better image processing effects of restoring details and maintaining colors. Summary of the invention

[0006] The purpose of the present invention is: The embodiment of the present invention provides a low-light image enhancement method based on deep learning, aiming to solve the problem in the existing low-light image enhancement technology that it is difficult to achieve both image processing effects of restoring details and maintaining colors, so as to improve the visual perception effect.

[0007] The technical solution of the present invention is: an embodiment of the present invention provides a low-light image enhancement method based on deep learning, comprising:

[0008] Step 1, constructing a multi-exposure and multi-scale reflectance image extraction model, including: obtaining a multi-exposure image and multi-scale information according to an input low-light image, and obtaining a reflectance image;

[0009] Step 2, constructing a brightness weight matrix estimation model, wherein the brightness weight matrix estimation model is used to estimate the input image brightness weight matrix and the reflectivity image brightness weight matrix;

[0010] Step 3, constructing a lighting enhancement model, including: combining the outputs of the multi-exposure and multi-scale reflectance image extraction model and the brightness weight matrix estimation model to establish a lighting enhancement model, wherein the lighting enhancement model is used to predict the input low-light image to obtain a normal-light image;

[0011] Step 4, construct a loss function and perform training based on the gradient descent method to solve the optimal parameters of the overall model of low-light image enhancement; wherein the overall model of low-light image enhancement includes: a multi-exposure and multi-scale reflectance image extraction model, a brightness weight matrix estimation model and a light enhancement model.

[0012] Optionally, in the low-light image enhancement method based on deep learning as described above, step 1 includes:

[0013] First, a multi-exposure image is generated for the input low-light image, and then multi-scale information is extracted from the generated multi-exposure image, and finally the multi-scale information is fused to obtain an intrinsic reflectance image.

[0014] Optionally, in the low-light image enhancement method based on deep learning as described above, step 1 specifically includes:

[0015] Step 11, construct a multi-exposure image generation model, the constructed multi-exposure image generation model is based on the low exposure image I low As input, a multi-exposure image cluster P i is the output model;

[0016] Step 12: construct a multi-scale information extraction model and cluster the multi-exposure images P i As input, each input is respectively passed through the multi-scale information extraction model to complete the multi-scale information P′ i The extraction of i =BTF(P i );

[0017] Step 13, fusing multi-scale information to obtain an intrinsic reflectivity image, including: using multi-scale information P′ i It is input, and after convolution, it outputs the intrinsic reflectivity image of the three channels, denoted as P′.

[0018] Optionally, in the deep learning-based low-light image enhancement method described above, in step 11, the method of constructing a multi-exposure image generation model includes:

[0019] By randomly generating a series of exposure rates κ = {k1, k2, …, k n}, for the input low exposure image I low The exposure rate is adjusted by pixel, and the generated multi-exposure image cluster is represented as P i =Clip[I low (x,y)·k i ].

[0020] Optionally, in the above-mentioned low-light image enhancement method based on deep learning, in step 12, the manner of constructing a multi-scale information extraction model includes:

[0021] The multi-scale information extraction model is a coding-decoding network, which is composed of N convolutions, activation functions, downsampling and N deconvolutions, activation functions, and upsampling connected in sequence.

[0022] Optionally, in the low-light image enhancement method based on deep learning as described above, step 2 includes:

[0023] Step 21, construct a brightness weight matrix estimation model, and transform the low exposure image I low As input, the brightness weight matrix W can be obtained through the brightness weight matrix estimation model;

[0024] Step 22, estimating the reflectivity image weight matrix, includes: performing an inverse operation on the brightness weight matrix W obtained in step 21 to obtain an intrinsic reflectivity image weight matrix Expressed as

[0025] Optionally, in the low-light image enhancement method based on deep learning as described above, in step 21, the method of constructing a brightness weight matrix estimation model includes:

[0026] The brightness weight matrix estimation model is a coding-decoding network, which is composed of M convolutions, activation functions, downsampling and M deconvolutions, activation functions, and upsampling connected in sequence; M is not equal to N.

[0027] Optionally, in the low-light image enhancement method based on deep learning as described above, step 3 includes:

[0028] Construct an illumination enhancement model, and the illumination enhancement model is based on the low-light image I low , intrinsic reflectance image P′, low light image brightness weight matrix W and reflectance image brightness weight matrix As input, the low-light image I is predicted by the illumination enhancement model low Normal lighting image.

[0029] Optionally, in the low-light image enhancement method based on deep learning as described above, the mathematical representation of the illumination enhancement model is:

[0030] Among them, α and β are constants determined by model training. is the predicted normal illumination image.

[0031] The beneficial effects of the present invention are as follows: an embodiment of the present invention provides a low-light image enhancement method based on deep learning, which utilizes deep learning to mine multi-exposure, multi-scale details and color information in low-light images, so that the enhanced image details and colors are restored with high fidelity, thereby enhancing the visual perception experience; at the same time, the method has fast training and processing speeds. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of a process of a low-light image enhancement method based on deep learning provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of an overall network model of a low-light image enhancement method based on deep learning provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of the principle of constructing a multi-scale information extraction model in the low-light image enhancement method provided by an embodiment of the present invention;

[0035] Figure 4 A schematic diagram comparing common typical image enhancement results and details using the low-light image enhancement method provided by an embodiment of the present invention and an existing method;

[0036] Figure 5 a is an actual photograph of the low-light image enhancement method provided by the embodiment of the present invention in the “helicopter blade motion parameter measurement”;

[0037] Figure 5 b is a diagram showing the enhancement results of the low-light image enhancement method provided by an embodiment of the present invention in the “helicopter blade motion parameter measurement”. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present invention more clear, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0039] The steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. Also, although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a sequence different from that shown here.

[0040] As described in the above background technology, among the existing low-light image enhancement technologies, the traditional image enhancement method based on Retinex theory and the new image enhancement method based on neural network are difficult to achieve both better image processing effects of restoring details and maintaining colors.

[0041] Vision accounts for about 80% of the various information received by humans, but it is difficult to read information in dark images. In addition, with the rapid development of machine vision, it is widely used in target detection, unmanned driving, and aerial reconnaissance and patrol. Image enhancement technology can play a significant role at night and in extreme weather. Therefore, low-light image enhancement methods are particularly important, and how to better restore details and maintain colors during the enhancement process is a new difficulty and hot spot. In response to the above requirements, an embodiment of the present invention provides a low-light image enhancement method based on deep learning.

[0042] The present invention provides the following specific embodiments which can be combined with each other. The same or similar concepts or processes may not be described in detail in some embodiments.

[0043] Figure 1 A flowchart of a method for low-light image enhancement based on deep learning provided by an embodiment of the present invention is provided. The method for low-light image enhancement based on deep learning provided by an embodiment of the present invention can use TensorFlow to establish a network structure model, such as Figure 2 , which is a schematic diagram of the overall network model of the low-light image enhancement method based on deep learning provided by an embodiment of the present invention. The low-light image enhancement method based on deep learning provided by an embodiment of the present invention may include the following steps:

[0044] Step 1, constructing a multi-exposure and multi-scale reflectance image extraction model, including: obtaining a multi-exposure image and multi-scale information according to an input low-light image, and obtaining a reflectance image;

[0045] Step 2, constructing a brightness weight matrix estimation model, the brightness weight matrix estimation model is used to estimate the input image brightness weight matrix and the reflectivity image brightness weight matrix;

[0046] Step 3, constructing a lighting enhancement model, including: combining the outputs of the multi-exposure and multi-scale reflectance image extraction model and the brightness weight matrix estimation model to establish a lighting enhancement model, wherein the lighting enhancement model is used to predict the input low-light image to obtain a normal-light image;

[0047] Step 4, construct a loss function and perform training based on the gradient descent method to solve the optimal parameters of the overall model of low-light image enhancement; wherein the overall model of low-light image enhancement includes: a multi-exposure and multi-scale reflectance image extraction model, a brightness weight matrix estimation model and a light enhancement model.

[0048] The specific implementation of the above steps of the low-light image enhancement method based on deep learning provided by an embodiment of the present invention is described in detail below.

[0049] The specific implementation of step 1 of the embodiment of the present invention is:

[0050] First, multi-exposure images are generated for the input low-light image, and then multi-scale information is extracted from the generated multi-exposure images. Finally, the multi-scale information is fused to obtain the intrinsic reflectance image.

[0051] In practical applications, the specific implementation process of step 1 may include:

[0052] Step 11, construct a multi-exposure image generation model (MEG Block), the constructed multi-exposure image generation model is: low As input, a multi-exposure image cluster P i The output model.

[0053] In step 11, the method of constructing a multi-exposure image generation model may include:

[0054] By randomly generating a series of exposure rates κ = {k1, k2, …, k n}, for the input low exposure image I low The exposure rate is adjusted by pixel, and the generated multi-exposure image cluster is represented as P i =Clip[I low (x,y)·k i ].

[0055] Step 12: construct a multi-scale information extraction model and cluster the multi-exposure images P i As input, each input is respectively passed through the multi-scale information extraction model to complete the multi-scale information P′ i The extraction of i =BTF(P i ).

[0056] In step 12, the method of constructing the multi-scale information extraction model may include:

[0057] The multi-scale information extraction model is a coding-decoding network, which is composed of N convolutions, activation functions, downsampling and N deconvolutions, activation functions, and upsampling connected in sequence.

[0058] Step 13, fusing multi-scale information to obtain an intrinsic reflectivity image, including: using multi-scale information P′ i It is input and after convolution, it outputs a 3-channel intrinsic reflectivity image, denoted as P′.

[0059] The specific implementation process of step 2 of the embodiment of the present invention may include:

[0060] Step 21, construct a brightness weight matrix estimation model, and transform the low exposure image I low As input, the brightness weight matrix W can be obtained through the brightness weight matrix estimation model.

[0061] In step 21, the method of constructing the brightness weight matrix estimation model may include:

[0062] The brightness weight matrix estimation model is a coding-decoding network, which is composed of M convolutions, activation functions, downsampling and M deconvolutions, activation functions, and upsampling connected in sequence; M is not equal to N.

[0063] Step 22, estimating the reflectivity image weight matrix, includes: performing an inverse operation on the brightness weight matrix W obtained in step 21 to obtain an intrinsic reflectivity image weight matrix Expressed as

[0064] The specific implementation of step 3 of the embodiment of the present invention may include:

[0065] Construct an illumination enhancement model, and the illumination enhancement model is based on the low-light image I low , intrinsic reflectance image P′, low light image brightness weight matrix W and reflectance image brightness weight matrix As input, the low-light image I is predicted by the illumination enhancement model low Normal lighting image.

[0066] The mathematical representation of the illumination enhancement model finally formed by the low-light image enhancement method provided by the embodiment of the present invention is:

[0067] Among them, α and β are constants determined by model training. is the predicted normal illumination image.

[0068] The low-light image enhancement method based on deep learning provided by the embodiment of the present invention utilizes deep learning to mine multi-exposure, multi-scale details and color information in low-light images, so that the enhanced image details and colors are restored with high fidelity, thereby enhancing the visual perception experience; at the same time, the method has fast training and processing speeds.

[0069] The specific implementation of the above steps of the low-light image enhancement method based on deep learning provided by an embodiment of the present invention is described in detail below.

[0070] This specific embodiment uses TensorFlow to establish a network structure model. During the specific implementation, the overall network model is as follows Figure 2 shown.

[0071] (1) Construct a multi-exposure and multi-scale reflectance image extraction model;

[0072] In this step, firstly, the MEG Block is used to generate a multi-exposure image for the input low-light image, then the BTF Block is used to extract multi-scale information from the multi-exposure image, and finally the multi-scale information is fused to obtain the intrinsic reflectance image. Specifically, the following steps are included:

[0073] (1.1) Construct a multi-exposure image generation model (MEG Block) by randomly generating a series of different exposure rates κ = {k1, k2, …, k n}, for the input low exposure image I low Adjust the exposure ratio by pixel to generate a multi-exposure image cluster P i , denoted as P i =Clip[I low (x,y)·k i ]; In this specific embodiment, for example, n=2, k1=0.5, k2=1.0 is adopted for implementation.

[0074] (1.2) Construct a multi-scale information extraction model (BTF Block) by: i As input, the multi-scale information P′ is completed through BTF Block respectively i The extraction of is mathematically expressed as: P′ i =BTF(P i BTFBlock is a coding-decoding network, which is composed of N convolutions, activation functions, downsampling, and N deconvolutions, activation functions, and upsampling.

[0075] like Figure 3 As shown in FIG. 1 , it is a schematic diagram of the principle of constructing a multi-scale information extraction model in the low-light image enhancement method provided by an embodiment of the present invention. In this specific embodiment, N=4 is used for implementation, the convolution kernel is 3×3, the activation function is LReLU, and the downsampling is maximum pooling. The specific implementation structure is as follows: Figure 3 As shown, the mathematical expression of LReLU is shown in the following formula.

[0076]

[0077] (1.3) Fusion of multi-scale information to obtain reflectance image, the specific implementation method is: i As input, after convolution, a 3-channel reflectance image is output, denoted as P′; it should be noted that color images generally use 3 channels.

[0078] (2) constructing a brightness weight matrix estimation model, which is used to estimate the input image brightness weight matrix and the reflectance image brightness weight matrix. The specific implementation process of step 2 may include the following steps:

[0079] (2.1) Construct the brightness weight matrix estimation model (W Block), and transform I low As input, the brightness weight matrix W can be obtained through W Block. W Block is a coding-decoding network, which is composed of M convolutions, activation functions, downsampling and M deconvolutions, activation functions, and upsampling connected in sequence.

[0080] In this specific embodiment, for example, M=2 is used, LReLU is used as the activation function, maximum pooling is used for downsampling, and the connection structure is consistent with the BTF Block.

[0081] (2.2) Estimating the reflectance image weight matrix. The specific implementation method is: inverting the above-obtained matrix W to obtain the reflectance image weight matrix Expressed as

[0082] (3) Construct an illumination enhancement model, which uses low-light image I low , reflectance image P′, low light image brightness weight matrix W and reflectance image brightness weight matrix As input, the model predicts the normal lighting image, which is mathematically expressed as

[0083] In this specific embodiment, for example, α=1 and β=1 are used to establish the model.

[0084] (4) constructing a loss function and performing training based on the gradient descent method to solve the optimal parameters of the overall model for low-light image enhancement. The specific implementation process of step 4 may include the following steps:

[0085] (4.1) Construct the loss function Loss. During the training process, the gradient descent direction is determined by the change of the loss function Loss, and the overall parameters are iteratively updated until Loss converges. Its mathematical expression is:

[0086] In this specific embodiment, for example, λ=1, μ=-1, ν=1 is used to establish the model.

[0087] (4.2) Based on the gradient descent method, the network is trained for m iterations on the Q pairs of data sets (pairs of low-light images and normal-light images). First, the low-light image I is input, and the model outputs the corresponding normal-light prediction image Then The normal illumination image R corresponding to the dataset HAs input, the corresponding Loss is calculated, and finally the overall model parameters are updated iteratively through gradient descent and back propagation until the Loss converges. The obtained parameters are the optimal parameters of the overall model.

[0088] In this specific embodiment, for example, 500 pairs of public dataset images are used, 485 pairs of images are used for network training, and 15 pairs of images are used for network testing. All network parameters are initialized to 0, and after 3700 training iterations, the loss converges. At this time, the network parameters are the optimal parameters, and the network model is saved for image enhancement that has not participated in the training.

[0089] like Figure 4 As shown, it is a schematic diagram comparing the common typical image enhancement results and details using the low-light image enhancement method provided by an embodiment of the present invention and the existing methods; it can be seen that the low-light image enhancement method provided by an embodiment of the present invention is significantly better than the enhanced images obtained by various existing image processing methods.

[0090] like Figure 5 As shown in a, it is an actual picture taken in "helicopter blade motion parameter measurement" using the low-light image enhancement method provided by an embodiment of the present invention; Figure 5 FIG. 2 shows an enhanced result diagram of the low-light image enhancement method provided by an embodiment of the present invention in the “helicopter blade motion parameter measurement”. It can be seen that Figure 5 In the actual shot in a, a photocopy was added to the image due to the extremely low image brightness.

[0091] Although the embodiments disclosed in the present invention are as above, the contents described are only embodiments adopted to facilitate understanding of the present invention and are not intended to limit the present invention. Any technician in the field to which the present invention belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present invention, but the patent protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. A low-light image enhancement method based on deep learning, characterized in that: include: Step 1, constructing a multi-exposure and multi-scale reflectance image extraction model, including: obtaining a multi-exposure image and multi-scale information according to an input low-light image, and obtaining a reflectance image; Step 2, constructing a brightness weight matrix estimation model, wherein the brightness weight matrix estimation model is used to estimate the input image brightness weight matrix and the reflectivity image brightness weight matrix; Step 3, constructing an illumination enhancement model, including: combining the outputs of the multi-exposure and multi-scale reflectance image extraction model and the brightness weight matrix estimation model to establish an illumination enhancement model, wherein the illumination enhancement model is used to predict the input low-light image to obtain a normal-light image; wherein the illumination enhancement model is based on the low-light image I low , intrinsic reflectance image P′, low light image brightness weight matrix W and reflectance image brightness weight matrix As input, the low-light image I is predicted by the illumination enhancement model low Normal lighting image of Step 4, construct a loss function and perform training based on the gradient descent method to solve the optimal parameters of the overall low-light image enhancement model; wherein the overall low-light image enhancement model includes: a multi-exposure and multi-scale reflectance image extraction model, a brightness weight matrix estimation model and a light enhancement model.

2. The method for low-light image enhancement based on deep learning according to claim 1, characterized in that: The step 1 comprises: First, a multi-exposure image is generated for the input low-light image, and then multi-scale information is extracted from the generated multi-exposure image, and finally the multi-scale information is fused to obtain an intrinsic reflectance image.

3. The method for low-light image enhancement based on deep learning according to claim 2, characterized in that: The step 1 specifically includes: Step 11, construct a multi-exposure image generation model, the constructed multi-exposure image generation model is based on the low exposure image I low As input, a multi-exposure image cluster P i is the output model; Step 12: construct a multi-scale information extraction model and cluster the multi-exposure images P i As input, each input is respectively passed through the multi-scale information extraction model to complete the multi-scale information P i ′, mathematically expressed as P i ′=BTF(P i ); Step 13, fusing multi-scale information to obtain an intrinsic reflectivity image, including: using the multi-scale information P i ′ is input, and after convolution, the 3-channel intrinsic reflectivity image is output, which is denoted as P′.

4. The method for low-light image enhancement based on deep learning according to claim 3, characterized in that: In step 11, the method of constructing a multi-exposure image generation model includes: By randomly generating a series of exposure rates κ = {k1, k2, …, k n }, for the input low exposure image I low The exposure rate is adjusted by pixel, and the generated multi-exposure image cluster is represented as P i =Clip[I low (x,y)·k i ].

5. The method for low-light image enhancement based on deep learning according to claim 3, characterized in that: In step 12, the method of constructing a multi-scale information extraction model includes: The multi-scale information extraction model is a coding-decoding network, which is composed of N convolutions, activation functions, downsampling and N deconvolutions, activation functions, and upsampling connected in sequence.

6. The method for low-light image enhancement based on deep learning according to claim 1, characterized in that: The step 2 comprises: Step 21, construct a brightness weight matrix estimation model, and transform the low exposure image I low As input, the brightness weight matrix W can be obtained through the brightness weight matrix estimation model; Step 22, estimating the reflectivity image weight matrix, includes: performing an inverse operation on the brightness weight matrix W obtained in step 21 to obtain an intrinsic reflectivity image weight matrix Expressed as 7. The method for low-light image enhancement based on deep learning according to claim 6, characterized in that: In step 21, the method of constructing a brightness weight matrix estimation model includes: The brightness weight matrix estimation model is a coding-decoding network, which is composed of M convolutions, activation functions, downsampling and M deconvolutions, activation functions, and upsampling connected in sequence; M is not equal to N.

8. The method for low-light image enhancement based on deep learning according to any one of claims 1 to 7, characterized in that: The mathematical representation of the illumination enhancement model is: Among them, α and β are constants determined by model training. is the predicted normal illumination image.

Images