Low-illumination image enhancement method based on low-light feature prior

Abstract

The invention discloses a low-illumination image enhancement method based on low-light feature prior, and belongs to the technical field of deep learning. A low-light feature map, an attention mechanism and a diffusion model are adopted to enhance a low-illumination image existing in an actual application scene, firstly, a feature sensing module is used for extracting the low-light feature map from the low-light image, and then a convolutional network is used for performing local recovery on a low-light area; and performing global recovery on the extremely dark area in combination with an attention mechanism guided by a low-light feature map, fusing two recovery results to obtain a coarse-grained enhancement result, and finally performing fine-grained detail recovery through a diffusion model to obtain a final enhancement result. When the method provided by the invention is used for enhancing the low-illumination image, the low-light area and the extremely dark area of the image are effectively recovered, artifacts and noise caused by enhancement are greatly reduced, and the visual perception quality of the image is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and specifically designs a low-light image enhancement method based on low-light feature priors. Background Art

[0002] In recent years, as various devices equipped with cameras have become increasingly popular, these devices are not only used to record and share daily life, but also play a key role in fields such as autonomous driving, industrial inspection, medical imaging, and scientific research. However, although camera sensors have made significant technological breakthroughs at the hardware level to provide users with a better image experience, various scenarios in real-world environments, especially under low-light conditions, still lead to unstable performance. For example, occlusions may cast complex shadows on objects, and backlights may form unnecessary dark areas in front of bright backgrounds. The problems caused by low-light environments are not limited to affecting the viewing quality of images. In the field of security monitoring, low-light conditions may cause important details in images captured by surveillance cameras to become blurred, thereby affecting the timely identification and response to potential security threats. In autonomous driving systems, low-light environments may make it difficult for cameras to accurately identify road signs, pedestrians, or obstacles, increasing driving safety risks. Traditional image enhancement techniques such as gamma correction and histogram equalization are mainly based on domain knowledge and statistical characteristics. Although they have improved low-light images with reduced global performance, they are difficult to cover the complexity of real-world environments, especially those dark areas that are spatially non-uniform and nonlinear.

[0003] Deep learning-based methods have been widely used in the field of low-light image enhancement, with the goal of improving the visibility of images while minimizing the distortion of the basic content of the image. For example, the SID network is designed for extremely low-light conditions and can recover clear and moderately bright images from almost completely dark environments. In addition, the CBAM network based on the attention mechanism can focus on key areas of the image, improve the visibility of low-light areas, and maintain image details. However, deep learning models have certain limitations, such as the possibility of over-enhancing specific areas, resulting in increased noise or loss of details. These problems still need further optimization and research to ensure image quality. Summary of the invention

[0004] In order to solve the problems of poor restoration effect of extremely dark areas, residual artifacts and noise in the enhancement results in the current low-light image enhancement, the present invention provides a low-light image enhancement method based on low-light feature priors; the present invention combines low-light feature maps, attention mechanisms and diffusion models to meet the needs of actual application scenarios. By accurately extracting and distinguishing the brightness features in low-light images through low-light feature maps, the method can effectively distinguish low-light areas and extremely dark areas and perform targeted enhancements. Specifically, for relatively bright low-light areas, traditional convolutional networks are used for local enhancement; for extremely dark areas, the attention mechanism guided by the low-light feature map is used for global enhancement. Finally, the two enhancement effects are fused and fine-grained adjustments are made through the diffusion model to ensure that the final enhancement results are significantly improved in brightness, details and overall visual effects to meet the needs of different application scenarios. The technical problem proposed by the present invention is solved as follows:

[0005] Specific content:

[0006] A low-light image enhancement method based on low-light feature priors comprises the following steps:

[0007] Step 1: Obtain the required paired low-light data sets, where each pair of data includes a low-light image and a normal-light image in the same scene, and perform preliminary processing on the images in the data set to divide them into a training set and a test set;

[0008] Step 2, preprocessing the obtained low-light data and performing normalization processing;

[0009] Step 3: Use the convolutional network to extract the brightness features of the image data obtained in step 2, use the VGG feature perception quality loss to constrain the brightness features, and complete the extraction of the low-light feature map;

[0010] Step 4: Input the original data into the convolutional network and the Transformer network respectively. In the convolutional network, the low-light area in the low-light image is restored according to the size of the convolution kernel. In the Transformer network, the low-light feature map extracted in step 3 is used to guide the attention mechanism to focus on the extremely dark area in the image. The global information of the low-light image is used to help restore the extremely dark area and complete the coarse-grained enhancement.

[0011] Step 5, the coarse-grained enhancement result obtained in step 4 is input into the diffusion model to continuously add noise and denoise, and fine-grained enhancement is performed in an iterative manner;

[0012] Step 6: Train the low-light image enhancement network constructed in steps 3, 4, and 5 on the low-light dataset.

[0013] Step 1 of the present invention comprises:

[0014] Step 1-1, obtain a low-light paired data set, including two categories of low light and corresponding normal light, and perform preliminary processing on the data set, delete irrelevant pictures, and crop the pictures into a uniform size;

[0015] Step 1-2: After step 1-1, the data set obtained is divided into a training set and a test set to facilitate subsequent training and testing of the network.

[0016] Step 2 of the present invention comprises:

[0017] Step 2-1, use imageio and opencv to complete the reading of the picture, read the corresponding picture file according to the input file name, file path and picture size, and return the array containing the picture;

[0018] Step 2-2, use shuffle and file name to batch read the data obtained in step 2-1, shuffle its index according to the number of data files, and then complete the data reading according to the file corresponding to the index;

[0019] Step 2-3, call the method in transforms to process the data and normalize the data read in step 2-2.

[0020] Step 3 of the present invention comprises:

[0021] Step 3-1: extract the low-light feature map through the feature perception module, and use the VGG feature perception quality loss to constrain the brightness feature. The feature perception module is composed of a convolutional network. The specific process is as follows:

[0022] I fea =Conv(I lq )

[0023]

[0024] Among them, I lq and I fea represents the input low-light image and the extracted low-light feature map, Conv represents the convolutional network, including four convolutional blocks, each of which consists of a padding layer, a 3x3 convolutional layer, a normalization layer and an activation layer, and Φ represents the VGG network;

[0025] Step 3-2: For the low-light feature map extracted in step 3-1, in order to reduce the complexity of the model, the low-light feature map is grayed out and converted into a single-channel feature attention map. The conversion process is as follows:

[0026] T=0.299R+0.587G+0.114B

[0027] Among them, T represents the grayscale single-channel feature attention map, and R, G, and B represent the red, green, and blue channels of the low-light feature map, respectively.

[0028] Step 4 of the present invention comprises:

[0029] Step 4-1, before image enhancement, the present invention adopts the UNet network framework to first downsample and compress the image data into a latent space and extract the feature vector of the input image;

[0030] Step 4-2, using a convolutional network to restore the low-light area for the feature vector extracted in step 4-1. For a low-light area with certain information, the convolutional network can effectively restore the brightness information of the local area;

[0031] Step 4-3: Use the Transformer network to restore the extremely dark area for the feature vector extracted in step 4-1, and use the feature attention map extracted in step 3 to guide the attention mechanism in the Transformer network, so that the network can use global information to restore the extremely dark area that lacks information;

[0032] Step 4-4, for the local features and global features recovered in step 4-2 and step 4-3, the present invention interpolates and fuses the two feature maps according to the feature attention map extracted in step 3. The above process can be expressed as follows:

[0033] F local =Conv(F)

[0034] F global =Trans(F)

[0035] T′=Normalize(T)

[0036] F final =F local ·T′+F global ·(1-T′)

[0037] Among them, F represents the feature vector extracted in step 4-1, Conv and Trans represent the convolutional network and Transformer network respectively, and F local and F global Represent the restored local features and global features respectively, T represents the feature attention map extracted in step 3, T′ represents the normalized feature map, and F final It represents the final fused feature map;

[0038] Step 4-5: The fused feature map obtained in step 4-4 is restored to the image space through upsampling of the UNet network to obtain a coarse-grained enhancement result.

[0039] Step 5 of the present invention comprises:

[0040] Step 5-1, sample random noise ∈, ∈~N(0, I) from a standard normal distribution, and sample time step t from a uniform distribution from 1 to T, t~Uniform{1,...,T};

[0041] Step 5-2, input the coarse-grained result obtained in step 4 into the diffusion model for Gaussian diffusion noise addition, until the result is infinitely close to the Gaussian distribution. In the present invention, a diffusion model based on block decomposition is adopted. Specifically, before image noise addition, the image is first decomposed into blocks of a specified size. Block-level processing allows the diffusion process to more finely target the detailed texture features of the image, thereby improving the image quality at a fine granularity. The specific process is as follows:

[0042]

[0043]

[0044]

[0045] Among them, X0 represents the coarse-grained result of step 4, represents the i blocks without noise after X0 cropping, represents the result of decomposing into i blocks and adding noise after t steps, a s ~(0,1) is a hyperparameter that controls the noise injection rate;

[0046] Step 5-3: Input the noise-added result in step 5-2 into the noise prediction network to perform reverse iterative denoising, reassemble the denoised result to restore the original image size, and obtain the final fine-grained enhancement result. The reverse process is as follows:

[0047] β t =1-α t

[0048]

[0049]

[0050] in, represents the enhanced result, θ represents the noise prediction network, ∈ θ represents the predicted noise, z is the random noise sampled from the standard normal distribution, σ t To fix the variance, Concat represents a concatenation operation, which concatenates the decomposed blocks back to their original size to obtain the final enhanced result.

[0051] The present invention proposes a low-light image enhancement method based on low-light feature priors, which has the following advantages and beneficial effects compared with the prior art:

[0052] The present invention combines the attention mechanism and the diffusion model, and can perform low-light image enhancement based on the collected data; compared with the existing methods, the method proposed in the present invention utilizes the global information aggregation capability of the Transformer attention mechanism to restore the extremely dark areas during image enhancement, effectively performs targeted enhancement on the low-light areas and extremely dark areas with uneven illumination in the low-light images, and reduces the influence of noise and artifacts in the enhancement results through the refinement of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The overall flow chart of the method of the present invention is

[0054] Figure 2 This is a schematic diagram of the overall network structure

[0055] Figure 3 This is a low light feature diagram

[0056] Figure 4 This is a schematic diagram of the diffusion model.

[0057] Figure 5 This is part of the dataset image DETAILED DESCRIPTION

[0058] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0059] The present invention is implemented including:

[0060] The overall process of the method proposed by the present invention is as follows Figure 1 As shown, the specific construction steps of low-light image enhancement in the embodiment of the present invention are as follows:

[0061] Step 1: Obtain the required paired low-light data sets, where each pair of data includes a low-light image and a normal-light image in the same scene, and perform preliminary processing on the images in the data set to divide them into a training set and a test set;

[0062] Step 2, preprocessing the obtained low-light data and performing normalization processing;

[0063] Step 3: Use the convolutional network to extract the brightness features of the image data obtained in step 2, use the VGG feature perception quality loss to constrain the brightness features, and complete the extraction of the low-light feature map;

[0064] Step 4: Input the original data into the convolutional network and the Transformer network respectively. In the convolutional network, the low-light area in the low-light image is restored according to the size of the convolution kernel. In the Transformer network, the low-light feature map extracted in step 3 is used to guide the attention mechanism to focus on the extremely dark area in the image. The global information of the low-light image is used to help restore the extremely dark area and complete the coarse-grained enhancement.

[0065] Step 5, the coarse-grained enhancement result obtained in step 4 is input into the diffusion model to continuously add noise and denoise, and fine-grained enhancement is performed in an iterative manner;

[0066] Step 6: Train the low-light image enhancement network built in steps 3, 4, and 5 on the low-light dataset.

[0067] Step 1 of the present invention comprises:

[0068] Step 1-1, obtain a low-light paired data set, including two categories of low light and corresponding normal light, and perform preliminary processing on the data set, delete irrelevant pictures, and crop the pictures into a uniform size;

[0069] Step 1-2: After step 1-1, the data set obtained is divided into a training set and a test set to facilitate subsequent training and testing of the network.

[0070] Step 2 of the present invention comprises:

[0071] Step 2-1, use imageio and opencv to complete the reading of the picture, read the corresponding picture file according to the input file name, file path and picture size, and return the array containing the picture;

[0072] Step 2-2, use shuffle and file name to batch read the data obtained in step 2-1, shuffle its index according to the number of data files, and then complete the data reading according to the file corresponding to the index;

[0073] Step 2-3, call the method in transforms to process the data and normalize the data read in step 2-2.

[0074] Step 3 of the present invention comprises:

[0075] Step 3-1: extract the low-light feature map through the feature perception module, and use the VGG feature perception quality loss to constrain the brightness feature. The feature perception module is composed of a convolutional network. The specific process is as follows:

[0076] I fea =Conv(I lq )

[0077]

[0078] Among them, I lq and I fea represents the input low-light image and the extracted low-light feature map, Conv represents the convolutional network, including four convolutional blocks, each of which consists of a padding layer, a 3x3 convolutional layer, a normalization layer and an activation layer, and Φ represents the VGG network;

[0079] Step 3-2: For the low-light feature map extracted in step 3-1, in order to reduce the complexity of the model, the low-light feature map is grayed out and converted into a single-channel feature attention map. The conversion process is as follows:

[0080] T=0.299R+0.587G+0.114B

[0081] Among them, T represents the grayscale single-channel feature attention map, and R, G, and B represent the red, green, and blue channels of the low-light feature map, respectively.

[0082] Step 4 of the present invention comprises:

[0083] Step 4-1, before image enhancement, the present invention adopts the UNet network framework to first downsample and compress the image data into a latent space and extract the feature vector of the input image;

[0084] Step 4-2, using a convolutional network to restore the low-light area for the feature vector extracted in step 4-1. For a low-light area with certain information, the convolutional network can effectively restore the brightness information of the local area;

[0085] Step 4-3: Use the Transformer network to restore the extremely dark area for the feature vector extracted in step 4-1, and use the feature attention map extracted in step 3 to guide the attention mechanism in the Transformer network, so that the network can use global information to restore the extremely dark area that lacks information;

[0086] Step 4-4, for the local features and global features recovered in step 4-2 and step 4-3, the present invention interpolates and fuses the two feature maps according to the feature attention map extracted in step 3. The above process can be expressed as follows:

[0087] F local =Conv(F)

[0088] F global =Trans(F)

[0089] T′=Normalize(T)

[0090] F final =F local ·T′+F global ·(1-T′)

[0091] Among them, F represents the feature vector extracted in step 4-1, Conv and Trans represent the convolutional network and Transformer network respectively, and F local and F global Represent the restored local features and global features respectively, T represents the feature attention map extracted in step 3, T′ represents the normalized feature map, and F final It represents the final fused feature map;

[0092] Step 4-5: The fused feature map obtained in step 4-4 is restored to the image space through upsampling of the UNet network to obtain a coarse-grained enhancement result.

[0093] Step 5 of the present invention comprises:

[0094] Step 5-1, sample random noise ∈, ∈~N(O, I) from a standard normal distribution, and sample time step t from a uniform distribution from 1 to T, t~Uniform{1,...,T};

[0095] Step 5-2, input the coarse-grained result obtained in step 4 into the diffusion model for Gaussian diffusion noise addition, until the result is infinitely close to the Gaussian distribution. In the present invention, a diffusion model based on block decomposition is adopted. Specifically, before image noise addition, the image is first decomposed into blocks of a specified size. Block-level processing allows the diffusion process to more finely target the detailed texture features of the image, thereby improving the image quality at a fine granularity. The specific process is as follows:

[0096]

[0097]

[0098]

[0099] Among them, X0 represents the coarse-grained result of step 4, represents the i blocks without noise after X0 cropping, represents the result of decomposing into i blocks and adding noise after t steps, a s ~(0,1) is a hyperparameter that controls the noise injection rate;

[0100] Step 5-3: Input the noise-added result in step 5-2 into the noise prediction network to perform reverse iterative denoising, reassemble the denoised result to restore the original image size, and obtain the final fine-grained enhancement result. The reverse process is as follows:

[0101] β t =1-α t

[0102]

[0103]

[0104] in, represents the enhanced result, θ represents the noise prediction network, ∈ θ represents the predicted noise, z is the random noise sampled from the standard normal distribution, σ t To fix the variance, Concat represents a concatenation operation, which concatenates the decomposed blocks back to their original size to obtain the final enhanced result.

[0105] The present invention combines the attention mechanism and the diffusion model, and can perform low-light image enhancement based on the collected data; compared with the existing methods, the method proposed in the present invention utilizes the global information aggregation capability of the Transformer attention mechanism to restore the extremely dark areas during image enhancement, effectively performs targeted enhancement on the low-light areas and extremely dark areas with uneven illumination in the low-light images, and reduces the influence of noise and artifacts in the enhancement results through the refinement of the diffusion model.

[0106] The present invention provides a low-light image enhancement method based on low-light feature priors. The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A low-light image enhancement method based on low-light feature prior, characterized in that: The following steps are involved: Step 1: Obtain the required paired low-light data sets, where each pair of data includes a low-light image and a normal-light image in the same scene, and perform preliminary processing on the images in the data set to divide them into a training set and a test set; Step 2, preprocessing the obtained low-light data and performing normalization processing; Step 3: Use the feature perception module to extract the brightness feature of the image data obtained in step 2, use the VGG feature perception quality loss to constrain the brightness feature, and complete the extraction of the low-light feature map; Step 4: Input the original data into the convolutional network and the Transformer network respectively. In the convolutional network, the low-light area in the low-light image is restored according to the size of the convolution kernel. In the Transformer network, the low-light feature map extracted in step 3 is used to guide the attention mechanism to focus on the extremely dark area in the image. The global information of the low-light image is used to help restore the extremely dark area and complete the coarse-grained enhancement. Step 5, the coarse-grained enhancement result obtained in step 4 is input into the diffusion model to continuously add noise and denoise, and fine-grained enhancement is performed in an iterative manner; Step 6: Train the low-light image enhancement network built in steps 3, 4, and 5 on the low-light dataset.

2. The method according to claim 1, characterized in that Step 2 includes: Step 2-1, use imageio and opencv to complete the reading of the image, read the corresponding image file according to the input file name, file path and image size, and return an array containing the image; Step 2-2, use shuffle and file name to batch read the data obtained in step 2-1, shuffle its index according to the number of data files, and then complete the data reading according to the file corresponding to the index; Step 2-3, call the method in transforms to process the data and normalize the data read in step 2-2.

3. The method according to claim 1, characterized in that Step 3 includes: Step 3-1: extract the low-light feature map through the feature perception module, and use the VGG feature perception quality loss to constrain the brightness feature. The feature perception module is composed of a convolutional network. The specific process is as follows: I fea =Conv(I lq ) Among them, I lq and I fea represents the input low-light image and the extracted low-light feature map, Conv represents the convolutional network, including four convolutional blocks, each of which consists of a padding layer, a 3x3 convolutional layer, a normalization layer and an activation layer, Φ represents the VGG network, is the final calculated loss value; Step 3-2: For the low-light feature map extracted in step 3-1, in order to reduce the complexity of the model, the low-light feature map is grayed out and converted into a single-channel feature attention map. The conversion process is as follows: T=0.299R+0.587G+0.114B Among them, T represents the grayscale single-channel feature attention map, and R, G, and B represent the red, green, and blue channels of the low-light feature map, respectively.

4. The method according to claim 1, characterized in that Step 4 includes: Step 4-1, before image enhancement, the present invention adopts the UNet network framework to first downsample and compress the image data into a latent space and extract the feature vector of the input image; Step 4-2, using a convolutional network to restore the low-light area for the feature vector extracted in step 4-1. For a low-light area with certain information, the convolutional network can effectively restore the brightness information of the local area; Step 4-3: Use the Transformer network to restore the extremely dark area for the feature vector extracted in step 4-1, and use the feature attention map extracted in step 3 to guide the attention mechanism in the Transformer network, so that the network can use global information to restore the extremely dark area that lacks information; Step 4-4, for the local features and global features recovered in step 4-2 and step 4-3, the present invention interpolates and fuses the two feature maps according to the feature attention map extracted in step 3. The above process can be expressed as follows: F local =Conv(F) F global =Trans(F) T′=Normalize(T) F final =F local ·T′+F global ·(1-T′) Among them, F represents the feature vector extracted in step 4-1, Conv and Trans represent the convolutional network and Transformer network respectively, and F local and F global They represent the restored local features and global features respectively, T represents the feature attention map extracted in step 3, Normalize represents the normalization operation, T′ represents the normalized feature map, and F final It represents the final fused feature map; Step 4-5: The fused feature map obtained in step 4-4 is restored to the image space through upsampling of the UNet network to obtain a coarse-grained enhancement result.

5. The method according to claim 1, characterized in that Step 5 includes: Step 5-1, sample random noise ∈, ∈~N(0, I) from a standard normal distribution, and sample time step t from a uniform distribution from 1 to T, t~Uniform{1,...,T}; Step 5-2, input the coarse-grained result obtained in step 4 into the diffusion model for Gaussian diffusion noise addition, until the result is infinitely close to the Gaussian distribution. In the present invention, a diffusion model based on block decomposition is adopted. Specifically, before image noise addition, the image is first decomposed into blocks of a specified size. Block-level processing allows the diffusion process to more finely target the detailed texture features of the image, thereby improving the image quality at a fine granularity. The specific process is as follows: Among them, X0 represents the coarse-grained enhancement result of step 4, represents the i blocks without noise after X0 is cropped, t represents the time step of the diffusion process, s ranges from 0 to t and represents the current time step, a s ~(0,1) is a hyperparameter that controls the noise injection rate, Represents a S The cumulative result of represents the result of decomposing into i blocks and then adding noise for t steps; Step 5-3: Input the noise-added result in step 5-2 into the noise prediction network to perform reverse iterative denoising, reassemble the denoised result to restore the original image size, and obtain the final fine-grained enhancement result. The reverse process is as follows: β t ＝1-α t in, and t are the same as in step 5-2, represents the enhanced result, θ represents the noise prediction network, ∈ θ represents the noise predicted by the network, z is the random noise sampled from the standard normal distribution, σ t is the preset variance, Concat represents the splicing operation, which splices the decomposed blocks back to their original size to obtain the final enhanced result.

Images