A Low-Light Image Enhancement Method Based on Low-Light Feature Prior

Abstract

This invention discloses a low-light image enhancement method based on low-light feature priors, belonging to the field of deep learning technology. This invention employs low-light feature maps, an attention mechanism, and a diffusion model to enhance low-light images present in real-world application scenarios. First, a feature perception module extracts low-light feature maps from the low-light image. Then, a convolutional network is used to locally restore the low-light regions. Next, an attention mechanism guided by the low-light feature maps is combined to globally restore the extremely dark regions. The two restoration results are then fused to obtain a coarse-grained enhancement result. Finally, a diffusion model is used for fine-grained detail restoration to obtain the final enhancement result. The proposed method effectively restores both low-light and extremely dark regions of the image during low-light image enhancement, significantly reducing artifacts and noise introduced after enhancement, and substantially improving the visual perception quality of the image.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and specifically designs a low-light image enhancement method based on low-light feature prior. Background Technology

[0002] In recent years, with the increasing prevalence of various devices equipped with cameras, these devices are not only used for recording and sharing daily life, but also play a crucial role in fields such as autonomous driving, industrial inspection, medical imaging, and scientific research. However, although camera sensors have achieved significant technological breakthroughs at the hardware level in order to provide users with a better image experience, various scenarios in the real environment, especially under low-light conditions, still lead to performance instability. For example, occlusions may cast complex shadows on objects, while backlighting may create unnecessary dark areas in front of a bright background. The problems caused by low-light environments are not limited to affecting the visual appeal of images. In the field of security monitoring, low-light conditions may cause important details in images captured by surveillance cameras to become blurred, thereby affecting the timely identification and response to potential security threats. In autonomous driving systems, low-light environments may make it difficult for cameras to accurately identify road signs, pedestrians, or obstacles, increasing driving safety risks. Traditional image enhancement techniques such as gamma correction and histogram equalization are mainly based on domain knowledge and statistical characteristics. Although they improve the performance of low-light images with overall performance degradation, they are difficult to cover the complexity of real-world environments, especially those spatially non-uniform and non-linear dark areas.

[0003] Deep learning-based methods have been widely used in low-light image enhancement, aiming to improve image visibility while minimizing severe distortion of the image's basic content. For example, the SID network is designed for extremely low-light conditions and can recover clear and moderately bright images from almost completely dark environments. Furthermore, the CBAM network, based on an attention mechanism, can focus on key regions of an image, improving the visibility of low-light areas while preserving image details. However, deep learning models have certain limitations, such as the potential for over-enhancing specific regions, leading to increased noise or loss of detail. These issues require further optimization and research to ensure image quality. Summary of the Invention

[0004] To address the problems of poor recovery of extremely dark areas, artifacts, and noise in current low-light image enhancement methods, this invention provides a low-light image enhancement method based on low-light feature priors. This invention combines low-light feature maps, an attention mechanism, and a diffusion model to adapt to the needs of practical application scenarios. By accurately extracting and distinguishing brightness features in low-light images using low-light feature maps, this method can effectively differentiate between low-light and extremely dark areas and perform targeted enhancement. Specifically, for relatively bright low-light areas, a traditional convolutional network is used for local enhancement; while for extremely dark areas, an attention mechanism guided by low-light feature maps is used for global enhancement. Finally, these two enhancement effects are fused and fine-grained adjustments are made using a diffusion model to ensure a significant improvement in brightness, detail, and overall visual effect in the final enhancement result, meeting the needs of different application scenarios. The technical problem proposed by this invention is solved as follows:

[0005] Specific content:

[0006] A low-light image enhancement method based on low-light feature priors includes the following steps:

[0007] Step 1: Obtain the required paired low-light datasets. Each pair of data includes a low-light image and a normal-light image from the same scene. Perform preliminary processing on the images in the dataset to divide them into training and test sets. This process consists of the following two steps.

[0008] Step 1-1: Obtain a low-light paired dataset, which includes two categories: low light and corresponding normal light. Perform preliminary processing on the dataset by deleting irrelevant images and cropping the images to a uniform size.

[0009] Step 1-2: The dataset obtained after processing in step 1-1 is divided into a training set and a test set, which facilitates the subsequent training and testing of the network.

[0010] Step 2: Preprocess the obtained low-light data and perform normalization; this process consists of the following three steps.

[0011] Step 2-1: Use imageio and opencv to read the image. Read the corresponding image file according to the input file name, file path and image size, and return an array containing the images.

[0012] Step 2-2: Use shuffle and filename to read the data obtained in step 2-1 in batches. Shuffle the index according to the number of data files, and then read the data according to the file corresponding to the index.

[0013] Step 2-3: Call the methods in transforms to process the data and normalize the data read in step 2-2.

[0014] Step 3: Extract the brightness features of the image data obtained in Step 2 using the feature perception module, constrain the brightness features using VGG feature perception quality loss to obtain a low-light feature map, and simultaneously normalize the low-light feature map to obtain a feature attention map; this process is divided into the following two steps.

[0015] Step 3-1: Extract low-light feature maps using the feature perception module, and constrain the brightness features using VGG feature perception quality loss. The feature perception module is composed of a convolutional network, and the specific process is as follows:

[0016] I fea =Conv(I lq )

[0017]

[0018] Among them, I lq and I fea Represents the input low-light image and the extracted low-light feature map. Conv represents the convolutional network, which includes four convolutional blocks. Each convolutional block consists of a padding layer, a 3x3 convolutional layer, a normalization layer, and an activation layer. Φ represents the VGG network.

[0019] Step 3-2: For the low-light feature map extracted in Step 3-1, in order to reduce model complexity, the low-light feature map is converted to grayscale and transformed into a single-channel feature attention map. The conversion process is as follows:

[0020] T = 0.299R + 0.587G + 0.114B

[0021] Where T represents the grayscale single-channel feature attention map, and R, G, and B represent the red, green, and blue channels of the low-light feature map, respectively.

[0022] Step 4: Input the original data into the convolutional network and the Transformer network respectively. The convolutional network produces local enhancement features, and the Transformer network produces global enhancement features. The feature attention map from step 3 is used as the interpolation weight to fuse the local and global enhancement features to obtain a coarse-grained enhancement result. This process consists of the following 5 steps.

[0023] Step 4-1: Before performing image enhancement, this invention uses the UNet network framework to first downsample and compress the image data into the latent space and extract the feature vector of the input image.

[0024] Step 4-2: Use a convolutional network to recover the low-light region from the feature vector extracted in step 4-1. For low-light regions with certain information, the convolutional network can effectively recover the brightness information of the local region.

[0025] Step 4-3: The feature vectors extracted in Step 4-1 are used to restore the extremely dark areas using the Transformer network. The attention map extracted in Step 3 is used to guide the attention mechanism in the Transformer network, so that the network can use global information to restore the extremely dark areas lacking information.

[0026] Step 4-4: For the local and global features recovered in steps 4-2 and 4-3, this invention performs interpolation and fusion on the two feature maps based on the feature attention map extracted in step 3. The above process can be represented as follows:

[0027] F local =Conv(F)

[0028] F global =Trans(F)

[0029] T′=Normalize(T)

[0030] F final =F local ·T′+F global ·(1-T′)

[0031] Where F represents the feature vector extracted in step 4-1, and Conv and Trans represent the convolutional network and the Transformer network, respectively. local and F global These represent the recovered local and global features, respectively. T represents the feature attention map extracted in step 3, and T′ represents the normalized feature map. F final This represents the final fused feature map;

[0032] Steps 4-5: The fused feature maps obtained in step 4-4 are upsampled by the UNet network to restore the image space, resulting in coarse-grained enhancement results.

[0033] Step 5: Input the coarse-grained enhancement result obtained in Step 4 into the diffusion model and continuously add and remove noise, performing fine-grained enhancement through iteration; this process is divided into the following 3 steps;

[0034] Step 5-1: Sample random noise ∈ from the standard normal distribution, ∈ ~ N(0, I), and sample time step t from the uniform distribution from 1 to T, t ~ Uniform{1, ..., T};

[0035] Step 5-2: Input the coarse-grained result obtained in Step 4 into the diffusion model for Gaussian diffusion noise addition until the result is infinitely close to a Gaussian distribution. In this invention, a block-based diffusion model is used. Specifically, before adding noise to the image, the image is decomposed into blocks of a specified size. Block-level processing allows the diffusion process to target the detailed texture features of the image more precisely, thereby improving image quality at a fine-grained level. The specific process is as follows:

[0036]

[0037]

[0038]

[0039] Where X0 represents the result of the coarse-grained step 4. This represents the i blocks that were not denoised after being clipped from X0. This represents the result of decomposing the data into i blocks and adding noise in t steps, where a s ~(0,1) is a hyperparameter controlling the noise injection rate;

[0040] Step 5-3: The noise-added result from Step 5-2 is input into the noise prediction network for inverse iterative denoising. The denoised result is then reassembled to restore the original image size, yielding the final fine-grained enhancement result. The inverse process is shown below:

[0041] β t =1-α t

[0042]

[0043]

[0044] in, This represents the enhanced result, θ represents the noise prediction network, ∈ θ This represents the predicted noise, where z is the random noise sampled from the standard normal distribution, and σ is the noise level. t With a fixed variance, Concat represents a splicing operation that reassembles the decomposed blocks back to their original size to obtain the final enhanced result.

[0045] Step 6: Train the low-light image enhancement network built in steps 3, 4, and 5 on the low-light dataset.

[0046] This invention proposes a low-light image enhancement method based on low-light feature priors, which has the following advantages and beneficial effects compared with existing technologies:

[0047] This invention combines an attention mechanism and a diffusion model to enhance low-light images based on collected data. Compared with existing methods, the proposed method utilizes the global information aggregation capability of the Transformer attention mechanism to recover extremely dark areas during image enhancement, effectively enhancing low-light and extremely dark areas with uneven illumination in low-light images. Furthermore, the diffusion model is used to refine the enhancement results and reduce the impact of noise and artifacts. Attached Figure Description

[0048] Figure 1 This is the overall flowchart of the method of the present invention.

[0049] Figure 2 This is a schematic diagram of the overall network structure.

[0050] Figure 3 This is a schematic diagram of low-light characteristics.

[0051] Figure 4 This is a schematic diagram of the diffusion model.

[0052] Figure 5 These are images from a portion of the dataset. Detailed Implementation

[0053] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0054] The present invention includes:

[0055] The overall process of the method proposed in this invention is as follows: Figure 1 As shown, the specific construction steps for low-light image enhancement in this embodiment of the invention are as follows:

[0056] The extracted feature vectors are used to recover low-light areas using a convolutional network. For low-light areas with certain information, the convolutional network can effectively recover the brightness information of the local area.

[0057] Step 4-3: The feature vectors extracted in Step 4-1 are used to restore the extremely dark areas using the Transformer network. The attention map extracted in Step 3 is used to guide the attention mechanism in the Transformer network, so that the network can use global information to restore the extremely dark areas lacking information.

[0058] Step 4-4: For the local and global features recovered in steps 4-2 and 4-3, this invention performs interpolation and fusion on the two feature maps based on the feature attention map extracted in step 3. The above process can be represented as follows:

[0059] Flocal =Conv(F)

[0060] F global =Trans(F)

[0061] T′=Normalize(T)

[0062] F final =F local ·T′+F global ·(1-T′)

[0063] Where F represents the feature vector extracted in step 4-1, and Conv and Trans represent the convolutional network and the Transformer network, respectively. local and F global These represent the recovered local and global features, respectively. T represents the feature attention map extracted in step 3. Normalize represents the normalization operation, and T′ represents the feature map after normalization. F final This represents the final fused feature map;

[0064] Steps 4-5: The fused feature maps obtained in step 4-4 are upsampled by the UNet network to restore the image space, resulting in coarse-grained enhancement results.

[0065] Step 5: Input the coarse-grained enhancement results obtained in steps 4-5 into the diffusion model and continuously add and remove noise, performing fine-grained enhancement through iteration; this process consists of the following 3 steps.

[0066] Step 5-1: Sample random noise ∈ from the standard normal distribution, ∈ ~ N(0, I), and sample time step t from the uniform distribution from 1 to T, t ~ Uniform{1, ..., T};

[0067] Step 5-2: Input the coarse-grained result obtained in Step 4 into the diffusion model for Gaussian diffusion noise addition until the result is infinitely close to a Gaussian distribution. In this invention, a block-based diffusion model is used. Specifically, before adding noise to the image, the image is decomposed into blocks of a specified size. Block-level processing allows the diffusion process to target the detailed texture features of the image more precisely, thereby improving image quality at a fine-grained level. The specific process is as follows:

[0068]

[0069]

[0070]

[0071] Where X0 represents the coarse-grained enhancement result of step 4. Let represent the i blocks without noise after X0 clipping, t represent the time step of the diffusion process, s range from 0 to t represents the current time step, and a s ~(0,1) is a hyperparameter controlling the noise injection rate. It represents a s The result of the cumulative product, This represents the result of decomposing the data into i blocks and then adding noise in t steps.

[0072] Step 5-3: The noise-added result from Step 5-2 is input into the noise prediction network for inverse iterative denoising. The denoised result is then reassembled to restore the original image size, yielding the final fine-grained enhancement result. The inverse process is shown below:

[0073] β t =1-α t

[0074]

[0075]

[0076] in, And t is the same as in step 5-2, This represents the enhanced result, θ represents the noise prediction network, ∈ θ This represents the noise predicted by the network, where z is random noise sampled from a standard normal distribution, and σ t With a preset variance, Concat represents a splicing operation that reassembles the decomposed blocks back to their original size to obtain the final enhanced result.

[0077] This invention combines an attention mechanism and a diffusion model to enhance low-light images based on collected data. Compared with existing methods, the proposed method utilizes the global information aggregation capability of the Transformer attention mechanism to recover extremely dark areas during image enhancement, effectively enhancing low-light and extremely dark areas with uneven illumination in low-light images. Furthermore, the diffusion model is used to refine the enhancement results and reduce the impact of noise and artifacts.

[0078] This invention provides a low-light image enhancement method based on low-light feature priors. The above description is only an embodiment of this invention and does not limit the patent scope of this invention. Any equivalent structural or procedural transformations made using the content of this specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this invention.

Claims

1. A low-light image enhancement method based on low-light feature prior, characterized in that, Includes the following steps: Step 1: Obtain the required paired low-light datasets. Each pair of data includes a low-light image and a normal-light image from the same scene. Perform preliminary processing on the images in the dataset to divide them into training and test sets. Step 2: Preprocess the low-light data from Step 1 to match the model input, and then perform normalization processing; the process consists of the following 3 steps. Step 2-1: Use imageio and opencv to read the image. Read the corresponding image file according to the input file name, file path and image size, and return an array containing the images. Step 2-2: Use shuffle and filename to read the data obtained in step 2-1 in batches. Shuffle the index according to the number of data files, and then read the data according to the file corresponding to the index. Step 2-3: Call the methods in transforms to process the data and normalize the data read in step 2-2; Step 3: Extract the brightness features of the image data obtained in Step 2 using the feature perception module, constrain the brightness features using VGG feature perception quality loss to obtain a low-light feature map, and simultaneously normalize the low-light feature map to obtain a feature attention map; this process is divided into the following two steps. Step 3-1: Extract low-light feature maps using the feature perception module, and constrain the brightness features using VGG feature perception quality loss. The feature perception module is composed of a convolutional network, and the specific process is as follows: I fea =Conv(I lq ) Among them, I lq and I fea This represents the input low-light image and the extracted low-light feature map. Conv represents the convolutional network, which consists of four convolutional blocks. Each convolutional block consists of a padding layer, a 3x3 convolutional layer, a normalization layer, and an activation layer. Φ represents the VGG network. This is the final calculated loss value; Step 3-2: For the low-light feature map extracted in Step 3-1, in order to reduce model complexity, the low-light feature map is converted to grayscale and transformed into a single-channel feature attention map. The conversion process is as follows: T = 0.299R + 0.587G + 0.114B Where T represents the grayscale single-channel feature attention map, and R, G, and B represent the red, green, and blue channels of the low-light feature map, respectively. Step 4: Input the original data into the convolutional network and the Transformer network respectively. The convolutional network produces local enhancement features, and the Transformer network produces global enhancement features. The feature attention map from step 3 is used as the interpolation weight to fuse the local and global enhancement features to obtain a coarse-grained enhancement result. This process consists of the following 5 steps. Step 4-1: Before performing image enhancement, this invention uses the UNet network framework to first downsample and compress the image data into the latent space and extract the feature vector of the input image. Step 4-2: Use a convolutional network to recover the low-light region from the feature vector extracted in step 4-1. For low-light regions with certain information, the convolutional network can effectively recover the brightness information of the local region. Step 4-3: The feature vectors extracted in Step 4-1 are used to restore the extremely dark areas using the Transformer network. The attention map extracted in Step 3 is used to guide the attention mechanism in the Transformer network, so that the network can use global information to restore the extremely dark areas lacking information. Step 4-4: For the local and global features recovered in steps 4-2 and 4-3, this invention performs interpolation and fusion on the two feature maps based on the feature attention map extracted in step 3. The above process can be represented as follows: F local =Conv(F) F global =Trans(F) T′=Normalize(T) F final =F local ·T′+F global ·(1-T′) Where F represents the feature vector extracted in step 4-1, and Conv and Trans represent the convolutional network and the Transformer network, respectively. local and F global These represent the recovered local and global features, respectively. T represents the feature attention map extracted in step 3. Normalize represents the normalization operation, and T′ represents the feature map after normalization. F final This represents the final fused feature map; Steps 4-5: The fused feature maps obtained in step 4-4 are upsampled by the UNet network to restore the image space, resulting in coarse-grained enhancement results. Step 5: Input the coarse-grained enhancement results obtained in steps 4-5 into the diffusion model and continuously add and remove noise, performing fine-grained enhancement through iteration; this process consists of the following 3 steps. Step 5-1: Sample random noise ∈ from the standard normal distribution, ∈ ~ N(0, I), and sample time step t from the uniform distribution from 1 to T, t ~ Uniform{1, ..., T}; Step 5-2: Input the coarse-grained result obtained in Step 4 into the diffusion model for Gaussian diffusion noise addition until the result is infinitely close to a Gaussian distribution. In this invention, a block-based diffusion model is used. Specifically, before adding noise to the image, the image is decomposed into blocks of a specified size. Block-level processing allows the diffusion process to target the detailed texture features of the image more precisely, thereby improving image quality at a fine-grained level. The specific process is as follows: Where X0 represents the coarse-grained enhancement result of step 4. Let represent the i blocks without noise after X0 clipping, t represent the time step of the diffusion process, s range from 0 to t represents the current time step, and a s ~(0,1) is a hyperparameter controlling the noise injection rate. It represents a s The result of the cumulative product, This represents the result of decomposing the data into i blocks and then adding noise in t steps. Step 5-3: The noise-added result from Step 5-2 is input into the noise prediction network for inverse iterative denoising. The denoised result is then reassembled to restore the original image size, yielding the final fine-grained enhancement result. The inverse process is shown below: β t ＝1-α t in, And t is the same as in step 5-2, This represents the enhanced result, θ represents the noise prediction network, ∈ θ This represents the noise predicted by the network, where z is random noise sampled from a standard normal distribution, and σ t With a preset variance, Concat represents a splicing operation that reassembles the decomposed blocks back to their original size to obtain the final enhanced result; Step 6: Train the low-light image enhancement network built in steps 3, 4, and 5 on the low-light dataset.

Images