Progressive generative adversarial network-based low-light image enhancement method and device
By using a progressive generative adversarial network-based approach, combined with a Retinex model and an image brightness region perception module, reflectance maps and saliency maps are generated, solving the problem of uneven illumination in low-light image enhancement and achieving high-quality image enhancement results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2023-11-30
- Publication Date
- 2026-07-31
AI Technical Summary
Existing low-light image enhancement methods are inadequate in handling dark areas after image enhancement and do not make full use of prior data, resulting in uneven illumination and affecting image quality.
A progressive generative adversarial network-based approach is adopted, which decomposes the image through a Retinex model and an image brightness region perception module, generates a reflectance map by combining a signal-driven model, and generates a saliency map and enhances the image through a saliency network and an attention-inspired model to ensure illumination uniformity.
It achieves enhanced uniform illumination, provides optimal subjective performance, and improves image quality.
Smart Images
Figure CN117689550B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more particularly to a low-light image enhancement method based on progressive generative adversarial networks (PGNs), and a low-light image enhancement apparatus based on PGNs. Background Technology
[0002] With the increasing prevalence of image acquisition equipment, low-light enhancement technology plays an increasingly important role in image processing. Due to unavoidable extreme environments during image acquisition, such as backlighting and nighttime conditions, the acquired images may be underexposed due to lighting and viewing angle issues, making the image content difficult to recognize and affecting visual quality. Therefore, how to enhance low-light images to obtain corresponding normally lit images is a crucial research area. Low-light enhancement technology refers to enhancing the illumination of an image while preserving its original content, turning dark areas into bright areas and clearly displaying the image content. Low-light enhancement technology is widely used in tasks such as target recognition, autonomous driving, and face detection, facilitating image and video data acquisition, processing, and analysis, thereby further improving data utilization.
[0003] Low-light image enhancement has been a focus of researchers for decades. Traditional low-light enhancement algorithms can be divided into histogram equalization (HE) based methods and Retinex theory based methods. HE-based methods make dark images visible by stretching the dynamic range of the image and applying several constraints. This method has more edge information and constraints, improving the local adaptability of the enhancement process. However, the adjustment of local regions is still not flexible enough, leading to glare, underexposure, and noise amplification. The Retinex model can decompose the image into reflection and illumination components, and combine the two components after image decomposition to complete low-light enhancement. However, the performance of this method is closely related to previously used methods, producing unsatisfactory results on a wide range of test data.
[0004] In recent years, deep learning-based low-light enhancement methods have gained popularity among researchers due to their superior performance and flexibility. Deep learning methods were initially proposed by LLNet, and then MSR-Net combined CNN and Retinex models for image enhancement. Today, no-reference methods, including Zero-DCE, have achieved great success, demonstrating the feasibility of deep learning-based low-light enhancement. GANs, a novel type of network consisting of a generator and a discriminator, have been increasingly used in image processing tasks as GAN technology has improved. EnlightenGAN and low-light GAN have successfully achieved low-light enhancement by utilizing local information and GANs. Despite the significant success of deep learning-based low-light enhancement methods in recent years, many methods still suffer from poor handling of dark areas after image enhancement, and the models do not fully utilize prior data. Summary of the Invention
[0005] To overcome the shortcomings of existing technologies, the technical problem to be solved by the present invention is to provide a low-light image enhancement method based on progressive generative adversarial networks, which can achieve uniform illumination enhancement results and provide the best subjective performance.
[0006] The technical solution of this invention is: a low-light image enhancement method based on progressive generative adversarial networks, which includes the following steps:
[0007] (1) Input a pair of low-light images and normal-light images;
[0008] (2) The reflection map and the mask map are obtained by the Retinex model-based image decomposition module P1 and the image brightness region perception module P2, and then input into the network;
[0009] (3) Generate the reflectance map of the low-light image through the signal-driven model;
[0010] (4) Input the generated reflection map into the attention-inspired model;
[0011] (5) Generate and fuse the feature images of the two through a saliency network and a generator;
[0012] (6) Generate saliency maps and enhanced images through attention-inspired models.
[0013] This invention combines model priors with data-driven approaches. It learns the relationship between the reflectance components of low-light and normal-light images through a network, generates reflectance maps of low-light images through signal-driven models, introduces a saliency map generation network, and fuses the generated feature maps with the feature maps of the transferred illumination network. By ensuring the consistency of saliency maps between input and output images, the complete image structure is preserved. Attention-inspired models generate saliency maps and enhance images, thus achieving uniform illumination enhancement results and providing optimal subjective performance.
[0014] A low-light image enhancement device based on a progressive generative adversarial network is also provided, which includes:
[0015] The first input module is configured to input pairs of low-light and normal-light images;
[0016] The image processing module is configured to obtain a reflection map and a mask map through the image decomposition module P1 and the image brightness region perception module P2, and input them into the network;
[0017] The first generation module is configured to generate a reflectance map of a low-light image using a signal-driven model;
[0018] The second input module is configured to input the generated reflection map into the attention-heuristic model;
[0019] The fusion module is configured to fuse the feature image generated by the saliency network with the feature image generated by the generator.
[0020] The second generation module is configured to generate saliency maps and augmented images using an attention-inspired model. Attached Figure Description
[0021] Figure 1 This is a framework diagram of the low-light image enhancement method based on progressive generative adversarial networks according to the present invention.
[0022] Figure 2 This is a flowchart of a low-light image enhancement method based on a progressive generative adversarial network according to the present invention.
[0023] Figure 3 This is a comparison of the visual quality of image decomposition on the RAISE training dataset. From (a) to (h): results for the input image, JieP, STAR, KinD, KinD_plus, RetinexNet, ProGAN, and the RAISE-based method of this invention.
[0024] Figure 4 This is a comparison of the visual quality of image decomposition on the Sy-LOL training dataset. From (a) to (h): results for the input image, JieP, STAR, KinD, KinD_plus, RetinexNet, ProGAN, and the RAISE-based method of this invention.
[0025] Figure 5This is a comparison of the visual quality of the Raise training set enhancement results. From (a) to (l): Input images, results from JieP, STAR, SDD, KinD, KinD_plus, Enlightengan, Zero-DCE, Zero-DCE++, RetinexNet, ProGAN, and the Raise-based method of this invention.
[0026] Figure 6 This is a comparison of the visual quality of the Sy-LOL training set enhancement results. From (a) to (l): Input images, results from JieP, STAR, SDD, KinD, KinD_plus, Enlightengan, Zero-DCE, Zero-DCE++, RetinexNet, ProGAN, and the Raise-based method of this invention.
[0027] Figure 7 These are images from the RAISE training dataset.
[0028] Figure 8 This is a table showing the average objective augmentation results of 11 contrast methods on the RAISE dataset.
[0029] Figure 9 This is a table showing the average objective augmentation results of the comparison method on the Sy-LOL dataset. Detailed Implementation
[0030] like Figure 1-2 As shown, this low-light image enhancement method based on progressive generative adversarial networks includes the following steps:
[0031] (1) Input a pair of low-light images and normal-light images;
[0032] (2) The reflection map and the mask map are obtained by the Retinex model-based image decomposition module P1 and the image brightness region perception module P2, and then input into the network;
[0033] (3) Generate the reflectance map of the low-light image through the signal-driven model;
[0034] (4) Input the generated reflection map into the attention-inspired model;
[0035] (5) Generate and fuse the feature images of the two through a saliency network and a generator;
[0036] (6) Generate saliency maps and enhanced images through attention-inspired models.
[0037] This invention combines model priors with data-driven approaches. It learns the relationship between the reflectance components of low-light and normal-light images through a network, generates reflectance maps of low-light images through signal-driven models, introduces a saliency map generation network, and fuses the generated feature maps with the feature maps of the transferred illumination network. By ensuring the consistency of saliency maps between input and output images, the complete image structure is preserved. Attention-inspired models generate saliency maps and enhance images, thus achieving uniform illumination enhancement results and providing optimal subjective performance.
[0038] Preferably, in step (3), the signal-driven model uses a reflection decomposition network to generate a reflection image. In order to effectively estimate the reflection of low-light and normal-light images, P1 is integrated into the following architecture for image decomposition: the image pairs are input into the generator to generate reflection components with complete content, texture and accurate color; then the reflection images generated by the real P1 and the generated reflection images are input into the discriminator for discrimination; a symmetric encoder-decoder network structure is used as the generator and discriminator; the generator includes standard convolutional blocks, DSConv, residual blocks, downsampling and upsampling modules, and the generator follows a symmetric U-net architecture.
[0039] Preferably, in step (3), in order to effectively utilize reflection prior, global and regional image information is incorporated into the establishment of the loss function, introducing a loss based on global and local color consistency and a regional regularization loss; the loss function is expressed as:
[0040]
[0041] Where ω adv ω col ω con These are the weights of the loss function, used to balance the various losses; L adv (G, D1) represents the resistance to loss; This results in a loss of local color consistency.
[0042] For global color consistency loss; L con (G, D1) represents the region regularization loss.
[0043] Preferably, in step (3), in order to capture the regional features of the image, a mask image is generated using P2; first, the brightness channel of the image is extracted, and then these brightness channels are... Figure 2 Value-enhanced to generate a mask image;
[0044] A content loss is formulated based on the bright areas from the low-light image and the dark areas from the normal-light image to impose constraints:
[0045]
[0046] Where M(*), M1(l i ) and M2(h i () All of these represent masked images, VGG k (*) indicates that VGG was used to extract the feature map.
[0047] Preferably, in step (3), the color consistency loss is divided into two parts: based on global color consistency loss and local color consistency loss, the global color consistency loss maintains the color consistency between the reflected and generated reflections in the normal light image, as shown in the following formula:
[0048]
[0049] Local color consistency loss establishes the relationship between the three channels of an RGB format image, balances the colors in the generated reflectance image, and reduces color cast:
[0050]
[0051] Where ε = {(R, G), (R, B), (G, B)}, J p represents the average value of the channel, and (p, q) represents the channel pair.
[0052] Preferably, in step (3), the loss function of the discriminator is as follows:
[0053]
[0054]
[0055]
[0056]
[0057] Preferably, in step (6), a saliency network is introduced to generate a saliency map as an attention map to ensure structural consistency between the input and output images; the features of the first few convolutional blocks of the saliency network are combined with the features generated by the corresponding layers in the generator network; the saliency network adjusts the size and number of channels in the feature map to fuse the output feature map of each block with the feature map of the generator.
[0058] Preferably, in step (6), to generate an enhanced image with uniform illumination, illumination loss and saliency-guided content loss are added to the model, and the model is trained using the following loss function:
[0059]
[0060] Where μ adv , μ con ,μ col,μ lig ,μ s These are the weights of the loss function, used to balance the various losses; L adv (F, D2) represents the resistance to loss; To guide content loss based on saliency; L con (F, D2) represents the content loss; L col (F, D2) represents the color consistency loss; L lig (F, D2) represents the illumination loss; L(S) represents the saliency map loss.
[0061] L col (F, D2) is defined in the RGB space, using the l2 norm, and its expression is:
[0062]
[0063] L con The eigenmaps of (F, D2) using VGG19 and l1 norms are as follows:
[0064]
[0065] The Y channel of an image represents its brightness; the Y channel serves as a brightness map, and the illumination loss L... lig (F, D2) is represented as:
[0066]
[0067] To maintain the structural integrity of the generated image, a saliency-guided content loss is applied using salient feature maps as attention maps and feature maps extracted by VGG. The saliency map loss L(S) is expressed as:
[0068]
[0069] Where S′=1-S(z) i ), where z is the true salient image.
[0070] Preferably, in step (6), a saliency network is introduced to generate a saliency map as an attention map to ensure structural consistency between the input and output images; the features of the first few convolutional blocks of the saliency network are combined with the features generated by the corresponding layers in the generator network; the saliency network adjusts the size and number of channels in the feature map to fuse the output feature map of each block with the feature map of the generator.
[0071] Preferably, in step (6), the loss function of the discriminator is as follows:
[0072]
[0073]
[0074]
[0075] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a low-light image enhancement device based on a progressive generative adversarial network. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:
[0076] The first input module is configured to input pairs of low-light and normal-light images;
[0077] The image processing module is configured to obtain reflection and mask images through modules P1 and P2, and input them into the network;
[0078] The first generation module is configured to generate a reflectance map of a low-light image using a signal-driven model;
[0079] The second input module is configured to input the generated reflection map into the attention-heuristic model;
[0080] The fusion module is configured to fuse the feature image generated by the saliency network with the feature image generated by the generator.
[0081] The second generation module is configured to generate saliency maps and augmented images using an attention-inspired model.
[0082] To validate the effectiveness of the proposed GAN-based region-aware progressive low-light enhancement method, experiments were conducted using different training datasets, including the synthetic dataset RAISE and the simulated dataset Sy-LOL. The model was compared with several state-of-the-art methods, including JieP, STAR, SDD, RetinexNet, KinD, KinD_plus, Zero-DCE, Zero-DCE++, EnlightenGAN, and ProGAN. Enhancement performance was compared using three widely used image quality metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and No-Reference Image Quality (NIQE). Higher PSNR, SSIM, or lower NIQE values indicate better image quality.
[0083] Figure 8 and Figure 9The average augmentation performance of 10 comparative methods on the RAISE and Sy-LOL datasets is presented respectively. It can be seen that after training on different datasets, the method of this invention has the highest PSNR and a relatively high SSIM result, while the NIQE result is at a mid-to-high level. Figure 3 and Figure 4 The images show the visualization results of reflection images after training on different datasets. Figure 5 and Figure 6 The images shown are visualizations of the reflected images trained on different datasets. It can be seen that the method of this invention can generate enhanced results with uniform illumination, providing optimal subjective performance.
[0084] The following is a more detailed explanation.
[0085] 1 Dataset
[0086] Experiments were conducted on the RAISE and Sy-LOL datasets. The RAISE dataset includes 1000 pairs of synthetic low-light / normal-light images, encompassing buildings, plants, people, landscapes, and natural features, such as... Figure 7 As shown. These 1000 image pairs were divided into two groups: 80% were used for training and the remainder for testing. On the other hand, Sy-LOL contained 745 image pairs, including real image pairs from LOL, 240 synthetic image pairs from RAISE, and some completely black images.
[0087] 2. Comparison Algorithm
[0088] The proposed method was compared with traditional methods JieP, STAR, SDD, and deep learning methods RetinexNet, KinD, KinD_plus, Zero-DCE, Zero-DCE++, EnlightenGAN, and ProGAN on the RAISE and Sy-LOL datasets. The training datasets used in the deep learning-based comparison methods were the RAISE and Sy-LOL datasets, respectively.
[0089] 3. Experimental Setup
[0090] In the experiment, the image size of the dataset for this invention was adjusted to 256×256×3. The framework of this invention was implemented using TensorFlow on an NVIDIA 3090 GPU. The batch size was 4. ω was set accordingly. adv =300, ω col =10, ω con =5, μ adv =150, μ con =5, μ col =10, μ lig=3, μ s =10, to balance the scale of the loss.
[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.
Claims
1. A low-light image enhancement method based on progressive generative adversarial networks, characterized in that: It includes the following steps: (1) Input a pair of low-light images and normal-light images; (2) Image decomposition module based on Retinex model and image brightness area perception module Obtain the reflection map and mask map, and input them into the network; (3) Generate the reflectance map of the low-light image through a signal-driven model; (4) Input the generated reflection map into the attention-inspired model; (5) Generate and fuse the feature images of the two through a saliency network and a generator; (6) Generate saliency maps and augmented images using attention-inspired models; In step (3), in order to capture the regional features of the image, the following steps are taken: Generate a mask image; first, extract the brightness channels of the image, then binarize these brightness maps to generate a mask image; A content loss is formulated based on the bright areas from the low-light image and the dark areas from the normal-light image to impose constraints: ， in , and All represent masked images. This indicates that VGG was used to extract feature maps; In step (6), to generate an enhanced image with uniform illumination, saliency-guided content loss and saliency map loss are added to the model. To maintain the structural integrity of the generated image, a saliency-guided content loss is applied using salient feature maps as attention maps and feature maps extracted by VGG. and saliency map loss The loss function is expressed as: ， ， in , A true and prominent image.
2. The low-light image enhancement method based on progressive generative adversarial networks according to claim 1, characterized in that: In step (3), the signal-driven model generates a reflection image using a reflection decomposition network. To effectively estimate the reflection of low-light and normal-light images, the reflection is... Image decomposition is performed within the following architecture; the image pairs are used to generate reflection components with complete content, texture, and accurate color in the input generator; then the real image is processed... Both the generated reflection image and the generated reflection image are input into the discriminator for discrimination; a symmetric encoder-decoder network structure is used as the generator and discriminator; the generator includes standard convolutional blocks, DSConv, residual blocks, downsampling and upsampling modules, and the generator follows a symmetric U-net architecture.
3. The low-light image enhancement method based on progressive generative adversarial networks according to claim 2, characterized in that: In step (3), in order to effectively utilize reflection priors, global and regional image information is incorporated into the establishment of the loss function, introducing loss based on global and local color consistency and regional regularization loss; the loss function is expressed as: ， in , , These are the weights of the loss function, used to balance the various losses; To combat the losses; This results in a loss of local color consistency. This results in a loss of global color consistency. This represents the loss due to regional regularization.
4. The low-light image enhancement method based on progressive generative adversarial networks according to claim 3, characterized in that: In step (3), the color consistency loss is divided into two parts: global color consistency loss and local color consistency loss. The global color consistency loss maintains the color consistency between the reflected and generated reflections in the normal light image, as shown in the following formula: ， Local color consistency loss establishes the relationship between the three channels of an RGB format image, balances the colors in the generated reflectance image, and reduces color cast: ， in , Represents the average value of the channel. Represents a channel pair; The loss function of the discriminator is as follows: ， 。 5. The low-light image enhancement method based on progressive generative adversarial networks according to claim 4, characterized in that: In step (6), a saliency network is introduced to generate a saliency map as an attention map to ensure structural consistency between the input and output images; the features of the first few convolutional blocks of the saliency network are combined with the features generated by the corresponding layers in the generator network; the saliency network adjusts the size and number of channels in the feature map to fuse the output feature map of each block with the feature map of the generator.
6. The low-light image enhancement method based on progressive generative adversarial networks according to claim 5, characterized in that: In step (6), the model is trained using the following loss function: ， in , , , , , These are the weights of the loss function, used to balance the various losses; To combat the losses; To salience guide content loss; For content loss; For loss of color consistency; Loss of light; For saliency map loss; Defined in RGB space, using Norm, expressed as: ， Using VGG19 and The eigenmaps of the norm are as follows: ， The Y channel of an image represents its brightness; the Y channel serves as a brightness map, and illumination loss... Represented as: 。 7. The low-light image enhancement method based on progressive generative adversarial networks according to claim 6, characterized in that: In step (6), the loss function of the discriminator is as follows: ， 。 8. A low-light image enhancement device based on progressive generative adversarial networks, which performs the low-light image enhancement method based on progressive generative adversarial networks according to claim 1, characterized in that: It includes: The first input module is configured to input pairs of low-light and normal-light images; The image processing module is configured to use the image decomposition module. and image brightness area perception module Obtain the reflection map and mask map, and input them into the network; The first generation module is configured to generate a reflectance map of a low-light image using a signal-driven model; The second input module is configured to input the saliency map and the generated reflectance map into the attention-heuristic model; The fusion module is configured to fuse the feature image generated by the saliency network with the feature image generated by the generator. The second generation module is configured to generate saliency maps and augmented images using an attention-inspired model.