Generative structure low-illumination image enhancement method based on Retinex constraint
By combining Retinex constraints and generating diffusion models, the shortcomings of low-illumination image enhancement methods in terms of robustness and interpretability are solved, and efficient enhancement and detail recovery of low-illumination images are achieved.
Patent Information
- Application Number
- CN202510596099.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2025-08-15
AI Technical Summary
Existing low-illumination image enhancement methods perform poorly in generalization ability and robustness, lack physical interpretability, difficult to recover weak or lost image details, and the training process of GAN-based methods is unstable.
The generative structure low-illumination image enhancement method based on Retinex constraints is adopted, combining the generated diffusion model and the GAN discriminator. Through the decomposition module, denoising module and the generative structure, Retinex decomposition provides an interpretable illumination regulation path, and combining the multi-step progressive generation ability of the diffusion model and the adversarial supervision of the GAN discriminator, detail reasoning and structure completion are achieved.
Systematic optimization of complex degradation problems such as uneven illumination, noise pollution and missing content in low-illumination images is achieved, weak information is restored, and image quality and interpretability are improved.
Smart Images
Figure CN120495108A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to the technical field of a generative structured low-illumination image enhancement method based on Retinex constraints. Background Art
[0002] Images captured in low-light environments often suffer from a variety of degradation issues, such as uncertain noise, low contrast, and variable color cast. Among these degradation phenomena, the loss of scene structure information is undoubtedly the most challenging problem, not only negatively impacting visual effects but also significantly reducing the amount of information contained in the image. Image enhancement technology, as an effective means, can significantly reduce the interference caused by degradation factors on human visual perception and subsequent visual tasks, thereby achieving high-quality image presentation. To address these degradation issues, numerous low-light image enhancement (LLIE) methods have emerged. At the same time, extensive and in-depth research has also been conducted in areas such as contrast enhancement, noise removal, and texture preservation.
[0003] Currently, mainstream LLIE methods are categorized into two main categories: traditional methods and learning-based methods. Traditional algorithms are typically designed based on prior knowledge of images or simple physical models. For example, grayscale transformation and histogram equalization both adjust the image's intensity distribution using linear or nonlinear methods; the Retinex model decomposes the image into an illumination component image and a reflectance component image. Traditional optimization methods have also played an important role in solving such problems. However, due to the limitations of manual design and optimization, these traditional methods lack generalization and robustness, significantly hindering their widespread adoption in practical applications.
[0004] In light of this, deep learning techniques have been introduced to this field, used to construct complex mappings from low-light images to normal-light images. Some deep learning-based methods treat low-light image enhancement purely as a holistic restoration task, but these methods often lack the theoretical support and interpretability of physical models. Compared with methods based on physical models, they often fall short in terms of targeted enhancement performance, primarily manifesting in issues such as uneven illumination and poor robustness to noise. This is primarily due to the lack of specific and clear definitions of certain degradation phenomena and effective, targeted treatment strategies for these methods.
[0005] The method based on physical models decomposes the image into components with clear physical meanings, and then performs specific processing on these components to achieve more targeted image enhancement effects. However, various existing methods are difficult to get rid of the nature of fitting. Specifically, in these methods, although the denoising operation can better present the distorted scene to a certain extent, it is powerless for the scene content that has been lost. Taking the low-light image enhancement method based on the Retinex model (URetinex-Net) as an example, this method cannot restore those faint or even lost image details, and in some cases it may even further aggravate the distortion of information.
[0006] Considering that LLIE is essentially a process of restoring a normal-illuminance image under the guidance of a low-illuminance image, we decided to employ a generative diffusion model for LLIE. This approach aims to recover or even infer information that is weak or even lost in the original low-illuminance image, thereby transforming LLIE beyond simply restoring a fitting function and transforming it into a conditionally constrained image generation task. For generative models, generative adversarial networks (GANs) train generators and discriminators through adversarial mechanisms. However, GANs suffer from numerous unstable factors during training, which can easily lead to problems such as mode collapse, non-convergence, and exploding or vanishing gradients. Furthermore, GAN-based LLIE methods also suffer from the problem of directly generating normal-illuminance images through overall fitting, lacking physical interpretability. Summary of the Invention
[0007] In response to the problems of overfitting, lack of physical interpretability and extremely unstable training process in existing low-light enhancement technologies, the present invention discloses a generative structured low-light image enhancement method based on Retinex constraints, which relates to the field of image enhancement technology. It integrates the advantages of physical models and generative networks, and uses the probabilistic generation characteristics of diffusion models to achieve detail reasoning and structure completion. At the same time, it uses the GAN discriminator to construct dual supervision at the pixel level and distribution level, and ultimately achieves systematic optimization of complex degradation problems such as uneven illumination, noise pollution and content missing in low-light images.
[0008] The method comprises the following steps:
[0009] S1. Acquire and preprocess an image dataset, where the image dataset includes N pairs of low-illumination images and normal-illumination images;
[0010] S2, the preprocessed data set is divided into training set and validation set according to the ratio of 97:3;
[0011] S3. Build a generative structured low-light image enhancement model based on Retinex constraints and name it the ReGDiff model.
[0012] The ReGDiff model includes: a decomposition module, a denoising module, a generative structure and a loss function;
[0013] S4. Iteratively train the ReGDiff model using the training set and the validation set to obtain a trained low-light image enhancement model;
[0014] S5. Input the low-light image S to be processed into the trained ReGDiff model;
[0015] S51, the decomposition module decomposes S into a reflectance map R and an illumination I;
[0016] S52, the denoising module performs denoising on the reflection image R to obtain the denoised reflection image R ′ ;
[0017] S53, the generative structure restores the illumination I to obtain the restored illumination I ′ ;
[0018] S6, R ′ and I ′ Multiply element by element to get the enhanced image S ′ .
[0019] Furthermore, the method for preprocessing the image dataset is: resizing.
[0020] Furthermore, the decomposition module includes a low illumination decomposition branch and a normal illumination decomposition branch;
[0021] The low-light decomposition branch passes through six 3×3 convolution modules and Sigmoid activation functions from input to output. The second 3×3 convolution module, the third 3×3 convolution module, the fourth 3×3 convolution module, and the fifth 3×3 convolution module all use the ReLU activation function.
[0022] The structure of the normal illumination decomposition branch is consistent with that of the low illumination decomposition branch.
[0023] Furthermore, the denoising module uses the BM3D method to perform denoising.
[0024] Furthermore, the generative structure is specifically as follows: the generator module of the generative adversarial network is replaced by the multi-step denoising generation mechanism of the diffusion model to obtain an improved generative adversarial network, and the improved generative adversarial network is named the generative structure.
[0025] Furthermore, the loss function is specifically: L = L recon +μ ir L ir +μ is L is , where L reconrepresents the reconstruction loss, μ ir represents the equilibrium reflectance consistency coefficient, μ is Indicates the illumination smoothness coefficient, L ir represents the constant reflectivity loss, L is Indicates the loss of illumination smoothness.
[0026] Furthermore, the processor implements the steps of the above method when executing the computer program.
[0027] Furthermore, the computer instructions implement the steps of the above method when executed by a processor.
[0028] The beneficial effects of the present invention are:
[0029] The method described in the present invention proposes a physically interpretable generative model for low-light image enhancement, called the ReGDiff model. The goal of this model is to integrate the advantages of physical models and generative networks. The ReGDiff model has made a breakthrough in constructing an innovative framework for deep collaboration between physical modeling and generative reasoning. It organically integrates the physical priors of illumination and reflection images obtained by Retinex decomposition and the multi-step progressive generation capability of the diffusion model with the adversarial authenticity supervision of the GAN discriminator to form a cross-paradigm enhancement mechanism - providing an interpretable illumination control path through Retinex decomposition, realizing detail reasoning and structure completion with the help of the probabilistic generation characteristics of the diffusion model, and using the GAN discriminator to construct dual supervision at the pixel level and distribution level, ultimately achieving systematic optimization of complex degradation problems such as uneven illumination, noise pollution and content missing in low-light images. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a flow chart of the method described in Example 1 of the present invention;
[0031] Figure 2 This is a schematic diagram of the decomposition module structure described in Example 1 of the present invention;
[0032] Figure 3 This is a schematic diagram of the ReGDiff model structure described in Example 1 of the present invention;
[0033] Figure 4 (a) is the low-light image (1) described in Example 2 of the present invention, (b) is the low-light image (2) described in Example 1 of the present invention;
[0034] Figure 5 (a) is a low-light image (1) processed by the zero-reference depth curve estimation method described in Example 2 of the present invention, and (b) is a low-light image (2) processed by the zero-reference depth curve estimation method described in Example 2 of the present invention;
[0035] Figure 6(a) is the low-light image (1) processed by the Kind method described in Example 2 of the present invention, and (b) is the low-light image (2) processed by the Kind method described in Example 2 of the present invention;
[0036] Figure 7 (a) is the low-light image (1) processed by the ReGDiff model described in Example 2 of the present invention, and (b) is the low-light image (2) processed by the ReGDiff model described in Example 2 of the present invention;
[0037] Figure 8 (a) is the normal illumination image (1) described in Example 2 of the present invention, and (b) is the normal illumination image (2) described in Example 2 of the present invention. DETAILED DESCRIPTION
[0038] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0039] This embodiment provides a generative structured low-light image enhancement method based on Retinex constraints. The flowchart of the method is as follows: Figure 1 As shown, the method includes the following steps:
[0040] S1. Acquire and preprocess an image dataset, where the image dataset includes N pairs of low-illumination images and normal-illumination images.
[0041] This example uses existing public low-light image datasets, such as the LOL (Low-Light dataset). This dataset consists of 500 pairs of low-light and normal-light images, making it suitable for image enhancement tasks. If existing datasets do not meet your needs, you can create a custom low-light image dataset by collecting your own data. The images are categorized by scene and illumination conditions, annotated with metadata (such as capture time, illumination intensity, camera parameters, etc.), and saved in a suitable format (such as JPEG, PNG, or RAW).
[0042] According to the input requirements of the model, such as the number of channels of the image, the size of the image and other parameters, the dataset is preprocessed by resizing.
[0043] S2. Divide the preprocessed dataset into a training set and a validation set at a ratio of 97:3.
[0044] S3. Build a generative structured low-light image enhancement model based on Retinex constraints and name it the ReGDiff model.
[0045] The core idea of the Retinex algorithm is to separate the reflection and illumination components of an image and then recombine them to eliminate or reduce the effects of uneven illumination in the image, thereby improving image quality and realism. This paper proposes a more effective model based on the Retinex principle, named the ReGDiff model.
[0046] like Figure 3 As shown in FIG, the ReGDiff model includes: a decomposition module (Decomposition), a denoising module (Denoising), a generative structure and a loss function.
[0047] like Figure 2 As shown, the decomposition module includes a low illumination decomposition branch and a normal illumination decomposition branch;
[0048] The low-light decomposition branch passes through six 3×3 convolution modules and Sigmoid activation functions from input to output. The second 3×3 convolution module, the third 3×3 convolution module, the fourth 3×3 convolution module, and the fifth 3×3 convolution module all use the ReLU activation function.
[0049] The structure of the normal illumination decomposition branch is consistent with that of the low illumination decomposition branch.
[0050] The decomposition module first sets up a 3×3 convolution layer to extract features from low-light images and normal-light images. Secondly, four 3×3 convolution layers with ReLU activation functions are used to map the RGB image into features related to reflectivity R and illumination I. Another 3×3 convolution layer is used to project the feature space into reflectivity R and illumination I, and the sigmoid activation function is used to constrain the values of reflectivity R and illumination I to [0,1]. The ReLU activation function is used to increase the nonlinear expression ability of the network.
[0051] The decomposition module receives a pair of low-light images and normal-light images each time and decomposes them into a reflectance map R and an illumination map I;
[0052] R includes: Low light reflectance map R low And the normal illumination reflection diagram R normal ;
[0053] I Includes: Low-light lighting diagram I low And normal illumination lighting diagram I normal ;
[0054] Let S denote the low-light image to be processed, which can be expressed as S = R·I. Under the constraint that low-light and normal-light images have the same reflectivity and illumination smoothness, the decomposition module learns the consistent reflectivity R of the image under different illuminations in a data-driven manner.
[0055] The denoising module uses the BM3D method to perform denoising.
[0056] The denoising module uses the BM3D method to denoise the low-light reflection image R low The BM3D method first generates a preliminary denoised image using a conservative thresholding method (basic estimation). It then performs a refined estimation, using the basic estimation result as a reference, and then performs block matching and three-dimensional transform filtering to obtain a higher-quality denoised image. This two-step process effectively reduces noise while maximally preserving image detail and edge information. Because noise is unevenly amplified across reflectivity, the BM3D algorithm uses a relative illumination strategy.
[0057] like Figure 3 As shown, the generative structure is specifically: the generator module of the generative adversarial network is replaced by the multi-step denoising generation mechanism of the diffusion model (Retinex diffusion model) to obtain an improved generative adversarial network, and the improved generative adversarial network is named the generative structure.
[0058] The principle of the multi-step denoising generation mechanism is as follows:
[0059] Forward diffusion: Let the data point Among them, x0 represents the original sample (such as image pixel), Represents the real number domain, d represents the dimension of x0, and defines a "noise-added" Markov chain: x0→x1→…→x T , where t={1,2,…,T}, T represents the total number of data points, and each step follows β t represents the noise scheduling sequence, is the noise intensity of the t-th data point, I represents the identity matrix, which is used to determine the form of the covariance matrix in the parameters of the normal distribution, and here it means that the noises in each dimension are independent of each other and have the same variance. Represents the normal distribution, which is used in the given formula to define the probability distribution of the data point during the noise addition or denoising process. q represents the probability distribution of the data point during the forward diffusion process, given the previous data point x. t-1 When the current data point x t The conditional probability distribution of .
[0060] Adding noise step by step to the tth data point can be written as in, Represents the distance from the first data point to the tth data point α s The cumulative product of α t =1-β t ,and α s represents the noise intensity parameter of the sth data point, and ∏() represents the cumulative multiplication function. This means that as long as the image x0 of the 0th data point is known, the noisy image of the tth data point can be generated in one go, reducing the number of iterations.
[0061] Inverse denoising: In the inference phase, from pure random noise x T Start by denoising in reverse step by step to get x0. If we define p θ (x t-1 |x t ), t={T,…,2,1}, then a “reverse Markov chain” can be sampled.
[0062] Among them, p θ represents the probability distribution defined by a neural network with parameters θ.
[0063] Since the forward process is Gaussian noise, we can use the Bayesian method to get
[0064] This is the true distribution, but it depends on x0, which is unknown when it is actually generated. Indicates that x is known t and x0, the mean of the q normal distribution, express The variance of p can be approximated using a neural network θ. θ (x t-1 |x t )≈q(x t-1 |x t ,x0), and rewrite it as Among them, μ θ It represents the prediction obtained by the neural network with parameter θ, given x t and t, the denoised data x t-1 The mean of the normal distribution.
[0065] Training goal: Let the multi-step denoising mechanism learn how to denoise at each t data point so that x t-1 Approximate the real x t-1 , minimize the loss L, for convenience, often written as the MSE form of noise prediction: ∈ represents the real noise added to the original data during the forward diffusion process, ∈ θ represents the noise parameter predicted by the neural network with parameter θ, represents the expected operation on t, x0 and the real noise.
[0066] Overview of the implementation process of the generative structured low-light image enhancement method based on Retinex constraints:
[0067] 1. Training phase
[0068] (1) Data preparation: Obtain a high-quality training dataset and perform preprocessing to convert the data into a format acceptable to the model;
[0069] (2) Forward diffusion: define the noise scheduling sequence β t (e.g. linear or cosine strategies), gradually adding noise to the data to obtain noisy samples at each time step;
[0070] (3) Reverse learning: Design a network for predicting noise, such as UNet and Transformer; use loss functions such as mean square error to force the network to learn how to remove noise; use an optimizer to iteratively update model parameters.
[0071] 2. Generation phase
[0072] (1) Noise sampling: Pure noise x is obtained by sampling from a standard normal distribution. T
[0073] (2) Reverse generation: Using the trained network, gradually generate T Perform denoising and generate x at each step T-1 ; After multiple iterations, the final high-quality sample ∑x0 is obtained.
[0074] The multi-step denoising generation mechanism takes the illumination image I as input, and through a step-by-step reverse diffusion process, it gradually denoises from random noise to generate the real normal illumination image. Figure 1 Detailed lighting diagram I ′ The entire denoising process is modeled by a deep neural network (such as UNet) to learn the noise removal operation at each step. The discriminator D maintains the common PatchGAN form in the GAN structure and is used to discriminate the generated illumination image I ′ Whether it comes from the real data distribution. During training, the multi-step denoising generation mechanism not only optimizes the traditional noise prediction loss (such as mean squared error), but also minimizes the adversarial loss, making the generated image closer to the real image in terms of structure, contrast, and details.
[0075] During training, the ground truth of reflectance and illumination does not need to be provided. Only necessary knowledge such as reflectance consistency and illumination smoothness is embedded into the ReGDiff model as a loss function.
[0076] The loss function is specifically: L = L recon +μir L ir +μ is L is , where L recon represents the reconstruction loss, μ ir represents the equilibrium reflectance consistency coefficient, μ is Indicates the illumination smoothness coefficient, L ir represents the constant reflectivity loss, L is Indicates the loss of illumination smoothness.
[0077] Reconstruction loss L recon =∑ i＝low,normal ∑ j＝low,normal μ ij ||R i I j -S j ||1, where i and j represent the cases of traversing low-illumination images and normal-illumination images, respectively. Low represents low-illumination images, and Normal represents normal-illumination images. ij Represents the weight coefficient, which adjusts the contribution of the corresponding item to the reconstruction loss under different illumination combinations, R i Indicates the reflectivity under different illumination, I j Indicates the lighting component under different illumination, S j Represents the corresponding target image, ||||1 represents the L1 norm, and the difference between the reconstructed image and the target image is calculated.
[0078] Constant reflectivity loss L ir =||R low -R normal ||1, where R low Represents the low-light reflection map, R normal Represents a normal illumination reflection diagram.
[0079] In order to make the loss understand the image structure, the total variation function TV (Total Variation) is weighted by the gradient of the reflection map, and the final expression is L is =∑ i＝low,normal ||I i ·exp(-λ g R i )||1, where represents the gradient (including: h (horizontal) and v (vertical), λ g The coefficient representing the perceived strength of the balanced structure is calculated by weighting exp(-R g R i ), breaking through the fixed smoothness assumption of the traditional Retinex model, suppressing noise while retaining edge details, and significantly improving the decomposition accuracy in complex scenes.
[0080] S4. Use the training set and validation set to iteratively train the ReGDiff model to obtain a trained low-light image enhancement model.
[0081] The trained model is evaluated and validated using the validation set to examine its generalization and predictive performance. Based on the evaluation results, the model is optimized and adjusted. This process includes detailed analysis and comparison of the model-generated results with the actual images in the validation set to meticulously assess the model's accuracy and stability. Based on the evaluation results, detailed model optimization and adjustments are then performed, including but not limited to adjustments to the neural network architecture, training algorithm optimization, regularization strategies, and hyperparameters such as the learning rate. Model optimization can be achieved in a variety of ways, including adjusting the neural network structure, such as increasing or decreasing the number of hidden layers and neurons, or changing the activation function, to improve the model's expressiveness and adaptability. Adjusting hyperparameters, such as the learning rate, batch size, and number of training iterations, is also a common optimization technique. This can optimize the training process and improve model convergence speed and performance. Furthermore, regularization techniques such as L1 and L2 regularization can help prevent overfitting and improve the model's generalization and stability. Data augmentation techniques also enhance the diversity of the training set data by increasing the sample size to improve the model's generalization and robustness. Furthermore, ensemble learning methods such as bagging and boosting combine the prediction results of multiple models to reduce the risk of overfitting and improve overall prediction performance. These optimization methods can continuously improve the performance and effectiveness of the model, making it more suitable for various practical application scenarios.
[0082] S5. Input the low-light image S to be processed into the trained ReGDiff model;
[0083] S51, the decomposition module decomposes S into a reflectance map R and an illumination I;
[0084] S52, denoising module for reflection image R low Perform denoising to obtain the denoised reflection map λ ′ ;
[0085] S53, the generative structure restores the illumination I to obtain the restored illumination I ′ ;
[0086] S6, R ′ and I ′ The enhanced image S is obtained by element-by-element multiplication. ′ .
[0087] Example 2
[0088] This embodiment is a further limitation of embodiment 1. This embodiment selects low illumination image (1), low illumination image (2), normal illumination image (1) and normal illumination image (2) according to the LOLv1 dataset to conduct comparative experiments on the zero reference depth curve estimation method, Kind method and ReGDiff model. Figure 4-8 As shown in the figure, the ReGDiff model of the present invention has a significantly higher ability to enhance low-light images than the zero-reference depth curve estimation method and the Kind method. The low-light images enhanced by the ReGDiff model of the present invention are closer to normal-light images.
Claims
1. A generative structured low-light image enhancement method based on Retinex constraints, characterized by: The method comprises the following steps: S1. Acquire and preprocess an image dataset, where the image dataset includes N pairs of low-illumination images and normal-illumination images; S2, the preprocessed data set is divided into training set and validation set according to the ratio of 97:3; S3. Build a generative structured low-light image enhancement model based on Retinex constraints and name it the ReGDiff model. The ReGDiff model includes: a decomposition module, a denoising module, a generative structure and a loss function; S4. Iteratively train the ReGDiff model using the training set and the validation set to obtain a trained low-light image enhancement model; S5. Input the low-light image S to be processed into the trained ReGDiff model; S51, the decomposition module decomposes S into a reflectance map R and an illumination I; S52, the denoising module performs denoising on the reflection image R to obtain the denoised reflection image R ′ ; S53, the generative structure restores the illumination I to obtain the restored illumination I ′ ; S6, R ′ and I ′ Multiply element by element to get the enhanced image S ′ .
2. The generative structured low-light image enhancement method based on Retinex constraints according to claim 1, characterized in that: The method for preprocessing the image data set is: resizing.
3. The generative structured low-light image enhancement method based on Retinex constraints according to claim 2, characterized in that: The decomposition module includes a low illumination decomposition branch and a normal illumination decomposition branch; The low-light decomposition branch passes through six 3×3 convolution modules and Sigmoid activation functions from input to output. The second 3×3 convolution module, the third 3×3 convolution module, the fourth 3×3 convolution module, and the fifth 3×3 convolution module all use the ReLU activation function. The structure of the normal illumination decomposition branch is consistent with that of the low illumination decomposition branch.
4. The generative structured low-light image enhancement method based on Retinex constraints according to claim 3, characterized in that: The denoising module uses the BM3D method to perform denoising.
5. The generative structured low-light image enhancement method based on Retinex constraints according to claim 4, characterized in that: The generative structure is specifically: the generator module of the generative adversarial network replaces the multi-step denoising generation mechanism of the diffusion model to obtain an improved generative adversarial network, and the improved generative adversarial network is named the generative structure.
6. The generative structured low-light image enhancement method based on Retinex constraints according to claim 5, characterized in that: The loss function is specifically: L = L recon +μ ir L ir +μ is L is , where L recon represents the reconstruction loss, μ ir represents the equilibrium reflectance consistency coefficient, μ is Indicates the illumination smoothness coefficient, L ir represents the constant reflectivity loss, L is Indicates the loss of illumination smoothness.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
8. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.