An adaptive low-light noise image enhancement method and system based on zero-order learning

Abstract

The application discloses a kind of self-adapting low-light noise image enhancement method and system based on zero-order learning.First, according to the original low-light image decomposition obtained by Retinex theory, the illumination component and the reflection component are obtained.Then, the noise in the reflection component is estimated and removed, and the illumination component is enhanced by a mixed illumination enhancement curve combining gamma transformation and linear transformation.Finally, the enhanced illumination component and the reflection component after removing noise are used for image reconstruction to obtain the enhanced image.By introducing noise estimation, the noise in the low-light image can be removed to some extent.By introducing the pixel-by-pixel enhancement curve, the low-light image with uneven illumination can be processed, the application scenarios of the application are expanded, and the application is more consistent with the real situation.

Description

Technical Field

[0001] This invention belongs to the field of image enhancement technology, specifically relating to an adaptive low-light noise image enhancement method and system based on zero-learning. Background Technology

[0002] When taking photos in low-light environments, the resulting images often suffer from low brightness, insufficient contrast, and noise due to the undesirable environment and equipment. This conveys incorrect information to people and also affects high-level visual tasks such as object detection and semantic segmentation. Therefore, low-light image enhancement is a technology of great significance.

[0003] Traditional low-light image enhancement methods include histogram equalization (HEM), which enhances the image by increasing its dynamic range, resulting in a pixel histogram that is uniformly distributed within the range [0, 255] (for an 8-bit image). However, this method tends to increase the contrast of background noise, decrease the contrast of useful signals, and is prone to local oversaturation. Another traditional method is based on the Retinex model. A typical Retinex-based method decomposes a low-light image into a reflection component and an illumination component. The reflection component represents the inherent properties of the object and is independent of the environment, while the illumination component represents the surrounding light level. The reflection component is generally used as the enhanced result. This method typically cannot limit the level of the reflection component and cannot effectively maintain the naturalness of the image illumination, especially when the image illumination is uneven, resulting in poor performance. Therefore, Retinex-based image enhancement methods must consider preserving the naturalness of non-uniformly illuminated images to achieve satisfactory perceptual quality.

[0004] Compared with traditional methods, deep learning-based methods have advantages such as high speed and high accuracy. In recent years, many deep learning-based methods have emerged, mainly including supervised methods, such as LLNet (Lore KG, Akintayo A, Sarkar S. LLNet: A Deep Autoencoder Approach to Natural Low-light Image Enhancement[J]. Pattern Recognition, 2017, 61: 650-662.), Retinex-Net (Wei C, Wang W, Yang W, et al. Deep Retinex Decomposition for Low-Light Enhancement[J]. 2018.), KinD (YZhang Y, Zhang J, Guo X. Kindling the Darkness: A Practical Low-light Image Enhancer:, 10.1145 / 3343031.3350926[P]. 2019.), LLFlow (Wang Y, Wan R, Yang W, et al. Low-Light Image Enhancement with Normalizing Supervised methods, such as Flow (J). 2021., require paired datasets, making data collection and comparison difficult, and the models have poor generalization ability. Deep learning-based methods also include unsupervised methods, such as EnlightenGAN (Jiang Y, Gong X, Liu D, et al. EnlightenGAN: Deep Light Enhancement Without Paired Supervision [J]. IEEE Transactions on Image Processing, 2021.), which also require unpaired datasets. Recently, the field of low-light image enhancement has focused on Zero-Shot methods. These methods do not require paired or unpaired datasets, learn enhancements only on test images, do not require dataset collection, save training time, and have better generalization ability.However, typical Zero-Shot methods, such as Zero-DCE (Guo C, Li C, Guo J, et al. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement: IEEE, 10.1109 / CVPR42600.2020.00185[P].2020.), directly enhance the entire image, which is prone to color deviation and ignores noise in the image. RRDNet (Zhu A, Zhang L, Shen Y, et al. Zero-Shot Restoration of Underexposed Images via Robust Retinex Decomposition[C] / / 2020IEEE International Conference on Multimedia and Expo(ICME).IEEE,2020.) and RetinexDIP (Zhao Z, Xiong B, Wang L, et al. RetinexDIP: A Unified Deep Framework for Low-light Image Enhancement[J].IEEE Transactions on Circuits and Systems for Video Technology,2021,PP(99):1-1.) use Retinex decomposition. Although they only enhance the illumination component, they use conventional gamma transform, which cannot handle images with uneven illumination. Summary of the Invention

[0005] To address the problems described above, this invention proposes an adaptive low-light noise image enhancement method and system based on zero-learning. Firstly, considering that images captured under low-light conditions often contain noise and the lighting is relatively smooth, with noise typically present in reflections, the present invention estimates and removes noise from reflections. Secondly, considering that low-light images in many scenarios are unevenly illuminated, it is necessary not only to enhance underexposed areas but also to suppress overexposed areas, thus introducing a pixel-by-pixel hybrid illumination enhancement curve.

[0006] To achieve the above-mentioned objectives, the technical solution of this invention is as follows:

[0007] An adaptive low-light-noise image enhancement method based on zero-learning includes the following steps:

[0008] The input raw image is decomposed into a smooth illumination component and a noisy reflection component.

[0009] Noise in the reflection component is estimated and removed;

[0010] The illumination component is enhanced using a hybrid illumination enhancement curve;

[0011] The enhanced image is obtained by reconstructing the image using the enhanced illumination component and the noise-removed reflection component.

[0012] Furthermore, the formula used to decompose the input raw image into a smooth illumination component and a noisy reflection component is as follows:

[0013] I = R n ·L (1)

[0014] The original low-light image I is input into the decomposition network to generate a noisy reflection component R. n And the illumination component L.

[0015] Furthermore, the formula used to estimate the noise in the reflection component is as follows:

[0016] R n =R+N (2)

[0017] Specifically, the input is the noisy reflection component R. n Then, the noise component N and the reflection component R are obtained through a noise estimation network.

[0018] Furthermore, the method of enhancing the illumination component using a hybrid illumination enhancement curve specifically involves inputting the illumination component of an image under low illumination, passing it through an illumination enhancement network to obtain two parameter maps A and B of the same size as the image, and combining Gamma transformation and linear transformation to perform illumination enhancement. The enhancement is performed pixel-by-pixel and can handle images with uneven illumination. The image is enhanced according to the following formula:

[0019]

[0020] in, This represents the enhanced image, and normalize means normalizing the values ​​to between 0 and 1.

[0021] Furthermore, the formula used to reconstruct the enhanced image is:

[0022]

[0023] Furthermore, the decomposition network uses the following loss function:

[0024] 1) Illumination Consistency Loss: Real lighting is not exactly the same as the initial lighting, but it should have similar characteristics. To constrain the generated lighting, illumination consistency loss is used, typically measured by the difference between the decomposed lighting map and the initial lighting map. Figure 1 It is generally defined as the maximum or average value of each channel of the image.

[0025] 2) Illumination smoothing loss: Initial illumination often has a lot of texture, while real illumination should be relatively smooth. TV (Total Variation) loss has a good form of smoothing constraint and is generally used to define illumination smoothing loss.

[0026] Furthermore, the noise estimation network uses the following loss function:

[0027] 1) Noise Loss: Noise loss constrains the noise components, making noise greater in darker areas. Low-light images often contain noise, requiring noise constraint. The darker the image, the greater the noise. Noise loss can be defined as the product of the image and the estimated noise, where the pixel values ​​of the image and the noise change inversely. Specifically, it can be defined as:

[0028] loss n =||I·N|| F (5)

[0029] Where loss n This indicates noise loss.

[0030] 2) Reflection smoothing loss: It is necessary to smooth the reflection component to transfer the noise in the reflection component to the noise component. TV constraint is also used, but the weight should be set much smaller than the illumination uniformity loss.

[0031] 3) Reconstruction Loss: A reconstruction loss is used to constrain the image content, ensuring that the content of the image remains unchanged before and after enhancement. It is generally measured by the distance between the product of the decomposed illumination and reflection components and the original image. To remove noise without over-denoising, the following reconstruction loss is designed:

[0032] loss r =||IL·R||1+||IL·(R+N)||1 (6)

[0033] Where loss r This represents the reconstruction loss; one option contains no noise, while the other contains noise, serving a balancing function.

[0034] Furthermore, the illumination enhancement network uses the following loss function:

[0035] 1) Enhanced Illumination Smoothing Loss: This loss constrains the enhanced illumination components. The enhanced illumination image needs to be smoothed, which can be achieved by smoothing parametric maps A and B. Specifically, it can be defined as:

[0036] loss illen-smooth =(|Δ x A|+|Δ y A|+|Δ x B|+|Δ y B|) 2 (7)

[0037] Where, Δ x Δ y These represent the gradients in the x and y directions, respectively.

[0038] 2) Spatial consistency loss: This is generally measured by how well the relative differences between adjacent regions of the image before and after enhancement are preserved. This spatial consistency loss constrains the enhanced illumination components, preserving the relative differences between adjacent regions as much as possible while enhancing the illumination.

[0039] Furthermore, the reconstruction stage utilizes exposure control loss to control the exposure level of the enhanced image, thereby suppressing overexposure and enhancing underexposed areas. Exposure control loss is generally defined as the difference between the exposure level of the enhanced image and the normal exposure level E.

[0040] An adaptive low-light-noise image enhancement system based on zero-learning, comprising:

[0041] The image decomposition module is used to decompose the input raw image into a smooth illumination component and a noisy reflection component;

[0042] The noise estimation module is used to estimate and remove noise in the reflection component;

[0043] The illumination enhancement module is used to enhance the illumination component using a hybrid illumination enhancement curve;

[0044] The image reconstruction module is used to reconstruct the image using the enhanced illumination component and the noise-removed reflection component to obtain the enhanced image.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0046] 1) The illumination enhancement method of the present invention is based on zero-learning, that is, it does not require paired or unpaired datasets and can learn enhancement from a single test image. Compared with the supervised methods of the prior art, the method of the present invention does not require the collection of training datasets and has better generalization than supervised methods.

[0047] 2) This invention estimates and removes noise from the decomposed reflection components, which can enhance the image while removing noise.

[0048] 3) This invention designs a new illumination enhancement method. This method learns pixel-by-pixel parameter maps through a convolutional neural network, which can enhance typical low-light images and images with uneven illumination, and can be applied to more scenarios. Attached Figure Description

[0049] Figure 1 This is a flowchart of the adaptive low-light noise image enhancement method based on zero-learning according to the present invention;

[0050] Figure 2 This is a network architecture diagram of the present invention;

[0051] Figure 3 A schematic diagram showing the image enhancement result using the adaptive enhancement method of the present invention;

[0052] Figure 4 This diagram shows a comparison between the enhanced method of the present invention and other methods. Detailed Implementation

[0053] To better understand the method of the present invention, the invention will be further described below by way of examples and in conjunction with the accompanying drawings, but these examples are not intended to limit the scope of the invention in any way. The described examples are only a part of the examples of the present invention, and not all of them.

[0054] Combination Figure 1 As shown, the adaptive low-light noise image enhancement method and system based on zero-learning of the present invention includes performing Retinex decomposition on the input low-light image to decompose it into a smooth illumination component and a noisy reflection component; estimating the noise in the noisy reflection component and removing the noise; enhancing the illumination component; and reconstructing the image.

[0055] The following will combine Figure 2 The network architecture diagram of the present invention is described in more detail to illustrate the implementation method.

[0056] Step 1: Input the original image and its processed version based on prior physical knowledge. Figure 1 Using it as input, multiple inputs can increase the amount of information in the input, while integrating physics into the neural network, which is a combination of physics and deep learning, and then decomposed into a smooth illumination component and a noisy reflection component.

[0057] Furthermore, various processing steps are performed on the original image (low-light image), and the results are added to the input, such as adding the initial illumination map as input, as shown in the following formula:

[0058]

[0059] Here, x represents each pixel, c represents the channel, L0(x) represents the initial illumination map, and I c (x) represents the pixel value of the c channel.

[0060] If an initial color map C(x) is added, the formula is as follows:

[0061]

[0062] Where, mean c I(x) represents the calculation of the average value of each pixel in each channel, and I(x) represents the original input image.

[0063] To add noise to the graph, the formula is as follows:

[0064]

[0065] Where N(x) represents the noise map, and abs represents taking the absolute value. This represents the gradient in the x-direction. This represents the gradient in the y-direction.

[0066] It can also add images such as histogram equalization of the input image.

[0067] Specifically, this invention inputs a low-light image and its processing result into a decomposition network. The decomposition network can be a U-Net network with 11 convolutional layers. Skip connections are added between layers 3 and 8, layers 4 and 7, and layers 5 and 6. Each layer consists of a 3×3 convolutional kernel followed by a ReLU activation function. The last convolutional layer is followed by a Sigmoid activation function, generating a four-channel image, which is then divided into a noisy reflection component R. n And the illumination component L.

[0068] Step 2: Estimate the noise in the reflection component.

[0069] This part takes a noisy reflection component as input, which is then passed through a five-layer convolutional neural network to obtain the noise component N. Each layer consists of a 3×3 kernel followed by a ReLU activation function, and the last layer uses a Tanh activation function. The reflection component R can be obtained using a U-Net network with nine convolutional layers. Skip connections are added between layers 2 and 7, layers 3 and 6, and layers 4 and 5. Each skip connection consists of a 3×3 convolutional kernel followed by a ReLU activation function. The last convolutional layer is followed by a Sigmoid activation function.

[0070] Step 3: Enhance the illumination using a mixed illumination enhancement curve.

[0071] In the illumination adjustment stage, the input is the illumination component of the low-light image. A four-layer convolutional neural network can be used for parameter estimation. Each layer consists of a 3×3 convolutional kernel followed by a ReLU activation function, and the last layer does not use an activation function. Two parameter maps A and B with the same size as the image are obtained. The gamma transformation and linear transformation are combined to enhance the illumination, and the illumination is enhanced according to the formula (3) mentioned above.

[0072] Step 4: Reconstruct the enhanced image.

[0073] Specifically, the enhanced image consists of an enhanced illumination component and a clean reflection component. That is, the reflection component is obtained by removing the noise estimated in step two and then multiplying it with the enhanced illumination component obtained in step three, as shown in the formula (4) above.

[0074] In an optional example, a preferred setting is E = 0.6, where E represents the normal exposure level, while setting the weight λ for noise loss. n =5000, the weight λ of exposure loss e =10, and the weights of the remaining losses are all 1.

[0075] Based on the above implementation details and parameter settings, in Figure 3 The figure demonstrates the low-light image enhancement effect of the present invention. As can be seen from the figure, the present invention can not only process typical low-light images, but also images with uneven lighting, enhance underexposed areas, and suppress overexposed areas, making it applicable to a wide range of scenarios.

[0076] Comparison of the present invention with other methods Figure 4 As shown, the "enhanced image" is the enhanced image obtained using the method of the present invention. It can be seen that the enhancement method of the present invention minimizes noise while ensuring the enhancement effect.

[0077] Another embodiment of the present invention provides an adaptive low-light noise image enhancement system based on zero-learning, comprising:

[0078] The image decomposition module is used to decompose the input raw image into a smooth illumination component and a noisy reflection component;

[0079] The noise estimation module is used to estimate and remove noise in the reflection component;

[0080] The illumination enhancement module is used to enhance the illumination component using a hybrid illumination enhancement curve;

[0081] The image reconstruction module is used to reconstruct the image using the enhanced illumination component and the noise-removed reflection component to obtain the enhanced image.

[0082] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0083] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0084] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive low-light noise image enhancement method based on zero-learning, characterized in that, Includes the following steps: The input raw image is decomposed into a smooth illumination component and a noisy reflection component. Noise in the reflection component is estimated and removed; The illumination component is enhanced using a hybrid illumination enhancement curve; The enhanced image is obtained by reconstructing the image using the enhanced illumination component and the noise-removed reflection component. The process of decomposing the input raw image into a smooth illumination component and a noisy reflection component uses the following formula: I=R n ·L The original low-light image I is input into the decomposition network to generate a noisy reflection component R. n and illumination component L; The method of enhancing the illumination component using a hybrid illumination enhancement curve includes: inputting the illumination components of an image under low illumination; passing the image through an illumination enhancement network to obtain two parameter maps A and B of the same size as the image; combining Gamma transformation and linear transformation to perform illumination enhancement; the enhancement is performed pixel-by-pixel and can handle images with uneven illumination; the enhanced illumination component is: in, This represents the enhanced image, and normalize means normalizing the values ​​to between 0 and 1.

2. The method according to claim 1, characterized in that, The loss functions used in the training process of the decomposed network include: using illumination consistency loss and illumination smoothing loss to constrain the decomposed illumination components.

3. The method according to claim 1, characterized in that, The noise in the reflected component is estimated using the following formula: R n =R+N Among them, the input noisy reflection component R n Then, the noise component N and the reflection component R are obtained through a noise estimation network.

4. The method according to claim 3, characterized in that, The loss function used by the noise estimation network during training includes: Noise loss is used to constrain the noise components, making the noise greater in areas with dimmer lighting. The reflection component is constrained by a reflection smoothing loss, and the noise in the reflection component is transferred to the noise component. The reconstruction loss is used to constrain the image content, so that the content of the image remains unchanged before and after enhancement.

5. The method according to claim 1, characterized in that, The loss function used by the illumination enhancement network during training includes: The enhanced illumination components are constrained by the enhanced illumination smoothing loss; Spatial consistency loss is used to constrain the enhanced illumination components, so as to maintain the relative differences between adjacent regions as much as possible while enhancing the illumination.

6. The method according to claim 1, characterized in that, The formula used for image reconstruction is as follows: Enhanced image It consists of the enhanced illumination component and the noise-removed reflection component.

7. The method according to claim 6, characterized in that, The loss function used in the image reconstruction includes: using exposure control loss to control the exposure level of the enhanced image, so as to suppress overexposure and enhance underexposure areas.

8. A zero-learning-based adaptive low-light-noise image enhancement system employing the method described in any one of claims 1 to 7, characterized in that, include: The image decomposition module is used to decompose the input raw image into a smooth illumination component and a noisy reflection component; The noise estimation module is used to estimate and remove noise in the reflection component; The illumination enhancement module is used to enhance the illumination component using a hybrid illumination enhancement curve; The image reconstruction module is used to reconstruct the image using the enhanced illumination component and the noise-removed reflection component to obtain the enhanced image.

Images