A method and device for low-light image enhancement considering real noise and artifacts
By decomposing images into reflectance and illuminance using Retinex theory, and training a network using a dark frame database and a noise model, the problems of overexposure and artifacts in low-light image enhancement are solved, thereby improving image quality and detail preservation.
Patent Information
- Application Number
- CN202311005200.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-10
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2043-08-10
AI Technical Summary
Existing low-light image enhancement techniques have poor generalization ability in real-world environments, resulting in overexposure in bright areas and artifacts due to noise amplification in dark areas, which affects visual applications.
Based on Retinex theory, a decomposition network is used to decompose an image into reflectance and illuminance. By constructing decomposition, illuminance optimization, reflectance recovery, and noise models, and training the neural network with a dark frame database, the image quality is optimized.
It effectively reduces image overexposure, reduces high-frequency artifacts, improves image quality and detail retention, and enhances the image viewing experience.
Smart Images
Figure CN117115018B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of low-light image enhancement technology, specifically relating to a low-light image enhancement method and apparatus that takes into account real noise and artifacts. Background Technology
[0002] With the continuous development of camera technology, the demand for capturing images in low-light environments has increased significantly, including in fields such as visual surveillance, autonomous driving, and computational photography. However, due to insufficient light and sensor size limitations, low-light images are often of poor quality, riddled with noise and artifacts. While existing methods can enhance low-light images to some extent, their generalization under real low-light conditions is poor. Enhanced images often suffer from overexposure in bright areas and amplified noise in dark areas, resulting in artifacts and loss of original details and textures, thus affecting subsequent visual applications. Therefore, enhancing low-light images through technical means is extremely necessary when capturing such images.
[0003] The existing deep learning techniques for enhancing low-light images mainly include the following: (1) Retinex-Net decomposes low-light images into reflectance and illuminance through prior knowledge and regularization, and then enhances them separately to obtain the final enhancement result. This simple model often leads to unrealistic enhancement results and ignores the noise problem of low-light images. (2) The KindD method uses the U-Net network to denoise the reflectance component. Although it has a good enhancement effect, it produces over-sharpening and artifact problems. (3) URetinex-Net proposes to expand the network to adaptively fit the implicit prior in a data-driven manner to achieve noise suppression and detail preservation. However, its generalization in real environments is poor, and it often results in overexposure of bright areas of the image. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a low-light image enhancement method that considers real noise and artifacts, aiming to solve the overexposure and noise problems that occur during the enhancement process of low-light images. To address this issue, the method proposed in this invention is inspired by Retinex theory, decomposing the image into two components and following the divide-and-conquer principle: one component is responsible for improving lighting conditions, and the other is responsible for eliminating image degradation problems. The proposed neural network model effectively improves overexposure and artifact problems by training the denoiser with synthesized real low-light image noise, thereby improving the quality of low-light image enhancement.
[0005] The technical solution of this invention consists of four parts: image decomposition, illumination optimization, reflectivity restoration, and dark frame database. The specific details are as follows:
[0006] Step 1: Data Acquisition
[0007] 1.1 Acquire multiple original images with long and short exposure times, with each long exposure image corresponding to multiple short exposure images, and then uniformly adjust all images to a fixed-size RGB image;
[0008] 1.2 Construct a dark frame database. For each ISO, take n dark frames using the same aperture and shutter speed. Save the noise value on average for n dark frames, where n≥10.
[0009] Step 2: Construct the decomposition network:
[0010] The decomposition network includes a first convolutional block and two parallel branches; its input is a short-exposure image, and its output is the illumination of the input image.
[0011] The two parallel branches include an uplink branch and a downlink branch;
[0012] The uplink branch is used to decompose the reflectance of the low-light image, and it includes a first max pooling layer, a second convolutional block, a second max pooling layer, a third convolutional block, a first deconvolutional layer, a first connection layer, a fourth convolutional block, a second deconvolutional layer, a second connection layer, a fifth convolutional block, a first convolutional layer, and a first sigmoid layer, which are cascaded in sequence. The first connection layer concatenates the output of the first deconvolutional layer and the output of the second convolutional block; the second connection layer concatenates the output of the second deconvolutional layer and the output of the first convolutional block.
[0013] The downlink branch is used to decompose the illuminance of the low-light image, and it includes a sixth convolutional block, a third connection layer, a second convolutional layer, and a second sigmoid layer cascaded in sequence; wherein the third connection layer concatenates the output of the sixth convolutional block and the output of the fifth convolutional block.
[0014] A jump connection is added between the uplink and downlink branches;
[0015] Step 3: Construct an illumination optimization network:
[0016] The illuminance optimization network includes a fourth connection layer, a seventh convolutional block, an eighth convolutional block, a ninth convolutional block, a fourth convolutional layer, and a third sigmoid layer; its input is the illuminance output by the decomposition network.
[0017] Step 4: Construct a noise model:
[0018] The noise source D of the original image is modeled as signal-dependent noise and signal-independent noise, as shown below:
[0019] D=K d K a (I+N p )+K d K a N1+Kd N2+K d N q
[0020] Where K d K a (I+N p The ) represents the signal-dependent noise. Since the incident photons follow a Poisson distribution, the signal-dependent noise is modeled as a Poisson distribution, as shown below:
[0021]
[0022] Where Y represents a clean image, K a K represents the analog gain of the camera system. d The digital gain of the camera system is represented by I, and the number of incident photons is represented by N. p This represents signal-dependent photon shot noise. K represents the correlated noise of the sampled signal. d K a N1+K d N2+K d N q This represents signal-independent noise, including dark current noise, fixed-pattern noise, thermal noise, and quantization noise. For signal-independent noise, synthetic noise is sampled directly from the dark frame database, such as:
[0023] K d K a N1+K d N2+K d N q =N independent
[0024] N independent ←RandomSampling(B)
[0025] Where B represents the dark frame database, RandomSampling represents random sampling; N1 represents the total noise generated before analog gain (including dark current noise, fixed-mode noise, reset noise, etc.); N2 represents the total noise before digital gain and after analog gain (including thermal noise, column fixed-mode noise); N q N represents quantization noise; independent This indicates noise that is irrelevant to the signal.
[0026] Step 5: Obtain the noise source D of the short exposure image based on the noise model, and add it to the original long exposure image to obtain the short exposure image with real noise.
[0027] Step 6: Construct a reflectivity recovery network and train it using short-exposure images with real noise;
[0028] The reflectivity recovery network uses a conventional U-Net network as its framework, replacing all max pooling layers of the framework network with average pooling layers, and removing the skip connections of the topmost U-Net, retaining only the skip connections of the bottommost layer.
[0029] Step 7: Combine the optimized illuminance output from the illuminance optimization network with the restored reflectance output from the reflectance recovery network to obtain an enhanced low-light image.
[0030]
[0031] Where ⊙ denotes element-wise multiplication, R t I represents the output of the reflectivity recovery network. t This represents the output of the illumination optimization network.
[0032] The beneficial effects of this invention are:
[0033] 1) The illumination optimization network proposed in this invention adjusts the illumination from a hierarchical perspective while taking into account the consistency of global illumination, thereby enhancing the image while reducing overexposure.
[0034] 2) The reflectivity recovery network proposed in this invention adopts a U-Net network that replaces the max pooling layer with the average pooling layer and removes skip connections, which effectively reduces the high-frequency artifact problem.
[0035] 3) The dark frame database constructed by this invention contains noise under different ISO conditions in the captured dark images. By random sampling, dark noise and fixed pattern noise can be effectively synthesized, reducing the tedious steps of synthesizing data and noise modeling.
[0036] 4) This invention introduces a noise model to construct a low-light image with real noise, and then uses it as the input to the reflectance recovery network, thereby improving the denoising effect on real low-light images.
[0037] In summary, the method of this invention addresses the inconsistencies and degradation that occur during low-light image enhancement. By combining an improved deep Retinex network framework and training the network using synthesized real low-light noise data, it solves the problems of local overexposure and artifact noise in low-light image enhancement under specific scenarios. This method is easy to implement, and the enhanced images offer a high level of viewing experience and quality. Attached Figure Description
[0038] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.
[0039] Figure 2 This is a diagram showing the decomposed network construction of the present invention;
[0040] Figure 3 This is the illumination optimization construction diagram of the present invention;
[0041] Figure 4 This is a reflection recovery construction diagram of the present invention;
[0042] Figure 5 The images show the test results of various low-light image enhancement methods. Detailed Implementation
[0043] The present invention will be further analyzed and described below with reference to the accompanying drawings and specific embodiments.
[0044] like Figure 1 As shown, the low-light image enhancement method based on Retinex theory and noise synthesis consists of four parts: image decomposition, illumination optimization, reflectivity restoration, and dark frame database.
[0045] Step 1: Data Collection Phase
[0046] 1.1 The Retinex network selects the SID dataset, which includes original images with long and short exposure times. One long exposure image corresponds to multiple short exposure images (i.e., low-light images), which are uniformly adjusted to 1200×800px RGB images. The exposure time of the long exposure images is controlled between 20 and 30 seconds and is taken by fixing with a tripod. The exposure time of the short exposure images is less than 1 second.
[0047] 1.2 Constructing a dark frame database: Dark frames were captured in a darkroom to construct the dark frame database. In each shot of a dark frame, all incident light was blocked. Ten dark frames were shot for each ISO setting, using the same aperture and shutter speed. The noise value was saved on average for 10 dark frames.
[0048] Step 2: Construct the decomposition network:
[0049] 2.1 As Figure 2 As shown, the decomposed network includes a first convolutional block and two parallel branches; the two parallel branches include an up branch and a down branch;
[0050] The uplink branch is used to decompose the reflectance of the low-light image, and it includes a second convolutional block, a first max pooling layer, a third convolutional block, a second max pooling layer, a first deconvolutional layer, a first connection layer, a fourth convolutional block, a second deconvolutional layer, a second connection layer, a fifth convolutional block, a first convolutional layer, and a first sigmoid layer, all cascaded in sequence. The first connection layer concatenates the output of the first deconvolutional layer and the output of the second convolutional block; the second connection layer concatenates the output of the second deconvolutional layer and the output of the first convolutional block.
[0051] The downlink branch is used to decompose the illumination of the low-light image, and it includes a sixth convolutional block, a third connection layer, a second convolutional layer, and a second sigmoid layer cascaded in sequence; wherein the third connection layer concatenates the output of the sixth convolutional block and the output of the fifth convolutional block; wherein the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, the fifth convolutional block, and the sixth convolutional block have the same structure, all including a third convolutional layer and an LReLU activation layer;
[0052] Skip connections are added to the two parallel branches to preserve the overall features of the image and alleviate the gradient explosion and gradient vanishing problems that occur during training. The decomposed reflectance and illuminance are obtained by using a Sigmoid binary classifier.
[0053] 2.2 Construct the loss function for decomposing the network training, as shown in formula (1), which consists of three parts.
[0054] Loss = L rec +μL ic +L sp (1)
[0055] The first item L rec The reconstruction loss function is shown in Equation (2).
[0056] L rec =||S-R0·L0||1 (2)
[0057] Where S represents the input low-light image, R0 represents the reflectance of the model output, and L0 represents the illuminance of the model output.
[0058] Second item L ic Let μ be the illuminance consistency loss function, which aims to maintain the overall illuminance structure of S, as shown in Equation (3), where μ is a hyperparameter.
[0059]
[0060] Where c represents the RGB color channel, S (c) This represents a low-light image of the c color channel.
[0061] The third item L sp Spatial smoothing of illumination is performed using the structure-aware loss function, as shown in Equation (4).
[0062]
[0063] in This represents a normally lit image (i.e., a long exposure image). Indicates the illuminance of a normally lit image. ∈ represents gradient operation, and ∈ represents hyperparameter.
[0064] Step 3: Construct an illumination optimization network:
[0065] 3.1 such as Figure 3 As shown, the illuminance optimization network is lightweight and consists of a fourth connection layer, a seventh convolutional block, an eighth convolutional block, a ninth convolutional block, a fourth convolutional layer, and a third sigmoid layer; its input is the illuminance output by the decomposed network.
[0066] The seventh, eighth, and ninth convolutional blocks have the same structure, all including a fifth convolutional layer and an LReLu activation layer;
[0067] 3.2 Construct the illuminance optimization network loss function, as shown in formula (5).
[0068]
[0069] in Use the L2 norm to maintain overall consistency of illuminance. The gradient loss is calculated to ensure that the illumination is sufficiently smooth. L represents the illuminance of the normally lit image, and L0 represents the illuminance of the low-illuminance image output by the decomposition network. t This represents the optimized illuminance output by the illuminance optimization network.
[0070] Step 4: Pre-train the noise model:
[0071] 4.1 Constructing a noise model: The noise source D of the original image can be modeled as signal-dependent noise and signal-independent noise, as shown in formula (6).
[0072] D=K d K a (I+N p )+K d K a N1+K d N2+K d N q (6)
[0073] 4.2 The first term K in the noise model d K a (I+N p ) represents the signal-dependent noise. Since the incident photons follow a Poisson distribution, the signal-dependent noise can be modeled as a Poisson distribution, as shown in Equation (7).
[0074]
[0075] Where Y represents a clean image, K a K represents the analog gain of the camera system. d The digital gain of the camera system is represented by I, and the number of incident photons is represented by N.p This represents signal-dependent photon shot noise. This represents the associated noise of the sampled signal;
[0076] 4.3 Residual Term K of the Noise Model d K a N1+K d N2+K d N q For signal-independent noise, there are dark current noise, fixed pattern noise, thermal noise and quantization noise, etc. For signal-independent noise, the synthesized noise is sampled directly from the dark frame database and can be formulated as shown in formula (9).
[0077] K d K a N1+K d N2+K d N q =N independent (8)
[0078] N independent ←RandomSampling(B) (9)
[0079] Where B represents the database of captured dark frames, and RandomSampling represents random sampling;
[0080] By using steps 4.2 and 4.3, the noise source D of the low-light image is obtained and added to the original normal-exposure image to obtain a low-light image with real noise.
[0081] Step 5: Construct a reflectivity recovery network and train it using low-light images with real noise.
[0082] 5.1 such as Figure 4 As shown, the reflectivity restoration network uses the conventional U-Net network as a framework, replaces all max pooling layers of the framework network with average pooling layers, and removes the skip connections of the topmost U-Net (i.e., the skip connections between the convolutional block and the last upsampling layer), retaining only the skip connections of the bottommost layer (i.e., the skip connections between the convolutional block and the first upsampling layer), thereby achieving the effect of denoising and eliminating high-frequency artifacts.
[0083] 5.2 Construct the loss function for the reflectivity recovery network, as shown in formula (10).
[0084]
[0085] in R represents the reflectance under normal lighting conditions. t The reflected reflectance, which represents the reflected reflectance output of the reflectance recovery network, can be R0 or... SSIM is used to calculate the structural similarity between the two, as shown in formula (11).
[0086]
[0087] Where μ represents the mean, σ represents the variance, and c is a constant to prevent the denominator from being zero.
[0088] Step 6: Combine the optimized illuminance output from the illuminance optimization network with the restored reflectance output from the reflectance recovery network to obtain an enhanced low-light image.
[0089]
[0090] Example Comparison and Verification
[0091] To illustrate the effectiveness of the present invention, this embodiment compares and analyzes various methods under the same experimental conditions.
[0092] Image enhancement results in low-light scenes are as follows Figure 5 As shown in the figure, the present invention can restore degraded images better. Compared with other methods, it avoids local overexposure and amplifies noise, retains more details and textures, and has a better visual experience.
[0093] Table 1 compares other low-light image enhancement methods.
[0094]
[0095] As shown in Table 1, this invention achieves better results in both Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) metrics. PSNR measures the mean squared error between the original image and the test image, while SSIM measures the similarity between the two images. Both metrics are widely used in image enhancement tasks; higher PSNR and SSIM values indicate that the test image is closer to the original image. The results in Table 1 demonstrate that this invention is superior to other methods in quantitative evaluation.
[0096] The above description is merely a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A low-light image enhancement method considering real noise and artifacts, characterized in that The method comprises the following steps: Step one, data acquisition: S1.1 Obtain multiple original images of long and short time exposure, and each long time exposure image corresponds to multiple short exposure images, and then unify all images into fixed size RGB images; S1.2 Construct a dark frame database, take n dark frames for each photosensitivity, use the same aperture and shutter time, average n dark frames to save noise values, n≥10; Step two, construct a decomposition network: The decomposition network comprises a first convolutional block and two parallel branches; the input is a short exposure image, and the output is the illumination of the input image; The two parallel branches comprise an uplink branch and a downlink branch; a jump connection is added between the uplink branch and the downlink branch; The uplink branch is used to decompose the reflectivity of the low-illumination image, and comprises a first max-pooling layer, a second convolutional block, a second max-pooling layer, a third convolutional block, a first deconvolutional layer, a first connection layer, a fourth convolutional block, a second deconvolutional layer, a second connection layer, a fifth convolutional block, a first convolutional layer and a first Sigmoid layer which are sequentially connected; wherein the first connection layer splices the output of the first deconvolutional layer and the output of the second convolutional block; the second connection layer splices the output of the second deconvolutional layer and the output of the first convolutional block; The downlink branch is used to decompose the illumination of the low-illumination image, and comprises a sixth convolutional block, a third connection layer, a second convolutional layer and a second Sigmoid layer which are sequentially connected; wherein the third connection layer splices the output of the sixth convolutional block and the output of the fifth convolutional block; Step three, construct an illumination optimization network: The illumination optimization network comprises a fourth connection layer, a seventh convolutional block, an eighth convolutional block, a ninth convolutional block, a fourth convolutional layer and a third Sigmoid layer; the input is the illumination output by the decomposition network, and the output is the optimized illumination; Step four, construct a noise model: The noise source D of the original image is modeled as signal-dependent noise and signal-independent noise, as shown in formula (1): D = K d K a (I + N p ) + K d K a N1+K d N2+K d N q (1) where K d K a (I+N p ) represents signal-dependent noise, which is modeled as a Poisson distribution due to the incident photons following a Poisson distribution, as shown in equation (2): where Y denotes the clean image, K a denotes the analog gain of the camera system, K d denotes the digital gain of the camera system, I denotes the number of incident photons, N p denotes the signal-dependent photon shot noise, denotes the sampled signal-dependent noise; K d K a N1+K d N2+K d N q represents signal-independent noise, including dark current noise, fixed pattern noise, thermal noise, and quantization noise; for signal-independent noise, the synthetic noise is sampled directly from a dark frame database, as in equation (4): K d K a N1+K d N2+K d N q =N independent (3) N independent ← RandomSampling(B) (4) where B represents a dark frame database, RandomSampling represents random sampling; N1 represents the total sum of noises generated before analog gain; N2 represents the total sum of all noises before digital gain and after analog gain; N q represents quantization noise; N independent represents signal-independent noise; Step five, obtain the noise source D of the short exposure image according to the noise model of formula (1), add it to the original long exposure image, and obtain the short exposure image with real noise: Step six, construct a reflectivity recovery network and train it using the short exposure image with real noise; The reflectivity recovery network takes a conventional U-Net network as a framework, replaces all max-pooling layers of the framework network with average-pooling layers, and removes the jump connection of the uppermost U-Net, leaving only the jump connection of the lowermost U-Net; Step seven, the optimized illumination I fused with the output of the illumination optimization network t and the recovered reflectivity R output by the reflectivity recovery network t to obtain an enhanced low-light image Wherein represents element multiplication.
2. The method of claim 1, wherein The exposure time of the long exposure image in step 1.1 is controlled between 20-30 seconds, and the exposure time of the short exposure image is less than 1 second.
3. The method of claim 1, wherein The structures of the first convolutional block, the second convolutional block, the third convolutional block, the fourth convolutional block, the fifth convolutional block and the sixth convolutional block in the decomposition network of step two are the same, and all comprise a third convolutional layer and an LReLU activation layer.
4. The method of claim 1, wherein The loss function for training the decomposition network in step two is shown in formula (6): Loss = L rec + μL ic + L sp (6) Wherein μ is a hyperparameter; reconstruction loss function L rec As shown in equation (7): L rec = ||S - R0 · L0||1 (7) Wherein S represents the input short exposure image, R0 represents the reflectivity output by the decomposition network, and L0 represents the illumination output by the decomposition network; illumination consistency loss loss function L ic aiming to preserve the overall structure of the illumination of S, as shown in equation (8): where c denotes the RGB color channel, S (c) short exposure image representing the c color channel: Structural-aware loss function L sp Spatially smooth the illumination as shown in equation (9): wherein represents a long-exposure image, represents an illumination of a normal-illumination image, represents a gradient operation, ∈ represents a hyperparameter.
5. The method of claim 1 wherein The structures of the seventh convolutional block, the eighth convolutional block, and the ninth convolutional block in the illumination optimization network in step three are the same, and each includes a fifth convolutional layer and an LReLu activation layer.
6. The method of claim 1, wherein The loss function of the illumination optimization network in step three is shown in formula (10): wherein The overall consistency of the illumination is maintained by using a two-norm, The gradient loss is calculated to ensure the illumination is smooth enough, represents the illumination of the long-exposure image, L0 represents the illumination of the short-exposure image output by the decomposition network, L t represents the optimized illumination output by the illumination optimization network.
7. The method of claim 1 wherein The loss function of the reflectance recovery network in step six is shown in formula (11): wherein represents reflectance under normal lighting conditions, R t represents the recovered reflectance output by the reflectance recovery network: SSIM is and R t structural similarity, as shown in equation (12): wherein denotes the mean of R denotes the mean of R t the mean of R denotes the variance of R denotes the variance of R t the variance of R denotes the covariance between R t and R c1, c2 are constants.
8. A low-light-level image intensifier system implementing the method according to any one of claims 1 to 7, taking into account real noise and artefacts, characterized in that Comprising: a data acquisition module for acquiring a short-exposure image and a corresponding long-exposure image; an illumination extraction module for extracting the illumination of the image using a decomposition network; an illumination optimization module for optimizing the illumination output by the decomposition network using an illumination optimization network to obtain optimized illumination; a noise extraction module for obtaining the noise source D of the short-exposure image using a noise model; a real data acquisition module for adding the noise source D of the short-exposure image to the original long-exposure image to obtain a short-exposure image with real noise; a recovered reflectance extraction module for extracting the recovered reflectance of the short-exposure image with real noise using a constructed reflectance recovery network; an enhancement module for fusing the optimized illumination output by the illumination optimization network and the recovered reflectance output by the reflectance recovery network to obtain an enhanced low-light image.
9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, the computer executes the method of any one of claims 1-7.
10. A computing device comprising a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-7.