Progressive Retinex-guided low-light image enhancement method

Abstract

The invention provides a progressive Retinex-guided low-light image enhancement method, which comprises the following steps of: acquiring an original RGB low-light image Ilow, extracting an illumination component initial state # imgabs0 # and a reflection component initial state # imgabs1 # of the Ilow through a convolution block, and then performing iterative enhancement on # imgabs2 # # imgabs3 # by using a complementary coding module, a complete illumination component # imgabs4 # and a complete reflection component # imgabs5 # are generated, an illumination adjustment module is used for enhancing the complete illumination component # imgabs6 # of the Ilow, an enhanced illumination result # imgabs7 # is obtained, and a noise removal module is used for conducting denoising operation on the complete reflection component # imgabs8 # of the Ilow; in the reconstruction stage, an obtained illumination result # imgabs9 # and a denoising result # imgabs10 # are multiplied element by element to generate a final enhanced image result # imgabs11 #. According to the method, a multi-scale residual dense network is introduced to remove noise, and particularly unexpected particles are removed in a dark area of a real image. The reflectance map is recovered by associating local and global features, thereby better eliminating noise.

Description

Technical Field

[0001] The present invention relates to the technical field of low-light image enhancement, and in particular to a low-light image enhancement method based on progressive Retinex guidance. Background Art

[0002] When taking photos in low-light environments, the images captured generally present serious quality problems such as low contrast, color cast, low signal-to-noise ratio, and low dynamic range due to the limitations of the imaging system's photosensitive elements and insufficient external photons. This makes the acquired images or videos unable to record real natural scene information with high fidelity. In particular, due to the particularity of low-light illumination, the illumination sources are complex and vary in strength, resulting in insufficient light brightness in low-light environments, weakening the light reflected from the surface of objects, and causing less light entering the acquisition device. The brightness value of the captured image is extremely low, and the visibility and contrast are seriously reduced, accompanied by a large amount of random noise. On the other hand, the uneven distribution of illumination (local illumination, such as car lights, street lights, fluorescent lights, etc.) makes some areas well illuminated while other areas have extremely low brightness, affecting the visibility of details and local contrast in these areas, and generating more complex noise distribution in darker areas than in other areas. The non-uniform distribution of illumination and noise in space makes the problem even more difficult, which is a huge challenge for the application of image enhancement and recognition algorithms in computer vision systems in real scenes.

[0003] Imaging in poor light environments is always accompanied by harmful noise, which is caused by insufficient photons captured by the sensor. Increasing exposure time and setting higher ISO can alleviate this problem, but it often leaves blurred scenes and scattered granular noise in saturated images, so it is necessary to develop an effective and practical low-light image enhancement method to remove noise in images. Summary of the invention

[0004] According to the technical problems mentioned in the above background technology, a progressive Retinex-guided low-light image enhancement method is provided.

[0005] The technical means adopted by the present invention are as follows:

[0006] A progressive Retinex-guided low-light image enhancement method comprises the following steps:

[0007] Step 01: Get the original RGB low-light image I low , extract I through the convolution block low The initial state of the illumination component and the initial state of the reflected component Then use the complementary coding module to Perform iterative enhancement to generate complete lighting components and the complete reflection component

[0008] Step 02: Use the Lighting Adjustment Module to adjust the I low The complete lighting component Enhance and obtain the enhanced lighting result

[0009] Step 03: Use the noise removal module to low The complete reflection component De-noising operation is performed; the noise removal module includes: three residual dense blocks; the residual dense blocks respectively extract local feature information of different scales, and perform local feature fusion through feature splicing operation; the feature information of the current scale is combined with the local information of the previous scale to restore the texture detail information in the reflection component, and the information of different scales is spliced ​​to perform global feature fusion, so as to generate the final denoising result

[0010] Step 04: Reconstruction phase, the lighting result is obtained Denoising results Perform element-by-element multiplication to generate the final enhanced image result

[0011] Furthermore, in step 01, the initial state of the illumination component is obtained. and the initial state of the reflected component After that, the initial state of the illumination component and the initial state of the reflected component Input the cascaded component enhancement network CEM to generate the illumination component and the reflected component

[0012]

[0013] Among them, f 1,CEM (·) represents the first feature extraction function of the cascade component enhancement network, Indicates the initial state of the lighting component The illumination component result generated by the first feature extraction function of the component enhancement network is Represents the initial state of the reflection component The reflection component result generated by the first feature extraction function of the component enhancement network; the f 1,CEM (·) includes illumination component, reflection component and redundant complementary representation features, expressed as:

[0014]

[0015] in, represents the processing result of the i-th CEM block in the cascade network;

[0016] The two-component result features obtained through multiple cascaded CEM blocks cover the characteristic information of the original low-light image; the image resolution is restored using the deconvolution operation, and the initial module network forms a U-shaped structure to generate the final illumination and reflection components.

[0017] Furthermore, the component enhancement module includes: a feature extraction module and a complementary coding module CCM; the feature extraction module performs rough feature extraction on the illumination component and the reflection component to generate a rough feature CCM of the illumination component. i,L and the coarse feature C of the reflection component i,R , and then the two components are refined through the complementary coding module to generate the refined illumination component features And the refined reflection component features

[0018] The feature extraction module uses the channel attention mechanism to extract the complementary information between the illumination component and the reflection component for refined representation; in order to reduce the amount of network calculation, the depth-separable convolution is used to replace the convolution; the complementary encoding module uses C i,L and C i,R As input, it undergoes two convolutions to extract features and generate a refined component representation. Then, the two components are normalized by the Sigmoid function to generate a mask M. i,L and M i,R ; Calculate the two's complement 1-M i,L and 1-M i,R , constrained to the range [0,1], and combined with and Perform feature ratio allocation to achieve a refinement operation on the illumination and reflection components; the refinement operation is:

[0019]

[0020] in, represents the illumination component extracted from the rough reflection component, Represents the reflectance component extracted from the coarse illumination component; They represent the complete illumination and reflection component results obtained by the i-th CEM block respectively.

[0021] Furthermore, in step 02, a five-layer convolutional neural network enhances the illumination component, each convolution includes two convolutional layers and a ReLU activation function, and finally a sigmiod layer.

[0022]

[0023] in, Represents the original image I low The complete illumination component extracted after the initial module, f ligt (·) indicates the illumination adjustment module, express The result after enhancement by the Light Adjustment module.

[0024] Furthermore, the step 03 includes the following steps:

[0025] Step 031: extract features of the original reflection component, and use downsampling operation to obtain 1 / 2 ratio and 1 / 4 ratio of the original size;

[0026] Step 032: Send the input of 1 / 2 scale and 1 / 4 scale of the original size to the residual dense network to obtain local feature information of different scales;

[0027] F i =f RDB,i (F i +f Deconv (F i-1 ));

[0028] Among them, f RDB,i Represents the specific operation of the i-th RDB block, which is a functional representation of the operation. The RDB module includes convolution and ReLU activation functions. Deconv (F i-1 ) represents a deconvolution operation on the previous scale image. The generated F i is the local feature information;

[0029] Step 033: Fuse the features generated at three different scales: 1 / 4 of the original size, 1 / 2 of the original size, and the original size to generate the final reflection component result It manifests as:

[0030]

[0031] Among them, f ff (·) represents the concatenation operation of features of different scales, F N Represents the final result after feature fusion. denois (·) represents the noise removal module, Represents the final reflection component result.

[0032] Furthermore, in the reconstruction stage of step 04, the illumination result is obtained Performing element-by-element multiplication generates the final result:

[0033]

[0034] in, represents element-wise multiplication, Represents the final image enhancement result.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] Different from the traditional Retinex optimization model, the regularizer is used as an indicator of gradient distortion and image noise. Based on this, two error terms are proposed to measure the reflectivity difference between paired images of different brightness and the gradient aggregation deviation between reflectivity and illumination. In addition, in the alternative optimization process, a binary mask is considered to dynamically locate reflectivity and illumination.

[0037] The present invention introduces a multi-scale residual dense network to remove noise, especially to remove unexpected particles in dark areas of real images. The reflectance map is restored by correlating local and global features, thus better removing noise.

[0038] Based on the above reasons, the present invention can be widely promoted in the fields of image processing and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 This is the network architecture of the method of the present invention.

[0041] Figure 2 It is a schematic diagram of a component enhancement module of the present invention.

[0042] Figure 3 Schematic diagram of the residual dense network block of the present invention.

[0043] Figure 4It is a schematic diagram of the qualitative comparison of the present invention on the LOL synthetic data set; wherein, Figure (a) 1, Figure (a) 2, and Figure (a) 3 are original low-light images; Figure (b) 1, Figure (b) 2, and Figure (b) 3 are schematic diagrams after the method LIME is enhanced; Figure (c) 1, Figure (c) 2, and Figure (c) 3 are schematic diagrams after the method Retinex-Net is enhanced; Figure (d) 1, Figure (d) 2, and Figure (d) 3 are schematic diagrams after the method EnGan is enhanced; Figure (e) 1, Figure (e) 2, and Figure (e) 3 are schematic diagrams after the method Zero-DCE is enhanced; Figure (f) 1, Figure (f) 2, and Figure (f) 3 are schematic diagrams after the method KinD++ is enhanced; Figure (g) 1, Figure (g) 2, and Figure (g) 3 are schematic diagrams after the method DRBN is enhanced; Figure (h) 1, Figure (h) 2, and Figure (h) 3 are schematic diagrams after the present invention is enhanced. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0046] like Figure 1-4 As shown, the present invention provides a progressive Retinex-guided low-light image enhancement method, such as Figure 1 As shown in the figure, it includes three parts: initialization module, noise removal module and illumination enhancement module. The initialization module contains multiple cascade component enhancement modules, which use complementary redundant coding to improve the generated illumination and reflection maps; in the noise removal module, a multi-scale (1 / 4, 1 / 2, 1) residual dense network is established to denoise the reflection component from local to overall features, where the red arrows are deconvolution operations; respectively are the losses of each module. It includes the following steps:

[0047] Step 01: Get the original RGB low-light image I low , extract I through the convolution block low The initial state of the illumination component and the initial state of the reflected component Then use the complementary coding module to Perform iterative enhancement to generate complete lighting components and the complete reflection component In step 01, the initial state of the illumination component is obtained and the initial state of the reflected component After that, the initial state of the illumination component and the initial state of the reflected component Input the cascaded component enhancement network CEM to generate the illumination component and the reflected component

[0048]

[0049] Among them, f 1,CEM (·) represents the first feature extraction function of the cascade component enhancement network, Indicates the initial state of the lighting component The illumination component result generated by the first feature extraction function of the component enhancement network is Represents the initial state of the reflection component The reflection component result generated by the first feature extraction function of the component enhancement network; the f 1,CEM (·) includes illumination component, reflection component and redundant complementary representation features, expressed as:

[0050]

[0051] in, represents the processing result of the i-th CEM block in the cascade network;

[0052] The two-component result features obtained through multiple cascaded CEM blocks cover the characteristic information of the original low-light image; the image resolution is restored using the deconvolution operation, and the initial module network forms a U-shaped structure to generate the final illumination and reflection components.

[0053] As a preferred implementation, the component enhancement module includes: a feature extraction module and a complementary coding module CCM; the feature extraction module performs rough feature extraction on the illumination component and the reflection component to generate a rough feature CCM of the illumination component. i,L and the coarse feature C of the reflection component i,R, and then the two components are refined through the complementary coding module to generate the refined illumination component features And the refined reflection component features

[0054] The feature extraction module uses the channel attention mechanism to extract the complementary information between the illumination component and the reflection component for refined representation; in order to reduce the amount of network calculation, the depth-separable convolution is used to replace the convolution; the complementary encoding module uses C i,L and C i,R As input, it undergoes two convolutions to extract features and generate a refined component representation. Then, the two components are normalized by the Sigmoid function to generate a mask M. i,L and M i,R ; Calculate the two's complement 1-M i,L and 1-M i,R , constrained to the range [0,1], and combined with and Perform feature ratio allocation to achieve a refinement operation on the illumination and reflection components; the refinement operation is:

[0055]

[0056] in, represents the illumination component extracted from the rough reflection component, Represents the reflectance component extracted from the coarse illumination component; They represent the complete illumination and reflection component results obtained by the i-th CEM block respectively.

[0057] Further, Step 02: Use the Light Adjustment Module to adjust the I low The complete lighting component Enhance and obtain the enhanced lighting result In step 02, a five-layer convolutional neural network enhances the illumination component, each convolution includes two convolutional layers and a ReLU activation function, and finally a sigmiod layer.

[0058]

[0059] in, Represents the original image I low The complete illumination component extracted after the initial module, f ligt (·) indicates the illumination adjustment module, express The result after enhancement by the Light Adjustment module.

[0060] Further, step 03: use the noise removal module to lowThe complete reflection component De-noising operation is performed; the noise removal module includes: three residual dense blocks; the residual dense blocks respectively extract local feature information of different scales, and perform local feature fusion through feature splicing operation; the feature information of the current scale is combined with the local information of the previous scale to restore the texture detail information in the reflection component, and the information of different scales is spliced ​​to perform global feature fusion, so as to generate the final denoising result The step 03 comprises the following steps:

[0061] Step 031: extract features of the original reflection component, and use downsampling operation to obtain 1 / 2 ratio and 1 / 4 ratio of the original size;

[0062] Step 032: Send the input of 1 / 2 scale and 1 / 4 scale of the original size to the residual dense network to obtain local feature information of different scales;

[0063] F i =f RDB,i (F i +f Deconv (F i-1 ));

[0064] Among them, f RDB,i Represents the specific operation of the i-th RDB block, which is a functional representation of the operation. The RDB module includes convolution and ReLU activation functions. Deconv (F i-1 ) represents a deconvolution operation on the previous scale image. The generated F i is the local feature information;

[0065] Step 033: Fuse the features generated at three different scales: 1 / 4 of the original size, 1 / 2 of the original size, and the original size to generate the final reflection component result It manifests as:

[0066]

[0067] Among them, f ff (·) represents the concatenation operation of features of different scales, F N Represents the final result after feature fusion. denois (·) represents the noise removal module, Represents the final reflection component result.

[0068] Further, in step 04: reconstruction phase, the obtained lighting results Denoising results Perform element-by-element multiplication to generate the final enhanced image result

[0069] In the reconstruction phase of step 04, the illumination result is obtained. And the reflection result Performing element-by-element multiplication generates the final result:

[0070]

[0071] in, represents element-wise multiplication, Represents the final image enhancement result.

[0072] Example:

[0073] In the framework of the present invention, each module is trained separately. The batch size is set to 16, and the training data is cropped to 96x 96 pixels. The Adam optimizer is selected to optimize the training model, and the initial learning rate is 4×10-4, and the learning rate is halved after 100,000 times. λt,λs,λssim are set to 0.01,0.1 and 1, respectively. All experiments are performed on NVIDIA2080GPU under the PyTorch framework. Figure 4 As shown in Table 1, it is a quantitative comparison between the method of the present invention and various methods.

[0074] Table 1 Quantitative comparison on LOL synthetic dataset

[0075] · LIME Retinex-net EnGan Zero-DCE Kind++ DRBN Ours PSNR 16.769 18.530 16.572 20.380 21.303 23.322 24.452 SSIM 0.5696 0.8764 0.7338 0.7826 0.8226 0.9268 0.9272

[0076] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A progressive Retinex-guided low-light image enhancement method, characterized in that: The following steps are involved: Step 01: Get the original RGB low-light image I low , extract I through the convolution block low The initial state of the illumination component and the initial state of the reflected component Then use the complementary coding module to Perform iterative enhancement to generate complete lighting components and the complete reflection component Step 02: Use the Lighting Adjustment Module to adjust the I low The complete lighting component Enhance and obtain the enhanced lighting result Step 03: Use the noise removal module to low The complete reflection component Perform denoising operation; the noise removal module includes: three residual dense blocks; the residual dense blocks respectively extract local feature information of different scales, and perform local feature fusion through feature splicing operation; The feature information of the current scale is combined with the local information of the previous scale to restore the texture detail information in the reflection component, and the information of different scales is spliced ​​to perform global feature fusion to generate the final denoising result. Step 04: Reconstruction phase, the lighting result is obtained Denoising results Perform element-by-element multiplication to generate the final enhanced image result 2. The progressive Retinex-guided low-light image enhancement method according to claim 1, characterized in that: In step 01, the initial state of the illumination component is obtained and the initial state of the reflected component After that, the initial state of the illumination component and the initial state of the reflected component Input the cascaded component enhancement network CEM to generate the illumination component and the reflected component Among them, f 1,CEM (·) represents the first feature extraction function of the cascade component enhancement network, Indicates the initial state of the lighting component The illumination component result generated by the first feature extraction function of the component enhancement network is Represents the initial state of the reflection component The reflection component result generated by the first feature extraction function of the component enhancement network; the f 1,CEM (·) includes illumination component, reflection component and redundant complementary representation features, expressed as: in, represents the processing result of the i-th CEM block in the cascade network; The two-component result features obtained through multiple cascaded CEM blocks cover the characteristic information of the original low-light image; the image resolution is restored using the deconvolution operation, and the initial module network forms a U-shaped structure to generate the final illumination and reflection components.

3. The method for low-light image enhancement guided by progressive Retinex according to claim 2, characterized in that: The component enhancement module includes: a feature extraction module and a complementary coding module CCM; the feature extraction module performs rough feature extraction on the illumination component and the reflection component to generate a rough feature CCM of the illumination component. i,L and the coarse feature C of the reflection component i,R , and then the two components are refined through the complementary coding module to generate the refined illumination component features And the refined reflection component features The feature extraction module uses the channel attention mechanism to extract the complementary information between the illumination component and the reflection component for refined representation; in order to reduce the amount of network calculation, the depth-separable convolution is used to replace the convolution; the complementary encoding module uses C i,L and C i,R As input, it undergoes two convolutions to extract features and generate a refined component representation. Then, the two components are normalized by the Sigmoid function to generate a mask M. i,L and M i,R ; Calculate the two's complement 1-M i,L and 1-M i,R , constrained to the range [0,1], and combined with and Perform feature ratio allocation to achieve a refinement operation on the illumination and reflection components; the refinement operation is: in, represents the illumination component extracted from the rough reflection component, Represents the reflectance component extracted from the coarse illumination component; They represent the complete illumination and reflection component results obtained by the i-th CEM block respectively.

4. The progressive Retinex-guided low-light image enhancement method according to claim 1, characterized in that: In step 02, a five-layer convolutional neural network enhances the illumination component, each convolution includes two convolutional layers and a ReLU activation function, and finally a sigmiod layer. in, Represents the original image I low The complete illumination component extracted after the initial module, f ligt (·) indicates the illumination adjustment module, express The result after enhancement by the Light Adjustment module.

5. The method for low-light image enhancement guided by progressive Retinex according to claim 1, characterized in that: The step 03 comprises the following steps: Step 031: extract features of the original reflection component, and use downsampling operation to obtain 1 / 2 ratio and 1 / 4 ratio of the original size; Step 032: Send the input of 1 / 2 scale and 1 / 4 scale of the original size to the residual dense network to obtain local feature information of different scales; F i =f RDB,i (F i +f Deconv (F i-1 )); Among them, f RDB,i Represents the specific operation of the i-th RDB block, which is a functional representation of the operation. The RDB module includes convolution and ReLU activation functions. Deconv (F i-1 ) represents a deconvolution operation on the previous scale image. The generated F i is the local feature information; Step 033: Fuse the features generated at three different scales: 1 / 4 of the original size, 1 / 2 of the original size, and the original size to generate the final reflection component result It manifests as: Among them, f ff (·) represents the concatenation operation of features of different scales, F N Represents the final result after feature fusion. denois (·) represents the noise removal module, Represents the final reflection component result.

6. The progressive Retinex-guided low-light image enhancement method according to claim 1, characterized in that: In the reconstruction phase of step 04, the illumination result is obtained. And the reflection result Performing element-by-element multiplication generates the final result: in, represents element-wise multiplication, Represents the final image enhancement result.

Images