A faint light image enhancement method based on multi-stage laplace feature fusion

Abstract

The application discloses a kind of micro-light image enhancement methods based on multi-stage laplace feature fusion, will laplace pyramid be decomposed into four stages, adopt the strategy of from coarse to fine to restore micro-light image, large scale image is more concerned with high frequency information, small scale image is more concerned with low frequency information, the fusion of different scale features to obtain fusion image;After feature extraction, the features are propagated and corrected, combined with the attention and residual strategy mechanism, the context interdependence relationship in the long-term picture is captured;In the second stage and the third stage of laplace pyramid, further retain the local details of the image through the supervision correction module, and adjust the brightness of the image before passing to the next stage. This method can simultaneously alleviate the degradation difficulty, and balance between avoiding detail loss and local underexposure in the enhancement image process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image enhancement, and relates to a faint light image enhancement method, in particular to a faint light image enhancement method based on multi-stage Laplacian feature fusion. BACKGROUND

[0002] Image enhancement plays an important role in processing high-level tasks such as target detection, classification, segmentation, etc.

[0003] Due to the inevitable degradation of photos taken in low-light conditions, such as low visibility, low contrast, uneven lighting, etc., the performance of high-level tasks will be significantly affected. Such an ill-posed problem may not have a universal solution.

[0004] In order to solve the problem of enhancing the contrast of images, traditional methods generally solve in a relatively limited aspect. For example, histogram equalization and its variants expand the dynamic range of pixels in a rather effective way. However, in complex scenes, they often cannot restore the contrast well, resulting in overexposure or underexposure of images. Retinex-based methods decompose images into reflectance and illumination components using given physical priors, however, due to the challenge of finding a suitable prior, blurring of details and obvious color distortion often occur during the recombination process.

[0005] Some degradation problems are inevitable in the process of faint light image enhancement with existing methods. For example, KinD (Zhang, Zhang, & Guo, 2019) combines the idea of retinex decomposition and develops three sub-networks for layer decomposition, reflectance recovery and illumination adjustment, respectively. However, these methods do not specifically consider high-frequency information such as edges and textures. Therefore, the loss of details in the repair process is inevitable. Mining buried detailed texture information. Multi-scale-based methods have shown great potential in image restoration (W. Ren, Pan, Zhang, Cao, & Yang, 2020; G. Wang, Sun, & Sowmya, 2021; Zamir et al., 2022), Mahmoud et al. (Afifi, Derpanis, Ommer, & Brown, 2021) use a coarse-to-fine strategy to correct global color information and then refine image details. Single-lens reflex camera (Lim & Kim, 2020) recovers global illumination and local details from the original input, respectively, and gradually merges them in image space. However, since the information between different scales is ignored, they still cannot maintain the global structure. These methods only improve one of the problems, resulting in suboptimal results. SUMMARY

[0006] In view of the problems in the low-light image enhancement process of the prior art, the present application proposes a low-light image enhancement method based on multi-stage Laplacian feature fusion, which can simultaneously alleviate the degradation difficulty and balance the avoidance of detail loss and local overexposure in the enhancement image process.

[0007] The technical solution adopted by the present application to solve the technical problem is:

[0008] A low-light image enhancement method based on multi-stage Laplacian feature fusion, which restores hidden details hidden in the low-light image through a multi-level pyramid structure, the multi-level pyramid structure is a linear reversible image representation composed of a group of bandpass images, which can reconstruct the original input without losing any information, thereby accurately refining local details. The Laplacian pyramid is divided into four stages, a coarse-to-fine strategy is adopted to restore the low-light image, the Laplacian feature map is integrated as prior information, and it is fused with the refined features of the previous stage, the extracted features are used as additional information to propagate to the next stage, this strategy is beneficial to feature representation and contrast information learning using hidden information, i.e. large-scale images pay more attention to high-frequency information, and small-scale images pay more attention to low-frequency information, the fusion of features of different scales thus obtains a fused image; in addition, considering the propagation of the corrected features after feature extraction, the channel attention strategy can capture long-term mutual dependence. Compared with simply concatenating cross-layer features, in order to make up for the lack of context detail maintenance and exposure control, a supervised correction module (Style-based Recalibration Module, SRM) is used in the second and fourth stages of the Laplacian pyramid, which further preserves local details without obvious color distortion and adjusts the brightness appropriately before propagation.

[0009] The present application proposes a learning-based low-light image enhancement network structure, a multi-stage Laplacian feature fusion network (MLFFNet), which is different from existing methods in that it uses Laplacian maps of different scales and fuses them with fine features to realize the restoration of low-light images from coarse to fine; in order to extract features with rich details and appropriate brightness, the present application designs a supervised correction module, which calibrates details in the attention strategy and adjusts brightness in the residual strategy; in addition, the present application trains a multi-level target fusion network on both paired and unpaired datasets. Experimental results verify the effectiveness and superiority of the method, which has good image enhancement effect on the inevitable degradation problems such as detail loss and local overexposure in the low-light image enhancement process. Attached Figure Description

[0010] Figure 1 This is a flowchart of the low-light image enhancement method based on multi-stage Laplacian feature fusion according to the present invention. Detailed Implementation

[0011] The present invention will be further described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Example:

[0013] Low-light enhancement tasks are prone to degradation, such as loss of detail and low contrast. Therefore, this invention primarily designs a pipeline network architecture that simultaneously addresses these problems. The Laplacian feature map algorithm has demonstrated its capabilities in image representation, with existing methods directly recovering the Laplacian map to obtain results at each scale. However, this process reduces the amount of information in the input and ignores the correlation of information between different scales. Consequently, the recovered image is prone to losing some contextual details.

[0014] This invention proposes a multi-level network structure for low-light image enhancement, using Laplacian feature maps as priors and fusing them with fine-grained features from the previous stage. Therefore, the model of this invention possesses strong feature representation and multi-scale feature mining capabilities. Furthermore, this invention propagates the corrected features after feature extraction; the channel attention strategy can capture long-term interdependencies; and the residual learning strategy demonstrates the feasibility of obtaining brightness by learning the brightness maps of the input and target images.

[0015] Figure 1 The flowchart below shows the low-light image enhancement method based on multi-stage Laplacian feature fusion according to the present invention. The enhancement process is divided into four stages, namely stages S1 to S4. First, low-frequency information features are extracted, and then the extracted features are transmitted to the next stage. In the second stage S2 and the third stage S3, the features are connected with Laplacian feature mapping to obtain global image features. Then, the supervised correction module proposed in this invention is used to calibrate these features. Finally, the corrected features are transmitted to the fourth stage to obtain the enhanced image.

[0016] Due to the unclear edge, the details of the weak light image are often destroyed. In order to reveal the hidden information in the dark area, the present application proposes a multi-level hierarchy to capture hidden details step by step. Unlike the existing pyramid method, the existing method interacts on their respective scales and only infers degradation from the single output of the previous stage, the supervision correction module proposed by the present application includes two core features: 1) transmission feature mapping and using feature mapping to assist the enhancement process of the next layer; 2) the mechanism combining attention and residual strategy is beneficial to context preservation and brightness adjustment.

[0017] A multi-stage Laplace feature fusion-based faint light image enhancement method, comprising the following steps:

[0018] Step one, design an encoder-decoder subnetwork to extract information features in the input picture:

[0019] In a coarse-to-fine manner, a multi-level target fusion network multi-level Laplace feature fusion network (MLFFNet) is used to divide the enhancement process into S1, S2, S3 and S4 four stages, in the coarsest level of the pyramid, that is, stage S1, an encoder-decoder subnetwork is used to extract low-frequency information features from the input image with the lowest resolution, and then the extracted features and the input image are transmitted to the next stage;

[0020] Definition is the input of the MLFFNet; is the corresponding input, is the corresponding output, denotes the i+1 stage of the four stages of the enhancement process; the input image is decomposed into a three-level Laplace pyramid, first downsampling operation is performed on , , that is:

[0021]

[0022]

[0023] Where ↓↓ represents the average pooling operation, and ↑↑ represents the up-sampling factor of 2; the Laplace image L i is calculated at the i-th layer; each adjustment operation adopts bilinear interpolation; while obtaining all levels of Laplace images, the Laplace image is connected with the features of the previous level, and then the obtained Laplace image is transmitted to the feature extraction network after a 3*3 convolution layer and a channel attention block;

[0024] Before the image and feature connection operation, the lightweight general up-sampling operator (CARAFE) up-samples the features of the previous stage to the same size as the current Laplace image, and then obtains features with more information through the supervised correction module to assist the calibration of the next stage, that is:

[0025]

[0026] wherein is the output image of each stage, is the feature map after calibration which will be transmitted to the next stage, is the feature map of the previous stage, F srm i represents the feature map after calibration, L i represents the Laplace image calculated at the i-th layer, represents the connection operation, represents the encoder-decoder feature extraction process, which adopts the U-Net architecture and extracts latent information from low frequency to high frequency through a skip connection;

[0027] Step two, design a supervised correction module to calibrate the features and realize the image enhancement process:

[0028] After step one, the global image features are obtained, and the supervised correction module is used to calibrate these features. Finally, the calibrated features are transmitted to the last stage S4 for the final image enhancement process.

[0029] The supervised correction module contains two core features:

[0030] 1) Transmission of feature map and use of the feature to assist the next level of image enhancement process;

[0031] 2) Combination of attention and residual strategy mechanism, which is beneficial to context preservation and brightness adjustment;

[0032] represents the features obtained through the feature extraction process in step one; is the output of the multi-level target fusion network MLFFNet, and SRM takes as input, divides into two branches B1 and B2, B1 represents the supervised attention strategy, which processes the global information after a 3*3 convolution block, connects and , and then passes the connected result to the reconstruction block to obtain the output result , and then obtains the intermediate variable A through the feature extraction operation, and then Rescale to attention map; the obtained feature map guarantees spatial accuracy and context reliability by reconstruction loss with real monitoring images; the process of B1 can be written as:

[0033]

[0034] where, represents a 3*3 convolution block that reduces the number of feature channels of to 3, represents a 3*3 convolution block that increases the number of feature channels of from 3 to the number of feature channels; represents a reconstruction block in which a plurality of multi-channel attention blocks (CAB) are stacked, represents a concatenation operation; represents the input of SRM in each level.

[0035] B2 represents a residual strategy that controls the brightness of feature mapping by manipulating the identity mapping, thereby avoiding overexposure and underexposure; first, estimate the brightness change between the low-light and normal-light images and define it as a brightness map , then compress into a single channel map and define it as the predicted brightness map , capture the brightness change by minimizing the difference between and , obtain the intermediate variable R, and add R to the recalibrated features through B1, the process of B2 can be written as follows:

[0036]

[0037] where, and represent the same convolution operation as B1;

[0038] Finally, pass the context information and brightness information generated by SRM to the next layer, and the whole process of SRM is as follows:

[0039]

[0040] Step three, design loss function to constrain the training of multi-level target fusion network model:

[0041] Train the target fusion network model end-to-end to minimize the loss function , which consists of three parts: reconstruction loss, structural similarity loss, and brightness loss:

[0042]

[0043] wherein, is a reconstruction loss, is a structural similarity loss, is a brightness loss;

[0044] The reconstruction loss calculates the difference between the output of each scale and the effective value of the real value:

[0045]

[0046] wherein represents the effective value of the real value of each scale, and ε represents a constant, and ε=0.0001 is set;

[0047] The structural similarity loss is used to measure the similarity between the output image and the effective real image:

[0048]

[0049] The brightness loss is used to control the brightness of the output of each scale:

[0050]

[0051] In the formula, is a real brightness map, represents the predicted brightness map after encoding and decoding in each level.

[0052] In the experiments, MIT5K and LOL are used as training datasets. In the MIT5K dataset, it contains 5000 underexposed images and the corresponding 5 enhanced versions (A / B / C / D / E) made by different experts. 4500 images are used for training, and the remaining 500 images are verified and tested by the following experts (Hu, He, Xu, Wang, & Lin, 2018; R. Wang et al., 2019), the first 4500 images are used for training, and the remaining are used for testing. LOL includes 500 pairs of low / normal light images taken in real scenes, we use 485 photos for training and 15 photos for testing. To demonstrate the effectiveness of the method of the present application, the model is compared with various advanced methods, and then the results are asserted with PSNR, SSIM and LPIPS. In the MIT5K dataset, LIME, DeepUPE, LPNet, DSLR in LOL dataset, MIRNet-v2 and retinex net, JED, Zero-DCE, LPNet, URetinex are compared. Note that in order to make a fair comparison, the present application regenerates their results using their official source code. Table 1 shows the quantitative comparison with the most advanced methods on the MIT5K and LOL datasets. The best results on the corresponding test dataset are shown in bold.

[0053] Table 1 Quantitative comparison with art-art image enhancement methods on MIT5K, LOL datasets

[0054]

[0055] (1) Comprehensive comparison on the MIT5K dataset

[0056] From the quantitative evaluation performance of the MIT5K dataset shown in Table 1, it can be seen that the method of the present application obtains the highest PSNR, SSIM and LPIPS values. Specifically, the PSNR of the method proposed in this paper is 1.0978 dB higher than the suboptimal method, the SSIM is at least 0.0294 higher, and the LPIPS value is at least 0.0136 higher. Compared with the method that also uses Laplacian pyramid, the PSNR of the proposed method is improved by more than 1.1075 dB, the SSIM is at least 0.006 higher, and the LPIPS value is at least 0.0243 higher. Overall, the quantitative comparison shows that the present application has good contrast enhancement capability and high visibility.

[0057] (2) Comprehensive comparison on the LOL dataset

[0058] As shown in Table 1, the method proposed in the present application far exceeds the previous method. In particular, the PSNR of the suboptimal method is significantly improved by 1.7262 dB or more, and the SSIM is at least improved by 0.0153. Our method ranks second in LPIPS, slightly lower than URetinex, with a difference of no more than 0.01. In summary, the method has achieved good results in both noise suppression and detail preservation.

[0059] (3) Comparison of real-world scenes

[0060] In addition, in order to further evaluate the performance of MLFFNet in real-world low-light scenes, blind image quality assessment NIQE, PI and NIQMC are used to verify the naturalness of the enhanced image using several representative methods. As shown in Table 2, the NIQE, PI and NIQMC scores of different methods on four real-world datasets are calculated. For NIQE, the method of the present application performs best on DICM, LIME and average scores, and ranks second on MEF. In terms of PI, the method of the present application performs best on DICM and LIME, and the average score is lower than 0.1, ranking second compared with the first-level method. Finally, for NIQMC, the method of the present application performs better on DICM and LIME, and also achieves the best result on the average score. In summary, the above experiments can prove that the method of the present application always performs best on DICM and LIME, and the average score indicates the robustness of the method of the present application in real scenes.

[0061] (4) Network architecture verification

[0062] In this part, the performance of different number of Laplacian pyramid levels and the effectiveness of SRM are verified. It is noted that i is the number of Laplacian pyramid levels, and the running time on the MIT5K dataset with resolution corresponds to 512*340 of LLIE-Platform. The results, as shown in Table 2, show that when the pyramid layer number i = 3, the effect is best under the supervision correction module (SRM). In the method of the present application, as the number of layers decreases, the performance of the model also decreases, as shown in the 2nd row and the 4th row. At the same time, when the number of Laplacian pyramid increases to i = 4, the PSNR / SSIM result is relatively poor, and the running time increases greatly, as shown in the 7th row. From the 1st row and the 2nd row, the 3rd row and the 4th row, and the 5th row and the 6th row, we can see that the supervision correction module (SRM) proposed can significantly improve the performance of the model regardless of the number of pyramid layers, and the cost is only a small increase in running time. In summary, the Laplacian pyramid structure and the supervision correction module (SRM) significantly improve the performance of the model, indicating that each component of the network structure of the present application is closely related.

[0063] Table 2 Quantitative results of the detection images (DICM, LIME, VV, MEF). "↓" means the lower the index, the better the effect, and "↑" is the opposite.

[0064]

[0065] While embodiments of the present application have been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A multi-stage Laplacian feature fusion-based low-light image enhancement method, characterized in that, The Laplacian pyramid is decomposed into four stages S1, S2, S3 and S4, a coarse-to-fine strategy is adopted to restore the faint light image, the Laplace feature map is used as prior information for integration, and it is fused with the fine features of the previous stage, the extracted features are used as additional information to propagate to the next stage, the hidden information is used for feature representation and contrast information learning, that is, large-scale images pay more attention to high-frequency information, small-scale images pay more attention to low-frequency information, and the fusion of features of different scales obtains a fused image; after feature extraction, the corrected features are propagated, and the attention and residual strategy mechanism is combined to capture the context interdependence relationship in the long-term picture; in the second and third stages of the Laplacian pyramid, the supervised correction module SRM is further used to retain the local details of the image, and the brightness of the image is properly adjusted before being propagated to the next stage; The supervised correction module includes two core features: 1) transmission feature mapping, and using the feature to assist the image enhancement process of the next level; 2) combining the attention and residual strategy mechanism, which is beneficial to context preservation and brightness adjustment; represent the features obtained by the feature extraction process; For the input corresponding to the stage, the SRM takes as input, and divides into two branches B1 and B2, B1 represents a supervised attention strategy, which processes the global information after a 3*3 convolution block, and connects and , and then passes the connected result to the reconstruction block to obtain the output result , and then obtains the intermediate variable A through the feature extraction operation, and then rescales it to an attention map; the obtained feature map is ensured to have spatial accuracy and context reliability by the reconstruction loss using the real monitoring image; the process of B1 can be written as: wherein, denotes a 3*3 convolutional block that reduces the number of feature channels of from 64 to 3, denotes a 3*3 convolutional block that increases the number of feature channels of from 3 to the number of feature channels; denotes a reconstruction block in which a plurality of multi-channel attention blocks (CAB) are stacked, denotes a concatenation operation; denotes the input of each stage SRM, is the stage S i+1 corresponding input, i = 1,2; B2 represents the residual strategy, which controls the brightness of the feature map by manipulating the identifier map, thereby avoiding overexposure and underexposure. It first estimates the brightness change between low-light and normal-light images and defines it as a brightness map. Then Compress it into a single channel map and define it as the predicted brightness map. By minimizing and The difference is used to capture changes in brightness, resulting in an intermediate variable R, which is then added to the recalibrated feature via B1. The process in B2 can be written as follows: wherein and denotes the same convolution operation as B1; Finally, the context information and the brightness information generated by the SRM are included in the metadata The whole procedure of the SRM is as follows: 。

Images