Low-light image enhancement method based on deep learning and Laplacian pyramid

Abstract

The present invention discloses a low-light image enhancement method based on deep learning and Laplacian pyramid. The method comprises the following steps: decomposing the low-light image into a single LF component and N HF components using a Laplacian pyramid decomposition function; enhancing the LF component using a low-frequency enhancement module; fusing the pre-enhanced LF component, the enhanced LF component, and the Nth HF component, and inputting the fusion into a cross-layer feature enhancement module, where the LF component guides the enhancement of the Nth HF component, with i set to N; inputting the enhanced i-th HF component and the unenhanced i-1th HF component into the cross-layer feature enhancement module, where the i-th HF component guides the enhancement of the i-1th HF component; if i is not 1, then i is set to i-1 and the previous step is repeated; otherwise, all frequency components are enhanced, and a light-enhanced image is reconstructed using a Laplacian pyramid reconstruction function. The method achieves better performance than existing methods while maintaining a relatively small model size and low computational complexity.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and more specifically, relates to a low-light image enhancement method based on deep learning and Laplacian pyramid. Background Art

[0002] The task of low-light image enhancement (LLIE) technology is to correct and enhance low-light (LL) images captured under non-ideal lighting conditions, such as low light, dim light, and backlight, and ultimately obtain normal light (Ground Truth, GT) images. This technology has high practical value and broad application prospects.

[0003] Existing LLIE methods can be roughly divided into two categories: one is the traditional method, and the other is the method based on deep learning. In traditional enhancement methods, it is usually necessary to manually design image enhancement rules to specify the spatial mapping relationship between LL and GT. Common algorithms include methods based on histogram equalization (HE), methods based on statistical models, and methods based on the retinal cortex theoretical model (Retinex Model). In deep learning-based methods, prior knowledge of natural images is usually used to constrain the solution space. In order to obtain results close to GT, many types of low-light image enhancement methods have been proposed. Common methods include mapping-based methods, Retinex-based methods, generative adversarial network (GAN)-based methods, and unsupervised learning (UL)-based methods.

[0004] Although many LLIE methods have been proposed, the existing methods all have certain shortcomings. For example, they are still not ideal in image lighting enhancement and detail adjustment. At the same time, the model calculation complexity is large and the running time is long. There is still room for further improvement in the performance of the methods. Summary of the Invention

[0005] In view of this, the main purpose of the present invention is to provide a method for low-light image enhancement based on deep learning and Laplacian pyramid, which can achieve end-to-end enhancement of low-light images, overcome the problem that existing algorithms do not achieve a good balance between model complexity and algorithm performance, and at the same time obtain higher subjective and objective image quality.

[0006] To achieve the above objectives, the present invention provides a low-light image enhancement method based on deep learning and Laplacian pyramid, comprising the following steps:

[0007] a. Input the low-light image into the Laplacian pyramid decomposition function to decompose the image into one LF component and N HF components;

[0008] b. Use low frequency enhancement module to enhance LF component;

[0009] c. The LF component before enhancement, the LF component after enhancement, and the Nth HF component are fused and input into the cross-layer feature enhancement module, and the LF component is used to guide the enhancement of the Nth HF component, and i = N;

[0010] d. The enhanced i-th HF component and the unenhanced i-1-th HF component are input into the cross-layer feature enhancement module, and the i-th HF component guides the enhancement of the i-1-th HF component;

[0011] e. If i is not equal to 1, set i=i-1 and repeat step d; otherwise, complete the enhancement of all frequency components and use the Laplace pyramid reconstruction function to reconstruct the illumination enhanced image.

[0012] In one embodiment of the present invention, step a specifically includes:

[0013] The low-light image is decomposed into a LF component L and N HF components {H i}; The LF component contains global information such as image brightness and color and has a relatively low resolution, while the HF component contains image detail information and has a relatively high resolution; in the Laplace decomposition function, simple up-sampling and down-sampling and Gaussian blur operations are used to decompose the low-light image; subsequently, different enhancement processing modules are designed for different frequency components, which helps to reduce the difficulty of establishing a nonlinear mapping from the low-light image to the normal-light image; wherein, N is a preset value, {H i} represents the set of HF components.

[0014] In one embodiment of the present invention, in step b:

[0015] The low-frequency enhancement module adopted first uses N1 convolutional layers to extract shallow features from the LF component of the low-light image; next, the feature resolution is reduced to 1 / 2 of the original through the MobileNetv2 module to reduce the computational complexity; then N2 convolutional layers are used to upscale the downsampled features, and the upscaled features are further enhanced by mining the global correlation on the feature map through a cascade of N3 Transformers; then, the number of channels is reduced through a convolutional layer and the features before upscaling are input into the feature cascade block, processed through a convolutional layer and the resolution of the enhanced features is improved by a deconvolution layer; finally, N4 convolutional layers are used to restore the features with the increased resolution to an image with 3 channels and obtain the enhanced LF component; wherein, the N1, N2, N3 and N4 have preset values.

[0016] In one embodiment of the present invention, in step c:

[0017] The enhanced low-frequency components The low-frequency component L before enhancement and the Nth HF component H N Cascade fusion is performed in the channel dimension, and then N5 3×3 convolutions are used to extract shallow features. Finally, N6 residual blocks are used to extract deep features, which are then input into the cross-layer feature enhancement module to guide the enhancement of the Nth HF component with the LF component; wherein, N5 and N6 are preset values.

[0018] In one embodiment of the present invention, in step d, a cross-layer feature enhancement module is used to fuse and enhance the different frequency components decomposed by the Laplacian pyramid; ordinary convolution is used in the cross-layer feature enhancement module to enhance the content details of the image; it first uses a convolution layer to obtain the high-frequency component H i The shallow features of the image are then gradually fused through N7 cross-layer feature enhancement blocks, and finally N8 convolutional layers are used to restore the features to obtain the final enhanced high-frequency components; wherein, N7 and N8 are preset values.

[0019] In one embodiment of the present invention, the cross-layer feature enhancement block uses affine transformation to fuse the lower layer guide features, that is, Among them, ⊙ represents element-by-element multiplication, F i is the feature to be processed before affine transformation, is the feature after affine transformation, α i and β i They are the next layer features of the current pyramid layer It is obtained by using deconvolution to perform double upsampling and then using N9 convolutional layers to learn; Further adopting N 10The feature enhancement within a single cross-layer feature enhancement block can be completed by performing feature refinement with a cascade of 3×3 convolutions and a residual convolution attention module; wherein, the N9 and N 10 is the default value.

[0020] In one embodiment of the present invention, in step e, a Laplacian pyramid reconstruction function is required to fuse all enhanced frequency components to obtain a final enhanced image; the reconstruction function only needs to iteratively upsample the enhanced features of the lower layer by a factor of two and add them to the enhanced components of that layer to reconstruct the enhanced frequency components into an enhanced image.

[0021] In general, the above technical solutions conceived by the present invention have the following beneficial effects compared with the prior art:

[0022] Non-machine learning methods use manual setting of various parameters and constraints, and their enhancement effects usually rely on careful adjustment of parameters. Existing deep learning methods are still not ideal in enhancing image illumination and adjusting details, and there is a problem of not achieving a good balance between model complexity and algorithm performance. The present invention combines deep learning with the Laplace pyramid. The use of Transformer at the bottom of the Laplace pyramid can effectively process the global information of the image. At the same time, since the resolution of the LF component at the bottom of the pyramid is relatively small, a large amount of calculation will not be generated due to the use of Transformer. At the same time, since there is a strong correlation at the same spatial coordinates of different frequency components, the present invention designs a fusion enhancement module for different frequency components to achieve step-by-step enhancement and restoration of the detail information of the high-frequency components. By combining the respective advantages of deep learning and the Laplace pyramid, the present invention achieves better performance than existing methods while maintaining a relatively small model size and low computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the principle of the low-light image enhancement method based on deep learning and Laplacian pyramid of the present invention;

[0024] Figure 2 This is a schematic diagram of the structure of the low frequency enhancement network module LFEM used in the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of the downsampling module MobileNetv2 in the low-frequency enhancement network module LFEM;

[0026] Figure 4 This is a schematic diagram of the Transformer structure in the low-frequency enhancement network module LFEM;

[0027] Figure 5This is a schematic diagram of the structure of the cross-layer feature enhancement module used in the present invention;

[0028] Figure 6 Schematic diagram of the structure of the cross-layer feature enhancement block in the cross-layer feature enhancement module;

[0029] Figure 7a is an original low-light image selected from the LOL-v1 dataset;

[0030] Figure 7b This is the low-light enhanced image obtained using the RetinexNet method;

[0031] Figure 7c This is the low-light enhanced image obtained using the MBLLEN method;

[0032] Figure 7d is the low-light enhanced image obtained using the KinD method;

[0033] Figure 7e This is the low-light enhanced image obtained using the EnlightenGAN method;

[0034] Figure 7f is the low-light enhanced image obtained using the Zero-DCE method;

[0035] Figure 7g This is the low-light enhanced image obtained using the DRBN method;

[0036] Figure 7h This is the low-light enhanced image obtained using the MSEC method;

[0037] Figure 7i This is a low-light enhanced image obtained using the DSLR method;

[0038] Figure 7j This is the low-light enhanced image obtained using the SNR-net method;

[0039] Figure 7k This is a low-light enhanced image obtained by the method of the present invention. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0041] In order to solve the problems existing in the prior art, the present invention provides a low-light image enhancement method based on deep learning and Laplacian pyramid, comprising the following steps:

[0042] a. Input the low-light image into the Laplacian pyramid decomposition function to decompose the image into a LF component and multiple HF components;

[0043] Specifically, the low-light image is decomposed into one LF component and N HF components {H i}; The LF component contains global information such as brightness, contrast, and color of the image and has a relatively low resolution, while the HF component contains more detailed information of the image and has a relatively high resolution;; The Laplace decomposition function uses simple up-sampling and down-sampling and Gaussian blur operations to decompose the low-light image; Subsequently, different enhancement processing modules are designed for different frequency components, which helps to reduce the difficulty of establishing a nonlinear mapping from the low-light image to the normal-light image; Wherein, N is a preset value, {H i} represents a set of HF components; preferably, N is an integer between 1 and 5.

[0044] b. Use low frequency enhancement module to enhance LF component;

[0045] Compared with ordinary convolution, Transformer is better at extracting global information by using self-attention mechanism. The low-frequency components obtained by Laplace decomposition function have a smaller resolution and contain global information such as brightness and color of the image. Therefore, low-frequency components are very suitable for expression and enhancement by Transformer. The LFEM module consists of convolution layer, Transformer module, upsampling module and downsampling module. By inputting L into the LFEM module, the enhanced LF component is obtained. The purpose of enhancing image brightness and contrast and correcting image color is achieved.

[0046] In one embodiment, in step b, the low-frequency enhancement module (LFEM) used first uses N1 convolutional layers to extract shallow features from the LF component of the low-light image; next, the feature resolution is reduced to 1 / 2 of the original through the MobileNetv2 module to reduce computational complexity; then, N2 convolutional layers are used to upscale the downsampled features, and the upscaled features are further enhanced by mining the global correlation on the feature map through a cascade of N3 Transformers; then, the channels are reduced through a convolutional layer and the upscaled features are input into a feature cascade block together with the pre-upscaled features, processed through a convolutional layer, and the resolution of the enhanced features is increased through a deconvolution layer; finally, N4 convolutional layers are used to restore the upscaled features to an image with 3 channels and obtain the enhanced LF component; compared to ordinary convolution, the Transformer is better at extracting global information using the self-attention mechanism; the low-frequency components obtained through the Laplace decomposition function have a lower resolution and contain global information such as the brightness and color of the image, so the low-frequency components are very suitable for expression and enhancement using the Transformer.

[0047] Among them, N1, N2, N3 and N4 are preset values; preferably, N1, N2, N3 and N4 are integers between 1 and 5 respectively.

[0048] Furthermore, in the Transformer module, the input is connected to the multi-head attention module after one layer normalization, and the multi-head attention module is jump-connected to the input. The output is normalized again and then jump-connected to the input using a fully connected layer. Finally, the output result is processed by the Transformer module to achieve interdependency modeling of global image content.

[0049] Furthermore, N1 = 3, N2 = 2, N3 = 3, and N4 = 3 are selected. That is, the LFEM module includes 10 convolutional layers, 3 Transformer modules, 1 downsampling module, and 1 upsampling module.

[0050] c. The LF component before enhancement, the LF component after enhancement, and the Nth HF component are fused and input into the cross-layer feature enhancement module, and the LF component is used to guide the enhancement of the Nth HF component, and i = N;

[0051] In one embodiment, in step c, in order to effectively guide the LF component to enhance the Nth HF component, the enhanced low frequency component The low-frequency component L before enhancement and the Nth HF component H NCascade fusion is performed in the channel dimension, and then N5 3×3 convolutions are used to extract shallow features. Finally, N6 residual blocks are used to extract deep features, which are then input into the Cross-Layer Feature Enhancement Module (CFEM) to guide the enhancement of the Nth HF component with the LF component.

[0052] Wherein, N5 and N6 are preset values; preferably, N5 is an integer between 1 and 3, and N6 is an integer between 1 and 6.

[0053] Preferably, select N5=1, N6=3, and let i=N.

[0054] d. The enhanced i-th HF component and the unenhanced i-1-th HF component are input into the cross-layer feature enhancement module, and the i-th HF component guides the enhancement of the i-1-th HF component.

[0055] In one embodiment, in step d, since there is a strong correlation at the same spatial coordinates of different frequency components, we need to fuse and enhance the features of these frequency components. The most commonly used feature fusion methods are addition or cascade, but these methods can only provide limited representation capabilities for the network. At the same time, the resolution of high-frequency components is relatively large and contains local information such as image content details. For this reason, a cross-layer feature enhancement module is used to fuse and enhance the different frequency components decomposed from the Laplacian pyramid. Ordinary convolution is used in the cross-layer feature enhancement module to enhance the content details of the image. The enhanced i-th HF component and the unenhanced i-1-th HF component are input into the cross-layer feature enhancement module together, and the i-th HF component is used to guide the enhancement of the i-1-th HF component, thereby achieving the purpose of step-by-step fusion and enhancement of features. Specifically, in CFEM, first, a convolution layer is used to obtain the high-frequency component H i The shallow features of the network are then gradually fused through N7 cross-layer feature enhancement blocks (CFEBs), and finally N8 convolutional layers are used to restore the features to obtain the final enhanced high-frequency components.

[0056] Wherein, N7 and N8 are preset values. Preferably, N7 is an integer between 1 and 4, and N8 is an integer between 1 and 3.

[0057] Furthermore, CFEB applies affine transformation to fuse the enhanced features from the previous layer. Then, in order to make the network pay more attention to the useful channel information and spatial information of the fused features and enhance the learning ability of the features, we use a residual convolution attention module after the affine transformation, which consists of N 10The network consists of 1 convolutional layer and 1 Convolutional Block Attention Module (CBAM) using skip connections. Since CBAM is an existing technology in convolutional neural networks, we will not elaborate on it here.

[0058] Specifically, the CFEB adopts affine transformation to fuse the underlying guidance features, namely Among them, ⊙ represents element-by-element multiplication, F i is the feature to be processed before affine transformation, is the feature after affine transformation, α i and β i They are the next layer features of the current pyramid layer It is obtained by using deconvolution to perform double upsampling and then using N9 convolutional layers to learn; Further adopting N 10 The feature enhancement within a single cross-layer feature enhancement block can be completed by performing feature refinement with a cascade of 3×3 convolutions and a residual convolution attention module; wherein, the N9 and N 10 is the default value. Preferably, N9 and N 10 It is an integer between 1 and 3.

[0059] Further, select N7=2, N8=2, N9=1, N 10 =2.

[0060] e: If i is not equal to 1, set i = i-1 and repeat step d; otherwise, enhance all frequency components and reconstruct the illumination-enhanced image using the Laplacian pyramid reconstruction function.

[0061] In one embodiment, in step e, enhanced high-frequency components are obtained by repeatedly performing step d. Subsequently, all enhanced components are fused using a Laplacian pyramid reconstruction function to obtain a final enhanced image. The reconstruction function simply iteratively upsamples the enhanced features of the lower layer by a factor of two and adds them to the enhanced components of that layer to reconstruct the enhanced frequency components into an enhanced image. By sequentially performing these five steps, low-light image enhancement is ultimately achieved, resulting in higher subjective and objective image quality.

[0062] The technical solution of the present invention is described below with reference to a specific embodiment:

[0063] a: in Figure 1 As shown in the figure, the low-light image is first input into the Laplace decomposition function, which decomposes the low-light image into low-frequency components and N high-frequency components This scheme uses a low-frequency component L and three high-frequency components (H1, H2, and H3) as an example. Here, 1 ≤ i ≤ N, where N represents the number of high-frequency component layers; W and H represent the width and height of the image, respectively. The following describes the specific method for the Laplacian pyramid decomposition function.

[0064] L=G N+1 (I) (1)

[0065] H i =G i (I)-B(G i+1 (I)↑2) (2)

[0066] Here N+1 represents the total number of decomposition layers. Indicates the use of a 5×5 two-dimensional Gaussian kernel, ↑2 means upsampling by twice; G i (I) represents the image at the i-th layer of the Gaussian pyramid. Specifically, it can be expressed as

[0067]

[0068] In the above formula, ↓2 means downsampling by two times. From formulas (1) to (3), it can be seen that the Laplace pyramid decomposition process is a reversible process.

[0069] From formula (3), we can see that the image at level N+1 of the Gaussian pyramid is blurred N times, resulting in the lowest image resolution. Therefore, L in the Laplacian pyramid is the LF component, which contains the global information of the image, such as brightness, hue, and contrast. On the other hand, according to formula (2), H i It is composed of the residual details of HF and has a larger resolution. From high decomposition level to low decomposition level, the image detail information from coarse to fine is stored in {H i}middle.

[0070] b: Process the decomposed low-frequency component L and construct a low-frequency enhancement module LFEM.

[0071] Its network structure is as follows Figure 2 The low-frequency component L first extracts shallow features through three convolutional layers. Then, in order to further reduce the computational complexity of the network, the MobileNetv2 block is used to downsample the shallow features by a factor of 2. Figure 3 As can be seen, MobileNetv2 uses a 1×1 convolution to increase the number of channels in the input feature map by six times. It then uses a 3×3 depthwise separable convolution with a stride of 2. Finally, a 1×1 convolution performs dimensionality reduction, achieving downsampling with lower computational effort while retaining more useful information. The downsampled features are then mapped to a higher-dimensional space by two convolutional layers.

[0072] After the above high-dimensional features are expanded by the Transformer operation, they are input into three series-connected Transformer modules. The features are then folded and restored to their original size by the Transformer. The number of channels is reduced by one convolutional layer and the features are input into the feature cascade block together with the features before dimensionality increase. After one convolutional layer, the features are upsampled twice using an upsampling module to improve the feature resolution. The upsampling module is implemented using deconvolution.

[0073] like Figure 4 As shown in the figure, in the Transformer module, the input undergoes layer normalization and is connected to the multi-head attention module. The multi-head attention module has a jump connection to the input. The output undergoes layer normalization again and a jump connection to the input using a fully connected layer, and finally the result is output. By processing three series-connected Transformer modules, the algorithm can fully exploit the interdependencies of global image content.

[0074] After upsampling, it is necessary to pass through 3 convolutional layers and add a jump connection of the low-frequency component L to finally obtain the enhanced low-frequency component Furthermore, the number of output channels of each module in the low frequency enhancement module LFEM is 16, 32, 64, 96, 96, 144, 144, 96, 192, 96, 64, 32, 16, and 3 respectively.

[0075] c: Enhancement of the Nth HF component guided by the LF component.

[0076] Use deconvolution to separately enhance the low-frequency components The low frequency component L before enhancement is twice upsampled and combined with the Nth HF component H N Cascading is performed in the channel dimension, and then a 3×3 convolution is used to extract shallow features. Finally, three residual blocks are used to extract deep features, and they are input into the cross-layer feature enhancement module, so as to achieve the enhancement of the Nth HF component guided by the LF component, and let i=N.

[0077] d: Since there is a strong correlation in the same spatial coordinates of different frequency components obtained after decomposition, the enhanced i-th HF component and the unenhanced i-1-th HF component are input into the cross-layer feature enhancement module together, and the i-th HF component is used to guide the enhancement of the i-1-th HF component to achieve step-by-step fusion enhancement of features.

[0078] Since the HF component H i Compared with the LF component, the high frequency component has more local features and higher resolution, so the present invention uses standard CNN to process the high frequency component. Figure 5 As shown, CFEM consists of 3 convolutional layers and 2 CFEBs (structure as shown in Figure 6 shown) and skip connections.

[0079] exist Figure 5 Medium and high frequency components H i First, a convolutional layer is used to obtain the shallow feature F i , F i Input to CFEB( Figure 6 ), then the affine transformation is applied, which is defined as:

[0080]

[0081] like Figure 6 As shown, here ⊙ represents element-by-element multiplication, F i is the feature to be processed before affine transformation, is the feature after affine transformation, α i and β i They are the next layer features of the current pyramid layer It is obtained by using deconvolution to perform double upsampling and then using 1 convolution layer to learn. The enhanced features from the next pyramid layer are obtained by upsampling them twice using deconvolution. Note that after the affine transformation, two convolutional layers and one CBAM are used to allow the network to focus more on the channel and spatial information of the fused features and enhance feature learning. Since CBAM is an existing convolutional neural network technology, it will not be discussed in detail here.

[0082] Furthermore, in CFEM, after the fusion enhancement of two CFEB modules, the enhanced features can be obtained after one convolution layer. Finally, use another convolutional layer to enhance Restored to 3-channel enhanced component image. Note that the high-frequency component H is added in CFEM i The jump connection finally obtains the enhanced high-frequency components Moreover, in the CFEM module, except for the last convolutional layer with 3 output channels, the output channels of all other modules are 32.

[0083] e: If i is not equal to 1, set i=i-1 and repeat step d; otherwise, complete the enhancement of all frequency components and use the Laplacian pyramid reconstruction function to reconstruct the illumination-enhanced image.

[0084] Repeat step d until all frequency components are enhanced. As described in step a, the Laplace decomposition process is a reversible process. The reconstruction function only needs to iteratively upsample the enhanced features from the lower layer by a factor of two and add them to the enhanced components of the layer to reconstruct all enhanced frequency components into an enhanced image. The specific formula of the reconstruction function is:

[0085]

[0086] here represents the intermediate result obtained by iteratively performing the process of doubly upsampling and adding the enhanced low-frequency component and the corresponding enhanced high-frequency component, wherein the order of iteration is i=4→3→2→1, and ↑2 represents doubly upsampling; The final enhanced image result

[0087] Furthermore, the network adopts a supervised training method to train and evaluate the performance of the present invention on low-light images; at the same time, the loss function used by the network is:

[0088]

[0089] W and H represent the width and height of the image respectively; and are the output enhanced image and normal image respectively.

[0090] In summary, after the above five steps, on the one hand, the LFEM module is designed to apply the Transformer to the bottom layer of the Laplacian pyramid, so that the Transformer can achieve lower computational complexity in LF component processing and establish the interdependence of the global content of the image; on the other hand, the high-frequency component has a relatively large resolution, and there is pixel correlation and similar texture between different frequency components. The CFEM module is designed to explore the inter-layer correlation of different components, so that the image details can be enhanced and refined layer by layer.

[0091] To illustrate the performance of the present invention, experiments were conducted using Python under the pytorch framework. Figure 1 The method is used to enhance low-light images and compared with existing RetinexNet, MBLLEN, KinD, EnlightenGAN, Zero-DCE, DRBN, MSEC, DSLR, and SNR-net methods.

[0092] Specifically, the performance of the present invention is evaluated by taking the open source low-light datasets LOL-v1 and LOL-v2 of real scenes as examples. LOL-v1 contains 485 and 15 pairs of low-light / normal-light images for training and testing, respectively. LOL-v2 contains 689 and 100 pairs of low-light / normal-light images for training and testing, respectively. Table 1 shows the quantitative comparison results on the LOL-v1 and LOL-v2 datasets. The compared indicators are: average peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), number of parameters (Parameters), number of floating-point operations (FLOPs) and runtime (Runtime), where FLOPs and Runtime are measured on images with a resolution of 512×512. All results were obtained by running on an NVIDIA GTX 3090 GPU.

[0093] The results in Table 1 show that the proposed method outperforms existing low-light image enhancement algorithms. Compared to the suboptimal SNR-net algorithm, the proposed method requires 39.467M fewer parameters, and its computational complexity (FLOPs) is approximately one-third that of the SNR-net algorithm.

[0094] Figure 7a to Figure 7k A low-light image on LOL-v1 is given, and the subjective quality comparison of the enhancement results of RetinexNet, MBLLEN, KinD, EnlightenGAN, Zero-DCE, DRBN, MSEC, DSLR, SNR-net and the proposed method on the LOL-v1 dataset is given. Figure 7b 、 7c As can be seen from Figures 7d, 7e, 7f, 7g, 7h, 7i, 7j, and 7k, compared with other methods, the method of the present invention can maintain more texture structures and details, produce less distortion, and has a better effect on enhancing image brightness.

[0095] In summary, the method described in the present invention combines the Laplacian pyramid and the Transformer to design a low-frequency enhancement module (LFEM) and a different frequency component fusion enhancement module (CFEM), thereby combining the respective advantages of CNN and Transformer. This allows the present invention to achieve better performance than existing methods while maintaining a relatively small model size and low computational complexity.

[0096] Table 1. Quantitative comparison results of different low-light enhancement algorithms on LOL-v1 and LOLv2 datasets

[0097]

[0098] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A low-light image enhancement method based on deep learning and Laplacian pyramid, characterized in that: The following steps are involved: a. Input the low-light image into the Laplace pyramid decomposition function to decompose the image into a LF component and N HF components; b. Using a low frequency enhancement module to enhance the LF component; in step b: the low frequency enhancement module used first uses N A convolutional layer extracts shallow features from the LF component of the low-light image; then, the feature resolution is reduced to 1 / 2 of the original through the MobileNetv2 module to reduce the computational complexity; and then the N The two convolutional layers increase the dimension of the downsampled features, and then concatenate the increased dimension features. N 3 Transformers mine the global correlation on the feature map to enhance the features; then it is passed through a convolution layer to reduce the number of channels and input into the feature cascade block with the features before dimensionality increase, processed through a convolution layer and the deconvolution layer to improve the resolution of the enhanced features; finally, it is used N The four convolutional layers restore the features with enhanced resolution to an image with three channels and obtain the enhanced LF components; wherein, N 1、 N 2、 N 3 and N 4 is the default value; c. The LF component before enhancement, the LF component after enhancement, and the N The HF components are fused and input into the cross-layer feature enhancement module, and the LF component is used to guide the N The HF component is enhanced and i = N ; d. i HF components and the unenhanced i -1 HF components are input into the cross-layer feature enhancement module together. i The HF component guides the i -1 HF component enhancement; e. If i If it is not equal to 1, then let i = i -1 and repeat step d, otherwise complete the enhancement of all frequency components and use the Laplace pyramid reconstruction function to reconstruct the illumination enhanced image.

2. The low-light image enhancement method based on deep learning and Laplacian pyramid according to claim 1, characterized in that: The step a specifically includes: Decompose the low-light image into a LF component using the Laplacian pyramid decomposition function and N HF components ; Among them, the LF component contains the global information of the brightness and color of the image and has a relatively low resolution, while the HF component contains the image detail information and has a relatively high resolution; in the Laplace decomposition function, simple up-sampling and down-sampling and Gaussian blur operations are used to decompose the low-light image; then, different enhancement processing modules are designed for different frequency components, which helps to reduce the difficulty of establishing a nonlinear mapping from the low-light image to the normal-light image; wherein, the N is the default value, Represents the set of HF components.

3. The low-light image enhancement method based on deep learning and Laplacian pyramid according to claim 1, characterized in that: The step c specifically includes: The enhanced low-frequency components , low-frequency components before enhancement and N HF components Cascade fusion is performed in the channel dimension, and then N 5 3×3 convolutions extract shallow features, and finally use N 6 residual blocks extract deep features, which are then input into the cross-layer feature enhancement module to guide the first N The enhancement of the HF component; wherein the N 5 and N 6 is the default value.

4. The low-light image enhancement method based on deep learning and Laplacian pyramid according to claim 1, characterized in that: The step d specifically includes: The cross-layer feature enhancement module is used to fuse and enhance the different frequency components decomposed by the Laplace pyramid; ordinary convolution is used in the cross-layer feature enhancement module to enhance the content details of the image; it first uses a convolution layer to obtain high-frequency components The shallow features of N 7 cross-layer feature enhancement blocks gradually fuse the lower layer features and finally use N 8 convolutional layers restore the features to obtain the final enhanced high-frequency components; wherein, N 7 and N 8 is the default value.

5. The low-light image enhancement method based on deep learning and Laplacian pyramid according to claim 4, characterized in that: The cross-layer feature enhancement block uses affine transformation to fuse the lower layer guidance features, that is, ,in, represents element-wise multiplication, is the feature to be processed before affine transformation, is the feature after affine transformation, and They are the next layer features of the current pyramid layer Use deconvolution to perform two times upsampling and then use N 9 convolutional layers are learned; Further adoption N 10 The feature enhancement within a single cross-layer feature enhancement block can be completed by performing feature refinement with a cascade of 3×3 convolutions and a residual convolution attention module; wherein, N 9 and N 10 is the default value.

6. The low-light image enhancement method based on deep learning and Laplacian pyramid according to claim 1 or 2, characterized in that: In the step e, all enhanced frequency components are fused using a Laplace pyramid reconstruction function to obtain a final enhanced image; The reconstruction function only needs to iteratively upsample the enhanced features of the lower layer by a factor of two and add them to the enhanced components of the current layer, so as to reconstruct the enhanced frequency components into an enhanced image.

Images