A weak light image enhancement deep learning method based on channel attention

Abstract

This invention relates to the field of low-light image enhancement technology, and more particularly to a deep learning method for low-light image enhancement based on channel attention. First, for the input low-light image, a pixel illumination estimation module is fed into it for feature extraction. After gain estimation, the illumination adjustment coefficient required for each pixel is adjusted to obtain the gain coefficient for each pixel. The extracted features are multiplied by the gain coefficient to obtain a preliminary illumination map. The obtained preliminary illumination map is then fed into a channel attention block, which enhances the structural and pattern information in important channels and performs noise reduction. The results of the channel attention block are input into a fine-tuning module, which fuses and fine-tunes the features of different receptive fields with the original features to obtain the final features. The obtained features are then subjected to convolution, activation, and other operations. The low-light image and the normal illumination image are supervised using the L1 loss function and the similarity loss function, ultimately resulting in a high-precision normal illumination image.

Description

Technical Field

[0001] This invention relates to the field of low-light image enhancement technology, and in particular to a deep learning method for low-light image enhancement based on channel attention.

[0002] Background technology.

[0003] Low-light images represent images acquired under insufficient illumination conditions that do not fully characterize the sensor. Therefore, the resulting images do not have ideal histogram distributions, necessitating an algorithm to enhance them and achieve better image quality. Global and local histogram equalization (GEM) methods are commonly used solutions for low-light image enhancement. Furthermore, Retinex Theory adds a theoretical dimension to the understanding of the problem. According to Retinex Theory, the global solution in low-light image enhancement is based on the Illumination Map Estimation (IME) method. While the IME method has proven effective in low-light image enhancement, as a traditional approach, it does not generalize well to different scenarios.

[0004] Weak images are characterized by insufficient light representation on the sensor, resulting in dark and low dynamic range projections. Most existing methods use gain methods to recover illumination intensity, employing a simple approach of multiplying a weak-light image by a single gain coefficient to obtain a strong-light image. However, the ISP framework introduces non-linear effects, such as local histogram enhancement, where each pixel experiences different illumination. Therefore, multiplying a single gain coefficient by the weak-light image does not enhance the illumination intensity of different pixels, requiring individual correction. Simply adjusting the gain coefficient is insufficient, as it may lead to over-illumination in some areas while other areas remain dark.

[0005] In recent years, deep learning-based low-light image enhancement methods have developed rapidly, mainly focusing on four categories: supervised learning, semi-supervised learning, zero-shot learning, and unsupervised learning. Supervised methods generally utilize a single end-to-end network to extract enhanced features to recover low-light images. Secondly, there are also networks designed based on retinal theory to reconstruct the illumination and reflow components of the enhanced image. Deep learning-based solutions have provided good results for low-light image restoration tasks. Based on this, this invention starts with pixel-level scene illumination estimation. While processing the image histogram of uneven illumination, it addresses the inherent noise in the input, which becomes more prominent after illumination estimation. To address the noise and color inconsistencies, channel attention is designed to solve the denoising and color correction problems.

[0006] In summary, this invention designs a deep learning method for low-light image enhancement based on channel attention, using pixel-level scene illumination estimation, channel attention denoising and color correction to ultimately obtain a high-precision normal lighting image. Summary of the Invention

[0007] The purpose of this invention is to provide a deep learning method for low-light image enhancement based on channel attention, so as to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A deep learning method for low-light image enhancement based on channel attention, the specific steps of which are as follows:

[0010] Step S1, Data Construction and Data Preprocessing: The main open-source datasets for low-light image enhancement are LOL-v1Dataset, LOL-v2-real Dataset and Rellisur Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above three datasets.

[0011] Step S2, Pixel Lighting Estimation (PLE): For the input low-light image, the pixel lighting estimation (PLE) module is sent to extract features. After gain estimation, the lighting adjustment coefficient required for each pixel is adjusted to obtain the gain coefficient of each pixel. The extracted features are multiplied by the gain coefficient to obtain the preliminary lighting map.

[0012] Step S3, Channel Attention Network (CAN): The obtained preliminary illumination map is fed into the Channel Attention Network (CAN), which enhances the structural and pattern information in important channels and performs noise reduction. The results of the channel attention blocks are then input into the fine-tuning module, which fuses and fine-tunes the features of different receptive fields with the original features to obtain the final features.

[0013] Step S4, Decoding Head: Perform convolution, activation, and other operations on the obtained features, and use the L1 loss function and similarity loss function to supervise the low-light image and the normal lighting image to obtain the final normal lighting image.

[0014] As a preferred embodiment of the present invention, the image in S1 is an RGB low-light image with a length of 1280, a width of 720, and a depth of 3.

[0015] As a preferred embodiment of the present invention, the low-light image input in S2 is sent to the Pixel Lighting Estimation (PLE) module for feature extraction. First, the image is initially extracted through a 3*3 convolution and two 1*1 convolutions. Then, the obtained features are subjected to double-layer feature extraction. The first branch is basic feature extraction, and the other branch is to first perform average pooling and then linear estimation. Finally, the result of multiplying the two branches is sent to the gain estimation module. The gain estimation module performs two 5*5 convolutions, two 3*3 convolutions, one 1*1 convolution, as well as average pooling and activation functions. The gain estimation adjusts the illumination adjustment coefficient required for each pixel to obtain the gain coefficient of each pixel. The input image is converted into latent features, and the extracted features are multiplied by the gain coefficient to obtain the preliminary illumination map.

[0016] In a preferred embodiment of the present invention, in step S3, the preliminary illumination map is fed into a Channel Attention Network (CAN). The Channel Attention Network is divided into an attention feature extraction block and a hierarchical feature fusion block. The attention feature extraction block first uses two 3*3 convolutions and one 1*1 convolution to extract features, and then includes three self-attention blocks. Finally, the result of adding the self-attention blocks and the 1*1 convolutions is fed into the hierarchical feature fusion block. The hierarchical feature fusion block first uses a 7*7 convolution and two 1*1 convolutions to extract channel features, and then divides the features into three parts so that features with different receptive fields, activation functions and kernel sizes can be transformed during the convolution operation, which enhances the structural and pattern information in important channels, and also performs denoising processing to obtain the final features.

[0017] In a preferred embodiment of the present invention, in step S4, the obtained features are subjected to convolution, tanh activation, and other operations. The low-light image and the normal-light image are then supervised using the L1 loss function and a similarity loss function to obtain the final normal-light image. The loss function is: (1)

[0018] in and To balance the hyperparameters.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] To address the current state of low-light image enhancement, this invention provides a deep learning method for low-light image enhancement based on channel attention. First, the input low-light image is fed into a pixel illumination estimation module for feature extraction. Gain estimation is then used to adjust the illumination adjustment coefficients required for each pixel, resulting in a gain coefficient for each pixel. The extracted features are multiplied by the gain coefficients to obtain a preliminary illumination map. This preliminary illumination map is then fed into a channel attention block, which enhances structural and pattern information in important channels while also performing noise reduction. The results from the channel attention block are then input into a fine-tuning module, where features from different receptive fields are fused with the original features to obtain the final features. Finally, the obtained features are subjected to convolution and activation operations, and L1 loss and similarity loss functions are used to supervise the comparison between the low-light image and the normal illumination image, ultimately yielding a high-precision normal illumination image. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the pixel illumination estimation module of the present invention;

[0022] Figure 2 This is a schematic diagram of the channel attention network structure of the present invention;

[0023] Figure 3 This is a schematic diagram of the overall network structure of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0025] This invention provides a deep learning method for low-light image enhancement based on channel attention. First, for the input low-light image, it is fed into a pixel illumination estimation module for feature extraction. After gain estimation, the illumination adjustment coefficient required for each pixel is adjusted to obtain the gain coefficient for each pixel. The extracted features are multiplied by the gain coefficient to obtain a preliminary illumination map. This preliminary illumination map is then fed into a channel attention block, which enhances the structural and pattern information in important channels while also performing noise reduction. The results of the channel attention block are input into a fine-tuning module, where features from different receptive fields are fused and fine-tuned with the original features to obtain the final features. The obtained features are then subjected to convolution and activation operations, and L1 loss and similarity loss functions are used to supervise the comparison between the low-light image and the normal illumination image, ultimately resulting in a high-precision normal illumination image.

[0026] The present invention mainly includes the following steps: data construction, data preprocessing, pixel lighting estimation (PLE), channel attention network (CAN), and decoding head.

[0027] The technical solution adopted is described below:

[0028] Data construction, data preprocessing, pixel lighting estimation (PLE), channel attention network (CAN), and decoding head.

[0029] Step S1: Data construction and data preprocessing:

[0030] The main open-source datasets for low-light image enhancement are LOL-v1 Dataset, LOL-v2-real Dataset, and Rellisur Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above three datasets.

[0031] Step S2: Pixel Lighting Estimation (PLE)

[0032] The input low-light image is 1280*720*3 pixels in size and is fed into the Pixel Lighting Estimation (PLE) module for feature extraction. First, it undergoes a 3*3 convolution and two 1*1 convolutions for initial feature extraction. Then, the obtained features are subjected to a two-layer feature extraction: the first branch is basic feature extraction, and the second branch performs average pooling followed by linear estimation. Finally, the result of multiplying the two branches is fed into the gain estimation module. The gain estimation module uses two 5*5 convolutions, two 3*3 convolutions, one 1*1 convolution, average pooling, and activation functions to adjust the lighting adjustment coefficient for each pixel, resulting in the gain coefficient for each pixel. The input image is then converted into latent features. Multiplying the extracted features by the gain coefficient yields the preliminary illumination map. The pixel lighting estimation module is as follows: Figure 1 ;

[0033] Step S3: Channel Attention Network (CAN)

[0034] The channel attention network is divided into an attention feature extraction block and a hierarchical feature fusion block. The attention feature extraction block first extracts features using two 3x3 convolutions and one 1x1 convolution. It then includes three self-attention blocks. Finally, the sum of the self-attention blocks and the 1x1 convolutions enters the hierarchical feature fusion block. The hierarchical feature fusion block first extracts channel features using a 7x7 convolution and two 1x1 convolutions, then divides the features into three parts. This allows for transformation of features with different receptive fields, activation functions, and kernel sizes during convolution operations, enhancing the structural and pattern information in important channels. It also performs denoising to obtain the final features. The network structure diagram is shown below. Figure 2 :

[0035] Step S4: Decoder

[0036] The features obtained above are convolved and decoded using the tanh activation function. The L1 loss function and similarity loss function are then used to supervise the comparison between the low-light image and the normal-light image. The loss function is as follows: (The loss function is:) (1)

[0037] in and To balance hyperparameters, a high-precision normal illumination image is obtained through pixel estimation, channel attention network, and supervision. The overall network structure diagram is shown below. Figure 3 .

[0038] A specific implementation of a deep learning method for low-light image enhancement based on channel attention:

[0039] Based on the specific implementation, it is divided into the following parts:

[0040] Step 1: Data Preparation

[0041] The main open-source datasets for low-light image enhancement are LOL-v1 Dataset, LOL-v2-real Dataset, and Rellisur Dataset. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above three datasets.

[0042] Step 2: Training Phase

[0043] The input low-light image is 1280*720*3 pixels in size and is fed into the Pixel Lighting Estimation (PLE) module for feature extraction. First, the image is initially extracted through a 3*3 convolution and two 1*1 convolutions. Then, the obtained features are subjected to two-layer feature extraction. The first branch is basic feature extraction, and the other branch is to perform average pooling followed by linear estimation. Finally, the result of multiplying the two branches is fed into the gain estimation module. The gain estimation module performs two 5*5 convolutions, two 3*3 convolutions, one 1*1 convolution, as well as average pooling and activation functions. The gain estimation adjusts the illumination adjustment coefficient required for each pixel, resulting in the gain coefficient for each pixel. The input image is converted into latent features, and the extracted features are multiplied by the gain coefficient to obtain the preliminary illumination map. The initial illumination image is fed into a channel attention network. First, feature extraction is performed using two 3x3 convolutions and one 1x1 convolution. Next, three self-attention blocks are included. Finally, the sum of the self-attention blocks and the 1x1 convolutions is fed into a hierarchical feature fusion block. This block first extracts channel features using a 7x7 convolution and two 1x1 convolutions, then divides the features into three parts. This allows for transformation of features with different receptive fields, activation functions, and kernel sizes during convolution operations, enhancing structural and pattern information in important channels. Noise reduction is also performed to obtain the final features. These features are then convolved and decoded using the tanh activation function to obtain a high-precision normal illumination image.

[0044] Step 3: Testing Phase

[0045] For the test low-light image, it is fed into the network for inference to obtain the enhanced normal lighting result image. The low-light image and the normal lighting image are supervised by the L1 loss function and the similarity loss function to obtain the final normal lighting image. The peak signal-to-noise ratio PSNR is calculated to obtain the final accuracy.

[0046] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A deep learning method for low-light image enhancement based on channel attention, the specific steps of which are as follows: Step S1, Data Construction and Data Preprocessing: The open-source datasets for low-light image enhancement are LOL-v1 Dataset, LOL-v2-real Dataset and Rellisur Dataset. When training on your own dataset, you need to prepare the data format to be consistent with the above three datasets. Step S2, Pixel Lighting Estimation (PLE): For the input low-light image, the pixel lighting estimation (PLE) module is sent to extract features. After gain estimation, the lighting adjustment coefficient required for each pixel is adjusted to obtain the gain coefficient of each pixel. The extracted features are multiplied by the gain coefficient to obtain the preliminary lighting map. Step S3, Channel Attention Network (CAN): The obtained preliminary illumination map is fed into the Channel Attention Network (CAN), which enhances the structural and pattern information in important channels and performs noise reduction. The results of the channel attention blocks are then input into the fine-tuning module, which fuses and fine-tunes the features of different receptive fields with the original features to obtain the final features. Step S4, Decoding Head: Perform convolution and activation operations on the obtained features, and use the L1 loss function and similarity loss function to supervise the low-light image and the normal lighting image, finally obtaining a high-precision normal lighting image.

2. The deep learning method for low-light image enhancement based on channel attention as described in claim 1, characterized in that, The image in S1 is an RGB low-light image with a length of 1280, a width of 720, and a depth of 3.

3. The deep learning method for low-light image enhancement based on channel attention as described in claim 1, characterized in that, The low-light image input in S2 is fed into the Pixel Lighting Estimation (PLE) module for feature extraction. First, the image is initially extracted through a 3*3 convolution and two 1*1 convolutions. Then, the obtained features are subjected to two-layer feature extraction. The first branch is basic feature extraction, and the other branch is to perform average pooling followed by linear estimation. Finally, the result of multiplying the two branches is fed into the gain estimation module. The gain estimation module performs two 5*5 convolutions, two 3*3 convolutions, one 1*1 convolution, as well as average pooling and activation functions. The gain estimation adjusts the illumination adjustment coefficient required for each pixel, resulting in the gain coefficient for each pixel. The input image is converted into latent features, and the extracted features are multiplied by the gain coefficient to obtain the preliminary illumination map.

4. The deep learning method for low-light image enhancement based on channel attention according to claim 1, characterized in that, In step S3, the preliminary illumination map is fed into the Channel Attention Network (CAN). The CAN is divided into an attention feature extraction block and a hierarchical feature fusion block. The attention feature extraction block first uses two 3*3 convolutions and one 1*1 convolution to extract features, then includes three self-attention blocks. Finally, the result of adding the self-attention blocks and the 1*1 convolutions is fed into the hierarchical feature fusion block. The hierarchical feature fusion block first uses a 7*7 convolution and two 1*1 convolutions to extract channel features, and then divides the features into three parts so that features with different receptive fields, activation functions and kernel sizes can be transformed during the convolution operation. This strengthens the structural and pattern information in important channels, and also performs denoising processing to obtain the final features.

5. The deep learning method for low-light image enhancement based on channel attention according to claim 1, characterized in that, In S4, the obtained features are convolutional and tanh activation operations. The L1 loss function and similarity loss function are used to supervise the comparison between the low-light image and the normal-light image, resulting in the final normal-light image. The loss function is: (1) in and To balance the hyperparameters.

Images