A low-light image enhancement method based on dual-channel attention

Abstract

This paper proposes a low-light image enhancement method based on dual attention. First, to better express the long-range spatial semantic features between pixels when extracting illumination features, this method introduces multi-scale dilated convolution with a dilation factor of [9 3 1 9 3 1] to expand the spatial receptive field and express the property that semantic dependencies between pixels weaken with increasing distance. Next, to address the noise problem that arises during brightness enhancement, we design an illumination fusion dual self-attention module. This module not only optimizes channel self-attention but also embeds spatial self-attention based on multi-scale dilated convolution to better express the long-range spatial dependencies between pixels. Finally, we adopt the multi-level skip connection method of Unet++ to better fuse the semantic information of damage repair layers at different scales. While maintaining acceptable computational complexity, this method can effectively express long-range dependencies between pixels and demonstrate excellent low-light image enhancement performance.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a low-illumination image enhancement method based on dual-channel attention. Background Art

[0002] Brightness information is a key indicator for expressing image content, and brightness enhancement of low-light images is an important research direction in the field of vision.

[0003] In the field of low-light enhancement, common traditional image processing methods include histogram equalization and gamma correction. These methods adhere to basic imaging principles and are highly interpretable. However, each algorithm has a limited scope of applicability, and enhanced images in complex lighting and dynamic scenes lack naturalness. Based on the principles of visual imaging, light strikes the surfaces of different objects, and the different materials of the objects reflect a portion of the light, projecting it onto the retina. The Retinex theory decomposes an image into a framework of illumination and reflection components, enhancing low-light images by adjusting the illumination components. This method provides theoretical support for more effective solutions to the challenging problem of low-light image enhancement in the future. However, this method's enhancement process ignores the effects of noise; simply enhancing brightness will result in the retention and amplification of noise.

[0004] In recent years, there have been continuous studies on applying convolutional neural networks and transformer models to the field of low-light image enhancement, and good results have been achieved. Convolutional networks can effectively capture the regional spatial contextual information of images, and have certain effects when applied to low-light image enhancement. However, convolutional networks have limitations in obtaining long-range image dependencies. On the other hand, transformer-based methods can express spatial long-range dependencies by establishing a self-attention mechanism, which can better restore the lighting details of low-light images. However, the computational complexity of transformers is generally the square of the size of the space, and there are problems with slow inference speed and high resource consumption. Therefore, the present invention aims to design a dual-channel attention model that can express long-range spatial semantic relationships while keeping the computational complexity controllable. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of Retinex-based self-attention methods in expressing long-range spatial semantic dependencies, thereby achieving more effective illumination restoration. To this end, this invention proposes a low-light image enhancement method based on dual-channel attention. This method extracts long-range spatial semantic features by designing multi-scale dilated convolutions. It then uses a self-attention mechanism to establish long-range spatial semantic dependencies between pixels, thereby achieving image illumination restoration and ultimately improving the effect of low-light image enhancement.

[0006] In order to achieve the above objectives, the present invention provides a low-light image enhancement method based on dual-channel attention, which mainly includes the following seven parts: the first part is to preprocess the image data set; the second part is to calculate the illumination and illuminate the image; the third part is to extract the illumination features of the multidimensional space based on the input image and the illumination prior; the fourth part is to perform illumination fusion on the illuminated image and the illumination features; the fifth part is to fuse the illumination based on the channel and spatial attention mechanism; the sixth part is to perform multi-scale illumination fusion on the fourth and fifth parts based on the UNet++ network structure to repair the illumination; the seventh part is to train and test the low-light image enhancement network model based on dual-channel attention, and finally obtain the image after illumination enhancement and restoration, specifically:

[0007] The first part consists of two steps:

[0008] Step 1: Download the low-light enhanced image datasets LOLv1 and LOLv2_real, and select 485 and 689 pairs of low-light images and their corresponding normal-brightness images as training data respectively;

[0009] Step 2: Each image pair in LOLv1 is randomly cropped to 128×128, and each image pair in LOLv2_real is randomly cropped to 256×256. The cropped images undergo geometric augmentation, including random flipping, rotation, scaling, and cropping. These images are then subjected to mixed data augmentation, using the Mixup technique to generate new training samples by randomly selecting image pairs.

[0010] The second part consists of one step:

[0011] In step 3, the low-light image obtained in step 2 is used as the input image, and the mean of the image channels is used as the illumination prior. After concatenating the two, they are fused through a 1×1 convolution and used as the input for a 3×3 convolution with a dilation factor of [9 3 1 9 3 1] to extract long-range spatial semantic relationships. At this point, the channel dimension increases from 3 to 40, resulting in the light-up feature.

[0012] The third part consists of a single step:

[0013] Step 4: The lighting features obtained in step 3 are fused through 1×1 convolution to generate lighting illumination, and then the input image is enhanced to obtain a lighting map;

[0014] The fourth part consists of a step:

[0015] In step 5, the lighting features from step 3 and the lighting map from step 4 are used as input features. The input passes through a normalization layer and an illumination fusion dual attention module to extract spatial and channel dimension information, and is further fused through a normalization layer and a feedforward network module to finally output the fused features.

[0016] The fifth part consists of two steps:

[0017] Step 6: Take the normalized output from step 5 as input and encode it into Q, K, and V matrices respectively. Take the light feature as input and perform channel self-attention operation with the Q, K, and V matrices to obtain the channel self-attention weight matrix.

[0018] In step 7, the normalized output from step 5 is used as input and encoded into a matrix K through multi-scale dilated convolution. The light features are used as input and spatial self-attention operations are performed on the K matrix and the Q and V matrices from step 6 to obtain a spatial self-attention weight matrix. This matrix is ​​concatenated with the channel self-attention weight matrix from step 6 to form a dual self-attention weight matrix of illumination fusion.

[0019] Part 6 consists of a single step:

[0020] In step 8, the lighting repairer is designed as a Unet++ network structure. It generates multi-scale encoded features through three encoding and feature fusion, and then decodes the image through three decoding and feature fusion. Skip connections are added to each layer to reduce information loss during the encoding and decoding process. Finally, a residual image is generated, and the lighting image is repaired to obtain an enhanced image.

[0021] Part 7 consists of two steps:

[0022] Step 9: Input the training set samples in step 2 into the network from step 3 to step 8, and set the network hyperparameters: the learning rate is 2e -4 The training epochs are 250, the batch size is 8, the optimizer is Adam (with a momentum term of 0.9 and a control parameter of 0.999), and the loss function is MAE. By training the network, the final dual-channel attention low-light image enhancement pre-training model is obtained;

[0023] In step 10, the low-light images in the LOLv1 and LOLv2_real public test sets are input into the pre-trained model obtained in step 9. The network will restore the illumination and enhance the image brightness.

[0024] The present invention provides a low-light image enhancement method based on dual-channel attention. First, in lighting estimation, in order to better contain the spatial position relationship of the extracted lighting features, especially to better express the long-range semantic features of space, a multi-scale dilated convolution structure is proposed. This structure can expand the receptive field while expressing the characteristic that the semantic dependency between pixels weakens with increasing distance in feature extraction. Then, in the lighting fusion module, in order to allow self-attention to express the long-range semantic dependency relationship in space, the computational complexity of spatial attention is reduced from the square of the space size to a multiple of the space size. A multi-scale dilated convolution spatial attention module is designed in the lighting fusion dual attention module. Finally, in order to better fuse the lighting fusion semantic information of different levels and scales to reduce information loss, the unet++ multi-level jump connection structure is adopted in lighting restoration. This paper uses a six-layer dilated convolutional structure with dilation coefficients of r = 9, 3, 1, 9, 3, 1 to better extract the distribution characteristics of multiple light sources. The present invention also incorporates a dilated convolution-based spatial attention module into its illumination fusion dual attention, which can express long-range spatial semantic dependencies while reducing the computational complexity of the transformer spatial attention from the square of the spatial size to a multiple of the spatial size. This invention can perform illumination enhancement on low-light images with excellent processing effects, high operational efficiency, and good robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the overall network framework diagram of the present invention;

[0026] Figure 2 The illumination fusion attention module of the present invention;

[0027] Figure 3 It is the multi-scale dilated convolution feature map used in the present invention;

[0028] Figure 4 Enhance the effect of low-light images;

[0029] Figure 5 It is based on HSV space enhancement effect. DETAILED DESCRIPTION

[0030] In order to better understand the present invention, the low-light image enhancement method based on dual-channel attention of the present invention is described in more detail below in conjunction with specific embodiments. In the following description, the detailed description of the current existing technology may dilute the subject content of the present invention, and these descriptions will be omitted here.

[0031] Step 1: Download the low-light enhanced image datasets LOLv1 and LOLv2_real, and select 485 and 689 pairs of low-light images and their corresponding normal-brightness images as training data respectively;

[0032] Step 2: Each image pair in LOLv1 is randomly cropped to 128×128, and each image pair in LOLv2_real is randomly cropped to 256×256. The cropped images undergo geometric augmentation, including random flipping, rotation, scaling, and cropping. These images are then subjected to mixed data augmentation, using the Mixup technique to generate new training samples by randomly selecting image pairs.

[0033] Figure 1 This is the overall network framework diagram of the low-light image enhancement method based on dual-channel attention of the present invention. In this embodiment, the following steps are performed:

[0034] Step 3, such as Figure 1 As shown in (a), the low-light image obtained in step 2 is used as the input image L p , and the mean of the image channels is used as the illumination prior I. After splicing these two, they are fused through a 1×1 convolution and used as the input of a 3×3 convolution with a dilation factor of [9 3 1 9 31] to extract spatial long-range semantic relationships. At this time, the channel dimension increases from 3 dimensions to 40 dimensions, and the lighting feature is obtained;

[0035] Step 4, such as Figure 1 As shown in (a), the light features obtained in step 3 are fused through 1×1 convolution to generate light illumination, and then the input image is enhanced to obtain the light map I lu ;

[0036] Step 5, such as Figure 2 As shown in (a), the lighting features in step 3 and the lighting image I in step 4 are combined. lu The input is passed through a normalization layer and an illumination fusion dual attention module to extract spatial and channel dimension information, and then further fused through a normalization layer and a feedforward network module to finally output the fused features.

[0037] In step 6, the normalized output from step 5 is used as input and encoded into the Q, K, and V matrices respectively. The light-up feature is used as input and the Q, K, and V matrices are subjected to the channel self-attention operation to obtain the channel self-attention weight matrix. The specific implementation is as follows:

[0038] Step 6.1, as Figure 2 As shown in (b), the normalized output in step 5 is taken as input and reshaped to HW×C, which is then encoded into Q, K, and V parameter matrices respectively;

[0039] Step 6.2: Cross-multiply the transpose of the key matrix K with the query matrix Q to obtain a channel self-attention score matrix of shape C×C;

[0040] Step 6.3, reshape the shape of the lit feature into HW×C, and perform dot multiplication with the value matrix V to obtain the feature weight value matrix;

[0041] Step 6.4, cross-multiply the feature weight value matrix with the channel self-attention score matrix to obtain the channel self-attention weight matrix;

[0042] In step 7, the normalized output from step 5 is used as input and encoded into a matrix K through multi-scale dilated convolution. The light features are used as input and spatial self-attention operations are performed on the K matrix and the Q and V matrices from step 6 to obtain a spatial self-attention weight matrix. This matrix is ​​concatenated with the channel self-attention weight matrix from step 6 to form a dual self-attention weight matrix of illumination fusion. The specific implementation is as follows:

[0043] Step 7.1, as Figure 2 As shown in (b), the normalized output in step 5 is used as input, and a 3×3 convolution with a dilation coefficient of [9 31 9 3 1] is performed to extract the spatial long-range semantic features and obtain the spatial key matrix K;

[0044] Step 7.2: Multiply the spatial key matrix K by the query matrix Q in step 6 to obtain the spatial self-attention score matrix to express the interdependence between pixels in space.

[0045] Step 7.3: Multiply the spatial self-attention score matrix by the feature weight value matrix in step 6 to obtain the spatial self-attention weight matrix.

[0046] In step 7.4, the spatial self-attention weight matrix is ​​concatenated with the channel self-attention weight matrix in step 6 to form the illumination fusion dual self-attention weight matrix.

[0047] Step 8, such as Figure 1 As shown in (b), the lighting repairer is designed as a Unet++ network structure. It generates multi-scale encoded features through three encoding and feature fusion, and then decodes them through three decoding and feature fusion. Skip connections are added to each layer to reduce information loss during the encoding and decoding process. Finally, a residual image is generated, and the lighting image is repaired to obtain an enhanced image. The specific implementation is as follows:

[0048] Step 8.1, as Figure 1 As shown in (b), the lighting repairer is designed as a Unet++ network structure. In the encoding part, light up Figure I lu After a 3×3 convolution with a stride of 2, a lighting fusion encoding, a 4×4 convolution downsampling with a stride of 2, two lighting fusion encodings, and a 4×4 convolution with a stride of 2, the multi-scale encoding features F0, F1, and F2 are generated. Subsequently, F2 undergoes two more lighting fusion encodings.

[0049] In step 8.2, the decoding part designs an upsampling branch that is symmetrical with the encoding part. After F1 and F2 undergo two illumination fusion encodings, they are upsampled using a 2×2 deconvolution with a stride of 2. At the same time, skip connections are added to each layer to reduce information loss during the encoding and decoding process.

[0050] Step 8.3, finally, the decoding part outputs the residual image I re , will light up Figure I lu With the residual image I re Add together to get the low illumination enhanced image I en ;

[0051] Step 9: Input the training set samples in step 2 into the network from step 3 to step 8, and set the network hyperparameters: the learning rate is 2e -4 The training epochs are 250, the batch size is 8, the optimizer is Adam (with a momentum term of 0.9 and a control parameter of 0.999), and the loss function is MAE. By training the network, the final dual-channel attention low-light image enhancement pre-training model is obtained;

[0052] Step 10: Input the low-light images in the LOLv1 and LOLv2_real public test sets into the pre-trained model obtained in step 9. The network will restore the illumination and enhance the image brightness.

[0053] This paper aims to balance the computational complexity of spatial self-attention and the expression of long-range spatial semantic dependencies to improve the enhancement effect of low-light images. A low-light image enhancement method based on dual-channel attention is proposed. First, during the illumination estimation process, this method uses a 3×3 convolution with a dilation factor of [9 3 1 9 3 1] to extract long-range spatial semantic features to generate illumination features. Second, a new convolution-based spatial attention module is introduced into the illumination fusion attention module, which can effectively express long-range spatial semantic relationships and reduce the computational complexity of transformer spatial attention from the square of the spatial size to a complexity proportional to the spatial size. Finally, in the illumination restoration phase, a Unet++ multi-level skip connection structure is adopted to effectively fuse semantic information at different levels and reduce information loss. While taking into account the computational complexity, this paper extracts illumination features of images through dual-channel attention, significantly improving the enhancement effect of low-light images.

[0054] Although the above describes the illustrative specific embodiments of the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For ordinary technicians in this technical field, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concepts of the present invention are protected.

Claims

1. A low-light image enhancement method based on dual-channel attention. Multi-scale dilated convolution is designed to extract long-range spatial semantic information. Then, a self-attention mechanism is used to establish long-range spatial semantic dependencies between pixels, thereby achieving image illumination restoration and ultimately improving the effect of low-light image enhancement. The method includes seven parts: dataset preprocessing, image lighting, multi-dimensional spatial lighting feature extraction, multi-scale lighting fusion, lighting fusion, dual-channel attention fusion lighting, and network model training and testing. The dataset preprocessing part includes two steps: Step 1: Download the low-light enhanced image datasets LOLv1 and LOLv2_real, and select 485 and 689 pairs of low-light images and their corresponding normal-brightness images as training data respectively; Step 2: Each pair of images in LOLv1 is randomly cropped to 128×128 size, and each pair of images in LOLv2_real is randomly cropped to 256×256 size. The cropped images are then geometrically enhanced, including random flipping, rotation, and scaling. Then, hybrid data enhancement is performed, and the Mixup technology is used to randomly select image pairs to generate training set samples; The image lighting part consists of one step: Step 3: Use the low-light image in the training set as the input image, and use the mean of the image channel as the illumination prior. After concatenating the input image and the illumination prior, the concatenated image and the illumination prior are fused through a 1×1 convolution and used as the input of a 3×3 convolution with a dilation coefficient of [9 3 1 9 3 1] to extract long-range spatial semantic relationships. At this time, the channel dimension increases from 3 to 40 dimensions, and the light-up feature is obtained. The multi-dimensional space lighting feature extraction part includes a step: Step 4: The light features are fused through 1×1 convolution to generate light illumination, and then the input image is enhanced to obtain the light map I lu ; The multi-scale illumination fusion part includes one step: In step 5, the lighting repairer is designed as a Unet++ network structure. It generates multi-scale encoding features through three encoding and feature fusion, and then decodes through three decoding and feature fusion. Skip connections are added to each layer to reduce information loss caused by the encoding and decoding process. Finally, a residual image is generated, and the lighting image is repaired to obtain an enhanced image. The details are as follows: (1) In the coding section, click on Figure I lu After a 3×3 convolution with a stride of 2, feature F0 is obtained, and then after one illumination fusion encoding and a 4×4 convolution with a stride of 2, feature F1 is obtained. After two illumination fusion encodings and a 4×4 convolution with a stride of 2, feature F2 is obtained. Subsequently, F2 undergoes two more illumination fusion encodings; (2) The decoding part designs an upsampling branch that is symmetrical with the encoding. After F1 and F2 undergo two illumination fusion encodings, they are upsampled using 2×2 deconvolution with a stride of 2. At the same time, skip connections are added to each layer to reduce information loss caused by the encoding and decoding process. (3) Finally, the decoding part outputs the residual image I re , will light up Figure I lu With the residual image I re Add together to get the low illumination enhanced image I en ; The lighting fusion part consists of one step: Step 6: Input the input features and the lighting features into the illumination fusion attention module to obtain the output features, including: inputting the lighting features into the illumination fusion dual self-attention module, passing the input features through the normalization layer and then inputting them into the illumination fusion dual self-attention module to extract spatial and channel dimension information, and fusing them through further normalization layers and feedforward network modules, and finally outputting the fused features; The dual-channel attention fusion illumination part consists of two steps: Step 7: Encode the output of the input feature after the normalization layer into Q, K, and V matrices. At the same time, input the light-up feature and perform the channel self-attention operation with the Q, K, and V matrices to obtain the channel self-attention weight matrix. The details are as follows: (1) The output shape of the input features after the normalization layer is reshaped to HW×C, and then encoded into Q, K, and V matrices respectively; (2) Cross-multiply the transpose of the key matrix K with the query matrix Q to obtain a channel self-attention score matrix of shape C×C; (3) Reshape the light feature into HW×C and perform dot multiplication with the value matrix V to obtain the feature weight value matrix; (4) Cross-multiply the feature weight value matrix with the channel self-attention score matrix to obtain the channel self-attention weight matrix; Step 8: Input the output of the normalized layer into the multi-scale dilated convolutional code to form the matrix K'. At the same time, input the lighting feature and the K', Q, and V matrices to perform spatial self-attention operations to obtain the spatial self-attention weight matrix. After splicing it with the channel self-attention weight matrix, the illumination fusion dual self-attention weight matrix is ​​obtained, as follows: (1) The output of the input feature after the normalization layer is subjected to a 3×3 convolution with a dilation coefficient of [9 3 1 9 3 1] to extract the spatial long-range semantic features and obtain the spatial key matrix K'; (2) Perform a dot product of the spatial key matrix K' and the query matrix Q to obtain the spatial self-attention score matrix to express the interdependence between pixels in space; (3) Multiply the spatial self-attention score matrix by the feature weight value matrix to obtain the spatial self-attention weight matrix; (4) After concatenating the spatial self-attention weight matrix and the channel self-attention weight matrix, the illumination fusion dual self-attention weight matrix is ​​obtained; The network model training and testing part includes two steps: Step 9: Input the training set samples into the network from step 3 to step 8 and set the network hyperparameters: the learning rate is 2e -4 , the training rounds are 250, the batch size is 8, the optimizer is Adam, the momentum term is 0.9, the control parameter is 0.999, and the loss function is MAE. By training the network, the final dual-channel attention low-light image enhancement pre-training model is obtained; Step 10: Input the low-light images in the LOLv1 and LOLv2_real public test sets into the pre-trained model to restore the illumination and enhance the image brightness.

Images