Low-illumination image enhancement method based on dual-channel attention

Abstract

The invention provides a low-illumination image enhancement method based on double attention. Firstly, in order to better express spatial long-range semantic features among pixels when illumination features are extracted, multi-scale expansion convolution with an expansion coefficient of [931931] is introduced in the method, so that a spatial receptive field is expanded, and the characteristic that semantic dependence among the pixels is weakened along with distance increase can be expressed. Then, aiming at the noise problem occurring in the brightness enhancement process, an illumination fusion dual-self-attention module is designed, channel self-attention is optimized, and spatial self-attention based on multi-scale expansion convolution is embedded, so that the spatial long-range dependency relationship between pixels is better expressed; and finally, a Unit + + multi-level jump connection method is adopted, and semantic information of damage repair layers with different scales is better fused. According to the method, on the premise of acceptable calculation complexity, the long-range dependency relationship between pixels can be effectively expressed, and excellent low-illumination image enhancement performance is shown.

Description

Technical field:

[0001] The present invention relates to image processing technology, and in particular to a low-illumination image enhancement method based on dual-channel attention. Technical background:

[0002] Brightness information is a key indicator for expressing image content, and brightness enhancement of low-light images is an important research direction in the field of vision.

[0003] In the field of low-light enhancement, common traditional image processing methods include histogram equalization and gamma correction. These methods follow basic image principles and are highly interpretable, but the scope of application of each algorithm is limited, and the enhanced image is not natural enough when processing complex lighting dynamic scenes. Based on the principle of visual imaging, light shines on the surface of different objects, and the different materials of the objects reflect part of the light and project it onto the retina for imaging. The Retinex theory decomposes the image into a framework of lighting and reflection components, and enhances the low-light image by adjusting the lighting components. This method provides theoretical support for more effectively solving the challenging low-light enhancement image problem in the future. The enhancement process of this method ignores the influence of noise, and simple brightness enhancement will result in the retention and increase of noise.

[0004] In recent years, there have been continuous studies on applying convolutional neural networks and transformer models to the field of low-light image enhancement, and good results have been achieved. Convolutional networks can effectively capture the regional spatial contextual information of images, and have certain effects on low-light image enhancement. However, convolutional networks have limitations in obtaining long-range image dependencies. On the other hand, transformer-based methods can express spatial long-range dependencies by establishing a self-attention mechanism, which can better restore the lighting details of low-light images. However, the computational complexity of transformers is generally the square of the size of the space, and there are problems of slow inference speed and high resource consumption. Therefore, the present invention aims to design a dual-channel attention model that can express spatial long-range semantic relationships while controlling the computational complexity. Summary of the invention:

[0005] The purpose of the present invention is to solve the shortcomings of the self-attention method based on Retinex in expressing long-range spatial semantic dependencies, so as to perform illumination restoration more effectively. To this end, the present invention proposes a low-light image enhancement method based on dual-channel attention. The method extracts spatial long-range semantic features by designing multi-scale dilated convolutions, and then uses the self-attention mechanism to establish spatial long-range semantic dependencies between pixels, thereby achieving image illumination restoration and ultimately improving the effect of low-light image enhancement.

[0006] In order to achieve the above-mentioned object, the present invention provides a low-light image enhancement method based on dual-channel attention, which mainly includes the following seven parts: the first part is to preprocess the image data set; the second part is to calculate the illumination and illuminate the image; the third part is to extract the illumination features of the multidimensional space based on the input image and the illumination prior; the fourth part is to perform illumination fusion on the illuminated image and the illumination features; the fifth part is to fuse the illumination based on the channel and spatial attention mechanism; the sixth part is to perform multi-scale illumination fusion on the fourth part and the fifth part based on the UNet++ network structure to repair the illumination; the seventh part is to train and test the low-light image enhancement network model based on dual-channel attention, and finally obtain the image after illumination enhancement and restoration, which is specifically:

[0007] The first part consists of two steps:

[0008] Step 1, download the low-light enhanced image datasets LOLv1 and LOLv2_real, and select 485 pairs and 689 pairs of low-light images and their corresponding normal brightness images as training data respectively;

[0009] Step 2: Each pair of images in LOLv1 is randomly cropped to 128×128 size, and each pair of images in LOLv2_real is randomly cropped to 256×256 size. The cropped images are geometrically enhanced, including random flipping, rotation, scaling, and cropping.

[0010] Mix up some images for data enhancement, and use Mixup technology to randomly select image pairs to mix and generate new training samples;

[0011] The second part consists of a single step:

[0012] Step 3: Use the low-light image obtained in step 2 as the input image, and use the mean of the image channels as the illumination prior. After concatenating the two, they are fused through a 1×1 convolution and used as the input of a 3×3 convolution with a dilation factor of [9 3 1 9 3 1] to extract spatial long-range semantics.

[0013] At this time, the channel dimension increases from 3 dimensions to 40 dimensions, and the light-up feature is obtained;

[0014] The third part consists of a step:

[0015] Step 4: The lighting features obtained in step 3 are fused through 1×1 convolution to generate lighting illumination, and then the input image is enhanced to obtain a lighting map;

[0016] The fourth part consists of a step:

[0017] Step 5: Take the lighting features in step 3 and the lighting map in step 4 as input features. The input is passed through the normalization layer and the illumination fusion dual attention module to extract the spatial and channel dimension information, and then further fused through the normalization layer and the feedforward network module to finally output

[0018] Fusion features;

[0019] Part 5 consists of two steps:

[0020] Step 6: Take the normalized output in step 5 as input and encode them into Q, K, and V matrices respectively.

[0021] The K and V matrices are subjected to channel self-attention operations to obtain the channel self-attention weight matrix;

[0022] Step 7: Take the normalized output in step 5 as input and encode it into matrix K through multi-scale dilated convolution. Take the lit features as input and perform spatial self-attention operation with K matrix and Q and V matrices in step 6 to obtain the spatial self-attention weight matrix.

[0023] After being concatenated with the channel self-attention weight matrix in step 6, it is fused into the illumination fusion dual self-attention weight matrix;

[0024] Part 6 includes a step:

[0025] Step 8: The lighting repairer is designed as a Unet++ network structure. Multi-scale encoding features are generated through three encodings and feature fusion, and then decoded through three decodings and feature fusion. Skip connections are added to each layer to reduce information loss caused by the encoding and decoding process.

[0026] The residual image is formed, and the lighted image is repaired to obtain the enhanced image;

[0027] Part 7 consists of two steps:

[0028] Step 9: Input the training set samples in step 2 into the network from step 3 to step 8, and set the network hyperparameters: the learning rate is 2e -4 , the training rounds (epochs) are 250, the batch size (batch size) is 8, the optimizer is Adam (the momentum term is 0.9, the control parameter is 0.999), and the loss function is MAE. By training the network, the final dual-channel attention low-light image enhancement pre-training model is obtained;

[0029] Step 10: Input the low-light images in the LOLv1 and LOLv2_real public test sets into the pre-trained model obtained in step 9. The network will restore the illumination and enhance the image brightness.

[0030] The present invention provides a low-illumination image enhancement method based on dual-channel attention. First, in illumination estimation, in order to better contain the spatial position relationship of the extracted lighting features, especially to better express the long-range semantic features of space, a multi-scale dilated convolution structure is proposed. This structure can expand the receptive field while expressing the characteristic that the semantic dependency between pixels weakens with the increase of distance in feature extraction. Then, in the illumination fusion module, in order to allow self-attention to express the long-range semantic dependency relationship in space, the computational complexity of spatial attention is reduced from the square of the space size to a multiple of the space size, and a multi-scale dilated convolution spatial attention module is designed in the illumination fusion dual attention module. Finally, in order to better fuse the illumination fusion semantic information of different levels and scales to reduce information loss, the unet++ multi-level jump connection structure is adopted in illumination restoration. The present invention adopts a six-layer dilated convolution structure with dilation coefficients of r = 9, 3, 1, 9, 3, 1 to better extract the distribution characteristics of multiple light sources; the present invention adds a spatial attention module based on dilated convolution to the illumination fusion dual attention, which can express the spatial long-range semantic dependency while reducing the transformer spatial attention calculation complexity from the square of the space size to a multiple of the space size. The present invention can perform illumination enhancement on low-illuminance images, has good processing effect, high operating efficiency, and good robustness. Description of the drawings:

[0031] Figure 1 It is the overall framework diagram of the network of the present invention;

[0032] Figure 2 The illumination fusion attention module of the present invention;

[0033] Figure 3 It is a multi-scale dilated convolution feature map used in the present invention;

[0034] Figure 4 Enhance the effect of low-light images;

[0035] Figure 5 It is based on HSV space enhancement effect. Specific implementation method:

[0036] In order to better understand the present invention, the low-light image enhancement method based on dual-channel attention of the present invention is described in more detail below in conjunction with specific embodiments. In the following description, the detailed description of the current prior art may dilute the subject content of the present invention, and these descriptions will be ignored here.

[0037] Step 1, download the low-light enhanced image datasets LOLv1 and LOLv2_real, and select 485 pairs and 689 pairs of low-light images and their corresponding normal brightness images as training data respectively;

[0038] Step 2: Each pair of images in LOLv1 is randomly cropped to 128×128 size, and each pair of images in LOLv2_real is randomly cropped to 256×256 size. The cropped images are geometrically enhanced, including random flipping, rotation, scaling, and cropping. Then, these images are mixed data augmented, and the Mixup technique is used to randomly select image pairs to mix and generate new training samples;

[0039] Figure 1 It is a network overall framework diagram of the low-light image enhancement method based on dual-channel attention of the present invention. In this embodiment, the following steps are performed:

[0040] Step 3, such as Figure 1 As shown in (a), the low-light image obtained in step 2 is used as the input image L p , and the mean of the image channels is used as the illumination prior I. After the two are concatenated, they are fused by a 1×1 convolution and used as the input of a 3×3 convolution with a dilation factor of [9 3 1 9 31] to extract the spatial long-range semantic relationship. At this time, the channel dimension increases from 3 dimensions to 40 dimensions, and the lighting feature is obtained;

[0041] Step 4, such as Figure 1 As shown in (a), the lighting features obtained in step 3 are fused through a 1×1 convolution to generate the lighting illumination, and then the input image is enhanced to obtain the lighting map I lu ;

[0042] Step 5, such as Figure 2 As shown in (a), the lighting features in step 3 and the lighting image I in step 4 are combined. lu As input features. The input passes through the normalization layer and the illumination fusion dual attention module to extract spatial and channel dimension information, and is further fused through the normalization layer and the feedforward network module to finally output the fused features;

[0043] Step 6: Take the normalized output in step 5 as input and encode them into Q, K, and V matrices respectively. Take the light feature as input and perform channel self-attention operation with Q, K, and V matrices to obtain the channel self-attention weight matrix. The specific implementation is as follows:

[0044] Step 6.1, such as Figure 2 As shown in (b), the normalized output in step 5 is taken as input and reshaped into HW×C, and then encoded into Q, K, and V parameter matrices respectively;

[0045] Step 6.2, cross-multiply the transpose of the key matrix K with the query matrix Q to obtain a channel self-attention score matrix of shape C×C;

[0046] Step 6.3, reshape the lightened feature into HW×C, and perform dot multiplication with the value matrix V to obtain the feature weight value matrix;

[0047] Step 6.4, cross-multiply the feature weight value matrix with the channel self-attention score matrix to obtain the channel self-attention weight matrix;

[0048] Step 7: Take the normalized output in step 5 as input and encode it into matrix K through multi-scale dilated convolution. Take the light feature as input and perform spatial self-attention operation with K matrix and Q and V matrices in step 6 to obtain spatial self-attention weight matrix. This matrix is ​​concatenated with the channel self-attention weight matrix in step 6 to form the illumination fusion dual self-attention weight matrix. The specific implementation is as follows:

[0049] Step 7.1, such as Figure 2 As shown in (b), the normalized output in step 5 is used as input, and a 3×3 convolution with a dilation coefficient of [9 31 9 3 1] is performed to extract the spatial long-range semantic features and obtain the spatial key matrix K;

[0050] Step 7.2, perform a dot multiplication of the spatial key matrix K and the query matrix Q in step 6 to obtain a spatial self-attention score matrix to express the interdependence between pixels in space;

[0051] Step 7.3, multiply the spatial self-attention score matrix by the feature weight value matrix in step 6 to obtain the spatial self-attention weight matrix;

[0052] Step 7.4, concatenate the spatial self-attention weight matrix with the channel self-attention weight matrix in step 6 to form an illumination fusion dual self-attention weight matrix;

[0053] Step 8, such as Figure 1 As shown in (b), the lighting repairer is designed as a Unet++ network structure, which generates multi-scale encoding features through three encodings and feature fusion, and then decodes through three decodings and feature fusion, and adds skip connections in each layer to reduce the information loss caused by the encoding and decoding process. Finally, the residual image is generated, and the lighting image is repaired to obtain an enhanced image. The specific implementation is as follows:

[0054] Step 8.1, such as Figure 1 As shown in (b), the lighting repairer is designed as a Unet++ network structure. In the encoding part, light up Figure I lu After a 3×3 convolution with a stride of 2, an illumination fusion encoding, a 4×4 convolution downsampling with a stride of 2, two illumination fusion encodings, and a 4×4 convolution with a stride of 2, multi-scale encoding features F0, F1, and F2 are generated. Subsequently, F2 undergoes two more illumination fusion encodings;

[0055] In step 8.2, the decoding part designs an upsampling branch that is symmetrical with the encoding. After F1 and F2 are encoded twice with illumination fusion, they are upsampled using 2×2 deconvolution with a step size of 2. At the same time, skip connections are added to each layer to reduce the information loss caused by the encoding and decoding process;

[0056] Step 8.3, finally, the decoding part outputs the residual image I re , will light up Figure I lu With the residual image I re Add together to get the low illumination enhanced image I en ;

[0057] Step 9: Input the training set samples in step 2 into the network from step 3 to step 8, and set the network hyperparameters: the learning rate is 2e -4 , the training rounds (epochs) are 250, the batch size (batch size) is 8, the optimizer is Adam (the momentum term is 0.9, the control parameter is 0.999), and the loss function is MAE. By training the network, the final dual-channel attention low-light image enhancement pre-training model is obtained;

[0058] Step 10: Input the low-light images in the LOLv1 and LOLv2_real public test sets into the pre-trained model obtained in step 9. The network will restore the illumination and enhance the image brightness.

[0059] The present invention aims to balance the computational complexity of spatial self-attention and the expression of spatial long-range semantic dependency to improve the enhancement effect of low-light images, and proposes a low-light image enhancement method based on dual-channel attention. First, in the illumination estimation process, the method uses a 3×3 convolution with a dilation coefficient of [9 3 1 9 3 1] to extract spatial long-range semantic features to generate illumination features. Secondly, a new convolution-based spatial attention module is introduced in the illumination fusion attention module, which can effectively express spatial long-range semantic relationships and reduce the computational complexity of transformer spatial attention from the square of the spatial size to a complexity proportional to the spatial size. Finally, in the illumination restoration link, the Unet++ multi-level jump connection structure is adopted, which can effectively fuse semantic information at different levels and reduce information loss. Under the premise of taking into account the computational complexity, the present invention extracts the illumination features of the image through dual-channel attention, which significantly improves the enhancement effect of low-light images.

[0060] Although the above describes an illustrative specific embodiment of the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiment. For ordinary technicians in this technical field, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A low-light image enhancement method based on dual-channel attention, which designs multi-scale dilated convolution to extract spatial long-range semantic information, and then uses the self-attention mechanism to establish spatial long-range semantic dependencies between pixels, thereby realizing image illumination restoration and ultimately improving the effect of low-light image enhancement. It includes seven parts: dataset preprocessing, image lighting, multi-dimensional space lighting feature extraction, lighting fusion, dual-channel attention fusion lighting, multi-scale lighting fusion, network model training and testing: The first part consists of two steps: Step 1, download the low-light enhanced image datasets LOLv1 and LOLv2_real, and select 485 pairs and 689 pairs of low-light images and their corresponding normal brightness images as training data respectively; Step 2: Each pair of images in LOLv1 is randomly cropped to 128×128 size, and each pair of images in LOLv2_real is randomly cropped to 256×256 size. The cropped images are geometrically enhanced, including random flipping, rotation, scaling, and cropping. Then, these images are mixed data augmented, and the Mixup technique is used to randomly select image pairs to mix and generate new training samples; The second part consists of a single step: Step 3: Use the low-light image obtained in step 2 as the input image, and use the mean of the image channels as the illumination prior. After concatenating the two, they are fused by a 1×1 convolution and used as the input of a 3×3 convolution with a dilation coefficient of [9 3 1 9 3 1] to extract spatial long-range semantic relationships. At this time, the channel dimension increases from 3 dimensions to 40 dimensions, and the light-up feature is obtained; The third part consists of a step: Step 4: The lighting features obtained in step 3 are fused through 1×1 convolution to generate lighting illumination, and then the input image is enhanced to obtain the lighting image I lu ; The fourth part consists of a step: Step 5: Combine the light-up features in step 3 and the light-up image I in step 4 lu As input features. The input passes through the normalization layer and the illumination fusion dual attention module to extract spatial and channel dimension information, and is further fused through the normalization layer and the feedforward network module to finally output the fused features; Part 5 consists of two steps: Step 6: Take the normalized output in step 5 as input and encode them into Q, K, and V matrices respectively. Take the light feature as input and perform channel self-attention operation with Q, K, and V matrices to obtain the channel self-attention weight matrix. The specific implementation is as follows: (1) Take the normalized output in step 5 as input and reshape it to HW×C, and then encode it into Q, K, and V parameter matrices respectively; (2) Cross-multiply the transpose of the key matrix K with the query matrix Q to obtain a channel self-attention score matrix of shape C×C; (3) Reshape the lightened feature into HW×C and perform dot multiplication with the value matrix V to obtain the feature weight value matrix; (4) Cross-multiply the feature weight value matrix with the channel self-attention score matrix to obtain the channel self-attention weight matrix; Step 7: Take the normalized output in step 5 as input and encode it into matrix K through multi-scale dilated convolution. Take the light feature as input and perform spatial self-attention operation with K matrix and Q and V matrices in step 6 to obtain spatial self-attention weight matrix. This matrix is ​​concatenated with the channel self-attention weight matrix in step 6 to form the illumination fusion dual self-attention weight matrix. The specific implementation is as follows: (1) Take the normalized output in step 5 as input, and perform a 3×3 convolution with a dilation coefficient of [9 3 1 9 3 1] to extract the spatial long-range semantic features and obtain the spatial key matrix K; (2) Multiply the spatial key matrix K by the query matrix Q in step 6 to obtain the spatial self-attention score matrix to express the interdependence between pixels in space; (3) Multiply the spatial self-attention score matrix by the feature weight value matrix in step 6 to obtain the spatial self-attention weight matrix; (4) Concatenate the spatial self-attention weight matrix with the channel self-attention weight matrix in step 6 to form an illumination fusion dual self-attention weight matrix; Part 6 includes a step: Step 8: The lighting repairer is designed as a Unet++ network structure. Multi-scale encoding features are generated through three encodings and feature fusion, and then decoded through three decodings and feature fusion. Skip connections are added to each layer to reduce information loss during the encoding and decoding process. Finally, a residual image is generated, and the lighting image is repaired to obtain an enhanced image. The specific implementation is as follows: (1) The lighting repairer is designed as a Unet++ network structure. In the encoding part, light up Figure I lu After a 3×3 convolution with a stride of 2, an illumination fusion encoding, a 4×4 convolution downsampling with a stride of 2, two illumination fusion encodings, and a 4×4 convolution with a stride of 2, multi-scale encoding features F0, F1, and F2 are generated. Subsequently, F2 undergoes two more illumination fusion encodings; (2) The decoding part designs an upsampling branch that is symmetrical with the encoding part. After F1 and F2 are encoded twice with illumination fusion, they are upsampled using a 2×2 deconvolution with a step size of 2. At the same time, skip connections are added to each layer to reduce information loss caused by the encoding and decoding process; (3) Finally, the decoding part outputs the residual image I re , will light up Figure I lu With the residual image I re Add together to get the low illumination enhanced image I en ; Part 7 consists of two steps: Step 9: Input the training set samples in step 2 into the network from step 3 to step 8, and set the network hyperparameters: the learning rate is 2e -4 , the training rounds (epochs) are 250, the batch size (batch size) is 8, the optimizer is Adam (the momentum term is 0.9, the control parameter is 0.999), and the loss function is MAE. By training the network, the final dual-channel attention low-light image enhancement pre-training model is obtained; Step 10: Input the low-light images in the LOLv1 and LOLv2_real public test sets into the pre-trained model obtained in step 9. The network will restore the illumination and enhance the image brightness.

2. The low-light image enhancement method based on dual-channel attention according to claim 1, characterized in that: In step 3, the illumination prior is concatenated with the input image, and then a 3×3 dilated convolution is performed to extract the spatial long-range semantic features as the lighting features.

3. The low-light image enhancement method based on dual-channel attention according to claim 1, characterized in that: In step 7(1), the features normalized in step 5 are subjected to 3×3 dilated convolution to extract spatial long-range semantic features as the construction matrix K.

4. The low-light image enhancement method based on dual-channel attention according to claim 1, characterized in that: In step 7(2), the spatial key matrix K is dot-multiplied with the query matrix Q in step 6 to obtain the spatial self-attention score matrix to express the long-range dependency between pixels in space.

5. The low-light image enhancement method based on dual-channel attention according to claim 1, characterized in that: In step 8(2), skip connections are added to each layer of the network to reduce the information loss caused by the illumination fusion encoding and decoding process.

Images