Low-illumination image enhancement method

Abstract

The invention discloses a low-illumination image enhancement method, which comprises the following steps of: estimating the illumination intensity of each RGB (Red, Green, Blue) channel for a low-illumination image by utilizing a dark channel prior principle, and carrying out color equalization processing to enable the illumination intensity of each channel to tend to be similar; training the network: calculating branches by using the illumination of the U-Net network and a field texture attention map, and learning a mapping mechanism between a minimum channel constraint Slow map of a low-illumination image and a minimum channel constraint Snormal of a normal exposure image; in an up-sampling stage, a bilinear up-sampling layer and a convolution layer are used to replace a standard deconvolution layer in a U-Net network, and CBMA attention and non-local attention are fused to form a double attention network; the mean square error, the structural similarity and the gradient loss are combined to serve as joint training loss of the image enhancement model; inputting the to-be-processed image into the training network to obtain an enhanced image; according to the invention, the practicability and definition of the low-illumination image can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and image processing, and particularly relates to a low-illumination image enhancement method. Background Art

[0002] With the popularization of digital imaging equipment, people can easily obtain a large number of digital images. However, when the lighting conditions of the shooting scene are poor or the performance of the image acquisition equipment is limited, low-light images are easily generated. The overall brightness and contrast of low-light images are low, which will affect their practical value. In order to improve the practicality and clarity of low-light images, low-light image enhancement methods need to be adopted for processing. Traditional low-light image enhancement algorithms mainly include histogram equalization methods, frequency domain methods, and Retinex calculation theory methods. Histogram equalization methods can be divided into global histogram equalization, local histogram equalization, adaptive histogram equalization, and limited contrast adaptive histogram equalization. This type of method is easily affected by artifacts in the image. The frequency domain method mainly uses wavelet transform and Fourier transform to enhance the processing of low-light images, but this type of method requires manual participation in the selection of transformation parameters. The Retinex calculation theory method uses the color constancy perception calculation theory to obtain the reflection component that reflects the inherent properties of the object to achieve image enhancement processing. This type of method can achieve good image enhancement effects, but its model calculation complexity is high and it is easy to cause overexposure in local areas. In addition, Yu Chunyan et al. and Dong et al. used defogging methods to achieve low-light image enhancement, but artifacts are prone to appear at the edges of objects in the enhanced images. These traditional algorithms have limited image enhancement capabilities and poor model generalization capabilities.

[0003] In recent years, low-light image enhancement methods based on deep learning have become a research hotspot. Lore et al. proposed a stacked sparse denoising encoder to enhance and denoise low-light images, but the details of the enhanced images are blurred and the model robustness is weak. Li et al. and Wei et al. combined the Retinex theory with the convolutional neural network and used supervised learning to train the network parameters to achieve low-light image enhancement, but it is easy to produce blurred edge contours. Lü et al. proposed the MBLLEN network, which uses multiple subnets for enhancement and produces output images with a multi-branch fusion strategy, but the enhanced image is blurred overall. Zhang et al. proposed the KIND method, which has good generalization ability, but its image is over-sharpened and local details are not clear. Jiang et al. proposed a night image enhancement model based on a generative adversarial network, which does not require image pairs for training, but its model robustness is still weak. Although deep learning methods are the main trend in current image enhancement processing research, they all extract features from feature maps of the same scale, resulting in insufficient expression of features of images of different scales, easy loss of image details, easy amplification of noise, and blurred edge contours. Summary of the invention

[0004] In view of this, the present invention provides a low-illumination image enhancement method, which can improve the practicality and clarity of low-illumination images.

[0005] The technical solution for implementing the present invention is as follows:

[0006] A low-light image enhancement method comprises the following steps:

[0007] Step 1: Use the dark channel prior principle to estimate the light intensity of each RGB channel of the low-light image to be enhanced, and then perform color balancing to make the light intensity of each channel close;

[0008] Step 2: Train the network: Use the illumination and field texture attention map calculation branches of the U-Net network to learn the mapping mechanism between the minimum channel constraint Slow map of low-illuminated images and the minimum channel constraint Snormal map of normal exposure images; in the upsampling stage, use a bilinear upsampling layer and a convolution layer to replace the standard deconvolution layer in the U-Net network, and fuse CBMA attention and non-local attention to form a dual attention network; combine the mean square error, structural similarity and gradient loss as the joint training loss of the image enhancement model;

[0009] Step 3: Input the image to be processed into the training network to obtain the enhanced image.

[0010] Furthermore, CBMA attention includes two modules: channel attention module and spatial attention module. After the input features pass through the above two modules in sequence, they will finally obtain recalibrated features, that is, important features are emphasized and unimportant features are compressed. The input features pass through the channel attention module to obtain the channel attention coefficient, which is multiplied element by element with the input features to obtain the intermediate features, and pass through the spatial attention module to obtain the spatial channel coefficient, which is multiplied with the intermediate features to obtain the output features.

[0011] Furthermore, in step 2, feature enhancement is first performed through CBAM, and then non-local attention is added, so that the network pays attention to global information on the basis of effectively utilizing the initial position information. The illumination and scene texture attention estimation maps are added to the network through jump connections, which helps the network to pay different attention and processing to areas with different image brightness.

[0012] Furthermore, the input feature map X is linearly mapped to generate θ, g three groups of features, after merging these three groups of feature channels, Perform matrix dot multiplication operations to calculate the autocorrelation between each pixel and other pixels to measure the degree of association between them; perform SoftMax normalization on the g autocorrelation feature to obtain the attention weight f of each pixel to other pixels, and finally multiply the obtained attention weight f with the feature g and expand it back to the original number of channels through a 1×1 convolution. Then perform a residual connection with the original input feature map X to obtain the output X of the non-local block.

[0013] Furthermore, in the local enhancement module, the globally enhanced image is cropped and divided into small image blocks, and then passed through multiple 3×3 small convolution kernels to extract image features, thereby increasing the network depth while effectively capturing the local features and edge information of the image, achieving local detail enhancement while further removing noise that affects image quality.

[0014] Beneficial effects:

[0015] 1. The present invention effectively avoids the problem that low-illumination image enhancement algorithms are prone to losing image details, noise is easily amplified, and edge contours are blurred.

[0016] 2. The present invention integrates the attention module and the non-local attention module in the upsampling stage of the image enhancement module to improve the phenomenon that the existing enhancement algorithm is prone to over-detail blurring and loss of image details.

[0017] 3. The present invention improves the loss function composition of the image enhancement algorithm and improves the edge contour blur phenomenon that is easily produced by the existing enhancement algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of the algorithm principle of the present invention.

[0019] Figure 2 Schematic diagram of the network architecture of the algorithm of the present invention. DETAILED DESCRIPTION

[0020] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0021] The present invention provides a low-light image enhancement method, and the specific implementation method is as shown in the attached Figure 1 shown.

[0022] 1. Generation of minimum channel constraint graph for low-light images

[0023] For images shot under normal lighting conditions, the illumination intensity of the three RGB channels is close to the same, but for images shot under low lighting conditions, the illumination intensity of the three RGB channels often shows obvious differences, resulting in color deviation. To this end, the dark channel prior principle is used to estimate the illumination intensity of each RGB channel, and then color balancing is performed to make the illumination intensity of each channel close. The specific process of the processing is as follows.

[0024] For the low-light image I to be enhanced low , first, estimate its dark channel map I dark In the dark channel map, the pixels whose brightness values ​​rank in the top 10% are selected according to the dark channel pixel distribution of the data set selected by the present invention. According to experimental experience, the increase of this ratio will reduce the overall brightness of the image and weaken some detail textures, which is not conducive to providing guidance information for the subsequent enhancement module. Let these pixels constitute the set Ω 1 , then the light intensity A of each color channel c , c∈{r,g,b} is calculated as

[0025]

[0026] In the formula, Represents a low-light image I low The value of the c color channel image at pixel p, num(Ω 1 ) represents the set Ω 1 Then, the light intensity A of each channel is used to calculate the number of elements in c Perform color balancing. Finally, the minimum channel constraint graph S of the low-light image is obtained by minimizing the function low , to achieve preliminary noise suppression and texture highlighting effect, the whole process can be expressed as

[0027]

[0028] For the normal exposure image I normal , without the need for color correction, its minimum channel constraint graph S normal It can be expressed as

[0029] S normal =min(1-I normal )(3)

[0030] 2. Illumination and scene texture attention map estimation

[0031] Image noise refers to the random signal interference that an image is subject to during the shooting and transmission process. Generally speaking, the higher the image brightness, the smaller the image noise, and vice versa. For images with particularly low illumination, the generated minimum channel constraint map contains obvious noise. In order to eliminate the misleading effect of noise on the subsequent enhancement stage, the minimum channel constraint S of low-illuminance images is learned based on the illumination and field texture attention map calculation branches of the U-Net network. low Figure 2 and the minimum channel constraint map S of the normal exposure image normal The mapping mechanism between them makes the attention map generated by the illumination and scene texture rough estimation module, namely S normal It contains rich scene texture information and illumination information similar to that of a normal exposure image, which is used for information guidance of subsequent enhancement modules, so that the network focuses on real scene information rather than noise, and guides the brightness adjustment degree of different areas of the image with uneven illumination, effectively avoiding the problem of under-enhancement and over-enhancement in local areas. The specific network architecture diagram of this module is shown in the figure. It consists of 9 convolution blocks and the last top 1×1 convolution layer. Each convolution block consists of 2 3×3 convolution layers followed by ReLU activation function and batch normalization processing function. In the downsampling stage, each convolution block is connected to a 2×2 maximum pooling layer. In the upsampling stage, the present invention uses 2×2 transposed convolution to extract feature maps, and splices and fuses them with the feature maps obtained in the downsampling step.

[0032] 3. Image Enhancement Network

[0033] First, a global enhancement network is used to adjust the overall brightness of the image. Then, the local brightness of the random blocks divided in the global enhanced image is adjusted, which avoids the problems of local over-enhancement and under-enhancement to a certain extent. The U-Net basic network architecture is used as the main body of the global enhancement part. It consists of 10 convolution blocks containing two 3×3 convolution layers. Each convolution block is followed by a LeakyReLU activation function layer and a batch normalization layer. Adding a batch normalization layer after the network convolution layer will greatly improve the network performance. In the upsampling stage, a bilinear upsampling layer and a convolution layer are used to replace the standard deconvolution layer in the original U-Net network to reduce the occurrence of image checkerboard and artifacts, and CBMA attention and non-local attention are fused to form a dual attention network, such as Figure 2As shown, feature enhancement is first performed through CBAM, and then non-local attention is added, so that the network pays attention to global information on the basis of effectively utilizing the initial position information. The illumination and scene texture attention estimation map is added to the network through a skip connection to help the network apply different attention and processing to areas with different image brightness. The original low-illumination image is spliced ​​and fused with the illumination and texture attention estimation map as the input of the global enhancement network, so that the attention map further provides information guidance. In the local enhancement module, the present invention divides the globally enhanced image into small image blocks by cropping, and then extracts image features through multiple 3×3 small convolution kernels, increasing the network depth while effectively capturing the local features and edge information of the image, achieving local detail enhancement while further removing factors such as noise that affect image quality. In addition, in order to make the training model more stable, a batch normalization layer is added thereafter.

[0034] In order to make the texture details of the image enhancement results clearer and the brightness distribution more reasonable, the mean square error (MSE), structural similarity index (SSIM) and gradient loss (GL) are combined as the joint training loss of the image enhancement model. The calculation formula is as follows:

[0035] L Loss =α×L MSE +β×L SSIM +γL GL (4)

[0036] The MSE loss function is a regression loss function commonly used in network training and has good convergence during the training process.

[0037]

[0038] Where: I represents the original normal image, whose size is S×T; K represents the image enhanced by the network model.

[0039] In order to measure the structural similarity between the enhanced low-light image and the normal-light image, the SSIM loss function is added to the joint training loss.

[0040]

[0041] In the formula, μ x and μ y are the means of images x and y, σ x 2 and σ y 2 are the variances of images x and y, σ xyrepresents the covariance of images x and y, c 1 and c 2 These are the default adjustment parameters.

[0042] In order to prevent the network from falling into the local optimum during training, a gradient loss function is added to the joint training loss.

[0043]

[0044] Where: It is a gradient operator, which includes two dimensions of image x and y; and Represents the gradient in the horizontal and vertical directions respectively; R high and R low Represents the pixel values ​​of normal illumination image and low illumination image respectively

[0045] 4. CBMA Attention Mechanism

[0046] The CBAM hybrid attention mechanism is a module that can focus on important local features of feature images. The core idea is to simulate the behavior of human selective attention in vision by assigning weights. It assigns weights to feature images in space and channels through adaptive learning, so that the output feature map is more inclined to the target area of ​​focus, thereby enhancing the features of the key areas. CBAM consists of two modules: channel attention module and spatial attention module. After the input features pass through the above two modules in sequence, the recalibrated features will be obtained. That is, important features are emphasized and unimportant features are compressed. The input features pass through the channel attention module to obtain the channel attention coefficient, which is multiplied element by element with the input features to obtain the intermediate features. The spatial attention module obtains the spatial channel coefficient, which is multiplied with the intermediate features to obtain the output features. The length, width and number of channels of the output features remain unchanged compared to the input features. G consists of 3 downsampling modules, 3 CBAM modules, 9 residual modules and 3 upsampling modules. Among them, the downsampling module, CBAM module and residual module belong to the encoder part, which are responsible for extracting the features of the input image. The upsampling block belongs to the decoder part, which is responsible for deconvolution of the acquired features to generate the output image.

[0047] 5. Non-local attention

[0048] Non-local attention convolution only operates on local areas in the image to obtain local feature information, ignoring the global image structure and context. The present invention introduces a non-local attention module to reduce information shrinkage and expand full pixel interaction. While paying attention to low-level feature edge information, it effectively utilizes the context information of multi-target semantic segmentation.

[0049] Linearly map the input feature map X to generate θ, g three groups of features, after merging these three groups of feature channels, Perform matrix dot multiplication to calculate the autocorrelation between each pixel and other pixels to measure the degree of association between them. Next, perform SoftMax normalization on the g autocorrelation feature to obtain the attention weight f of each pixel to other pixels. These weights represent the importance of the pixel. Finally, multiply the obtained attention weight f with the feature g and expand it back to the original number of channels through a 1×1 convolution. Then perform a residual connection with the original input feature map X to obtain the output X of the non-local block.

[0050] Three no-reference quality evaluation indicators are used to analyze the algorithm, including natural image quality evaluator (NIQE), blind tone-mapped quality index (BTMQI) and no-reference image quality metric for contrast distortion (NIQMC). NIQE compares the image with the designed natural image model. The lower the value, the higher the image quality. BTMQI evaluates the perceptual quality of the image after tone mapping by analyzing the naturalness and structure of the image. The lower the value, the more natural the image is and the higher the quality. NIQMC uses the local properties of the blocks in the image and the correlation properties between the blocks to evaluate the image quality by calculating the contrast between them. The higher the score, the better the image quality.

[0051] The evaluation results of the overall enhancement effect of a data set of 500 ordinary low-light images show that compared with the algorithm before improvement, the NIQE of the improved algorithm decreased by 0.2228, the BTMQI decreased by 0.2585, and the NIQMC increased by 0.1137.

[0052] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-light image enhancement method, characterized in that: The following steps are involved: Step 1: Use the dark channel prior principle to estimate the light intensity of each RGB channel of the low-light image to be enhanced, and then perform color balancing to make the light intensity of each channel close; Step 2: Train the network: Use the illumination and field texture attention map calculation branches of the U-Net network to learn the mapping mechanism between the minimum channel constraint Slow map of low-illumination images and the minimum channel constraint Snormal map of normal exposure images; in the upsampling stage, use a bilinear upsampling layer and a convolution layer to replace the standard deconvolution layer in the U-Net network, and fuse CBMA attention and non-local attention to form a dual attention network; Combine mean squared error, structural similarity and gradient loss as the joint training loss for image enhancement models; Step 3: Input the image to be processed into the training network to obtain the enhanced image.

2. The low-light image enhancement method according to claim 1, characterized in that: CBMA attention consists of two modules: channel attention module and spatial attention module. After the input features pass through the above two modules in sequence, they will finally obtain recalibrated features, that is, important features are emphasized and unimportant features are compressed. The input features pass through the channel attention module to obtain the channel attention coefficient, which is multiplied element by element with the input features to obtain the intermediate features. The spatial channel coefficient is obtained by passing through the spatial attention module, and multiplied with the intermediate features to obtain the output features.

3. The low-light image enhancement method according to claim 1 or 2, characterized in that: In step 2, feature enhancement is first performed through CBAM, and then non-local attention is added to enable the network to pay attention to global information on the basis of effectively utilizing the initial position information. The illumination and scene texture attention estimation maps are added to the network through jump connections to help the network pay different attention and processing to areas with different image brightness.

4. The low-light image enhancement method according to claim 3, characterized in that: The input feature map X is linearly mapped to generate three sets of features θ, φ, and g. After merging these three sets of feature channels, matrix dot multiplication is performed on θ and φ to calculate the autocorrelation between each pixel and other pixels to measure the degree of association between them. The g autocorrelation feature is SoftMax normalized to obtain the attention weight f of each pixel to other pixels. Finally, the obtained attention weight f is multiplied by the feature g and expanded back to the original number of channels through a 1×1 convolution. The result is then residually connected with the original input feature map X to obtain the output X of the non-local block.

5. The low-light image enhancement method according to claim 4, characterized in that: In the local enhancement module, the globally enhanced image is cropped and divided into small image blocks, and then passed through multiple 3×3 small convolution kernels to extract image features, increasing the network depth while effectively capturing the local features and edge information of the image, achieving local detail enhancement while further removing noise that affects image quality.

Images