Low illumination image enhancement method based on illumination and scene texture attention map

Abstract

The present invention discloses a low-illumination image enhancement method based on illumination and scene texture attention maps, performs color balancing processing on low-illumination images to obtain a minimum channel constraint map, and reduces the color deviation problem of the RGB three channels of the low-illumination image due to the difference in light intensity; constructs an image enhancement network, inputs the minimum channel constraint map of the low-illumination image, performs attention map estimation on the illumination and texture of the normal exposure image, obtains illumination and texture attention estimation maps, uses the illumination and texture attention estimation maps as information guidance to perform brightness enhancement and noise suppression on the image, divides the results obtained after brightness enhancement and noise suppression into small blocks for local enhancement, and obtains an enhanced image of the low-illumination image; solves the problem in existing image enhancement technology that it is necessary to display the estimated noise intensity and the enhanced image is overexposed or underenhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a low-illumination image enhancement method based on illumination and scene texture attention maps. Background Art

[0002] Low light is one of the most common low-visibility scenarios in daily life. Common types include backlight, uneven lighting, colored light sources, low indoor lighting, and extremely low lighting. Low-light images are primarily caused by the dim environment in which the target object is captured. Variations in the position, intensity, and color of the light source result in insufficient photons and a low signal-to-noise ratio (SNR). This results in captured images with low brightness, low contrast, noise, and a loss of detail. Image enhancement aims to improve the visual quality of low-light images, enabling computers to see them not only clearly but also accurately. It holds significant research significance and application value in fields such as security surveillance, autonomous driving, and smart healthcare. Noise suppression is a key consideration in low-light image enhancement. Existing image enhancement methods consider both noise suppression and brightness enhancement. They explicitly estimate noise and perform overall image enhancement by introducing a denoising module. However, noise distribution is complex and its intensity varies. Furthermore, in real low-light scenarios, noisy and non-noisey image data pairs are difficult to obtain, and inter-domain differences are significant. Therefore, overall enhancement of images with uneven illumination is prone to over- or under-enhancement. Summary of the Invention

[0003] The purpose of the present invention is to provide a low-illumination image enhancement method based on illumination and scene texture attention maps, which solves the problem in existing image enhancement technologies that the estimated noise intensity needs to be displayed and the enhanced image is overexposed or under-enhanced.

[0004] The technical solution adopted by the present invention is a low-light image enhancement method based on illumination and scene texture attention maps, which performs color equalization processing on low-light images to obtain a minimum channel constraint map, thereby reducing the color deviation problem of the RGB three channels of low-light images due to differences in light intensity; constructing an image enhancement network, inputting the minimum channel constraint map of the low-light image, and performing attention map estimation on the illumination and texture of the normal exposure image to obtain illumination and texture attention estimation maps, using the illumination and texture attention estimation maps as information guidance to perform brightness enhancement and noise suppression on the low-light image, dividing the results obtained after brightness enhancement and noise suppression into small blocks for local enhancement, and obtaining an enhanced image of the low-light image.

[0005] The present invention is also characterized in that:

[0006] Please follow the steps below to implement it:

[0007] Step 1: Select multiple pairs of low-light images and their corresponding normal-exposure images from the dataset, perform dark channel estimation on the low-light images to obtain dark channel estimation maps, perform color equalization on the dark channel estimation maps, and obtain the minimum channel constraint maps of the low-light images and normal-exposure images;

[0008] Step 2: Construct an image enhancement network. During training, constrain the image enhancement network through a loss function. Use low-light images and corresponding normal-exposure images selected from the LOL dataset to train the image enhancement network. During each training session, use the minimum channel constraint map to estimate the illumination and texture of the normal-exposure image. Generate illumination and texture attention estimation maps as guidance maps for the global enhancement module. The image enhancement network obtained when the value of the selected loss function no longer decreases is the optimal image enhancement network.

[0009] Step 3: The low-light image is concatenated with the illumination and texture attention estimation maps and fed into the image enhancement network. The illumination and texture attention estimation maps are used as guidance maps for the image enhancement network to guide global image enhancement.

[0010] Step 4: Divide the enhanced image into small blocks, perform local enhancement, determine the local texture similarity of the image, and obtain an enhanced image of the low-light image.

[0011] The specific process of step 1 is:

[0012] Step 1.1: Select low-light images from the LOL dataset and the corresponding normal exposure image , for low-light images Dark channel estimation obtains dark channel estimation map , select the pixels whose brightness values ​​​​are in the top 10% in the dark channel image, and let these pixels form a set , then the light intensity of each color channel can be calculated using formula (1) , :

[0013] (1)

[0014] in, Represents low-light images of Color channel map at pixel point The value at Representing a collection The number of elements in

[0015] Step 1.2: Using the light intensity of each channel Perform color balancing, the expression is: ;

[0016] Step 1.3: Use the minimization function formula (2) to obtain the minimum channel constraint graph of the low-light image. , for a normally exposed image Its minimum channel constraint map does not require a color correction process:

[0017] (2).

[0018] The image enhancement network includes an attention estimation network, a global enhancement network and a local enhancement network. The attention estimation network and the global enhancement network are connected in a cascade manner.

[0019] The attention estimation network consists of an encoder, a decoder, and a decoder. The encoder consists of 4 convolution blocks and 1 convolution layer, and the decoder includes The transposed convolution layer, each convolution block is composed of 2 The convolutional layer is composed of ReLu activation function and batch normalization function.

[0020] The loss function in step 2 includes the mean square error loss function of the attention estimation network and the joint loss function of the global and local enhancement networks. The mean square error loss function of the attention estimation network is expressed as:

[0021] (3)

[0022] in, is the number of samples;

[0023] The joint loss function of the global and local enhancement network includes:

[0024] (1) Multi-scale structural similarity loss function , expressed as:

[0025] (4)

[0026] in, They are brightness measurement module, contrast measurement module and structural similarity measurement module. and are scale parameters, and Represent the enhanced predicted image and the normal exposure image as the true value respectively;

[0027] (2) Mean absolute error loss function , expressed as:

[0028] (5)

[0029] (3) Perceptual loss function , expressed as:

[0030] (6)

[0031] in, represents the image after global enhancement, represents a normally exposed image, It represents the feature map extracted from the globally enhanced image after being processed by the convolutional neural network. t Represents the network t layer, 、 It is the length and width dimension of the extracted feature map;

[0032] Global and local enhancement networks will 、 and The weighted training loss function is expressed as:

[0033] (7).

[0034] Step 2 uses the minimum channel constraint map to estimate the illumination and texture of the normal exposure image. The specific process is as follows: the minimum channel constraint map is input into the attention estimation network for downsampling and upsampling. The upsampling is performed using The transposed convolution is used to extract the feature map, which is then concatenated with the feature map obtained in the downsampling stage, and finally a convolution operation is performed to obtain the illumination and scene texture attention estimation map.

[0035] The specific process of step 3 is as follows:

[0036] The low-light image is spliced ​​and fused with the illumination and texture attention estimation map as the input of the global enhancement module, and the illumination and texture attention estimation map is used as the guidance map of the global enhancement network to guide the global image enhancement. The global enhancement module adopts an end-to-end U-Net network structure, including feature extraction, upsampling and splicing. In the feature extraction part, four downsampling operations are used to continuously reduce the image size and extract the shallow feature information of the image. The convolution kernel size used is , then use LeakyReLu as the activation function, and the downsampling process uses a size of The average pooling operation is the opposite of the upsampling part. The image size continues to increase while the number of channels is halved. The upsampling part uses The convolution kernel is used, and 4 deconvolutions are used to restore the image to a high-resolution image. At the same time, the jump connection uses the inverted image of the illumination and texture attention estimation map as a guide, so that the global enhancement network allocates different attention to the bright and dark areas of the image, and superimposes the feature maps of the corresponding levels of upsampling and downsampling to obtain the enhanced image.

[0037] The local enhancement module includes multiple Small convolution kernel.

[0038] The specific process of step 4 is: divide the enhanced image into small blocks and input them into the local enhancement module. Each block goes through multiple The small convolution kernel is used to extract image features. By judging the local texture similarity of the image, deep semantic information is extracted while effectively capturing the local features and edge information of the image, local enhancement is achieved, and an enhanced image of the low-light image is obtained.

[0039] The beneficial effects of the present invention are:

[0040] The low-illumination image enhancement method based on illumination and scene texture attention maps does not require explicit noise estimation. It uses the minimum channel constraint map of the low-illumination image to suppress noise and highlight texture, and has a certain robustness advantage against changes in image illumination and noise intensity. The illumination and texture attention estimation maps of the normally exposed image are pre-estimated, and the illumination and texture attention estimation maps contain rich scene texture information and image illumination information. While enhancing image brightness and improving image quality, the detailed texture information is better maintained. In addition, we improve image brightness and contrast while avoiding insufficient enhancement and overexposure through a combination of global and local enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of the low-illumination image enhancement method based on illumination and scene texture attention map of the present invention;

[0042] Figure 2 This is a schematic diagram of obtaining the minimum channel constraint graph of the present invention;

[0043] Figure 3 Schematic diagram of the image enhancement network structure in the present invention;

[0044] Figure 4 Schematic diagram of the attention graph estimation subnetwork of the present invention;

[0045] Figure 5 is a schematic diagram of the global enhancer network of the present invention;

[0046] Figure 6 It is a schematic diagram of the enhancement effect on the LOL real scene test set in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1

[0049] The present invention proposes a low-illumination image enhancement method based on illumination and scene texture attention map, such as Figure 1 As shown, the low-light image is color-balanced to obtain the minimum channel constraint map, thereby reducing the color deviation problem of the RGB three channels of the low-light image due to the difference in light intensity; an image enhancement network is constructed, and the minimum channel constraint map of the low-light image is input to estimate the illumination and texture of the normal exposure image to obtain the illumination and texture attention estimation map, which is used as information to guide the brightness enhancement and noise suppression of the low-light image. The enhanced results after brightness enhancement and noise suppression are divided into small blocks for local enhancement to obtain an enhanced image of the low-light image.

[0050] Example 2

[0051] Based on Example 1, the low-illumination image enhancement method of this embodiment based on illumination and scene texture attention map is implemented in the following steps:

[0052] Step 1: Select multiple pairs of low-light images and their corresponding normal exposure images in the dataset, perform dark channel estimation on the low-light images to obtain a dark channel estimation map, perform color balancing on the dark channel estimation map, and obtain the minimum channel constraint map of the low-light images;

[0053] In this paper, in order to improve the generalization ability of the model, the low-light image dataset LOL (Lowlight dataset) is selected. It consists of 689 low-light images and corresponding normal exposure images. It is the first paired dataset collected in real scenes and is saved in RGB format with a size of 400. 600.

[0054] The specific process is:

[0055] Step 1.1: Select low-light images from the LOL dataset and the corresponding normal exposure image , for low-light images Dark channel estimation obtains dark channel estimation map ,like Figure 2 As shown, in the dark channel image, the pixels with brightness values ​​in the top 10% are selected and these pixels form a set , then the light intensity of each color channel can be calculated using formula (1) , :

[0056] (1)

[0057] in, Represents low-light images of Color channel map at pixel point The value at Representing a collection The number of elements in

[0058] Step 1.2: Using the light intensity of each channel Perform color balancing, the expression is: ;

[0059] Step 1.3: Use the minimization function formula (2) to obtain the minimum channel constraint graph of the low-light image. , for a normally exposed image Its minimum channel constraint map does not require a color correction process:

[0060] (2).

[0061] Step 2: Figure 3 As shown in the figure, an image enhancement network is constructed. During training, the image enhancement network is constrained by a loss function. Low-illumination images and corresponding normal-exposure images selected from the LOL dataset are used to train the image enhancement network. During each training, the minimum channel constraint map is used to estimate the illumination and texture of the normal-exposure image. The illumination and texture attention estimation maps are generated as the guidance maps of the global enhancement module. The image enhancement network obtained when the value of the selected loss function no longer decreases is the optimal image enhancement network.

[0062] The image enhancement network includes an attention estimation network, a global enhancement network and a local enhancement network. The attention estimation network and the global enhancement network are connected in a cascade manner.

[0063] like Figure 4 As shown in Figure 2, the attention estimation network consists of an encoder, a decoder, and a decoder. The encoder consists of 4 convolutional blocks and 1 convolutional layer, and the decoder includes The transposed convolution layer, each convolution block is composed of 2 The convolutional layer is composed of ReLu activation function and batch normalization function.

[0064] The minimum channel constraint graph of low-light images is learned through the above stages And the minimum channel constraint map of the normal exposure image The mapping mechanism between them makes the illumination and texture attention estimation maps generated by the attention map estimation module This contains rich scene texture information and illumination information similar to that of a normal exposure image, which is used to guide the subsequent global enhancement module, allowing the network to focus on real scene texture information rather than noise. Furthermore, the illumination and scene texture attention estimation maps obtained by the attention estimation network have the characteristic of decreasing with increasing brightness. Therefore, the obtained illumination and scene texture attention estimation maps need to be inverted to provide guidance information for the image enhancement network.

[0065] The loss function in step 2 includes the mean square error loss function of the attention estimation network and the joint loss function of the global and local enhancement networks. The mean square error loss function of the attention estimation network is expressed as:

[0066] (3)

[0067] in, is the number of samples;

[0068] The joint loss function of the global and local enhancement network includes:

[0069] (1) Multi-scale structural similarity loss function , expressed as:

[0070] (4)

[0071] in, They are brightness measurement module, contrast measurement module and structural similarity measurement module. and are scale parameters, and Represent the enhanced predicted image and the normal exposure image as the true value respectively;

[0072] (2) Mean absolute error loss function , expressed as:

[0073] (5)

[0074] (3) Perceptual loss function , expressed as:

[0075] (6)

[0076] in, represents the image after global enhancement, represents a normally exposed image, It represents the feature map extracted from the globally enhanced image after being processed by the convolutional neural network. t Represents the network t layer, 、 It is the length and width dimension of the extracted feature map;

[0077] Global and local enhancement networks will 、 and The weighted training loss function is expressed as:

[0078] (7).

[0079] The minimum channel constraint map is used to estimate the illumination and texture of the normal exposure image. The specific process is as follows: the minimum channel constraint map is input into the attention estimation network for downsampling and upsampling. The upsampling is performed using The transposed convolution is used to extract the feature map, which is then concatenated with the feature map obtained in the downsampling stage, and finally a convolution operation is performed to obtain the illumination and scene texture attention estimation map.

[0080] Step 3: Input the low-light image and the illumination and texture attention estimation map into the image enhancement network for splicing and fusion. The illumination and texture attention estimation map is used as the guidance map of the image enhancement network to guide global image enhancement. The specific process is as follows:

[0081] The low-light image is spliced ​​and fused with the illumination and texture attention estimation map as the input of the global enhancement module, and the illumination and texture attention estimation map is used as the guidance map of the global enhancement network to guide global image enhancement, such as Figure 5 As shown in the figure, the global enhancement module adopts an end-to-end U-Net network structure, which includes three parts: feature extraction, upsampling and splicing. In the feature extraction part, four downsampling operations are used to continuously reduce the image size and extract shallow feature information of the image. The convolution kernel size used is , then use LeakyReLu as the activation function, and the downsampling process uses a size of The average pooling operation is the opposite of the upsampling part. The image size continues to increase while the number of channels is halved. The upsampling part uses The convolution kernel is used, and 4 deconvolutions are used to restore the image to a high-resolution image. At the same time, the jump connection uses the inverted image of the illumination and texture attention estimation map as a guide, so that the global enhancement network allocates different attention to the bright and dark areas of the image, and superimposes the feature maps of the corresponding levels of upsampling and downsampling. This reduces the loss of shallow image features caused by the increase in network depth to a certain extent, and finally obtains the enhanced image.

[0082] Step 4: Divide the enhanced image into small blocks, perform local enhancement, determine the local texture similarity of the image, and obtain the enhanced image of the low-light image. The specific process is: divide the enhanced image into small blocks and distribute them into the local enhancement module. Each block undergoes multiple The small convolution kernel is used to extract image features, which can not only reduce network parameters but also increase the network's expressive power. By judging the local texture similarity of the image, deep semantic information is extracted while effectively capturing the local features and edge information of the image, local enhancement is achieved, and an enhanced image of the low-light image is obtained.

[0083] Example 3

[0084] like Figure 6 As shown in the figure, this method is compared with two mainstream traditional methods and three deep learning methods, including SDD (semi-decomposed decomposition), PnPR (plug-and-play Retinex model), EnlightenGAN, Zero-DCE (zero-reference deep curve estimation) and Retinex-Net (Deep Retinex Decomposition for Low-Light Enhancement). Figure 6 (a) to Figure 6 Middle (h) are the original low-light image, SDD, PnPR, EnlightenGAN, Zero-DCE, Retinex-Net, this method and normal exposure image respectively. Figure 6 The first two images are test samples without corresponding normal exposure reference images, and the last two are the results on the LOL real scene test set with corresponding normal exposure images. By comparing the results, it can be seen that SDD and PnPR have the problem of insufficient enhancement under the same conditions, Zero-DCE has the problem of overexposure, EnlightenGAN does not take the dark area noise of low-light images into consideration, so there is a lot of noise in the enhancement results, Retinex-Net needs to explicitly estimate the noise intensity, while this method can not only improve the brightness and contrast of low-light images, but also effectively suppress noise while highlighting the texture detail information of the image. In addition, on extremely low-light images, the present invention can also achieve ideal enhancement effects.

Claims

1. A low-light image enhancement method based on illumination and scene texture attention map, characterized in that: Please follow the steps below to implement it: Step 1: Select multiple pairs of low-light images and their corresponding normal exposure images in the dataset, estimate the dark channel of the low-light images to obtain a dark channel estimation map, perform color equalization on the dark channel estimation map, and obtain the minimum channel constraint map of the low-light images and the normal exposure images; Step 2: Construct an image enhancement network. During training, constrain the image enhancement network through a loss function. Use low-light images and corresponding normal-exposure images selected from the LOL dataset to train the image enhancement network. During each training session, use the minimum channel constraint map to estimate the illumination and texture of the normal-exposure image. Generate illumination and texture attention estimation maps as guidance maps for the global enhancement module. The image enhancement network obtained when the value of the selected loss function no longer decreases is the optimal image enhancement network. Step 3: The low-light image is concatenated with the illumination and texture attention estimation maps and fed into the image enhancement network. The illumination and texture attention estimation maps are used as guidance maps for the image enhancement network to guide global image enhancement. Step 4: Divide the enhanced image into small blocks, perform local enhancement, determine the local texture similarity of the image, and obtain an enhanced image of the low-light image; The specific process of step 1 is: Step 1.1: Select low-light images from the LOL dataset and the corresponding normal exposure image , for low-light images Dark channel estimation obtains dark channel estimation map , select the pixels whose brightness values ​​​​are in the top 10% in the dark channel image, and let these pixels form a set , then the light intensity of each color channel can be calculated using formula (1) , : (1) in, Represents low-light images of Color channel map at pixel point The value at Representing a collection The number of elements in Step 1.2: Using the light intensity of each channel Perform color balancing, the expression is: ; Step 1.3: Use the minimization function formula (2) to obtain the minimum channel constraint graph of the low-light image. , for a normally exposed image Its minimum channel constraint map does not require a color correction process: (2)； In step 2, the minimum channel constraint map is used to estimate the illumination and texture of the normal exposure image. The specific process is as follows: the minimum channel constraint map is input into the attention estimation network for downsampling and upsampling. The upsampling is performed using The transposed convolution is used to extract the feature map, which is then concatenated with the feature map obtained in the downsampling stage, and finally a convolution operation is performed to obtain the illumination and scene texture attention estimation map.

2. The low-illumination image enhancement method based on illumination and scene texture attention map according to claim 1, characterized in that: The image enhancement network includes an attention estimation network, a global enhancement network, and a local enhancement network. The attention estimation network and the global enhancement module are connected in a cascade manner.

3. The low-illumination image enhancement method based on illumination and scene texture attention map according to claim 2, characterized in that: The attention estimation network consists of an encoder and a decoder. The encoder consists of 4 convolution blocks and 1 convolution layer. The decoder includes The transposed convolution layer, each convolution block is composed of 2 The convolutional layer is composed of ReLu activation function and batch normalization function.

4. The low-illumination image enhancement method based on illumination and scene texture attention map according to claim 3, characterized in that: The loss function in step 2 includes the mean square error loss function of the attention estimation network and the joint loss function of the global and local enhancement networks. The mean square error loss function of the attention estimation network is expressed as: (3) in, is the number of samples; The joint loss function of the global and local enhancement network includes: (1) Multi-scale structural similarity loss function , expressed as: (4) in, They are brightness measurement module, contrast measurement module and structural similarity measurement module. and are scale parameters, and Represent the enhanced predicted image and the normal exposure image as the true value respectively; (2) Mean absolute error loss function , expressed as: (5) (3) Perceptual loss function , expressed as: (6) in, represents the image after global enhancement, represents a normally exposed image, It represents the feature map extracted from the globally enhanced image after being processed by the convolutional neural network. t Represents the network t layer, 、 It is the length and width dimension of the extracted feature map; Enhanced network will 、 and The weighted training loss function is expressed as: (7)。 5. The low-illumination image enhancement method based on illumination and scene texture attention map according to claim 2, characterized in that: The specific process of step 3 is as follows: The low-light image is spliced ​​and fused with the illumination and texture attention estimation map as the input of the global enhancement module, and the illumination and texture attention estimation map is used as the guidance map of the global enhancement network to guide the global image enhancement. The global enhancement module adopts an end-to-end U-Net network structure, including feature extraction, upsampling and splicing. In the feature extraction part, four downsampling operations are used to continuously reduce the image size and extract the shallow feature information of the image. The convolution kernel size used is , then use LeakyReLu as the activation function, and the downsampling process uses a size of The average pooling operation is the opposite of the upsampling part. The image size continues to increase while the number of channels is halved. The upsampling part uses The convolution kernel is used, and 4 deconvolutions are used to restore the image to a high-resolution image. At the same time, the jump connection uses the inverted image of the illumination and texture attention estimation map as a guide, so that the global enhancement network allocates different attention to the bright and dark areas of the image, and superimposes the feature maps of the corresponding levels of upsampling and downsampling to obtain the enhanced image.

6. The low-illumination image enhancement method based on illumination and scene texture attention map according to claim 2, characterized in that: The local enhancement network includes multiple Small convolution kernel.

7. The low-illumination image enhancement method based on illumination and scene texture attention map according to claim 6, characterized in that: The specific process of step 4 is: divide the enhanced image into small blocks and input them into the local enhancement module. Each block is processed by multiple The small convolution kernel is used to extract image features. By judging the local texture similarity of the image, deep semantic information is extracted while effectively capturing the local features and edge information of the image, local enhancement is achieved, and an enhanced image of the low-light image is obtained.

Images