A self-supervised low-light image enhancement method based on Retinex and CBAM

Abstract

This invention discloses a self-supervised low-light image enhancement method based on Retinex and CBAM, comprising the following steps: 1) obtaining the illuminance component of the image using RIENet; 2) dividing the input image by the corresponding illuminance component to obtain the reflectance component; 3) dividing the reflectance component R... low and R ho The images are input into the CA-DNNet network for denoising, and finally the enhanced image E is obtained. low and E ho This method does not require any normal lighting images during training, and will not suffer from color distortion, over-enhancement, insufficient enhancement of some details, or noise amplification. It effectively avoids problems such as feature distortion or feature redundancy in the enhanced image.

Description

Technical Field

[0001] This invention relates to the field of low-light image enhancement technology, specifically a self-supervised low-light image enhancement method based on Retinex and CBAM applicable to various scenarios, combining Retinex and CBAM. Background Technology

[0002] With the rapid development of information technology, images have become one of the important ways for us to acquire and disseminate information. However, due to unavoidable environmental or technical limitations, images taken under low-light conditions often suffer from low contrast, high noise, and loss of image details. This not only reduces the visual appeal of the image but also affects the performance of computer vision algorithms, such as image segmentation and object detection, because the incomplete information in low-light images can easily lead to the transmission of incorrect information. Therefore, low-light image enhancement has always been one of the methods to improve the performance of computer vision algorithms.

[0003] Most existing enhancement methods can be divided into two types: supervised learning methods and supervised learning methods. Examples include: KGLore, A. Akintayo, and S. Sarkar, “Llnet: A deep autoencoder approach to natural low-light image enhancement,” Pattern Recognition, vol. 61, pp. 650–662, 2017; Chen C, Chen Q, Xu J, et al. Learning to See in the Dark [C] / / 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018; Wei C., Wang WJ, Yang W.H., et al. Deep retinex decomposition for low-light enhancement [J]. arXiv:1808.04560, 2018; Wang J., Tan W., Niu X., et al. RDGAN: Retinex decomposition pasted adversarial learning for low-light enhancement[C] / / 2019IEEE International Conference on Multimedia and Expo(ICME),Shanghai,China,2019:1186-1191., which requires a large number of paired low-light / normal-light images; another is unsupervised learning methods, such as Wang R,Jiang B,Yang C,et al.MAGAN:Unsupervised Low-Light Image Enhancement Guided by Mixed-Attention[J].Big Data Mining and Analytics,2022,5(2):110-119.;Guo H,XuW,Qiu S.Unsupervised low-light Image Enhancement with Quality-Task-PerceptionLoss[C] / / 2021International Joint Conference on Neural Networks(IJCNN).0.;Jiang Y.,Gong X.,Liu D.=, et al. EnlightenGAN: deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing, vol, 2021, 30: 2340-2349.;Xiong W, Liu D, Shen X, et al. Unsupervised Real-world Low-light Image Enhancement with Decoupled Networks[J]. 2020., do not require paired datasets, but require a large number of carefully selected unpaired normal light images to train the model. Although these methods have achieved good results, the datasets are difficult to obtain. Therefore, to address the difficulty in obtaining training datasets, self-supervised learning has become a popular research topic. Examples include Guo C, Li C, Guo J, et al. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement[C] / / 2020IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2020.; and Jiang Z, Li H, Liu L, et al. A Switched View of Retinex: Deep Self-Regularized Low-Light Image Enhancement[J]. 2021. These methods can achieve enhancement results comparable to supervised learning methods, but they suffer from problems such as insufficient enhancement, over-enhancement, and feature or color distortion. Summary of the Invention

[0004] The purpose of this invention is to address the challenges of existing low-light image enhancement methods, which are mostly based on supervised, semi-supervised, or unsupervised learning. These methods require paired or unpaired image sets during training, making dataset acquisition difficult. Furthermore, many methods suffer from over-enhancement, insufficient enhancement of certain details, noise amplification, and color or feature distortion after enhancement. This invention proposes a self-supervised low-light image enhancement method based on Retinex and CBAM. This method does not require any normal light images during training, avoiding color distortion, over-enhancement, insufficient enhancement of certain details, and noise amplification. It effectively prevents feature distortion or feature redundancy in the enhanced image.

[0005] The technical solution to achieve the objective of this invention is:

[0006] A self-supervised low-light image enhancement method based on Retinex and CBAM includes the following steps:

[0007] 1) Use RIENet to obtain the illumination components of the image:

[0008] 1.1) Obtain the low-light image S using homomorphic filtering transformation. low Image S of uniform illumination in the same scene ho ;

[0009] 1.2) Convert the low-light image S low Image S with uniform illumination ho The corresponding illuminance components I are obtained by sequentially inputting them into the RIENet network. ho and I ho ,

[0010] An illuminance estimation loss is proposed, as shown in the following formula:

[0011]

[0012] in Used for smoothing illuminance, where I is the illuminance component obtained after network decomposition. Represents the gradient in the horizontal and vertical directions; The luminance is used to constrain the illuminance component I, where M is the number of 8×8 local regions segmented into the image, and I... k V k Let represent the decomposed illuminance and the mean of the kth local region of the luminance channel in the HSV color space of the input image, respectively, k∈[1,M]. This loss can not only smooth the illuminance component, but also effectively estimate the luminance of the low-light image, so that the network can decompose the illuminance component more accurately.

[0013] 2) Divide the input image by the corresponding illuminance component to obtain the reflectance component. In Retinex theory, the original image S(x,y) is composed of the illuminance image I(x,y) and the reflectance image R(x,y), and their relationship is expressed by the formula S(x,y) = R(x,y) * I(x,y). To ensure that the obtained reflectance component retains the features of the original image, a feature self-preservation loss L is introduced from EnlightenGAN. SFP The formula is shown below:

[0014]

[0015] Where R is the reflectance component, S is the input image, and φ i,j W represents the feature map extracted from the pre-trained VGG-16 model, where i represents the i-th max pooling layer, j represents the j-th convolutional layer of the i-th max pooling layer, and W represents the feature map extracted from the pre-trained VGG-16 model. i,j and Hi,j Indicates the dimension of the extracted feature map;

[0016] 3) The reflectivity component R low and R ho The images are input into the CA-DNNet network for denoising, and finally the enhanced image E is obtained. low and E ho ;

[0017] Even using feature self-preservation loss and reflectance consistency loss, feature self-preservation loss is used to ensure that the final image does not have feature redundancy or distortion. Reflectance consistency comes from Retinex theory. Reflectance components are inherent properties of objects, and the reflectance of images of the same scene under different lighting conditions should be consistent. The reflectance formula is as follows:

[0018] L rc =||E low -E ho ||1

[0019] Where E low and E ho This is the enhanced image obtained after denoising.

[0020] The low-light image S in step 1.1) low Training data taken from EnlightenGAN.

[0021] In step 3), the reflectivity component R low and R ho The acquisition process is as follows:

[0022] First, the low-light image S low and uniform illumination image S ho Inputting into RIENet yields the corresponding illuminance component I. low and I ho Then use S low and S ho Divide by the corresponding illuminance component I respectively low and I ho Obtain the reflectivity component R low and R ho The technical solution has the following beneficial effects:

[0023] First, to avoid color distortion in the enhanced image, based on Retinex theory, the reflectance component obtained by dividing the input image by the illuminance component estimated by RIENet is used as the enhanced image. Since the illuminance is smooth, the color and texture information of the original image are preserved in the reflectance component, and there will be no color distortion problem.

[0024] Secondly, to avoid over-enhancement or insufficient enhancement of details in the enhanced image, a homomorphic filtering transformation is introduced. Based on the principle of reflectance consistency, distinctive reflectance components can be obtained. Then, exposure control loss is used to control the brightness of the reflectance, so as to avoid over-enhancement or insufficient enhancement of details.

[0025] Then, to avoid noise amplification in the enhanced image, a CBAM module was added, combined with relevant loss, to guide the network to reasonably segment the noise component, thus avoiding the noise amplification problem.

[0026] Finally, to avoid feature distortion or redundancy in the enhanced image, feature-aware loss is introduced. This loss can further constrain the feature spatial distance between the reflectance component and the low-light image, thereby effectively avoiding feature distortion or redundancy in the enhanced image. Attached Figure Description

[0027] Figure 1 The model structure for this embodiment;

[0028] Figure 2 This is a structural diagram of the CBAM in the embodiment;

[0029] Figure 3 This is a structural diagram of the channel attention module in the embodiment;

[0030] Figure 4 This is a structural diagram of the spatial attention module in the embodiment;

[0031] Figure 5 The network model diagrams for RIENet and CA-DNNet in the embodiment are shown below;

[0032] Figure 6 This is a visual comparison of the examples with several other methods;

[0033] Figure 7 The following are the statistical results of the scores of the examples and several other methods on the objective evaluation metrics NIQE and BRISQUE. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0035] Example:

[0036] This example uses the PyTorch open-source framework for training on a Linux + NVIDIA TITAN XP environment. The network weights are initialized using a Gaussian distribution N(0, 0.02). The Adam optimizer is used during training, which lasts for 100 epochs. The first 50 epochs are trained with a learning rate of 0.0002, and the learning rate is linearly decayed from 0.0002 to zero in the last 50 epochs. The batch size is 8. The training set is the low-light image portion of the training data from EnlightenGAN (Jiang Y., Gong X., Liu D., et al. EnlightenGAN: deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing, vol, 2021, 30: 2340-2349.). Figure 1 S in low The test set consists of 914 images, which is also the test set used in EnlightenGAN, including 200 images such as DICM, LIME, MEF, NPE, and VV.

[0037] A self-supervised low-light image enhancement method based on Retinex and CBAM, such as Figure 1 As shown, this example method mainly consists of a Retinex-based illumination estimation network (RIENet) and a CBAM-based denoising network (CA-DNNet). Therefore, this example method is named (A self-supervised low-light image enhancement method based on Retinex and CBAM, abbreviated as RCA-SIE). First, the low-light image S... low and homomorphically filtered image S ho Inputting into RIENet yields the corresponding illuminance component I. low and I ho Then use the input image S low and S ho Divide by the corresponding illuminance component I low and I ho Obtain the reflectivity component R low and R ho Finally, CA-DNNet is used to denoise the reflectance component to obtain the enhanced image E. low and E ho The method includes the following steps:

[0038] 1) Use RIENet to obtain the illumination components of the image:

[0039] 1.1) Obtain the low-light image S using homomorphic filtering transformation. low Image S of uniform illumination in the same scene ho ;

[0040] 1.2) Convert the low-light image S low Image S with uniform illumination ho The corresponding illuminance components I are obtained by sequentially inputting them into the RIENet network. ho and I ho RIENet network model such as Figure 5 As shown;

[0041] In this step, an illuminance estimation loss is proposed, as shown in the following formula:

[0042]

[0043] in Used for smoothing illuminance, where I is the illuminance component obtained after network decomposition. Represents the gradient in the horizontal and vertical directions; The luminance is used to constrain the illuminance component I, where M is the number of 8×8 local regions segmented into the image, and I... k V k Let represent the decomposed illuminance and the mean of the kth local region of the luminance channel in the HSV color space of the input image, respectively, k∈[1,M]. This loss can not only smooth the illuminance component, but also effectively estimate the luminance of the low-light image, so that the network can decompose the illuminance component more accurately.

[0044] 2) Divide the input image by the corresponding illuminance component to obtain the reflectance component. In Retinex theory, the original image S(x,y) is composed of the illuminance image I(x,y) and the reflectance image R(x,y), and their relationship is expressed by the formula S(x,y) = R(x,y) * I(x,y). To ensure that the obtained reflectance component retains the features of the original image, a feature self-preservation loss L is introduced from EnlightenGAN. SFP The formula is shown below:

[0045]

[0046] Where R is the reflectance component, S is the input image, and φ i,j W represents the feature map extracted from the pre-trained VGG-16 model, where i represents the i-th max pooling layer, j represents the j-th convolutional layer of the i-th max pooling layer, and W represents the feature map extracted from the pre-trained VGG-16 model. i,j and H i,j This indicates the dimension of the extracted feature map; in this example, the value of i is 5 and the value of j is 1.

[0047] 3) The reflectivity component Rlow and R ho The images are input into the CA-DNNet network for denoising, and finally the enhanced image E is obtained. low and E ho The CA-DNNet network model is as follows: Figure 5 As shown;

[0048] Even using feature self-preservation loss and reflectance consistency loss, feature self-preservation loss is used to ensure that the final image does not have feature redundancy or distortion. Reflectance consistency comes from Retinex theory. Reflectance components are inherent properties of objects, and the reflectance of images of the same scene under different lighting conditions should be consistent. The reflectance formula is as follows:

[0049] L rc =||E low -E ho ||1

[0050] Where E low and E ho This is the enhanced image obtained after denoising. In this example, the method chosen is E. low As the final enhanced image.

[0051] The CBAM in the CA-DNNet network is described below:

[0052] like Figure 2 As shown, the Convolutional Block Attention Module (CBAM) is a module in convolutional neural networks used to enhance feature representations. It learns channel attention and spatial attention from feature maps and multiplies them to produce a final attention map, which guides the model's feature extraction process. The CBAM module mainly consists of two sub-modules: the Channel Attention Module (CAM) and the Spatial Attention Module (SAM), as shown... Figure 3 , Figure 4 As shown.

[0053] The CAM module learns the inter-channel relationships of feature maps, highlighting important channels and thus enabling the model to focus more on key features. It first performs global and average pooling on the input feature maps, then feeds them into a Multilayer Perceptron (MLP) to learn channel-dimensional features and the importance of each channel. The MLP output is then summed, and finally, a sigmoid activation function maps the output to a range of 0 to 1, resulting in a channel attention vector used to weight the features of each channel. The specific structure is as follows: Figure 3 As shown:

[0054] The SAM module learns the importance of different spatial locations in the feature maps, thus making the model more focused on useful spatial information. It first performs channel-based max pooling and channel-based average pooling on the feature maps output by the CAM module, resulting in two feature maps. These two feature maps are then concatenated and passed through a convolutional layer to obtain a 1-dimensional feature map. Finally, a sigmoid activation function is used to map the output to the range of 0 to 1, resulting in a feature map representing spatial attention, used to weight the features at each spatial location. The specific structure is as follows: Figure 4 As shown:

[0055] Figure 5 The network models of the illumination estimation network RIENet and the denoising network CA-DNNet in this example method are shown in Tables 1 and 2.

[0056] Table 1

[0057]

[0058] Table 2

[0059]

[0060]

[0061] Figure 6Shown are this example method and several typical methods RetinexNet (Wei C., Wang WJ, YangW.H., et al. Deep retinex decomposition for low-light enhancement [J].arXiv:1808.04560, 2018.), EnlightenGAN (Jiang Y., Gong X., Liu D., et al. EnlightenGAN: deep light enhancement without paired supervision[J].IEEE Transactions onImage Processing,vol,2021,30:2340-2349.), LIME(GUO al.ASwitched View of Retinex:Deep Self-Regularized Low-Light Image A comparison of visual quality between Enhancement[J]. 2021.) and Zero-DCE(Guo C, Li C, Guo J, et al. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement[C] / / 2020IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE, 2020.) is presented. Figure 6The first line shows that Retinex-SV has the worst enhancement effect, with insufficient enhancement on the face, resulting in an overall dark appearance, and color distortion in the sky. LIME, EnlightenGAN, and Zero-DCE have better brightness than Retinex-SV, but still insufficient enhancement, particularly on the face, and EnlightenGAN has color distortion in the sky. Retinex-Net and RCA-SIE proposed in this example have better brightness enhancement, but RCA-SIE enhances brightness more evenly and has the best visual effect. For example, in the cheek area, Retinex-Net enhances with patches, while RCA-SIE does not. In the second row of labeled area results, LIME, EnlightenGAN, and Zero-DCE all exhibit varying degrees of over-enhancement. Zero-DCE shows the most severe over-enhancement, resulting in excessive brightness at the eaves. EnlightenGAN shows slight over-enhancement, but color distortion occurs in the sky. Retinex-SV shows insufficient enhancement, resulting in an overall darker appearance. Retinex-Net shows good enhancement, but compared to RCA-SIE in this example, its brightness enhancement is lower, and the two sunlight rays are distorted after enhancement, while RCA-SIE's enhancement results in natural brightness and clear details. In the third row of results, LIME, EnlightenGAN, and Zero-DCE show relatively dark enhancement results in the labeled area. Retinex-Net, Retinex-SV, and RCA-SIE show good enhancement in the labeled area, but Retinex-SV's brightness enhancement is uneven and results in a darker sky. Retinex-Net's brightness enhancement in the sky is also worse than RCA-SIE's. In the last row of results, Retinex-Net showed the worst enhancement effect, appearing unnatural and exhibiting distortion. EnlightenGAN, on the other hand, suffered from insufficient enhancement, failing to adequately increase brightness. LIME and Zero-DCE produced the brightest results, but exhibited slight over-enhancement in field conditions, and Zero-DCE also showed slight color distortion, appearing somewhat washed out. RCA-SIE, however, produced uniform brightness, with color and texture features best matching the original low-light image. Therefore, compared to other enhancement results, the RCA-SIE method proposed in this example not only provides uniform and natural brightness but also avoids over-enhancement, color or texture distortion, achieving a more satisfactory visual effect. This further demonstrates the significant competitiveness and practicality of RCA-SIE.

[0062] To further evaluate the quality of the enhanced images, five methods—RetinexNet, EnlightenGAN, LIME, Retinex-SV, and Zero-DCE—along with the method RCA-SIE used in this example, were tested on five publicly available low-light image sets (DICM, LIME, MEF, NPE, and VV). NIQE and BRISQUE were then used to calculate the image scores for each method. The performance of the enhancement methods was judged based on the scores. The calculation results are shown in Tables 3 and 4. The smaller the score calculated by NIQE and BRISQUE, the better the image quality. The results in bold in the tables represent the best results.

[0063] Table 3

[0064]

[0065]

[0066] Table 4

[0067] Image set DICM LIME MEF NPE VV All Input 20.695 19.140 21.002 16.845 13.988 19.797 RetinexNet 28.251 26.216 21.719 22.487 19.668 26.586 EnlightenGAN 18.881 14.156 14.262 18.009 10.700 18.325 LIME 24.566 17.967 16.168 17.351 16.069 22.034 Retinex-SV 22.316 15.372 14.168 16.492 10.891 18.413 Zero-DCE 23.305 17.126 14.313 15.475 11.633 19.426 Retinex-SIE 16.494 13.283 13.525 14.864 11.204 17.247

[0068] Figure 7This paper presents the NIQE and BRISQUE scores of these six methods on 200 test images across the entire dataset All. In the NIQE statistics, RCA-SIE's scores are mainly concentrated in the range [1, 4.5]. The method closest to RCA-SIE in score is EnlightenGAN, which has slightly fewer images in the [1, 4.5] range than RCA-SIE, but the difference is small. The other four methods have significantly fewer images in lower score ranges than RCA-SIE and EnlightenGAN. Since lower NIQE scores indicate better image quality, RCA-SIE enhances the best image quality compared to the other methods, followed by EnlightenGAN, with Retinex-Net being the worst, as its image count is concentrated in the range [3.5, 9), and its image scores are generally high. The BRISQUE statistics clearly show that RCA-SIE's enhanced image scores are mainly concentrated in the [0,20] interval, with more images in this interval than the other five methods. Only a small number of images score in the (20,60) interval, and no image scores 60 or higher. Other methods have far more images scoring in the [20,100] interval than RCA-SIE, thus RCA-SIE has a clear advantage. Retinex-SV and EnlightenGAN perform well next. Although Retinex-SV has more images in the [0,5] interval than RCA-SIE, its number of images in the (5,20] interval is far lower, making it slightly worse than RCA-SIE overall, similar to EnlightenGAN, and better than the other three methods. Therefore, the method in this example has a clear advantage over the others.

Claims

1. A self-supervised low-light image enhancement method based on Retinex and CBAM, characterized in that, Includes the following steps: 1) Use RIENet to obtain the illumination components of the image: 1.1) Obtaining low-light images using homomorphic filtering transformation Uniformly lit images of the same scene ; 1.2) Low-light images Image with uniform illumination The corresponding illuminance components are obtained by sequentially inputting them into the RIENet network. and , An illuminance estimation loss is proposed, as shown in the following formula: ； in Used for smoothing illuminance, where I is the illuminance component obtained after network decomposition. Represents the gradient in the horizontal and vertical directions; The brightness is used to constrain the illuminance component I, where M is the number of 8×8 local regions segmented into the image. , Let represent the decomposed illuminance and the mean value of the k-th local region of the luminance channel in the HSV color space of the input image, respectively. This loss not only smooths the illuminance component, but also effectively estimates the brightness of low-light images, enabling the network to decompose the illuminance component more accurately. 2) Divide the input image by the corresponding illuminance component to obtain the reflectance component, because in Retinex theory, the original image... It is an illumination image and reflected images The composition and relationship between the three are given by the formula. This indicates that, in order to preserve the features of the original image in the obtained reflectance components, a feature self-preservation loss is introduced from EnlightenGAN. The formula is shown below: ； Where R is the reflectance component and S is the input image. This represents the feature map extracted from the pre-trained VGG-16 model, where i represents the i-th max pooling layer. This represents the j-th convolutional layer with the i-th max pooling layer. and Indicates the dimension of the extracted feature map; 3) Convert the reflectivity component and The images are then input into the CA-DNNet network for denoising, and finally the enhanced image is obtained. and ; Even using feature self-preservation loss and reflectance consistency loss, feature self-preservation loss is used to ensure that the final image does not have feature redundancy or distortion. Reflectance consistency comes from Retinex theory. Reflectance components are inherent properties of objects, and the reflectance of images of the same scene under different lighting conditions should be consistent. The reflectance formula is as follows: ； in and This is the enhanced image obtained after denoising.

2. The self-supervised low-light image enhancement method based on Retinex and CBAM according to claim 1, characterized in that, The low-light image in step 1.1) Training data taken from EnlightenGAN.

3. The self-supervised low-light image enhancement method based on Retinex and CBAM according to claim 1, characterized in that, The reflectivity component in step 3) and The acquisition process is as follows: First, the low-light image and uniformly illuminated images Inputting into RIENet yields the corresponding illuminance components. and Then use and Divide by the corresponding illuminance component and Obtain reflectivity components and .

Images