Low-illumination image enhancement method based on brightness guidance and color adjustment diffusion model

Abstract

The invention discloses a low-light image enhancement method based on brightness guidance and a color adjustment diffusion model. The enhancement performance is improved by deeply mining information of a low-light image. The specific method comprises the following three aspects: 1) brightness information extraction: in order to extract additional information in a low-light image, extracting an average value of pixels from a grey-scale map through linear mapping, and taking the average value as representation of image brightness information; and 2) detail and texture enhancement: in order to improve the detail and texture quality of an enhancement result, introducing a residual denoising diffusion model, and learning a mapping relation between a normal illumination image and a low-light image, so that the image content is restored more effectively. And 3) color recovery: providing an adaptive chromaticity adjustment algorithm for the color distortion problem of a low-illumination image, adaptively adjusting the chromaticity of an enhancement result through a trainable parameter, ensuring the accuracy and naturalness of color recovery, and significantly improving the performance of an enhancement model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a low-light image enhancement method based on brightness guidance and color adjustment diffusion model. Background Art

[0002] Low-light image enhancement aims to restore the corresponding normal-light image from the degraded low-light image. Traditional low-light image enhancement methods usually rely on optimization-based rules, but the effectiveness of these methods is highly dependent on hand-crafted prior models, and the accuracy of prior models is often difficult to guarantee under different lighting conditions, making it difficult to meet the changing actual needs.

[0003] With the development of deep learning, low-light image enhancement methods based on deep learning have gradually become the mainstream of research. These methods significantly improve the enhancement effect by learning the mapping relationship between low-light images and normal-light images. In particular, methods based on convolutional neural networks (CNNs) have made significant progress by effectively aggregating local information and have achieved significant improvements in enhancement effects. However, the limited receptive field and weight sharing strategy of CNNs have led to local restoration bias to a certain extent, which in turn affects the adaptability of the model to different inputs. On the other hand, the Transformer-based image enhancement method emphasizes long-term dependencies through the self-attention mechanism, which can achieve a wider adaptive receptive field and achieve better enhancement effects. However, it should be noted that the ordinary attention mechanism scales quadratically with the input size, resulting in a significant increase in computational overhead.

[0004] Diffusion models have demonstrated excellent performance in image denoising and reconstruction, but their direct application to low-light image enhancement faces certain challenges. Traditional diffusion models are usually designed for image generation tasks, and their inverse process starts from noise, while the low-light image itself is known. Therefore, using a residual diffusion model to learn the mapping between low-light images and normal-light images can more effectively improve the enhancement effect. In addition, in order to further improve the feature learning ability of the diffusion model, some methods have introduced technologies such as DA-CLIP to extract text features from images and embed them into the image restoration network, thereby guiding the model to learn high-fidelity image reconstruction. In the selection of color models, the color space of the image has an important influence on the image enhancement effect. Commonly used color spaces include RGB, YUV, and Lab, among which the YUV color space consists of three components: Y (brightness), U, and V (chrominance). The Y component represents the brightness information of the image, and the U and V components represent the chrominance signals, which can effectively capture the hue and saturation. Since brightness information plays a key role in images, YUV color space is often used in image restoration tasks. Compared with RGB color space, YUV can better handle the separation of brightness and chromaticity in images, so it has been widely used in image enhancement and restoration.

[0005] Chinese patent CN117557482A discloses a low-light image enhancement method based on illumination component optimization. This method obtains several groups of low-light image information, extracts the features of each image and constructs a judgment numerical model for low-light images. After the judgment numerical value is calculated and marked, the optimal value is selected as the standard value of the low-light image, and then the image is enhanced. This method effectively retains the detailed information of the image by constructing a Retinex neural network, and at the same time combines adaptive global mapping and homomorphic filtering for optimization, effectively reducing the degree of image distortion. This method can quickly complete the optimization of images at different illumination levels, and can avoid the image color being affected by the illumination component during the optimization process.

[0006] Low-light images usually face many degradation problems, such as loss of details, reduced contrast, and color distortion, which brings great challenges to downstream tasks such as object detection. At present, most low-light image enhancement models directly learn the mapping relationship between low-light images and normal-light images, but ignore how to effectively extract additional information from low-light images to guide the enhancement process. Therefore, existing low-light image enhancement methods may still have problems with insufficient information extraction, and may have problems with unrealistic or over-saturated colors, and there is still room for improvement. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a low-light image enhancement method based on brightness guidance and color adjustment diffusion model, which can improve the problems of insufficient information extraction and unrealistic or over-saturated colors existing in traditional low-light image enhancement models when processing low-light images.

[0008] In order to achieve the above object, the solution of the present invention is: A low-light image enhancement method based on brightness guidance and color adjustment diffusion model comprises the following steps: Step S1: The low-light image enhancement dataset contains a low-light image set and the normal lighting image set , in Represents the total number of images, Indicates pairs of low-light images, Indicates Paired normal-light images are used to sample the low-light image enhancement dataset. In each batch, M paired low-light images and M paired normal-light images are selected as the input of the model, where M>2; Step S2: randomly cropping the input paired images into image blocks of fixed size, and performing data enhancement on the cropped image blocks by random flipping and brightness enhancement; Step S3: extracting pixel values ​​from the grayscale image of the low-light image and mapping them to a normal distribution through linear transformation to obtain brightness levels; Step S4: After the brightness level is obtained, a brightness-guided residual diffusion model is trained. The training of the residual diffusion model includes two aspects: one is forward diffusion, using the residual map as a guide, gradually adding noise to the normal light image, and finally obtaining the accumulation of the low light image and the complete noise map, wherein the residual map is obtained by subtracting the normal light image from the low light image; the other is reverse diffusion, using the residual and the brightness level as conditions, gradually denoising from the final result of the forward diffusion, and obtaining the enhanced image; Step S5: Adaptive chromaticity adjustment is performed. U and V in the YUV color space represent chromaticity. The U and V channels of the image enhanced in step S4 are adjusted to adjust the color of the enhanced image. Step S6: During the entire training process of the residual diffusion model, the mean absolute error is used Perform pixel-level constraints on the enhanced image and normal-light image obtained by reverse diffusion of the residual diffusion model, as well as the predicted residual map (the output of the U-Net network in the residual diffusion model) and the true residual map (low-light image minus normal-light image); Step S7: Perform image quality evaluation and calculate the peak signal-to-noise ratio (PSNR) (dB) and the structural similarity index (SSIM) of the enhanced image obtained by reverse diffusion of the residual diffusion model and the normal illumination image.

[0009] Further, in step S3, the calculation of the brightness level includes the following steps: Step S31: For each input low-light image , convert it to a grayscale image Next, by calculating the grayscale image The overall brightness of a low-light image is determined by taking the average value of all pixels in :

[0010] in, Represents the width of the image, H represents the height of the image, and N represents the total number of pixels, that is, , are the 2D coordinates of pixels in the image; Step S32: Calculate the brightness level based on the overall brightness B , and then map it to a normal distribution with a mean of 5 and a standard deviation of 4 according to normalization and linear transformation:

[0011] in, , Indicates the brightness level of low-light images.

[0012] Further, the training of the brightness-guided residual diffusion model in step S4 includes the following steps: Step S41: The forward diffusion process is used to simulate the gradual decline of image quality and the increase of noise. The processing steps of a single forward diffusion are as follows:

[0013] in, represents the noisy image with t diffusion steps, , Indicates from the state arrive The residual diffusion and random perturbations of represents a normal distribution, , is the identity matrix, and is a table of independent coefficients that varies with the number of diffusion steps t, Can be obtained from The sampling is obtained, Represents a normal lighting image:

[0014] in, , . , Represents the cumulative coefficient table at time t, Represents random noise, which follows a normal distribution , is the unit matrix. When the diffusion step number t is long enough, that is, t=T, it can be simplified to , T represents the total number of diffusion steps; Step S42: Back diffusion process for low light images The image is enhanced by gradual noise reduction, and the sampling process is:

[0015] in, , Represents the cumulative coefficient table at time t-1, Residual map representing the prediction of the U-Net network, noise predicted by the U-Net network Calculated by the following formula:

[0016] Step S43: During the training phase, the goal of the diffusion model is to optimize the network Parameters , so that the estimated residual Close to real , the formula is: .

[0017] Further, step S5 specifically includes the following steps: Step S51: The image enhanced in step S4 is in RGB color space, and the RGB color space is converted into YUV color space. The conversion formula is as follows:

[0018] Step S52: In the training phase, first use the formula in step S42 The enhanced image is obtained, and then the enhanced image is converted into the YUV color space through the conversion formula from RGB color space to YUV color space. The U and V channels are adjusted using learnable parameters and finally converted into the RGB color space. The formula is as follows:

[0019] By its mean absolute error with the normal illumination image Optimize the model.

[0020] After adopting the above scheme, the low-light image enhancement method based on brightness guidance and color adjustment diffusion model of the present invention improves the enhancement performance by deeply mining the information of low-light images. The specific method includes the following three aspects: 1) Brightness information extraction: In order to extract additional information in low-light images, the average value of pixels is extracted from the grayscale image through linear mapping as a representation of the image brightness information. 2) Detail and texture enhancement: In order to improve the detail and texture quality of the enhanced results, a residual denoising diffusion model is introduced to learn the mapping relationship between normal light images and low-light images, so as to more effectively restore the image content. 3) Color restoration: In response to the color distortion problem of low-light images, an adaptive chromaticity adjustment algorithm is proposed, which adaptively adjusts the chromaticity of the enhanced results through trainable parameters to ensure the accuracy and naturalness of color restoration. The method of the present invention not only significantly improves the performance of the enhancement model by fully mining the brightness and texture information in low-light images, but also provides higher quality image input for downstream tasks such as target detection, semantic segmentation, and target tracking.

[0021] Compared with the prior art, the present invention has the following advantages: By introducing brightness information guidance and chromaticity adjustment mechanisms, the present invention can effectively extract additional information from low-light images and guide the model to perform enhancement based on this information. This method makes up for the problem of insufficient information extraction that may exist in traditional low-light image enhancement models when processing low-light images, and significantly improves the details and quality of the image.

[0022] The present invention proposes an adaptive chromaticity adjustment algorithm that can automatically adjust the chromaticity according to the image content. The algorithm not only ensures the accuracy of image color restoration, but also makes the restored image color more natural, consistent with the subjective visual perception of the human eye, and avoids the problem of unrealistic or over-saturated colors that may occur in traditional methods.

[0023] Through a large number of experimental verifications, the low-light image enhancement method based on brightness guidance and color adjustment diffusion model of the present invention shows obvious advantages in low-light image enhancement datasets, and can effectively improve image quality, especially in detail restoration, contrast enhancement and color correction, etc., which is significantly improved compared with existing enhancement methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the framework of low-light image enhancement based on brightness guidance and chromaticity adjustment diffusion model of the present invention. DETAILED DESCRIPTION

[0025] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0026] like Figure 1 As shown, the present invention discloses a low-light image enhancement method based on brightness guidance and color adjustment diffusion model, which includes the following steps: Step S1: Organize the low-light image enhancement dataset, which contains a low-light image set and the normal lighting image set , in Represents the total number of images, Indicates pairs of low-light images, Indicates Paired normal-light images are used to sample the low-light image enhancement dataset. In each batch, 5 paired low-light images and 5 paired normal-light images are selected as the input of the model. Step S2: The 10 input paired images are randomly cropped, the size of the image block is 192×288, and random flipping and brightness enhancement are used for data enhancement; Step S3: In order to use additional information to guide the diffusion model, pixel values ​​are extracted from the grayscale image of the low-light image and mapped to a normal distribution through a linear transformation to obtain a brightness level; the calculation of the brightness level includes the following steps: Step S31: For each input low-light image , convert it to a grayscale image Next, the overall brightness of the low-light image is determined by calculating the average value of all pixels in the grayscale image. :

[0027] in, , H represent the width and height of the image respectively, and N represents the total number of pixels, that is, , are the 2D coordinates of the pixels in the image.

[0028] Step S32: Calculate the brightness level based on the brightness , and then mapped it to a normal distribution with a mean of 5 and a standard deviation of 4 based on normalization and linear transformation.

[0029]

[0030] in, , Represents the brightness level of low-light images.

[0031] Step S4: After obtaining the brightness level of the low-light image, the residual diffusion model guided by brightness can be trained, including forward diffusion and backward diffusion. The forward diffusion uses the residual map as a guide. The residual map is obtained by subtracting the normal light image from the low-light image. The normal light image is gradually denoised to finally obtain the low-light image and noise. The backward diffusion uses the residual and brightness level as conditions to gradually denoise the complete noise map to obtain the enhanced image. The training of the brightness-guided residual diffusion model consists of the following steps: Step S41: The forward diffusion process is used to simulate the gradual decline of image quality and the increase of noise. The processing steps of a single forward diffusion are as follows:

[0032] in, represents the noisy image with t diffusion steps, , Indicates from the state arrive The residual diffusion and random perturbations of represents a normal distribution, , is the identity matrix, and is a table of independent coefficients that varies with the number of diffusion steps t, Can be obtained from The sampling is obtained, Represents a normal lighting image:

[0033] in, , . , Represents the cumulative coefficient table at time t, Represents random noise, which follows a normal distribution , is the unit matrix. When the diffusion step number t is long enough, that is, t=T, it can be simplified to , T represents the total number of diffusion steps.

[0034] Step S42: Back diffusion process for low light images The image is enhanced by gradual noise reduction, and the sampling process is:

[0035] in, , Represents the cumulative coefficient table at time t-1, Residual map representing the prediction of the U-Net network, noise predicted by the U-Net network Calculated by the following formula:

[0036] Step S43: During the training phase, the goal of the residual diffusion model is to optimize the network Parameters , so that the estimated residual near , the formula is: .

[0037] Step S5: Adaptive chromaticity adjustment is performed. The U and V channels in the YUV color space represent chromaticity. The U and V channels of the image enhanced in step S4 are adjusted to adjust the color of the enhanced image to solve the color deviation problem. Specifically, the following steps are included: Step S51: The image enhanced in step S4 is in RGB color space, and the RGB color space is converted into YUV color space. The conversion formula is as follows:

[0038] Step S52: In order to adaptively adjust the chromaticity, the result of the diffusion model sampling is adjusted by the learnable parameters. In the training stage, the enhanced image is first obtained by formula (5), and then the enhanced image is converted into the YUV color space by formula (8), and the U and V channels are adjusted by the learnable parameters, and finally converted into the RGB color space. The formula is as follows:

[0039] By its mean absolute error with the normal illumination image Optimize the model.

[0040] Step S6: During the entire training process of the residual diffusion model, the mean absolute error L_1 is used to perform pixel-level constraints on the enhanced image and the normal illumination image obtained by the reverse diffusion of the residual diffusion model, as well as the predicted residual map (the output of the U-Net network in the residual diffusion model) and the true residual map (low-light image minus normal illumination image); Step S7: Perform image quality evaluation and calculate the peak signal-to-noise ratio (PSNR) (dB) and the structural similarity index (SSIM) of the enhanced image obtained by reverse diffusion of the residual diffusion model and the normal illumination image.

[0041] The present invention is tested and analyzed as follows: In experiment 1, the present invention is used to conduct an ablation experiment on the LOL low-light image enhancement dataset. The LOL low-light image enhancement dataset consists of 500 pairs of low-light and normal-light images, which are divided into 485 training pairs and 15 test pairs.

[0042] In order to verify the effectiveness of each module of the present invention, an ablation experiment is conducted on the test set of the LOL low-light image enhancement dataset. Table 1 shows the results of experiment 1. Wherein, 'Br' represents the brightness condition of the low-light image described in step S3. , 'ACA' represents the adaptive chromaticity adjustment method described in step S5, '√' and '×' represent using and not using the corresponding modules, respectively, and 'PSNR (dB)' and 'SSIM' represent peak signal-to-noise ratio and structural similarity, respectively.

[0043] Table 1:

[0044] Experimental results show that the brightness condition and chromaticity adjustment module proposed in the present invention have greatly improved performance.

[0045] Experiment 2: The present invention is tested on a low-light image enhancement benchmark dataset.

[0046] In order to verify the effectiveness of the algorithm, the proposed model is tested on the low-light image enhancement benchmark datasets LOL, LOLv2-real and LOLv2-synthetic. Table 2 shows the results of Experiment 2.

[0047] Table 2:

[0048] It can be found from Table 2 that the low-light image enhancement method based on brightness guidance and chromaticity adjustment diffusion model proposed in the present invention achieves excellent performance on the low-light image enhancement benchmark dataset.

[0049] Combining Experiments 1 and 2, the present invention has significant performance advantages on the three existing low-light benchmark datasets, surpassing the highest level in the current academic field, and verifies the effectiveness of the brightness-guided and chromaticity-adjusted diffusion model proposed in the present invention.

[0050] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A low-light image enhancement method based on brightness guidance and color adjustment diffusion model, characterized in that: The following steps are involved: Step S1: The low-light image enhancement dataset contains a low-light image set and the normal lighting image set , in Represents the total number of images, Indicates pairs of low-light images, Indicates Paired normal-light images are used to sample the low-light image enhancement dataset. In each batch, M paired low-light images and M paired normal-light images are selected as the input of the model, where M>2; Step S2: randomly cropping the input paired images into image blocks of fixed size, and performing data enhancement on the cropped image blocks by random flipping and brightness enhancement; Step S3: extracting pixel values ​​from the grayscale image of the low-light image and mapping them to a normal distribution through linear transformation to obtain brightness levels; Step S4: After the brightness level is obtained, the brightness-guided residual diffusion model is trained. The training of the residual diffusion model includes two aspects: one is forward diffusion, using the residual map as a guide, gradually adding noise to the normal light image, and finally obtaining the accumulation of the low light image and the complete noise map; the other is reverse diffusion, using the residual map and the brightness level as conditions, gradually denoising from the final result of the forward diffusion, and obtaining the enhanced image; Step S5: Adaptive chromaticity adjustment is performed. U and V in the YUV color space represent chromaticity. The U and V channels of the image enhanced in step S4 are adjusted to adjust the color of the enhanced image. Step S6: During the entire training process of the residual diffusion model, the mean absolute error is used Pixel-level constraints are applied to the enhanced image obtained by reverse diffusion of the residual diffusion model, the normal illumination image, the predicted residual map, and the true residual map; the predicted residual map refers to the output of the U-Net network in the residual diffusion model, and the true residual map refers to the low-light image minus the normal illumination image; Step S7: Perform image quality evaluation and calculate the peak signal-to-noise ratio (PSNR) (dB) and the structural similarity index (SSIM) of the enhanced image obtained by reverse diffusion of the residual diffusion model and the normal illumination image.

2. The low-light image enhancement method based on brightness guidance and color adjustment diffusion model according to claim 1, characterized in that: In step S3, the calculation of the brightness level includes the following steps: Step S31: For each input low-light image , convert it to a grayscale image Next, by calculating the grayscale image The overall brightness of a low-light image is determined by taking the average value of all pixels in : in, Represents the width of the image, H represents the height of the image, and N represents the total number of pixels, that is, , is the 2D coordinate of the pixel in the image; Step S32: Calculate the brightness level based on the overall brightness B , and then map it to a normal distribution with a mean of 5 and a standard deviation of 4 according to normalization and linear transformation: in, , Indicates the brightness level of low-light images.

3. The low-light image enhancement method based on brightness guidance and color adjustment diffusion model as claimed in claim 1, characterized in that: The training of the brightness-guided residual diffusion model of step S4 includes the following steps: Step S41: The forward diffusion process is used to simulate the gradual decline of image quality and the increase of noise. The processing steps of a single forward diffusion are as follows: in, represents the noisy image with t diffusion steps, , Indicates from the state arrive The residual diffusion and random perturbations of represents a normal distribution, , represents the identity matrix, and is a table of independent coefficients that varies with the number of diffusion steps t, from The sampling is obtained, Represents a normal lighting image: in, , ; , Represents the cumulative coefficient table at time t, Represents random noise, which follows a normal distribution F , when t=T, we can simplify to get , T represents the total number of diffusion steps; Step S42: The reverse diffusion process is used to gradually reduce the noise of the result of the previous diffusion to obtain an enhanced image. The sampling process is: in, , Represents the cumulative coefficient table at time t-1, Residual map representing the prediction of the U-Net network, noise predicted by the U-Net network Calculated by the following formula: Step S43: During the training phase, the goal of the diffusion model is to optimize the network Parameters , so that the estimated residual Close to real , the formula is: 。 4. The low-light image enhancement method based on brightness guidance and color adjustment diffusion model according to claim 1, characterized in that: Step S5 specifically includes the following steps: Step S51: The image enhanced in step S4 is in RGB color space, and the RGB color space is converted into YUV color space. The conversion formula is as follows: Step S52: In the training phase, first use the formula in step S42 The enhanced image is obtained, and then the enhanced image is converted into the YUV color space through the conversion formula from RGB color space to YUV color space. The U and V channels are adjusted using learnable parameters and finally converted into the RGB color space. The formula is as follows: By its mean absolute error with the normal illumination image Optimize the model.

Images