Low-light image enhancement method based on wavelet diffusion model of fourier transform

Abstract

The present application relates to the technical field of low-light image enhancement, and particularly relates to a low-light image enhancement method based on a wavelet diffusion model of Fourier transform. The steps are as follows: S1: wavelet transform is performed on an input image, and a diffusion process is transferred to a low-frequency domain for processing; S2: prior information of a frequency spectrum of the input image is extracted through Fourier transform, and a sampling process is embedded; S3: a high-frequency detail enhancement module is used to reconstruct and enhance fine-grained structures in a high-frequency domain of the low-light image; and S4: recovered wavelet low-frequency domain x0 and high-frequency domain L H The low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application extracts Fourier prior information of an input image, embeds a sampling process of a wavelet diffusion model, effectively restricts the generation of redundant content, and further uses a high-frequency detail enhancement module to realize fine-grained recovery of high-frequency information of an image, so that a perception-oriented visual enhancement effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of low-light image enhancement, in particular to a low-light image enhancement method based on a wavelet diffusion model of Fourier transform. BACKGROUND

[0002] In the real world, dim lighting conditions often cause complex degraded environments, resulting in various adverse shooting factors and leading to serious quality damage of captured images. At the same time, this also brings great challenges to various computer vision applications, such as automatic driving, image segmentation, text detection, etc. Therefore, based on these challenges, how to conduct image enhancement in complex real environments has attracted great attention and has a significant impact on the development of visual intelligence.

[0003] Low-light image enhancement aims to learn the mapping relationship between the degradation domain and the normal domain, reverse various degradation domains, and restore the original clean image, so as to improve the performance of various industrial vision applications. With the rise of the deep learning era, data-driven performance has become the mainstream of research. They learn the prior knowledge of degradation parameters from large-scale data to bridge the gap between degraded images and normal images. Compared with traditional methods that rely on manual prior optimization models, this undoubtedly enhances the generalization and fault tolerance of the model. Despite this, existing methods still face the challenge of modeling and perception of complex environments.

[0004] Low-light image enhancement aims to reverse various degradation domains to achieve high-quality mapping. Unlike most linear image restoration tasks, the low-light image enhancement task has many unknown degradation parameters, and needs to construct unknown degradation models while enhancing images. Early low-light image enhancement techniques focus on using model-based methods to process by optimizing the parameter information of the image itself, but this is severely limited by the accuracy of manual verification. With the advent of the deep learning era, the above problems have been greatly alleviated through data-driven methods, which has brought major progress in image enhancement research. For example, a dataset with different exposure levels is constructed for night imaging problems. The signal-to-noise ratio transformer and CNN model with spatially varying operations are used for restoration. Neural representation is used to regulate degradation to reduce enhancement difficulty;

[0005] Due to the excellent visual perception of generated models on images, there has been a surge of interest in seeking more extensive image priors through generated models. In particular, diffusion models, as a recent research hotspot, have been widely used in the field of image enhancement by researchers;

[0006] In recent years, diffusion models have achieved remarkable results in image generation by learning parameter Markov chains, which also prompted them to have more accurate target distribution and generalization performance than other generative models (such as GAN). Inspired by recent diffusion models, many image restoration methods have also been studied. By constructing a stochastic differential equation to reverse image degradation, good recovery is obtained. It is proposed to generate real output using diffusion prior. It is proposed to use Fourier prior to guide diffusion. However, due to the limitations of the model, these methods have a large amount of time overhead. Therefore, it is proposed to reduce memory and time overhead by transferring the diffusion process to the wavelet domain. However, the above methods still cannot achieve effective mapping balance between the degradation domains, resulting in still biased enhanced performance.

[0007] Therefore, a wavelet diffusion model is proposed to effectively alleviate the above problems. However, as Figure 1 indicated, due to the lack of effective guidance to the model, the sampling results do not have excellent visual perception effect on image details. At the same time, due to the lack of effective prior constraint on the sampling process, the existing method is difficult to construct high-quality mapping in complex real environment.

[0008] Therefore, a low-light image enhancement method based on wavelet diffusion model of Fourier transform is designed to provide another technical solution to the above technical problems. SUMMARY

[0009] Based on this, it is necessary to provide a low-light image enhancement method based on wavelet diffusion model of Fourier transform to solve the technical problems proposed in the background art for the above technical problems.

[0010] In order to solve the above technical problems, the present application adopts the following technical solutions:

[0011] The low-light image enhancement method based on wavelet diffusion model of Fourier transform has the following steps:

[0012] S1: wavelet transform is performed on the input image, and the diffusion process is transferred to the wavelet low frequency domain for processing;

[0013] S2: extract the frequency spectrum prior information of the input image through Fourier transform, embed the sampling process, guide the generation and detail enhancement of the wavelet low frequency domain;

[0014] S3: through the high frequency detail enhancement module, the wavelet high frequency domain of the low-light image is reconstructed to enhance the fine-grained structure;

[0015] S4: through discrete wavelet inverse transform, the recovered wavelet low frequency domain x0 and high frequency domain L H transformed into the final enhanced image

[0016] As a preferred embodiment of the low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application, the wavelet transform is performed on the input image, and the diffusion process is transferred to the low-frequency domain for processing, and the steps are as follows:

[0017] The low-light image I L and the normal image L H are subjected to discrete wavelet transform to obtain respective low-frequency domains L L and H L .

[0018] As a preferred embodiment of the low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application, the frequency spectrum prior information of the input image is extracted by Fourier transform, the sampling process is embedded, and the generation and detail enhancement of the low-frequency domain of the wavelet are guided, and the steps are as follows:

[0019] The fast Fourier transform is performed on L L and H L to obtain their frequency spectrum graphs;

[0020] The amplitude and phase of L L are embedded into the sampling process, and pha is replaced to update the graph;

[0021] The inverse fast Fourier transform is used to obtain the new sampling result

[0022] As a preferred embodiment of the low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application, the Fourier decomposition is performed on the sampling result of each step of the sampling process to obtain the amplitude and phase, and the expression is as follows:

[0023] amp L ,pha L =FFT(L L );

[0024] amp H ,pha H =FFT(H L );

[0025]

[0026] Wherein, amp represents the amplitude, and pha represents the phase.

[0027] As a preferred embodiment of the low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application, the new sampling result is expressed as follows:

[0028]

[0029] As a preferred embodiment of the low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application, the wavelet high-frequency domain of the low-light image is reconstructed and enhanced by the high-frequency detail enhancement module, and the steps are as follows:

[0030] The L H features are extracted by the depth separable convolution.

[0031] The feature information in three directions is concentrated and supplemented by cross attention.

[0032] The progressive feature enhancement block combining dilated convolution and effective local attention is used to continuously capture local information.

[0033] The depth separable convolution is used to reduce the dimension, so as to obtain the reconstructed wavelet high-frequency domain.

[0034] It can be seen without doubt that the above technical solutions of the present application can solve the technical problems to be solved by the present application.

[0035] Meanwhile, through the above technical solutions, the present application at least has the following beneficial effects:

[0036] The low-light image enhancement method based on the wavelet diffusion model of Fourier transform provided by the present application extracts the Fourier prior of the input image, embeds the sampling process of the wavelet diffusion model, effectively constrains the generation of redundant content, guides the enhancement of details, and realizes the recovery of the high-frequency information of the image by the high-frequency detail enhancement module, so as to achieve the visual enhancement effect of perception guidance. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 The visual effect demonstration diagram of the most advanced low-light image enhancement method WCDM;

[0039] Figure 2 The overall working process diagram of the generation of the enhanced diffusion model (FWDM) of the present application;

[0040] Figure 3 The visual comparison diagram of the latest low-light image enhancement method of the present application on the LOLv1 data set;

[0041] Figure 4 Schematic diagram of visual comparison of the LIME (line 1) and DICM (line 2) data sets of the present application in the most advanced low-light image enhancement method. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In order to enable those skilled in the art to better understand the technical solutions of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings.

[0043] It should be noted that the embodiments in the present application and the features and technical solutions in the embodiments can be combined with each other without conflict.

[0044] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0045] Reference Figures 1-4 Low-light image enhancement method based on wavelet diffusion model of Fourier transform.

[0046] The diffusion model trains the Markov chain through variational inference. The forward process converts complex data into completely random data by adding noise. This process can be expressed as:

[0047]

[0048] where q is the joint probability distribution, x t and β t are the corrupted noisy data and the predefined variance at time step t, respectively. N represents the Gaussian distribution, and I is the identity matrix. In addition, each time step x t can be obtained directly by calculating the wavelet low-frequency domain x0;

[0049]

[0050] where α t = 1-β t I, α t is the attenuation factor, is the cumulative attenuation factor. The backward inference process is to remove noise from the Gaussian noise iteratively by relying on the optimized noise predictor until the randomly sampled noise becomes clean data x0. Formulated as:

[0051]

[0052] where u0is the diffusion model noise predictor, p θ is the conditional probability distribution of the model parameters θ, is the covariance matrix of the noise, denoted as σt, which is mainly optimized by the editing and data synthesis functions as a way to learn the conditional denoising process, as follows:

[0053]

[0054] where ∈ θ is the noise at time step t, is a function approximator to predict ∈, and the rest of the letters have the same meaning as above.

[0055] Since the low-frequency domain of wavelet decomposition contains most of the information of the image, the wavelet diffusion model relies on the framework of the conditional denoising probability model, and the diffusion process is completely transferred to the low-frequency domain of the wavelet by the discrete wavelet transform (DWT):

[0056] {L L , L H} = DWT(I L ) (8)

[0057] {H L , H H} = DWT(I H ) (7)

[0058] where I L , I H , respectively, represent the input low-light image and normal-light image, L L , H L , L H , H H , respectively, represent the wavelet low-frequency domain and high-frequency domain of I L , I H .

[0059] Therefore, we can use the following objective function L diff to optimize the noise predictor ∈ θ of the wavelet diffusion model:

[0060]

[0061] where T is the total step of the inference process. ∈ t is the true noise at time step t, is the recovered wavelet low-frequency domain, and the rest of the letters have the same meaning as above.

[0062] 1. Implementation

[0063] Perceptual oriented enhancement is achieved by restoring image details and constraining the generation of redundant features. Specifically, as shown in Figure 2 , first, the input image is wavelet transformed, then the diffusion process is transferred to the wavelet low frequency domain, the spectral prior information of the input image is extracted by Fourier transform, the sampling process is embedded to guide the generation of wavelet low frequency domain and detail enhancement. For the decomposed wavelet high frequency domain, a high frequency detail enhancement module (HDEM) is constructed by effective combination of multiple attention and neural networks to enhance the restoration of image fine-grained structure.

[0064] 1.1 Fourier guided wavelet diffusion

[0065] Most of the luminance information of the image obtained by Fourier transform is concentrated on the amplitude. In contrast, the structure and content information is closely related to the phase. At the same time, the diffusion model is committed to maintain the data distribution rather than the content consistency of pixels. Therefore, it is tried to use Fourier spectrum prior to constrain the guided wavelet diffusion model. Therefore, as shown in Figure 2 , first, the low light image I L and the normal image L H are discrete wavelet transformed (DWT) to obtain their respective wavelet low frequency domain (L L and H L ). Then, the fast Fourier transform (FFT) is performed on L L and H L to obtain their spectral graphs. At the same time, the sampling results of each step of the sampling process are Fourier decomposed (FFT) to obtain the amplitude and phase. It can be expressed as:

[0066] amp L ,pha L =FFT(L L ) (9)

[0067] amp H ,pha H =FFT(H L ) (10)

[0068]

[0069] In the formula, amp represents the amplitude, and pha represents the phase.

[0070] Then, in order to interfere with the generation of redundant content in the sampling process and preserve the original content structure of the image, the amplitude and phase of L L are embedded into the sampling process and replaced with pha to constrain the content structure of the sampled image. After updating the spectral graph, the new sampling result is defined as:

[0071]

[0072] In addition, in order to further guide the performance of the model, the difference between them is minimized by realizing the L1 distance between the amplitude and phase of H and H L , so as to obtain more realistic lighting conditions and content structure of the sampling results. It can be expressed as:

[0073]

[0074] Wherein, is the amplitude loss, phase loss and weighted parameter.

[0075] 1.2 High frequency detail enhancement module

[0076] The wavelet high frequency subband of the low light image, in general, represents the sparse representation of the image details in the vertical, horizontal and diagonal three directions. Therefore, in order to restore the high frequency details of the low light image to the same high frequency detail rich content structure as the normal light image, a high frequency detail enhancement module (HDEM) is proposed for reconstruction. As shown in the lower half of the figure, Figure 2 Firstly, the features of L H are extracted using depth separable convolution, and then the feature information of each direction is centralized, and the feature details of the normal image L H are supplemented by using cross attention layer.

[0077] Subsequently, by fusing the outputs of dilated convolution and effective local attention layer, a progressive feature enhancement block combining dilated convolution and effective local attention (ELA) is designed, which fuses the outputs of dilated convolution and effective local attention layer; for capturing local information constantly, and through the introduction of directional attention mechanism, significant directional enhancement is carried out, wherein the first and last convolution are used to extract local information. In contrast, the middle convolution improves the receptive field and better utilizes the long-range information. ELA enhances the dependency between local features and global features, and improves the enhancement of important local features. Embedding ELA in the dilated process effectively avoids the loss of local features and enhances the association with global information. Finally, depth separable convolution is used again to reduce the dimension, so as to obtain the reconstructed wavelet high frequency domain. By using inverse discrete wavelet transform (IDWT), the restored wavelet low frequency domain and wavelet high frequency domain are transformed into the final enhanced image

[0078]

[0079] Wherein, the meanings of the letters are consistent with the above.

[0080] 1.3 Network training

[0081] To further enhance the recovery of low-light images, the content loss L combining MSE loss and SSIM loss is used to minimize the content difference between the recovered image contnet and the content normal-light image I H :

[0082]

[0083] where ω1 is the weight of the first layer of the image encoder in the ResNet101 CLIP model, and the remaining letters have the same meaning as above.

[0084] Therefore, the total training loss can be defined as:

[0085] L total = L diff + L content + L fre (16)

[0086] 2. Experimental evaluation

[0087] 2.1 Experimental setup

[0088] The method of the present application is implemented in PyTorch on two NVIDIA RTX 3090 GPUs. The network is trained for a total of 1000 epochs using the Adam optimizer, with an initial learning rate set to 1x10 -4 , and the batch size and patch size are set to 4 and 256x256. For the training phase, the time step T is set to 200, and the implicit sampling step is set to 10, which is used for both the training phase and the inference phase.

[0089] Dataset and evaluation metrics. The network of the present application is trained and evaluated on the LOLv1 dataset, which contains 500 real-world low / normal-light image pairs, of which 485 image pairs are used for training and 15 image pairs are used for evaluation. In addition, tests are also conducted on two unpaired datasets, LIME and DICM. For paired datasets, the fidelity of the recovery performance is measured and the quality of the resulting image is measured by two distortion indicators (PSNR and SSIM) and two perceptual indicators (LPIPS and VSI). While for unpaired datasets, two non-reference indicators, i.e. the enhancement level (LOE) and the blind / referenceless image spatial quality evaluator (BRISQUE) are used for evaluation.

[0090] ​To verify the effectiveness of the method proposed in the present application, it is compared with the most advanced methods in recent years, including zero-dce++, heurgan, RUAS, SCI, SNRNet, Uformer, Restormer, UHDFour, CLIP-LIT, LLFormer, NeRCo, GDP, WCDM, etc.

[0091] 2.2 Evaluation results

[0092] The method of the present application is compared with all the most advanced methods on LOLv1. As shown in Table 1, the method of the present application achieves the most advanced quantitative performance in several indicators compared with all the methods. In particular, the significant improvement of PSNR provides convincing evidence for the superior image quality of our method. At the same time, the evaluation comparison of two unpaired data LIME 8 and DICM with competitive methods verifies the effectiveness and generalization of the method of the present application.

[0093] Table 1 Quantitative evaluation of different methods on LOLv1 dataset

[0094]

[0095] The effectiveness of the method of the present application in visual quality is evaluated by combining two non-reference perception indicators (BRISQUE and LOE), which are lower, thereby producing better visual quality. As shown in Table 2, the method of the present application meets the quantitative standard on both datasets compared with other competitive methods. Specifically, the best performance of all quantitative evaluations is obtained for BRISQUE. This further proves the better generality of the method of the present application in real-world scenarios, and the enhancement more consistent with human visual perception.

[0096] Table 2 Quantitative evaluation of different methods on LIME [8] and DICM

[14] datasets

[0097]

[0098] Figure 3 The visual comparison results of the method of the present application and the most advanced methods on LOLv1 dataset are shown. Through these comparisons, it is easy to see that the method of the present application can restore more realistic colors and more friendly visual effects without excessive enhancement or insufficient enhancement. At the same time, all other methods have the problem of color distortion. In addition, Figure 4 The visual comparison results of the method of the present application on LIME and DICM datasets are shown. It is obvious that the model of the present application skillfully adjusts the lighting conditions without causing redundant color enhancement to the environment. This successful balance further confirms the effectiveness of the method proposed in the present application and its friendliness to visual perception.

[0099] 5.3 Ablation Study

[0100] An ablation study is demonstrated on the LOLvI dataset to prove the effectiveness of the proposed method. To verify the effectiveness of the proposed components, the proposed components are added to the network one by one, and the rest remains unchanged for the ablation experiments.

[0101] Specifically, as shown in Table 3, #1 indicates that nothing is added. #2 represents the replacement stage operation performed during network sampling. #3 represents the prior guidance operation performed by adding the Ere loss to #2. #4 represents the use of the complete setting of the added high-frequency detail enhancement module to verify its effectiveness. It can be easily seen that when only one layer of the wavelet diffusion model is used on the network without performing any operation, the performance is at the lowest value. After adding the two Fourier prior bootstrapping methods to the network one by one, the network performance is constantly improved, fully confirming the positive bootstrapping effect of the Fourier prior on the sampling process. In addition, for the network using the complete setting

[0102] Table 3: Ablation study results of the proposed method

[0103]

[0104] The performance reaches the optimal value, verifying the effectiveness of the proposed HDEM.

[0105] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The preferred embodiments do not describe all the details, nor limit the present application to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and utilize the present application. The present application is limited only by the claims and their full scope and equivalents.

Claims

1. A low-light image enhancement method based on a wavelet diffusion model of Fourier transform, characterized in that, The steps are as follows: S1: wavelet transform is performed on the input image, and the diffusion process is transferred to the wavelet low-frequency domain for processing; S2: the spectral prior information of the input image is extracted through Fourier transform, the sampling process is embedded, and the generation and detail enhancement of the wavelet low-frequency domain are guided; S3: the wavelet high-frequency domain of the low-light image is reconstructed and enhanced in fine-grained structure through the high-frequency detail enhancement module. S4: the restored wavelet low frequency domain and high frequency domain are transformed into an enhanced image by inverse discrete wavelet transform ;​​ Wavelet transform is performed on the input image, and the diffusion process is transferred to the wavelet low-frequency domain for processing, and the steps are as follows: low light image and normal image performing discrete wavelet transform to obtain respective wavelet low frequency domain and ; The spectral prior information of the input image is extracted through Fourier transform, the sampling process is embedded, and the generation and detail enhancement of the wavelet low-frequency domain are guided, and the steps are as follows: To and perform a fast Fourier transform to obtain a frequency spectrum thereof; The amplitude and phase of are embedded into the sampling process and the updated map is obtained by replacing . obtaining new samples using an inverse fast fourier transform ; The wavelet high-frequency domain of the low-light image is reconstructed and enhanced in fine-grained structure through the high-frequency detail enhancement module, and the steps are as follows: extracting features by deep separable convolution ; The feature information of the three directions is concentrated, and cross-attention is used for supplementation; Through the progressive feature enhancement block combining dilated convolution and effective local attention, local information is continuously captured; Depth separable convolution is used to reduce the dimension, so as to obtain the reconstructed wavelet high-frequency domain. 2.The low-light image enhancement method based on Fourier transform and wavelet diffusion model according to claim 1, wherein, the sampling result of each step of the sampling process Fourier decomposition is performed to obtain the amplitude and phase, expressed as follows: ； ； ； wherein denotes the amplitude, denotes the phase. 3.The low-light image enhancement method based on Fourier transform and wavelet diffusion model according to claim 1, wherein, new sample result , the expression is as follows: 。

Images