Low-light image enhancement method of wavelet diffusion model based on Fourier transform

Abstract

The invention relates to the technical field of low-light image enhancement, in particular to a low-light image enhancement method of a wavelet diffusion model based on Fourier transform. The method comprises the following steps: S1, carrying out wavelet transformation on an input image, and transferring a diffusion process to wavelet low-frequency domain processing; s2, extracting frequency spectrum prior information of the input image through Fourier transform, and embedding a sampling process; s3, reconstructing an enhanced fine-grained structure for a wavelet high-frequency domain of the low-light image through a high-frequency detail enhancement module; and S4, transforming the recovered low-frequency domain x0 and high-frequency domain LH of the wavelet into an enhanced image # imgabs0 # through discrete wavelet inverse transformation. According to the low-light image enhancement method of the wavelet diffusion model based on Fourier transformation provided by the invention, the generation of redundant contents is effectively restrained by extracting the Fourier prior of the input image and embedding the Fourier prior into the sampling process of the wavelet diffusion model; and the high-frequency detail enhancement module is used for recovering the fine granularity of the high-frequency information of the image, so that the perception-oriented visual enhancement effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of low-light image enhancement, and in particular to a low-light image enhancement method based on a wavelet diffusion model of Fourier transform. Background Art

[0002] In the real world, dim lighting conditions often create complex and degraded environments, which bring about various unfavorable shooting factors and cause serious quality damage to the captured images. At the same time, this also brings huge challenges to various computer vision applications, such as autonomous driving, image segmentation, text detection, etc. Therefore, based on these challenges, the research on how to perform image enhancement in complex real-world environments has received great attention and has had a significant impact on the development of visual intelligence.

[0003] Low-light image enhancement aims to improve the performance of various industrial vision applications by learning the mapping relationship between degraded domains and normal domains, reversing various degraded domains, and restoring the original clean image. With the rise of the deep learning era, data-driven performance has become the mainstream of research. They learn prior knowledge of degradation parameters from large-scale data to bridge the gap between degraded and normal images, which undoubtedly enhances the generalization and fault tolerance of the model compared to traditional methods that rely on manual prior optimization models. Despite this, existing methods still face the challenge of modeling and perceiving complex environments.

[0004] Low-light image enhancement aims to reverse various degradation domains and achieve high-quality mapping. Unlike most linear image restoration tasks, low-light image enhancement tasks have many unknown degradation parameters, and it is necessary to build unknown degradation models while enhancing the image. Early low-light image enhancement techniques focused on using model-based methods to optimize the parameter information of the image itself, but this was severely limited by the accuracy of manual verification. With the advent of the deep learning era, the above problems have been greatly alleviated through data-driven methods, which has also brought about significant progress in image enhancement research. For example, a dataset with different exposure levels was constructed for night imaging problems. Restoration is performed using a signal-to-noise ratio transformer and a CNN model with spatial variation operations. A neural representation is used to normalize degradation to reduce the difficulty of enhancement;

[0005] Since generative models have excellent visual perception effects on images, people are increasingly interested in seeking more extensive image prior knowledge through generative models. In particular, diffusion models have become a recent research hotspot and have been widely used by researchers in the field of image enhancement.

[0006] In recent years, diffusion models have achieved remarkable results by learning parameterized Markov chains for image generation, which has also led to them having more accurate target distributions and generalization performance than other generative models (such as GAN). Inspired by recent diffusion models, many image restoration methods have also been studied. By constructing stochastic differential equations to reverse image degradation, good restoration is achieved. It is proposed to use diffusion priors to generate realistic outputs. It is proposed to use Fourier priors to guide diffusion. However, due to model limitations, these methods have a lot of time overhead. Therefore, it is proposed to reduce memory and time overhead by transferring the diffusion process to the wavelet domain. However, the above methods still cannot achieve an effective mapping balance between the degraded domains, resulting in an enhancement performance that is still biased.

[0007] Therefore, a wavelet diffusion model is proposed to effectively alleviate the above problems. Figure 1 As shown in Figure 1, due to the lack of effective guidance of the model, the sampling results do not achieve excellent visual perception effects on image details. At the same time, due to the lack of effective prior constraints on the sampling process, existing methods find it difficult to build high-quality mappings in complex real-world environments.

[0008] To this end, a low-light image enhancement method based on a wavelet diffusion model of Fourier transform is designed to provide another technical solution to the above technical problem. Summary of the invention

[0009] Based on this, it is necessary to provide a low-light image enhancement method based on a wavelet diffusion model of Fourier transform to solve the technical problems raised in the above background technology.

[0010] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0011] The low-light image enhancement method based on the wavelet diffusion model of Fourier transform has the following steps:

[0012] S1: Perform wavelet transform on the input image and transfer the diffusion process to the wavelet low-frequency domain;

[0013] S2: Extract the spectral prior information of the input image through Fourier transform, embed it into the sampling process, and guide the generation and detail enhancement of the wavelet low-frequency domain;

[0014] S3: Reconstruct and enhance the fine-grained structure of the wavelet high-frequency domain of the low-light image through the high-frequency detail enhancement module;

[0015] S4: Through the inverse discrete wavelet transform, the recovered wavelet low frequency domain x0 and high frequency domain L H Transformed into the final enhanced image

[0016] As a preferred implementation of the low-light image enhancement method based on the Fourier transform wavelet diffusion model provided by the present invention, the input image is subjected to wavelet transform, and the diffusion process is transferred to the wavelet low-frequency domain for processing, the steps are as follows:

[0017] For low light images L and the normal image L H Perform discrete wavelet transform to obtain the respective wavelet low-frequency domain L L and H L .

[0018] As a preferred implementation of the low-light image enhancement method based on the Fourier transform wavelet diffusion model provided by the present invention, the spectral prior information of the input image is extracted by Fourier transform, embedded in the sampling process, and guided to generate the wavelet low-frequency domain and enhance the details, the steps are as follows:

[0019] For L L and H L Perform fast Fourier transform to obtain its spectrum diagram;

[0020] L L The amplitude and phase are embedded into the sampling process and replaced with pha to update the spectrum;

[0021] Use inverse fast Fourier transform to get new sampling results

[0022] As a preferred implementation of the low-light image enhancement method based on the Fourier transform wavelet diffusion model provided by the present invention, the sampling results of each step of the sampling process are Perform Fourier decomposition to obtain the amplitude and phase, the expressions are as follows:

[0023] amp L ,pha L =FFT(L L );

[0024] amp H ,pha H =FFT(H L );

[0025]

[0026] Among them, amp represents amplitude and pha represents phase.

[0027] As a preferred implementation of the low-light image enhancement method based on the Fourier transform wavelet diffusion model provided by the present invention, the new sampling result The expression is as follows:

[0028]

[0029] As a preferred implementation of the low-light image enhancement method based on the Fourier transform wavelet diffusion model provided by the present invention, the high-frequency detail enhancement module reconstructs and enhances the fine-grained structure of the wavelet high-frequency domain of the low-light image, and the steps are as follows:

[0030] Extract L through depthwise separable convolution H Features;

[0031] The feature information of three directions is concentrated and supplemented by cross attention;

[0032] The local information is continuously captured by combining dilated convolution and progressive feature enhancement blocks with effective local attention;

[0033] Depthwise separable convolution is used to reduce the dimensionality and thus obtain the reconstructed wavelet high frequency domain.

[0034] It can be seen without a doubt that the above-mentioned technical solution of the present application can definitely solve the technical problem to be solved by the present application.

[0035] At the same time, through the above technical solutions, the present invention has at least the following beneficial effects:

[0036] The low-light image enhancement method based on the Fourier transform wavelet diffusion model provided by the present invention effectively constrains the generation of redundant content by extracting the Fourier prior of the input image and embedding it into the sampling process of the wavelet diffusion model, while guiding the enhancement of details. It also realizes the fine-grained recovery of the high-frequency information of the image through the high-frequency detail enhancement module, thereby achieving a perception-guided visual enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 Schematic diagram for visual effect demonstration using the most advanced low-light image enhancement method WCDM;

[0039] Figure 2 A schematic diagram of the overall workflow of generating an enhanced diffusion model (FWDM) of the present invention;

[0040] Figure 3 It is a schematic diagram of visual comparison of the latest low-light image enhancement method of the present invention on the LOLv1 dataset;

[0041] Figure 4 Schematic diagram of visual comparison of the LIME (1st row) and DICM (2nd row) datasets of the present invention in the most advanced low-light image enhancement methods. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In order to enable those skilled in the art to better understand the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0043] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.

[0044] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0045] Reference Figure 1-Figure 4 ,Low-light image enhancement method based on Fourier transform wavelet diffusion model.

[0046] The diffusion model trains the Markov chain through variational inference. The forward process transforms complex data into completely random data by adding noise. This process can be expressed as:

[0047]

[0048] Where q is the joint probability distribution, x t and β t are the corrupted noise data and the predefined variance at time step t, respectively. N represents the Gaussian distribution and I is the identity matrix. In addition, each time step x of the forward diffusion process t It can be directly obtained by calculating the wavelet low-frequency domain x0;

[0049]

[0050] Where α t =1-β t I, α t is the attenuation factor, is the cumulative attenuation factor. The inverse inference process is to iteratively remove noise from Gaussian noise by relying on the optimized noise predictor until the randomly sampled noise Become clean data x0. Formulated as:

[0051]

[0052] Where u0 is the diffusion model noise predictor, p θ is the conditional probability distribution with model parameter θ, is the covariance matrix of the noise, representing the variance of the noise at time step t, which is mainly optimized by the editing and data synthesis functions as a way to learn the conditional denoising process, as follows:

[0053]

[0054] Among them, ∈ θ is a The function approximator of prediction∈, the meanings of the remaining letters are the same as above.

[0055] Since the low-frequency domain of wavelet decomposition contains most of the information of the image, the wavelet diffusion model relies on the framework of the conditional denoising probability model and completely transfers the diffusion process to the wavelet low-frequency domain through discrete wavelet transform (DWT):

[0056] {L L , L H = DWT(I L ) (8)

[0057] {H L , H H = DWT(I H ) (7)

[0058] Among them, I L ,I H , respectively represent the input low-light image and normal-light image, L L ,H L ,L H ,H H I L ,I H Wavelet low frequency domain and high frequency domain.

[0059] Therefore, we can use the following objective function L diff To optimize the noise predictor ∈ of the wavelet diffusion model θ :

[0060]

[0061] Where T is the total step length of the reasoning process. ∈ t is the real noise at time step t, The restored wavelet low frequency domain, the meanings of the remaining letters are the same as above.

[0062] 1. Implementation

[0063] Perception-guided enhancement is achieved by restoring image details and constraining the generation of redundant features. Specifically, Figure 2 As shown, the input image is firstly transformed by wavelet, and then the diffusion process is transferred to the wavelet low-frequency domain. The spectral prior information of the input image is extracted by Fourier transform and embedded in the sampling process to guide the generation and detail enhancement of the wavelet low-frequency domain. For the decomposed wavelet high-frequency domain, a high-frequency detail enhancement module (HDEM) is constructed through the effective combination of multiple attention and neural networks to enhance the recovery of fine-grained structures of the image.

[0064] 1.1 Fourier-guided wavelet diffusion

[0065] Most of the brightness information of an image obtained by Fourier transform is concentrated on the amplitude. In contrast, the structure and content information are closely related to the phase. At the same time, the diffusion model is committed to maintaining the consistency of data distribution rather than pixel content. Therefore, we try to use the Fourier spectrum prior to constrain and guide the wavelet diffusion model. Figure 2 As shown, firstly, the low light image I L and the normal image L H Perform discrete wavelet transform (DWT) to obtain the respective wavelet low frequency domain (L L and H L ). Then, L L and H L Perform fast Fourier transform (FFT) to obtain its spectrum. At the same time, the sampling results of each step of the sampling process Perform Fourier decomposition (FFT) to obtain the amplitude and phase. It can be expressed as:

[0066] amp L ,pha L =FFT(L L ) (9)

[0067] amp H ,pha H =FFT(H L ) (10)

[0068]

[0069] Where amp represents amplitude and pha represents phase.

[0070] Subsequently, in order to interfere with the generation of redundant content during the sampling process and preserve the original content structure of the image, we transform L L The amplitude and phase of are embedded into the sampling process and replaced by pha to constrain the content structure of the sampled image. After updating the spectrogram, the inverse fast Fourier transform (IFFT) is used to obtain the new sampling result. Defined as:

[0071]

[0072] In addition, to further guide the performance of the model, by implementing The amplitude, phase and H L The L1 distance between them is used to minimize the difference between them, thereby obtaining a more realistic lighting condition and content structure for the sampling result. It can be expressed as:

[0073]

[0074] in, are the weighted parameters of amplitude loss and phase loss.

[0075] 1.2 High-frequency detail enhancement module

[0076] The wavelet high-frequency subbands of low-light images, in general, represent sparse representations of image details in the vertical, horizontal, and diagonal directions. Therefore, in order to restore the high-frequency details of low-light images to the same high-frequency detail-rich content structure as normal-light images, a high-frequency detail enhancement module (HDEM) is proposed for reconstruction. Figure 2 As shown in the lower part, we first use depthwise separable convolution to extract L H Then the feature information of each direction is concentrated and supplemented with the normal image L by using a cross attention layer H feature details.

[0077] Subsequently, a progressive feature enhancement block combining dilated convolution and effective local attention (ELA) is designed by fusing the outputs of dilated convolution and effective local attention layers; it is used to continuously capture local information and perform significant directional enhancement by introducing a directional attention mechanism, where the first and last convolutions are used to extract local information. In contrast, the intermediate convolution improves the receptive field and makes better use of long-range information. ELA enhances the dependency between local and global features and improves the enhancement of important local features. Embedding ELA in the dilation process effectively avoids the loss of local features and enhances the association with global information. Finally, depthwise separable convolution is used again to reduce the dimensionality to obtain the reconstructed wavelet high-frequency domain. The recovered wavelet low-frequency domain is transformed into the original image using the inverse discrete wavelet transform (IDWT). and wavelet high frequency domain Transformed into the final enhanced image

[0078]

[0079] The meanings of the letters are the same as above.

[0080] 1.3 Network Training

[0081] In order to further enhance the restoration of low-light images, we use a combination of MSE loss and SSIM loss. The content loss L contnet To minimize the recovery image Normal light image with content I H Content differences between:

[0082]

[0083] Among them, ω1 is the weight of the first layer of the image encoder in the ResNet101 CLIP model, and the meanings of the remaining letters are consistent with the above.

[0084] Therefore, the total training loss can be defined as:

[0085] L total =L diff +L content +L fre (16)

[0086] 2. Experimental evaluation

[0087] 2.1 Experimental Setup

[0088] The method of the present invention was implemented using PyTorch on two NVIDIA RTX 3090 GPUs. The network was trained for a total of 1000 epochs using the Adam optimizer, with the initial learning rate set to 1x10 -4 , the batch size and patch size are set to 4 and 256x256. For the training phase, the time step T is set to 200 and the implicit sampling step is set to 10 for both the training phase and the inference phase.

[0089] Datasets and evaluation metrics. The network of the present invention is trained and evaluated on the LOLv1 dataset, which contains 500 real-world low / normal illumination image pairs, of which 485 image pairs are used for training and 15 image pairs are used for evaluation. In addition, it is tested on two unpaired datasets, LIME and DICM. For the paired datasets, the fidelity of the restoration performance is measured and the quality of the resulting image is measured by two distortion metrics (PSNR and SSIM) and two perceptual metrics (LPIPS and VSI. For the unpaired datasets, two non-reference metrics, namely the level of enhancement (LOE) and the blind / no-reference image spatial quality evaluator (BRISQUE), are used for evaluation.

[0090] In order to verify the effectiveness of the method proposed in this paper, it is compared with the most advanced methods in recent years, including zero-dce++, heurgan, RUAS, SCI, SNRNet, Uformer, Restormer, UHDFour, CLIP-LIT, LLFormer, NeRCo, GDP, WCDM, etc.

[0091] 2.2 Evaluation results

[0092] The proposed method is compared with all state-of-the-art methods on LOLv1. As shown in Table 1, the proposed method achieves state-of-the-art quantization performance on several metrics compared with all methods. In particular, the significant improvement in PSNR provides convincing evidence for the superior image quality of our method. At the same time, two unpaired data LIME 8 and DICM are evaluated and compared with competing methods, verifying the effectiveness and generalization of the proposed method.

[0093] Table 1 Quantitative evaluation of different methods on the LOLv1 dataset

[0094]

[0095] The effectiveness of the proposed method in terms of visual quality is evaluated by combining two non-reference perceptual metrics (BRISQUE and LOE), which are lower and thus produce better visual quality. As shown in Table 2, the proposed method meets the quantitative criteria on both datasets compared to other competing methods. Specifically, the best performance of all quantitative evaluations is obtained for BRISQUE. This further demonstrates the better versatility of the proposed method in real-world scenarios and the enhancement that is more consistent with human visual perception.

[0096] Table 2: Quantitative evaluation of different methods on LIME[8] and DICM

[14] datasets

[0097]

[0098] Figure 3 The visual comparison results of the proposed method and the state-of-the-art methods on the LOLv1 dataset are shown. Through these comparisons, it is easy to see that the proposed method can restore more realistic colors and more friendly visual effects without over-enhancement or under-enhancement. At the same time, all other methods have the problem of color distortion. In addition, Figure 4 The visual comparison results of the proposed method on the LIME and DICM datasets are shown. It is obvious that the proposed model cleverly adjusts the lighting conditions without causing unnecessary color enhancement to the environment. This successful balance further confirms the effectiveness of the proposed method and its friendliness to visual perception.

[0099] 5.3 Ablation study

[0100] The ablation study on the LOLvI dataset is presented to demonstrate the effectiveness of the proposed method. In order to verify the effectiveness of the proposed components, the problematic components are added to the network one by one, and the rest of the content is kept unchanged for ablation experiments.

[0101] Specifically, as shown in Table 3, #1 indicates that nothing is added. #2 indicates that a replacement phase operation is performed during the network sampling process. #3 indicates that a priori bootstrapping operation is performed by adding the Ere loss to #2. #4 indicates that the full setting with the high-frequency detail enhancement module added is used to verify its effectiveness. It is easy to see that when only one layer of the wavelet diffusion model is used without performing any operation on the network, the performance is at the lowest value. After the two Fourier prior bootstrapping methods are added to the network in turn, the network performance continues to improve, which fully confirms the positive bootstrapping effect of Fourier prior on the sampling process. In addition, for the network using the full setting

[0102] Table 3: Ablation study results of the proposed method

[0103]

[0104] The performance reached the optimal value, verifying the effectiveness of the HDEM proposed in the present invention.

[0105] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the present invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A low-light image enhancement method based on a Fourier transform wavelet diffusion model, characterized in that: Here are the steps: S1: Perform wavelet transform on the input image and transfer the diffusion process to the wavelet low-frequency domain; S2: Extract the spectral prior information of the input image through Fourier transform, embed it into the sampling process, and guide the generation and detail enhancement of the wavelet low frequency domain; S3: Reconstruct and enhance the fine-grained structure of the wavelet high-frequency domain of the low-light image through the high-frequency detail enhancement module; S4: Through the inverse discrete wavelet transform, the recovered wavelet low frequency domain x0 and high frequency domain L H Transformed into enhanced image 2. The low-light image enhancement method based on the Fourier transform wavelet diffusion model according to claim 1, characterized in that: Perform wavelet transform on the input image and transfer the diffusion process to the wavelet low-frequency domain for processing. The steps are as follows: For low light images L and the normal image L H Perform discrete wavelet transform to obtain the respective wavelet low-frequency domain L L and H L .

3. The low-light image enhancement method based on Fourier transform wavelet diffusion model according to claim 2, characterized in that: The spectral prior information of the input image is extracted through Fourier transform, embedded in the sampling process, and guided to generate the wavelet low-frequency domain and enhance the details. The steps are as follows: For L L and H L Perform fast Fourier transform to obtain its spectrum diagram; L L The amplitude and phase are embedded into the sampling process and replaced with pha to update the spectrum; Use inverse fast Fourier transform to get new sampling results 4. The low-light image enhancement method based on Fourier transform wavelet diffusion model according to claim 3, characterized in that: Sampling results for each step of the sampling process Perform Fourier decomposition to obtain the amplitude and phase, the expressions are as follows: amp L ,pha L =FFT(L L ); amp H ,pha H =FFT(H L ); Among them, amp represents amplitude and pha represents phase.

5. The low-light image enhancement method based on Fourier transform wavelet diffusion model according to claim 3, characterized in that: New sampling results The expression is as follows:

6. The low-light image enhancement method based on Fourier transform wavelet diffusion model according to claim 1, characterized in that: The high-frequency detail enhancement module reconstructs and enhances the fine-grained structure of the wavelet high-frequency domain of the low-light image. The steps are as follows: Extract L through depthwise separable convolution H Features; The feature information of three directions is concentrated and supplemented by cross attention; The local information is continuously captured by combining dilated convolution and progressive feature enhancement blocks with effective local attention; Depthwise separable convolution is used to reduce the dimensionality and thus obtain the reconstructed wavelet high frequency domain.

Images