A low-illumination superimposed fog image enhancement method based on HFRC-Diff

Abstract

The application discloses a low-illumination superimposed fog image enhancement method based on HFRC-Diff, which can enhance and restore the scene image under poor illumination and fog, and restore it to a high-quality image with normal illumination and a clean surface. The method comprises the following steps: 1) processing an existing data set; 2) training an initial prediction module IP-Net; 3) training a frequency-domain and spatial-domain guided diffusion model; 4) training a parallel attention feature fusion module; and 5) testing the entire low-illumination superimposed fog image enhancement network. The method can make up for the gap between low-illumination images and normal-illumination images at the pixel level and the semantic level, and achieve a good defogging effect, thereby generating higher-quality images under complex scenes.

Description

Technical Field

[0001] This invention relates to the fields of image enhancement, digital image processing technology, image dehazing and noise reduction diffusion probability models, specifically a low-light overlay fog image enhancement method based on the High Frequency-domain Residual Condition Diffusion model. Background Technology

[0002] Fog is a product of water vapor condensation under certain atmospheric conditions and is a common weather phenomenon. Images captured by cameras when fog forms at night or in cloud-covered scenes are called low-light fog images. These images often exhibit low brightness, low contrast, and high noise. As an upstream task in image processing, it is essential to enhance the brightness and restore details of these special images to better reproduce their original appearance under normal lighting conditions.

[0003] Currently, low-light image enhancement methods are mainly divided into traditional methods and deep learning methods. Due to the limitations of traditional methods, the enhancement effect on low-light images is often unsatisfactory. Deep learning technology, relying on continuously improving datasets and powerful computing capabilities, can better extract features and restore details. Its enhancement effect has far surpassed that of traditional methods, and it can better handle enhancement tasks in complex scenes, thus becoming the mainstream technology for low-light image enhancement. Most existing low-light image enhancement methods focus only on enhancement tasks in simple scenes, that is, enhancement tasks that do not introduce additional image degradation problems. Furthermore, most existing image dehazing algorithms are based on image restoration tasks completed under normal illumination, making it difficult to effectively handle the dehazing problem in low-light scenes with superimposed fog. Low-light images with superimposed fog visually exhibit complex characteristics such as low brightness and objects being partially or completely obscured. In the frequency domain, fog exists in the low-frequency components of the image while also affecting the high-frequency components. As the fog concentration increases and the influence of low light intensifies, the degree of image degradation is aggravated, especially in edges and details. Analysis of low-light overlay fog images in both the spatial and frequency domains reveals significant differences in performance compared to single low-light images and normal-light fog images. Therefore, simply concatenating these two existing algorithms cannot effectively complete the restoration task for this type of special image. Based on this, this invention designs an integrated network combining low-light enhancement and defogging, specifically an HFRC-Diff-based low-light overlay fog image enhancement network.

[0004] Current mainstream image enhancement methods are mainly divided into two categories: improved methods based on Retinex theory and improved methods based on Generative Adversarial Networks (GANs). Retinex-based methods decompose the image into illumination and reflection components, then process these components separately and perform corresponding feature fusion operations. The enhanced result inevitably suffers from color distortion and loss of detail. GAN-based methods are unsupervised enhancement methods. A basic GAN consists of a generator and a discriminator. Due to the instability of its training, the enhancement effect is often not ideal, manifesting as loss of detail, introduction of additional noise, and some color deviation. Currently, advanced single-image dehazing methods are mainly divided into three categories: image dehazing based on physical models and prior knowledge, image dehazing based on pixel domain mapping, and image dehazing based on domain knowledge transfer learning. Although all have made some progress in their respective technical fields, most research is only conducted under normal illumination scenes.

[0005] In recent years, diffusion models, as a novel generative model, have been applied to image generation, image enhancement, and image restoration tasks. They have shown promise in low-light image enhancement and normal-light image dehazing, with stability and experimental results even surpassing GANs, demonstrating greater research and application potential. This invention addresses the aforementioned issues and leverages new technological advancements. Specifically, after data preprocessing, the invention first uses the IP-Net initial prediction module to obtain the high-frequency and low-frequency components of the image. Then, based on the original diffusion model, enhanced high-frequency components, extracted low-frequency components, and a mixed frequency domain information prediction map are added as conditions to the MFS-Unet network. This predicts more accurate pure noise images, guiding the model to generate the desired clear image. Finally, a parallel attention feature fusion module fuses the low-frequency component image output from IP-Net and the denoised high-frequency residual image output from FSG-Diff at both the channel and pixel levels, completing the low-light enhancement and dehazing tasks and generating a high-quality, clean image. Summary of the Invention

[0006] This invention addresses the poor performance of existing low-light image enhancement and dehazing algorithms in joint low-light dehazing tasks by providing a low-light image enhancement method based on a conditional diffusion model. This method makes the model pay more attention to the high-frequency and low-frequency information of the degraded image, and can more efficiently fuse these two types of information after completing the inverse denoising process, so that the enhanced result is closer to the state of normal illumination and no fog.

[0007] Reference Figure 1 A low-light overlay fog image enhancement method based on a conditional diffusion model includes the following steps:

[0008] 1) Processing existing public datasets:

[0009] (1) The LOL-V1 dataset contains 500 low-light / normal-light image pairs. 80% of these are randomly selected as the training set, and 20% as the test set. The LOL-V2 dataset includes two subsets: LOL-v2-real and LOL-v2-synthetic. LOL-v2-real contains 689 low-light / normal-light image pairs, and LOL-v2-synthetic contains 1000 low-light / normal-light image pairs. 80% of each subset is randomly selected as the training set, and 20% as the test set. Existing natural fog synthesis algorithms are used to fuse the low-light images in the datasets with fog of random sizes.

[0010] 2) Train the IP-Net (Initial prediction Network) initial prediction module, see... Figure 2 It includes the following steps:

[0011] (1) The training dataset in 1) is preprocessed. The width and height of all low-light superimposed fog images are uniformly scaled to 256*256 pixels. Then, the image brightness and contrast are improved by histogram equalization to obtain the preliminarily enhanced low-light superimposed fog images.

[0012] (2) The preliminarily enhanced low-light overlay fog image obtained in (1) is sent to the hybrid frequency-domain information extractor module, see... Figure 3 In addition, two extra loss functions are introduced: the Fast Fourier Transform loss and the Two-Dimensional Discrete Wavelet Transform loss, see [link to relevant documentation]. Figure 4 While obtaining the main low-frequency information, the high-frequency information is further recovered to obtain a prediction map that combines low-frequency and high-frequency information.

[0013] (3) The mixed frequency domain information prediction map obtained in (2) is sent to the frequency-domain information splitter module, see Figure 2 That is, by using the inverse fast Fourier transform, the mixed frequency domain information prediction map is input into an adaptive Gaussian high-pass filter and an adaptive Gaussian low-pass filter respectively, to obtain two initial prediction maps containing only high-frequency information and only low-frequency information.

[0014] 3) Training a conditional diffusion model guided by the spatial and frequency domains includes the following steps:

[0015] (1) Perform a forward Gaussian noise addition process on the normal illumination images corresponding to the preprocessed data in (2)-(1).

[0016] (2) Construct a channel attention module guided by high-frequency information (HF-guided CA), see Figure 7 In each downsampling layer and corresponding upsampling layer of Unet, HF-guided CA is embedded. That is, the output of the downsampling layer of Unet is used as the input of the channel attention module. The attention mechanism is applied to all channels of the input feature map for weighted operation, which can flexibly capture the dependencies between channels to capture information that has a prominent effect on color, thereby aligning semantic information.

[0017] (3) During the reverse denoising process, the MFS-Unet (Mix Frequency-domain and Spatial-domain Condition for Unet) network is used to estimate the noise. See Figure 6 The Unet network, which introduces a channel attention module guided by high-frequency information, takes the mixed frequency domain information prediction map and the three images of high-frequency and low-frequency components obtained in 2)-(2) as one input to the MFS-Unet network, and the other input is the noise image obtained at time step t. The added channel attention module guided by high-frequency details can improve the MFS-Unet network's ability to extract high-frequency features, help recover finer high-frequency details, and thus achieve the purpose of more accurate noise estimation.

[0018] (4) Subtract the pure noise image obtained in step (3) from the image with added Gaussian noise obtained at the current time step t to obtain a high-frequency residual image of the denoising process. In this way, the final denoised high-frequency residual image is obtained after the preset time step is completed.

[0019] 4) Train the parallel attention feature fusion module, including the following steps:

[0020] (1) First, perform channel-by-channel concatenation and element-by-element addition operations on the low-frequency component feature map output in 2)-(3) and the denoised high-frequency residual feature map output in 3)-(4), respectively. These two branches will each obtain an initial feature fusion map.

[0021] (2) Apply the channel attention mechanism to the output results of the channel-by-channel concatenation operation in (4)-(1), and apply the pixel attention mechanism to the output results of the element-by-element addition operation. The two operations are performed in parallel.

[0022] (3) Perform element-wise addition on the two attention maps obtained in (4)-(2) to complete the final feature fusion and output the enhanced result of the low-light superimposed fog image.

[0023] 5) Testing the entire low-light overlay fog image enhancement network includes the following steps:

[0024] 1) Low-light overlaid fog images from the LOL-V1, LOL-V2, and SID test sets are first image equalized to improve brightness and contrast, and then fed into the IP-Net initial prediction module with pre-trained weights to obtain a preliminary prediction map that mixes high-frequency and low-frequency information, as well as the decomposed high-frequency and low-frequency components. See [link to documentation]. Figure 2 .

[0025] (2) The three images obtained in (1) are stitched together channel by channel to form a conditional image. At the same time, random Gaussian noise is generated and fed into the trained conditional diffusion model for denoising and enhancement.

[0026] (3) The enhanced result and the low-frequency component image obtained in (1) are fed into the parallel attention feature fusion module.

[0027] (4) Perform PSNR and SSIM image quality assessment on the feature fusion results obtained in (3).

[0028] This technical solution addresses the challenge of image enhancement in low-light, foggy environments. Low-light conditions introduce degradation, leading to blurred images and significantly increased noise, rendering existing low-light image enhancement methods ineffective. First, this technique synthesizes natural fog in existing low-light images to create training and testing datasets. Second, the processed training dataset is fed into the IP-Net initial prediction module to obtain RGB prediction maps containing both high-frequency and low-frequency components. These prediction maps are then decomposed into two initial prediction maps: one containing only high-frequency information and the other only low-frequency information. Third, a spatial domain prediction map and two frequency domain prediction maps are concatenated along the channels and fed into a diffusion model combining spatial and frequency domains to obtain a denoised and enhanced image. Finally, the denoised and enhanced image is first fused with the low-frequency component image obtained from the IP-Net initial prediction module, followed by pixel attention and channel attention mechanisms to complete the final feature fusion, yielding the final enhanced result. Attached Figure Description

[0029] Figure 1 HFRC-Diff network structure diagram

[0030] Figure 2 IP-Net architecture diagram

[0031] Figure 3Structure diagram of the hybrid frequency domain information extraction module

[0032] Figure 4 : Three loss function description structure diagrams

[0033] Figure 5 FSGDiff structure diagram

[0034] Figure 6 MFS-Unet architecture diagram

[0035] Figure 7 HF-guided CA structure diagram

[0036] Figure 8 Pixel Attention Module Structure Diagram Detailed Implementation

[0037] Reference Figure 1 A low-light image enhancement method based on HFRC-Diff with overlay fog includes the following steps:

[0038] Step 1: Process the existing dataset

[0039] 1-1 The LOL-V1 dataset contains 500 low-light / normal-light image pairs. 80% of these are randomly selected as the training set and 20% as the test set. The LOL-V2 dataset consists of two subsets: LOL-v2-real and LOL-v2-synthetic. LOL-v2-real contains 689 low-light / normal-light image pairs, and LOL-v2-synthetic contains 1000 low-light / normal-light image pairs. 80% of each subset is randomly selected as the training set and 20% as the test set.

[0040] The 1-2SID dataset contains 5094 low-light images and their corresponding normal-light images. 70% were randomly selected as the training set and 30% as the test set.

[0041] 1-3 utilizes existing natural fog synthesis algorithms to fuse low-light images from the three datasets above with fog of random sizes. IP-Net is trained using a portion of the training set of low-light images overlaid with fog and clean images under normal lighting from the three datasets, generating two initial prediction images (high-frequency and low-frequency). These initial prediction images are then used to train the MFS-Unet network to estimate noise. A test set of low-light images overlaid with fog images is used to test the trained diffusion model.

[0042] Table 1. Source of the dataset

[0043]

[0044] Step 2: Train the IP-Net initial prediction module

[0045] Since simple convolutional neural networks can recover the main low-frequency content of an image, IP-Net was designed to recover the main low-frequency content while also predicting some high-frequency information. This allows the subsequent conditional diffusion model to use the high-frequency and low-frequency content predicted by IP-Net to guide noise into the residual between the high-frequency content of the normally lit clean image and the predicted image. This not only speeds up the model convergence time but also achieves better sample quality.

[0046] 2-1 Preprocessing was performed on the training datasets from the LOL-V1, LOL-V2, and SID datasets, uniformly scaling the width and height of all low-light overlaid fog images to 256*256 pixels. Then, histogram equalization was used to enhance the image brightness and contrast, resulting in preliminarily enhanced low-light images.

[0047] 2-2 Reference Figure 3 The preliminarily enhanced low-light image obtained in step 2-1 is fed into the frequency domain information extraction module of IP-Net for frequency domain information extraction. The input image first passes through a convolutional layer with a kernel size of 3*3 and a stride of 1, followed by a ReLU activation function layer; then it passes through another convolutional layer with a kernel size of 3*3 and a stride of 1, followed by another ReLU activation function layer; then, the feature map output from the first convolutional layer is connected to the feature map output from the second convolutional layer by skip connection, that is, channel-wise and pixel-wise addition is performed. This step integrates the extracted low-frequency information. The output of the skip connection layer serves as the input to the third convolutional layer, which also has a 3x3 kernel and a stride of 1. A ReLU activation layer is then added to integrate high- and low-frequency information. The fourth layer is a 1x1 convolutional layer with a stride of 1, used to adjust the feature map channel dimensions. Another ReLU activation layer is added. The third convolutional layer and the last ReLU layer then undergo a skip connection operation. Finally, a sigmoid activation layer is used to obtain the weights for each channel of the feature map, generating the initial prediction map. This prediction map recovers the main low-frequency information of the image and predicts some high-frequency information. (Refer to...) Figure 4 In addition to the Ground Truth loss, two extra loss functions are introduced: Fast Fourier Transform loss and Two-Dimensional Discrete Wavelet Transform loss. These are shown in Equations (1), (2), and (3) respectively:

[0048]

[0049] For L FFT , I and These represent the frequency domain images obtained by performing FFT on the ground-truth image and the predicted image, respectively. For L... DWT Wavelet coefficients of the high-frequency bands H, V, and D in the image frequency domain are extracted, representing the high-frequency components in the horizontal, vertical, and diagonal directions, respectively. H... i V i D i It is a sub-band of the real image in the i-th downsampling. Let L be the subband of the predicted image in the i-th downsampling, and L be the total downsampling level. The spatial domain loss is L. GT In the middle, Y and Let represent the real image and the predicted image, respectively. Finally, the three losses are summed to obtain the overall loss of the hybrid frequency domain information extraction module, as shown in the following formula (4):

[0050] L Total =L FFT +L DWT +L GT (4)

[0051] 2-3 The obtained mixed frequency domain information prediction map is sent to the frequency domain information separation module. First, for the RGB prediction image x with high-frequency and low-frequency information... mfi ∈R H×W×C First, we perform a 2DFFT transform along the spatial dimension to obtain the frequency domain feature map M, as shown in the following formula (5):

[0052] M = FFT(x) mfi )∈R H×W×C (5)

[0053] Next, the standard deviation is calculated using the formula shown in formula (6) below:

[0054]

[0055] Calculate the standard deviation of feature map M, where σ is the standard deviation, N is the total number of pixels, and x is the standard deviation. i Let be the value of the i-th pixel, and μ be the average value of the pixel values. Using the calculated standard deviation, an adaptive Gaussian high-pass filter and an adaptive Gaussian low-pass filter can be constructed, as shown in formulas (7) and (8), respectively:

[0056]

[0057] Where D(u,v) is the distance from the point (u,v) in the frequency domain to the center point. Then, the outputs of the Gaussian high-pass filter and the Gaussian low-pass filter are multiplied element-wise with M to obtain the adaptive high-pass filter feature map M. * h and adaptive low-pass filter feature map M* l As shown in formulas (9) and (10) respectively:

[0058]

[0059]

[0060] Finally, the inverse FFT transform is used to transform M * h and M * l Reversing to the spatial domain, we obtain feature maps x containing only high-frequency information. HF And prediction graph x containing only low-frequency information LF As shown in formulas (11) and (12) respectively:

[0061] x HF =FFT -1 (M * h )∈R H×W×C (11)

[0062] x LF =FFT -1 (M * l )∈R H×W×C (12)

[0063] To introduce spatial and frequency domain information into the conditional diffusion model, x mfi x HF and x LF By stitching the images together along the channel dimension, we obtain the final conditional image X. c As shown in the following formula (13):

[0064] X c =concat(x mfi ,x HF ,x LF (13)

[0065] Step 3: Train the conditional diffusion model guided by the spatial and frequency domains.

[0066] The original diffusion model, DDPM (Denoising Diffusion Probabilistic Models), is based on the Markov chain principle and includes two processes: forward diffusion for noise addition and backward inference for denoising. The forward diffusion for noise addition process of the conditional diffusion model is completely identical to DDPM. The difference lies in that its backward inference for denoising process requires additional input information as a condition to guide the generation direction of the diffusion model. This method utilizes the output of the IP-Net initial prediction module as a control condition to accelerate model convergence and generate higher-quality images.

[0067] 3-1 Reference Figure 5 A forward diffusion noise addition process is performed on normally lit images from the LOL-V1, LOL-V2, and SID datasets. Specifically, for the given real image samples x to q(x0), Gaussian noise is added to them 1000 times cumulatively to obtain X1, X2, ..., X... T Using a predefined Gaussian distribution with variance hyperparameter {β} t The noise level is controlled by ∈(0.0001,0.02)}, where t∈(1,T), i.e., T=1000. The noise addition of the model is actually a weighted fusion of the input image of the previous stage with a Gaussian distribution sample. The weight increases with the time step. It can be seen that the noise amplitude is relatively small at the beginning and gradually increases as time goes on. Since the noise distribution obtained at each time step t in the forward process is only related to time step t-1, it can also be equivalent to a Markov chain process, as shown in the following formulas (14) and (15):

[0068]

[0069] It can be seen that as t increases, X t As T approaches infinity, X becomes increasingly close to pure noise. T It is a completely noisy image. After 1000 time steps, the input normal lighting image X0 has become a completely noisy image X. T This completes the forward diffusion noise addition process of the diffusion model.

[0070] 3-2 Reference Figure 5 The Gaussian noise X obtained in 3-1 T ~N(0,I) serves as the input information for the inverse denoising process, referring to... Figure 6 Next, a noise predictor, namely the MFS-Unet network, is constructed to predict the noise added at each time step of the forward noise addition process, while using X... c As a condition for the reverse denoising process, X is then... c and X TThe data is concatenated channel by channel and fed into MFS-Unet. Through T iterations, the random noise X is gradually reduced... T Restore to enhanced result X * 0. Each denoising step in the reverse process is shown in the following formula (16):

[0071]

[0072] in The image X represents the defined condition. c X t This represents the Gaussian noise obtained from sampling, with invariant variance. Learning mean Here α t =1-β t , It can be seen As the only unknown variable, the MFS-Unet denoising network is needed to predict it.

[0073] 3-3 In the original U-net architecture, encoder features are directly concatenated with features obtained by the decoder. This fusion is beneficial for the network to effectively fuse high-level and low-level features, but it lacks the ability to extract high-frequency features. To address this issue, a channel attention module guided by high-frequency details (HF-guided CA) is designed to help recover finer high-frequency details. (See reference...) Figure 7 HF-guided CA is an improvement on the classic channel attention network SENet (Squeeze-and-Excitation Networks). Specifically, for the input H×W×C feature map, it first passes through a GAP (Global Average Pooling) layer, which sums and averages all pixel values ​​to obtain a 1×1×C feature map. Then, it passes through a Fully Connected (FC) layer for compression, resulting in a 1×1×C / r feature map, where r = 16 represents the compression ratio. Next, it passes through a ReLU layer for non-linear activation, followed by a convolutional layer with a 1*1 kernel and a stride of 1 instead of the original FC layer. This adjusts the channel dimensions while capturing the dependencies between channels, effectively reducing color deviation caused by the U-net network's downsampling operation. This results in a 1×1×C feature map, which then passes through another ReLU layer for non-linear activation. Finally, a sigmoid layer normalizes the values ​​of each layer to between 0 and 1, representing the weight of each channel. The feature fusion operation multiplies the original feature map by the output 1×1×C to obtain the final feature map, whose dimensions are consistent with the original input feature map, also H×W×C.

[0074] 3-4 Reference Figure 6 MFS-Unet is used to predict random noise, i.e., to predict... The value of . One of the inputs is the noise X obtained at time step T. T One input is X, and the other input is X. c As a condition. By X c and X T The data is concatenated channel by channel and then fed into MFS-Unet. The objective function of this process is shown in formula (17):

[0075]

[0076] The goal of MFS-Unet optimization is to fit the noise residual, that is, the residual image with high-frequency information predicted by IP-Net from a normally lit, clean image. This requires minimizing the "distance" between the noise distribution predicted in the backward process and the noise distribution applied in the forward process. (Refer to...) Figure 5 Specifically, for the current time step t, we take the random Gaussian noise X obtained from the forward noise addition process... t With X c The noise is fed into the MFS-Unet network to obtain the predicted noise ∈ θ Our goal is to make ∈ θ and ∈ t The difference is minimized, and X is simultaneously... t and ∈ θ Perform the subtraction operation to obtain the denoised restored state X corresponding to the current time step t. * t And so on, we define the restored state obtained through the MFS-Unet network at each step as X. * T X * T-1 , ..., X * t X * 0. Each step attempts to gradually reduce noise and restore an earlier state of the original image until it is restored to a state close to the initial sharpness X0. * 0 represents the final denoised high-frequency residual image.

[0077] Step 4: Train the parallel attention feature fusion module

[0078] From the above steps, it can be seen that the low-frequency component characteristic map M * l The high-frequency residual feature map X, already generated by the IP-Net network and denoised, is shown below. *The value 0 has already been generated by FSGDiff. Next, feature fusion is performed. Since fusing two feature maps using only addition or concatenation operations yields poor results, a parallel attention feature fusion module was designed to further fuse features through pixel and channel collaboration. These two attention mechanisms execute in parallel, allowing for flexible handling of different types of information: low-light and foggy pixel information, and, more importantly, channel information. This provides additional compensation during low-light enhancement and dehazing processes, improving the final image quality.

[0079] 4-1 Reference Figure 1 First, the low-frequency component feature map M output by IP-Net is processed. * l Denoising high-frequency residual feature map X of FSGDiff output * 0 performs channel-by-channel stitching and element-by-element addition operations respectively, resulting in two initial feature fusion maps F1 and F2.

[0080] 4-2 Reference Figure 8 Applying a pixel attention mechanism, the feature map F1 with shape H×W×C obtained in step 4-1 is input into a convolutional layer with a kernel size of 1*1 and a stride of 1, resulting in a feature map of H×W×C / 2. Then, it passes through a ReLU activation function layer, keeping the feature map shape unchanged. Next, it passes through another convolutional layer with a kernel size of 1*1 and a stride of 1, resulting in a feature map of H×W×1. Finally, it passes through a sigmoid layer, where all pixels on the feature map are weighted and summed to obtain a feature vector of H×W×1. Finally, F1 is multiplied element-wise with the output of the sigmoid layer to obtain F°1, which has the same dimension as F1, also H×W×C.

[0081] 4-3 Reference Figure 7 The channel attention mechanism is applied by reusing the HF-guided CA module. The feature map F2 with shape H×W×C obtained in 4-1 is used as the input of the channel attention module, and the output is the attention map F°2 with shape H×W×C.

[0082] 4-4 Perform element-wise addition on the two attention maps F°1 and F°2 to complete the final feature fusion and output the enhanced result of the low-light fog image.

[0083] Step 5: Testing the entire low-light overlay fog image enhancement network

[0084] (1) The low-light superimposed fog images in the LOL-V1, LOL-V2 and SID test sets are first image equalized to improve brightness and contrast, and then sent to the IP-Net initial prediction module with pre-trained weights to obtain the prediction map of the initial mixture of high-frequency and low-frequency information and the decomposed high-frequency and low-frequency components.

[0085] (2) The three images obtained in (1) are stitched together channel by channel to form a conditional image. At the same time, random Gaussian noise is generated and fed into the trained conditional diffusion model for denoising and enhancement.

[0086] (3) The enhanced result and the low-frequency component image obtained in (1) are fed into the parallel attention feature fusion module.

[0087] (4) Perform PSNR and SSIM image quality assessment on the feature fusion results obtained in (3).

Claims

1. A low-light overlay fog image enhancement method based on HFRC-Diff. Its features include the following method steps: 1) Processing existing public datasets: (1) The LOL-V1 dataset contains 500 low-light / normal-light image pairs. 80% of them are randomly selected as the training set and 20% as the test set. The LOL-V2 dataset includes two subsets: LOL-v2-real and LOL-v2-synthetic. LOL-v2-real contains 689 low-light / normal-light image pairs, and LOL-v2-synthetic contains 1000 low-light / normal-light image pairs. 80% of each subset is randomly selected as the training set and 20% as the test set. The low-light images in the dataset are all fused with fog of random size using existing natural fog synthesis algorithms. (2) The SID dataset contains 5094 low-light images and their corresponding normal-light images; 70% are randomly selected as the training set and 30% as the test set; existing natural fog synthesis algorithms are used to fuse the low-light images in the dataset with fog of random size range. 2) Train the initial prediction module of IP-Net, including the following steps: (1) Preprocess the training dataset in 1) to uniformly scale the width and height of all low-light superimposed fog images to 256*256 pixels, and then improve the image brightness and contrast through histogram equalization to obtain the preliminarily enhanced low-light superimposed fog images. (2) The preliminarily enhanced low-light overlay fog image obtained in (1) is sent to the hybrid frequency domain information extraction module. The input image first passes through a convolutional layer with a kernel size of 3*3 and a stride of 1, followed by a ReLU activation function layer; then it passes through another convolutional layer with a kernel size of 3*3 and a stride of 1, followed by another ReLU activation function layer; then the feature map output from the first convolutional layer is connected to the feature map output from the second convolutional layer by skip connection, that is, the corresponding channel-by-channel and pixel-by-pixel addition is performed. This step integrates the extracted low-frequency information; the output of the skip connection layer is used as the input of the third convolutional layer, which also has a kernel size of 3*3 and a stride of 1. Subsequently, a ReLU activation function layer is added to integrate high- and low-frequency information. The fourth layer is a convolutional layer with a kernel size of 1*1 and a stride of 1, used to adjust the channel dimension of the feature map. Another ReLU activation function layer is added, and the third convolutional layer and the last ReLU layer perform a skip connection operation. Finally, a sigmoid activation function layer is used to obtain the weights corresponding to each channel of the feature map, thereby generating the initial prediction map. At the same time, two additional loss functions are introduced, namely Fast Fourier Transform loss and 2D Discrete Wavelet Transform loss, to further recover high-frequency information while obtaining the main low-frequency information, thus obtaining a prediction map that mixes low-frequency and high-frequency information. (3) The mixed frequency domain information prediction map obtained in (2) is sent into the frequency domain information separation module. That is, the mixed frequency domain information prediction map is input into the adaptive Gaussian high-pass filter and the adaptive Gaussian low-pass filter respectively using the inverse fast Fourier transform to obtain two initial prediction maps containing only high-frequency information and only low-frequency information. 3) Train the conditional diffusion model guided by the spatial and frequency domains, including the following steps: (1) Perform a forward Gaussian noise addition process on the normal illumination images corresponding to the preprocessed data in (2)-(1); (2) Construct a channel attention module guided by high-frequency information. The input H×W×C feature map first passes through a GAP layer, which sums all pixel values ​​of the feature map and averages them to obtain a feature map with a shape of 1×1×C. Then, it passes through an FC layer for compression to obtain a feature map with a shape of 1×1×C / r, where r=16 represents the compression ratio. Next, it passes through a ReLU layer for non-linear activation, and then through a convolutional layer with a kernel size of 1*1 and a stride of 1 instead of the original FC layer. The purpose is to adjust the channel dimension while capturing the dependencies between channels, thereby effectively reducing the color deviation problem caused by the downsampling operation of the U-net network on the feature map. After that, a 1×1×C shape is obtained, and it passes through another ReLU layer for non-linear activation. Finally, it passes through a sigmoid layer to normalize the values ​​of each layer to between 0 and 1, thereby representing the weight of each channel. The feature fusion operation multiplies the original feature map by the output 1×1×C to obtain the final feature map, whose dimensions are consistent with the original input feature map, which are also H×W×C. In each downsampling layer and corresponding upsampling layer of Unet, HF-guided CA is embedded. That is, the output of the downsampling layer of Unet is used as the input of the channel attention module. The attention mechanism is applied to all channels of the input feature map for weighted operation, which can flexibly capture the dependencies between channels to capture information that has a prominent effect on color, thereby aligning semantic information. (3) In the reverse denoising process, the MFS-Unet network is used to estimate the noise. That is, the Unet network with the channel attention module guided by high frequency information is introduced. The three images of the mixed frequency domain information prediction map and the separated high frequency component and low frequency component in 2)-(2) are used as one input of the MFS-Unet network. The other input is the noise image obtained at time step t. The added channel attention module guided by high frequency information can enable the MFS-Unet network to improve its ability to extract high frequency features and help to recover more refined high frequency details, thereby achieving the purpose of more accurate noise estimation. (4) Subtract the pure noise image obtained in step (3) from the image with added Gaussian noise obtained at the current time step t to obtain a high-frequency residual image of the denoising process; and so on, to obtain the final denoised high-frequency residual image after the preset time step is completed. 4) Train the parallel attention feature fusion module, including the following steps: (1) First, perform channel-by-channel concatenation and element-by-element addition operations on the low-frequency component feature map output in 2)-(3) and the denoised high-frequency residual feature map output in 3)-(4), respectively. These two branches will each obtain an initial feature fusion map. (2) Apply the channel attention mechanism to the output results of the channel-by-channel concatenation operation in (4)-(1), and apply the pixel attention mechanism to the output results of the element-by-element addition operation. The two operations are performed in parallel. (3) Perform element-wise addition on the two attention maps obtained in (4)-(2) to complete the final feature fusion and output the enhanced result of the low-light superimposed fog image; 5) Testing the entire low-light overlay fog image enhancement network includes the following steps: (1) The low-light superimposed fog images in the LOL-V1, LOL-V2 and SID test sets are first image equalized to improve brightness and contrast, and then sent to the IP-Net initial prediction module with pre-trained weights to obtain the prediction map of the initial mixture of high-frequency and low-frequency information and the decomposed high-frequency and low-frequency components. (2) The three images obtained in (1) are stitched together channel by channel to form a conditional image. At the same time, random Gaussian noise is generated and fed into the trained conditional diffusion model for denoising and enhancement. (3) The enhanced result and the low-frequency component image obtained in (1) are fed into the parallel attention feature fusion module; (4) Perform PSNR and SSIM image quality assessment on the feature fusion results obtained in (3).

Images