An underwater image enhancement method based on a denoising diffusion probability model

Abstract

This invention discloses an underwater image enhancement method based on a denoising diffusion probability model. In underwater image enhancement tasks, this method effectively improves the quality of underwater images. This invention proposes an improved denoising diffusion probability model, trained on paired datasets. It utilizes two standard U-Net networks to construct a denoising network for image denoising and a transformation network for image distribution transformation, successfully completing image denoising and data distribution transformation functions. To improve the quality of enhanced images, this invention proposes an operation for image distribution normalization during the enhancement process, achieving correct fusion of the outputs of the denoising network and the distribution transformation network. Finally, through experimental comparison, in underwater image and low-light image enhancement tasks, this method achieves superior processing results compared to other existing methods, obtaining better visual effects and evaluation metrics.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an underwater image enhancement method based on a denoising diffusion probability model. Background Technology

[0002] In recent years, countries worldwide have placed increasing emphasis on exploring marine information. However, due to the difficulties of manual exploration, nations have vigorously developed underwater robots, with Autonomous Underwater Vehicles (AUVs) and Unmanned Underwater Vehicles (UUVs) becoming the mainstay. Imagery, as the most intuitive and efficient way for underwater intelligent agents to acquire information, plays an irreplaceable role in the exploration process. However, due to the complex underwater environment and low visibility, even with expensive image acquisition equipment, the images acquired by these vehicles are severely degraded, exhibiting serious color distortion, blurriness, and low contrast between the target and background. Blurred images not only affect the observation effect on the surface operator but also impact the underwater robot's application of the images. Therefore, research on underwater image enhancement methods is beneficial for underwater robots to complete their tasks. However, as water depth increases, the scattering and absorption of natural light in the water become increasingly severe. Images acquired in deeper waters often appear blue, green, or yellowish-green, significantly affecting image quality.

[0003] In their paper "Denoising diffusion probabilistic models," Ho et al. proposed a denoising diffusion probabilistic model, which consists of two parts: a diffusion process and a denoising process. The diffusion process progressively adds Gaussian noise to real images in the dataset, while the denoising process uses a network to progressively remove noise from the added images, thus restoring the true image. While this method produces clear images, directly applying it to image enhancement tasks can lead to insufficient or even ineffective enhancement.

[0004] In their paper "Underwater Image Enhancement Algorithm Combining Guided Filtering and Wavelet Transform," He Xiao et al. divided the image into a basal part and a detail part, and applied different processing methods to each part. After weighted fusion and contrast histogram-constrained equalization, they obtained an excellent underwater enhanced image, overcoming the problem of uneven illumination. However, this method processes individual pixels of the image and cannot effectively link the global information of the image, thus often resulting in over-enhancement or under-enhancement.

[0005] In their paper "Sea-thru: A Method For Removing Water From Underwater Images," Derya et al. proposed a physical model for underwater image enhancement that uses dark pixels and their known distance information to estimate backscattering. The enhanced image's colors are then recovered using an RGBD image containing depth information. However, this method often exhibits poor generalization ability and robustness in complex underwater environments.

[0006] The paper "Fast Underwater Image Enhancement for Improved Visual Perception" by Islam et al. mentions that by simultaneously considering both global and local information to form the loss function, it outperforms other models on the underwater image dataset EUVP and requires less training time. However, due to the unstable training effect, it can lead to inconsistent color in the generated images when processing some severely degraded images.

[0007] In their paper "MLFcGAN: Multilevel Feature Fusion-Based Conditional GAN ​​for Underwater Image Color Correction," Liu et al. proposed a method for color correction of underwater images. This method uses global features to enhance local features at each scale to achieve color correction and preserve image details. However, this method shows poor enhancement results when processing severely degraded images.

[0008] In their paper "Underwater Image Enhancement Using a Multiscale Dense Generative Adversarial Network," Guo et al. proposed a method that simultaneously uses unsupervised generative adversarial networks (GANs) and end-to-end autoencoder networks for color restoration and dehazing, achieving good performance. However, this method suffers from severe feature loss and color distortion when processing low-light and foggy images. Therefore, designing a model with good generalization and adaptability is crucial to the design of such a method. Summary of the Invention

[0009] The purpose of this invention is to provide an underwater image enhancement method based on a denoising diffusion probability model.

[0010] The objective of this invention is achieved through the following technical solution:

[0011] An underwater image enhancement method based on a denoising diffusion probability model, characterized by the following steps:

[0012] Step 1: Determine the training and test sets required for the network;

[0013] Step 2: Gradually add noise to the degraded image and the reference image;

[0014] Step 3: Improve the network structure of the denoising diffusion probability model into a dual U-Net network, input different parameters to the two networks, and complete the two functions of denoising and data distribution transformation;

[0015] Step 4: Complete the training of the denoising network and the distribution transformation network;

[0016] Step 5: Complete the model's inference process. The data sampling method during the process is improved to a distributed fusion sampling method that is more suitable for the dual U-Net network, thereby enhancing a single underwater degraded image.

[0017] Furthermore, the training set in step one uses paired portions of the SUID and UIEB datasets. The SUID dataset simulates 30 land images as degraded images with an underwater image style, while the UIEB dataset contains 890 pairs of underwater images and 60 unpaired real underwater images. The test set uses 60 unpaired real underwater images from UIEB and the U45 dataset, which includes green, blue degraded, and fogged underwater images.

[0018] Furthermore, step two provides input data. Where x i Given the input degraded image, y i For x i The corresponding reference image, defined x t and y t Let x and y be the images after t rounds of noise addition, respectively. The formula for the noise addition process is:

[0019]

[0020]

[0021] Where t∈(1,T), δ is random noise at time t that follows a standard Gaussian distribution. It is a satisfying α t The hyperparameters ∈ (0,1) determine the variance of the noise added in each iteration; at the same time, to ensure that it is bounded as t→∞, α t It gradually decreases as time increases; the same Gaussian noise is applied to x0 and y0.

[0022] Furthermore, step three uses a dual U-Net network structure to complete both denoising and data distribution transformation processes; the network consists of a denoising network f θ (x t (t,x0) and distributed transformation network f φ (x t The image is composed of x0, x0, and xt, where t is time t, x0 is the initial degraded image, and x0 is the initial degraded image. t The image of x0 after t denoising is shown. The denoising network is used to predict the random noise sampled at time t during the diffusion process, and the distribution transformation network is used to fit the two data distributions at time t.

[0023] Both networks employ the standard U-Net topology in the denoising diffusion probability model; to ensure the enhanced image content matches the input image, x0 is fed into the θ network as a prior condition; the θ network inputs x... t At time t and x0, the output noise δ θ φ network input noisy image x t And at time t, output the noise image y of the reference image y0 at time t. t .

[0024] Furthermore, in step four, the loss function of the denoising network can be expressed as:

[0025]

[0026] Where δ is random noise at time t that follows a standard Gaussian distribution;

[0027] The loss function of a distributed transformation network can be expressed as:

[0028]

[0029] Where y t This is the diffusion result of the reference image y0 at time t.

[0030] Furthermore, in step five, during the first iteration of the reasoning process, x... t The image is fed into a denoising network, and the denoised image x at time t-1 is obtained according to the parameter renormalization rule. t-1 Subsequently, a distributed transformation network is used to convert it into a noisy image y of the reference image y0 at time t-1. t-1 ,Right now:

[0031]

[0032] y t-1 =f φ (x t-1 )

[0033] During the second to the (T - 1)-th iteration of the inference process, the same denoising network is used for the denoising process of the degraded image and the reference image. At the same time, to improve the network enhancement effect, the distribution conversion network is used to convert x t to x ′ t and add it to y ′ t at the same time to obtain the total distribution Y at that moment, that is:

[0034] Y = y ′ t + f φ (x t , t) ~ N(2μ t , 2σ t 2 )

[0035] where 1 < t < T, y ′ t and f φ (x t , t) both follow the normal distribution N(μ t as the mean and σ t 2 as the variance N(μ t , σ t 2 ); to avoid distortion of the enhanced image, after the feature maps are superimposed, the distribution Y ~ N(2μ t , 2σ t 2 ) is converted to Y ~ N(μ t , σ t 2 ), and its operation is:

[0036]

[0037]

[0038] That is, when 1 < t < T:

[0039]

[0040] During the last iteration of the inference process, only the denoising network is used for y1, and the final enhanced image is obtained according to the following formula, Y = y ′ t follows the normal distribution N(μ t , σ t 2 ), substituting into the above formula, the formula for the data distribution of the reference image y0 is finally derived as:

[0041]

[0042] The beneficial effects of this invention are as follows:

[0043] To address the degradation of underwater images, this invention proposes an underwater image enhancement method based on a denoising diffusion probability model. Building upon this model, the network structure is improved, and a dual U-Net network is proposed for image-to-image tasks, enabling the conversion between the two image data distributions.

[0044] We propose an improved data sampling method for the model inference process, which is more suitable for the distributed fusion sampling method of dual U-Net networks.

[0045] This method is applicable to underwater image enhancement of various styles and has better generalization ability compared to traditional image enhancement methods, effectively solving the problem of insufficient model generalization ability caused by differences in water body and depth. Experiments show that this method can achieve excellent evaluation index values ​​and visual effects. Attached Figure Description

[0046] Figure 1 This is a flowchart of an underwater image enhancement method based on a denoising diffusion probability model, according to a specific embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the method;

[0048] Figure 3 This is a schematic diagram of the reasoning process of this method;

[0049] Figure 4 This is a comparison chart of the enhancement effects of different underwater image enhancement methods on the U45 test set;

[0050] Figure 5 This is a comparison chart of the enhancement effects of different underwater image enhancement methods on the Challenge60 test set;

[0051] Figure 6 This is an image showing the edge extraction results of the input and enhanced images;

[0052] Figure 7 It is a SIFT result image of the input image and the enhanced image. Detailed Implementation

[0053] The present invention will now be further described with reference to the accompanying drawings.

[0054] Combination Figure 1 The present invention provides an underwater image enhancement method based on a denoising diffusion probability model, the process of which includes:

[0055] Step 1: Determine the training and test sets required for the network;

[0056] Step 2: Gradually add noise to the degraded image and the reference image;

[0057] Step 3: Improve the network structure of the denoising diffusion probability model into a dual U-Net network, input different parameters to the two networks, and complete the two functions of denoising and data distribution transformation;

[0058] Step 4: Complete the training of the denoising network and the distribution transformation network;

[0059] Step 5: Complete the inference process of the model. The data sampling method in the process is improved to a distribution fusion sampling method that is more suitable for the dual U-Net network. The trained network is used to enhance a single underwater degraded image.

[0060] Step Six: Conduct underwater image enhancement experiments and compare them with other enhancement methods;

[0061] Step 7: Engineering Applicability Analysis.

[0062] The specific steps for using step one are as follows:

[0063] In this invention, the training set for the network uses paired portions of the SUID and UIEB datasets. The SUID dataset simulates 30 land images as degraded images with an underwater image style, while the UIEB dataset contains 890 pairs of underwater images and 60 unpaired real underwater images. The test set for the network in this invention uses 60 unpaired real underwater images (challenge60) from UIEB and the U45 dataset. The U45 dataset includes green, blue degraded, and foggy underwater images and has been widely used in testing underwater image enhancement models in recent years.

[0064] Step two is as follows:

[0065] The diffusion process is a continuous process of adding known noise to the input image, and the noisy image at each stage can be directly calculated from known information. This invention provides the input data... Where x i Given the input degraded image, y i For x i The corresponding reference image, defined x t and y t Let x and y be the images after t rounds of noise addition, respectively. The formula for the noise addition process is:

[0066]

[0067]

[0068] Where t∈(1,T), δ is random noise at time t that follows a standard Gaussian distribution. αt It is a satisfying α t The hyperparameters ∈ (0,1) determine the variance of the noise added in each iteration. Simultaneously, to ensure boundedness as t→∞, α... t It gradually decreases over time. In this invention, the same Gaussian noise is applied to x0 and y0;

[0069] Step three is as follows:

[0070] The dual U-Net network consists of a denoising network f θ (x t ,t,x0)(hereinafter referred to as θ network) and distributed transformation network f φ (x t The θ network is composed of two networks, θ0 and θt (hereinafter referred to as the φ network). Both networks adopt the standard U-Net topology in the denoising diffusion probability model. The difference is that, to ensure the enhanced image content is consistent with the input image, this invention feeds x0 as a prior condition into the θ network. The θ network input x0... t At time t and x0, the output noise δ θ φ network input x t And at time t, output the noise image y of the reference image y0 at time t. t ; Schematic diagram as follows Figure 2 As shown;

[0071] Step four is as follows:

[0072] The model training process proposed in this invention includes training the denoising network θ and training the data distribution transformation network φ; for the denoising network, the loss function can be expressed as:

[0073]

[0074] Where δ is random noise at time t that follows a standard Gaussian distribution;

[0075] The loss function of a distributed transformation network can be expressed as:

[0076]

[0077] Where y t This is the diffusion result of the reference image y0 at time t;

[0078] Step five is as follows:

[0079] In the first iteration of the reasoning process, x is... t The image is fed into a denoising network, and the denoised image x at time t-1 is obtained according to the parameter renormalization rule. t-1 Subsequently, a distributed transformation network is used to convert it into a noisy image y of the reference image y0 at time t-1.t-1 , that is:

[0080]

[0081] y t-1 = f φ (x t-1 )

[0082] In the second to the (T - 1)-th iteration of the inference process, the present invention uses the same denoising network for the denoising process of the degraded image and the reference image. At the same time, in order to improve the network enhancement effect, at each t moment, the distribution conversion network is used to convert x t to x ′ t and add it to y ′ t at the same moment to obtain the total distribution Y at this moment, that is:

[0083] Y = y ′ t + f φ (x t , t) ~ N(2μ t , 2σ t 2 )

[0084] where 1 < t < T, y ′ t and f φ (x t , t) both follow the normal distribution N(μ t as the mean and σ t 2 as the variance N(μ t , σ t 2 ); To avoid the enhanced image from being distorted, after the feature maps are superimposed, the distribution Y ~ N(2μ t , 2σ t 2 ) is converted to Y ~ N(μ t , σ t 2 ), and its operation is:

[0085]

[0086]

[0087] That is, when 1 < t < T:

[0088]

[0089] In the last iteration of the inference process, only the denoising network needs to be used for y1, and the final enhanced image is obtained according to the following formula, Y = y′ t N(μ) follows a normal distribution t ,σ t 2 Substituting these values ​​into the above formula, the final formula for the data distribution of the reference image y0 is derived as follows:

[0090]

[0091] The schematic diagram of the reasoning process of this invention is shown below. Figure 3 As shown.

[0092] Step six is ​​as follows:

[0093] In this invention, 1200 pairs of underwater images from the UIEB and SUID datasets were used within the TensorFlow framework to train the denoising network and the data distribution transformation network 100,000 times each. The batch size was set to 4, the learning rate to 0.00001, and the Adam optimizer was used, with α selected as the optimal value. t Sampling from a uniform distribution, α t The training parameters are defined as follows: ∈ [0.9999, 0.98], with 1000 noise addition attempts (T = 1000). After training, the optimal network weights from the two networks are selected. The selected optimal model is used to enhance the test set, and an evaluation metric is calculated.

[0094] Since the test datasets Challenge60 and U45 are unpaired datasets, the Underwater Image Quality Metric (UIQM) and the Underwater Color Image Quality Evaluation (UCIQE) metrics were used for their evaluation. UIQM is derived from a weighted sum of the Underwater Image Color Metric (UICM), the Underwater Image Sharpness Metric (UISM), and the Underwater Image Contrast Metric (UIConM). UCIQE uses a linear combination of chroma, saturation, and contrast to evaluate the enhanced image, quantifying uneven color casts, blurring, and low contrast.

[0095] The enhancement results of the method of this invention are compared with those of widely used underwater image enhancement methods in recent years on the U45 test set. The comparison results are as follows: Figure 4 As shown, from left to right, these methods are: Raw, FE, FUnIEGAN, MLFcGAN, Shallow-UWnet, UWGAN, and the method of this invention.

[0096] Image comparison analysis:

[0097] When processing blue-toned images, the FE method exhibits over-enhancement, reducing image realism. When processing green-toned images, FUNIEGAN, MLFcGAN, and Shallow-UWnet enhance the overall image tone to a yellowish tint, causing a certain degree of color distortion. When processing foggy images, MLFcGAN and Shallow-UWnet make the overall image tone pinkish, resulting in poor enhancement effects. UWGAN exhibits severe color distortion when processing foggy images.

[0098] In comparison, our proposed method performs well in enhancing images of all color styles, adapting to different image styles. Images enhanced using this method exhibit richer colors and better visual effects. The average UIQM and UCIQE metrics measured on the U45 dataset for different enhancement methods are shown in Table 1.

[0099] Table 1 UIQM and UCIQE metrics for different methods on the U45 dataset.

[0100] UISM UICM UICONM UIQM UCIQE Raw 7.3140 -74.2683 0.6479 2.3821 0.3754 FE 7.2924 -32.7786 0.8093 4.1226 0.4791 FUnieGAN 7.2989 -20.2163 0.7733 4.3502 0.3997 MLFcGAN 6.9808 -1.6147 0.6116 4.2029 0.3908 Shallow-UWnet 7.1271 -8.3827 0.6108 4.0523 0.3703 UWGAN 7.1329 -39.2864 0.8350 3.9840 0.4384 This method 7.1222 1.9244 1.1787 4.6031 0.4521

[0101] Note: Bold text indicates the optimal value of each column of indicators, and italic text indicates the suboptimal value of each column of indicators.

[0102] As can be seen from the table above, this method achieves the best value among the six methods in terms of UIQM, which is 5.8% higher than the second-best value, and achieves the second-best value in terms of UCIQE.

[0103] The enhancement results of this method are compared with those of underwater image enhancement methods that have been widely used in recent years on the Challenge60 test set. Figure 5 This is a comparison chart showing the enhancement effects of each method on the Challenge60 test set.

[0104] Image comparison analysis:

[0105] When processing low-light images, the enhancement effects of methods other than FE and UWGAN are not ideal. The enhanced images are still low in brightness and the information in the images cannot be distinguished. Although the UWGAN method improves the brightness of low-light images to some extent, feature loss occurs and the information in the images is blurred. In the processing of blue-green tones, the performance of each method is similar to the results tested on the U45 dataset.

[0106] In comparison, this method, when processing low-light images, improves image brightness and contrast while preserving image information, resulting in better visual effects; it also better avoids color distortion when processing blue-green images. The average UIQM and UCIQE metrics of different enhancement methods measured on the Challenge60 dataset are shown in Table 2.

[0107] Table 2 UIQM and UCIQE metrics for different methods on the Challenge60 dataset.

[0108] UISM UICM UICONM UIQM UCIQE Raw 7.2895 -49.7770 0.6125 2.9388 0.3590 FE 7.1831 -11.8096 0.7173 4.3528 0.4782 FUnieGAN 7.4700 -12.0836 0.6795 4.2946 0.3592 MLFcGAN 7.1103 -0.5603 0.6177 4.2923 0.3681 Shallow-UWnet 7.3278 -6.1696 0.5937 4.1125 0.3348 UWGAN 7.1946 -20.414 0.7678 4.2963 0.5043 This method 7.0980 2.1821 0.8944 4.4733 0.4198

[0109] As shown in Table 2, our proposed method still represents the best UIQM metric among the six methods. Considering both visual effects and the two metrics, our method achieves ideal results in underwater image enhancement, demonstrating superior enhancement performance compared to other current methods.

[0110] Step seven is as follows:

[0111] To demonstrate that the enhanced image effectively enriches edge and feature information, this method performs SIFT (Scale Invariant Feature Transform) feature point matching and Canny operator (a multi-level edge detection algorithm) edge extraction on the underwater degraded image and the enhanced image. By comparing the number of feature points and edge information before and after enhancement, the research effect of this method in the field of underwater image enhancement is demonstrated. Figure 6 This is a comparison image of the Canny edge extraction results. Figure 7 This image shows a comparison of SIFT feature point matching results between the degraded underwater image and the enhanced image. It is clear that this method effectively enriches the edge information and the number of feature points in the degraded image.

[0112] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for underwater image enhancement based on a denoising diffusion probability model, characterized in that: The steps are as follows: Step one: determine the training set and test set required by the network; the training set includes the degraded images with underwater image style in the SUID dataset, the paired underwater images in the UIEB dataset, and the non-paired real underwater images; the test set includes the non-paired real underwater images in the UIEB dataset, the green, blue degraded, and fogging underwater images in the U45 dataset; Step two: add noise to the degraded images and reference images step by step to obtain the noisy images; Step three: improve the network structure of the denoising diffusion probability model to a double U-Net network, input different parameters to the two networks, and complete the denoising and data distribution conversion functions; The two processes of de-noising and data distribution conversion are completed using a double U-Net network structure; the network is composed of a de-noising network and a distribution conversion network , wherein is a time point, is an initial degraded image, is an image after times of noise addition; The role of the denoising network is to predict the random noise sampled at time t, referred to as The role of the distribution conversion network is to complete the fitting of the two data distributions at time t, referred to as the network; Both networks employ the standard U-Net topology in the denoising diffusion probabilistic model; the enhanced image content is made consistent with the input image by feeding as a prior condition to the network; the network input , the time and , the output noise , the network input noise image and the time , the output reference image the noise image at time ; Step four: complete the training of the denoising network and the distribution conversion network; The loss function of the denoising network is: ； wherein is is a random noise satisfying a standard Gaussian distribution at the moment The loss function of the distribution conversion network is: ； wherein is a reference image at the diffusion result at the time instant; Step five: complete the inference process of the model, improve the data sampling method in the process to a distribution fusion sampling method more suitable for the double U-Net network, and complete the enhancement of the single underwater degraded image.

2. The underwater image enhancement method based on the denoising diffusion probability model according to claim 1, characterized in that: The training set of step one selects part of the paired SUID and UIEB datasets. The SUID dataset simulates 30 terrestrial images into degraded images with underwater image style, and the UIEB dataset contains 890 pairs of underwater images and 60 non-paired real underwater images. The test set selects 60 non-paired real underwater images in the UIEB dataset and the U45 dataset. The U45 dataset includes green, blue degraded, and fogging underwater images.

3. The method of claim 1, wherein the method is based on a denoising diffusion probabilistic model. The step two gives the input data wherein is the input degraded image, is the corresponding reference image, and is defined as and are respectively and images after times of adding noise, and the formula of the adding noise process is ；； ； where, , is a random noise satisfying standard Gaussian distribution at time , is a hyper-parameter satisfying , which determines the variance of the added noise at each iteration; at the same time, to ensure that is bounded at , it is gradually reduced with the increase of time; the same Gaussian noise is applied to and .

4. The method of claim 3, wherein the method is based on a denoising diffusion probabilistic model. In step five, during the first iteration of the reasoning process, Feed the denoising network and obtain the parameters according to the parameter renormalization rule. time Noisy images Then, a distributed transformation network is used to convert it into... Time reference image Noisy images ,Right now: ；； ； In the second to T-1th iteration of the inference process, the same denoising network is used for the denoising process of the degraded image and the reference image, and at the same time, in order to improve the network enhancement effect, the distribution conversion network is used to convert the degraded image distribution and the reference image distribution at each moment to the total distribution at the same moment, and then the total distribution at the same moment is obtained by adding the degraded image distribution and the reference image distribution at the same moment. That is:​​​​ ； wherein, , and are subject to a normal distribution with as mean and as variance ; to avoid distortion of the augmented image, the distribution is converted to after the feature map is superimposed, which operates as: ； ； That is, when time: ； At the last iteration of the inference process, only the Using the denoising network, the final enhanced image is obtained according to the following formula, Subject to normal distribution , substitute the above formula, and finally derive the reference image The formula of data distribution is: 。

Images