Image processing method, system and device based on covariance optimization diffusion model

Abstract

The present invention provides an image processing method, system, and device based on a covariance-optimized diffusion model for image reconstruction, image denoising, image restoration, image deblurring, or image super-resolution. The method comprises: obtaining measurement values; establishing, based on diffusion model theory, a guided unconditional pre-trained diffusion model to implement conditional probability sampling, with the final sampling result serving as the reconstructed image; and using the maximum likelihood method in the conditional probability sampling to obtain the optimal posterior covariance, which serves as the covariance for the guided Gaussian approximation. The present invention can improve the accuracy of existing image reconstruction and reduce sensitivity to model parameters, making it applicable to a wide range of reconstruction scenarios, including image denoising, restoration, and enhancement.

Description

Technical Field

[0001] The present invention relates to the field of signal processing, and in particular to an image processing method, system, device and storage medium. Background Art

[0002] With the development of media technology, the real-time transmission, analysis, and storage of massive amounts of data present significant challenges. For example, the widespread application of media images in various fields has placed higher demands on image resolution and imaging speed. Image sparse reconstruction can address the problems of partial signal loss and low image compression efficiency caused by the large amount of image information during compression, thereby reconstructing the images required by the application.

[0003] In recent years, with the rapid growth of data volumes, the development of high-performance parallel architectures (GPUs), and deep learning, a large number of data-driven deep learning image reconstruction methods have been proposed. Typical methods are based on supervised learning frameworks. These methods utilize training sample pairs from a training dataset, use low-quality image signals as input to a deep network model, and use high-quality signals as the target of the deep network model. The deep network model is then optimized using the back gradient propagation algorithm.

[0004] However, the models trained by these methods are usually only suitable for a single application scenario. For different image reconstruction problems, it is necessary to collect new training sample pairs and retrain the model. As a result, these methods consume a lot of resources, which limits their application scenarios.

[0005] In response to the technical defects of the above methods, researchers have proposed solutions, such as Plug-and-play, RED, deep generative prior methods, etc., which effectively alleviate the large amount of computing resources consumed by retraining models in different application scenarios. However, these methods are limited by the expressive power of the prior and their performance is usually inferior to that of supervised learning methods. In recent years, with the development of diffusion models, it has become possible to stably train high-quality deep image priors. Since 2022, using diffusion models as deep image priors to solve image restoration and enhancement problems has become a highly feasible direction. However, today's methods are based on Gaussian approximations, in which the covariance is heuristically selected, so the performance can be further improved when a more accurate covariance is used. Summary of the Invention

[0006] In response to the deficiencies of the prior art, the present invention provides an image processing method, system, terminal and storage medium, which use optimal covariance in the existing diffusion model-based image processing method to improve image processing performance.

[0007] According to a first aspect of the present invention, there is provided an image processing method based on a covariance optimized diffusion model for image reconstruction, image denoising, image restoration, image deblurring or image super-resolution, comprising:

[0008] A measurement value y is obtained, an image linear degradation model is established, and a guidance method based on Gaussian approximation is derived according to the image linear degradation model.

[0009] A U-Net architecture network is constructed using convolutional layers, nonlinear layers, and upsampling and downsampling layers as an unconditional diffusion model. The unconditional diffusion model is used to provide the mean and covariance of the Gaussian approximation in the guidance method. The unconditional diffusion model can be an untrained unconditional diffusion model or a pretrained unconditional diffusion model.

[0010] For an untrained unconditional diffusion model, a maximum likelihood method is used to train the unconditional diffusion model based on the acquired training image, and the output of the unconditional diffusion model serves as the mean and covariance of the Gaussian approximation in the bootstrapping method.

[0011] For a pre-trained unconditional diffusion model, the pre-trained unconditional diffusion model provides a mean prediction, and when an inverse diffusion covariance prediction branch is included, the optimal value of the inverse diffusion covariance is converted as the optimal covariance of the Gaussian approximation; when the inverse diffusion covariance prediction branch is not included, the optimal covariance of the Gaussian approximation is estimated by the Monte Carlo method.

[0012] The unconditional diffusion model is guided using a bootstrapping method based on the Gaussian approximation. The bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for sampling in a conditional inverse diffusion process. The final samples of the conditional inverse diffusion process obey the conditional distribution of the original image x0 given the measurement value, and the final samples are used as the reconstructed image.

[0013] Optionally, the image processing method based on the covariance optimized diffusion model is characterized in that the measurement value y is obtained, an image linear degradation model is established, and a guidance method based on Gaussian approximation is derived according to the image linear degradation model, including:

[0014] Establish image degradation mathematical models based on different image degradation scenarios, including any of the following:

[0015] -The general form of the image linear degradation model is: y = Ax0 + n, where the original image is x0, the measurement value is y, A is the measurement matrix, and n is additive Gaussian white noise with a standard deviation of σ.

[0016] - When the image processing is image sparse reconstruction: let the original image be x0, sample the original image using the sampling matrix A, and obtain the measurement signal y=Ax0+n;

[0017] When the image processing is image denoising: the original image is denoted as x0, and the original image x0 is added to the noise matrix to obtain a measurement signal y, which is a noisy image;

[0018] -When the image processing is compressed sensing image reconstruction: let the original image be x0, and compare the original image x0 with a random sampling matrix Multiply to obtain the measured value y=Ax0+n;

[0019] When the image processing is image restoration: Let the original image be x0. Multiply the original image x0 by a 0-1 diagonal sampling matrix diag(m) to obtain a measurement value y = diag(m)x0 + n. This measurement value is the image to be restored with missing pixels. The 0-1 diagonal sampling matrix diag(m) is a diagonal matrix, and the elements in m are either 0 or 1.

[0020] When the image processing is image deblurring: the original image is denoted as x0, and the original image x0 is convolved with a convolution kernel k to obtain a measurement value y=x0*k+n.

[0021] - When the image processing is image super-resolution: let the original image be x0, downsample the original image x0, and obtain the measurement value y=downsample(x0)+n

[0022] According to the different usage of Gaussian approximation in guidance, guidance can be divided into two categories, including:

[0023] -Describe the likelihood score function based on the conditional posterior mean and the posterior mean:

[0024]

[0025] Among them, σ t is the noise standard deviation of the noisy image, and the posterior expectation The output of the unconditional diffusion model can be used to obtain a closed-form solution to the score function approximation based on the conditions of the isotropic Gaussian approximation, or a numerical solution to the score function approximation based on the conjugate gradient;

[0026] -According to Bayes' theorem, p t (x0|x t ,y)∝p t (y|x0)p t (x0|x t ). Use q t (x0|x t ) approximate p t (x0|x t ), and get p t (x0|xt ,y)'s approximate q t (x0|x t ,y)∝p t (y|x0)q t (x0|x t ). Among them, q t (x0|x t ,y) is Gaussian distribution. Use q t (x0|x t ,y) is used as an approximation of the conditional posterior mean. Based on the isotropic Gaussian approximation condition, a closed-form solution for the Gaussian distribution mean is derived, or a numerical solution for the Gaussian distribution mean is obtained based on the conjugate gradient.

[0027] Optionally, the image processing method based on the covariance optimized diffusion model is characterized in that the closed-form solution of the score function approximation is derived based on the isotropic Gaussian approximation condition, or the numerical solution of the score function approximation is obtained based on the conjugate gradient, including:

[0028] Under the condition of isotropic Gaussian approximation, that is, According to the mathematical model of image degradation, an approximate formula of the score function based on Gaussian approximation is derived, including any of the following:

[0029] -The closed-form solution in general form is

[0030]

[0031] Avoid high-dimensional matrix inversion operations and obtain efficient closed-form solutions, including any of the following:

[0032] Define the symbol F as the discrete Fourier transform matrix, ⊙ as the Hadamard product, ⊙ s block-wise multiplication operator, is s times block average downsampling, is the Fourier transform coefficient of k, for The conjugate of F ↓s is the discrete Fourier transform matrix of the s-fold down-sampled signal.

[0033] -For image restoration, the closed-form solution of v is

[0034]

[0035] in,

[0036] -For image deblurring, the closed-form solution of v is

[0037]

[0038] -For image super-resolution, the closed-form solution of v is

[0039]

[0040] According to the closed-form solution under the general form, if the temporary variable u is defined to satisfy v=A T u, then u is the form of the inverse matrix multiplied by the vector, and the inverse matrix is ​​a symmetric positive definite matrix, u can be expressed as the solution of a symmetric positive definite linear equation, that is,

[0041]

[0042] Symmetric positive definite linear equations can be solved efficiently using the well-established conjugate gradient method.

[0043] Optionally, the condition based on the isotropic Gaussian approximation is used to derive q t (x0|x t ,y) by closed-form solution, or by obtaining q based on conjugate gradients t (x0|x t ,y) means, including:

[0044] Under the condition of isotropic Gaussian approximation, that is, According to the image degradation linear model, q is derived t (x0|x t ,y) includes any of the following:

[0045] -The closed-form solution in general form is

[0046]

[0047] Avoid high-dimensional matrix inversion operations and obtain efficient closed-form solutions, including any of the following:

[0048] -For image restoration, the closed-form solution is

[0049]

[0050] -For image deblurring, the closed-form solution is

[0051]

[0052] -For image super-resolution, the closed-form solution is

[0053]

[0054] According to the closed-form solution under the general form, it is observed that it is in the form of an inverse matrix multiplied by a vector, and the inverse matrix is ​​a symmetric positive definite matrix, which can be converted into q t (x0|x t ,y)mean It is expressed as the solution of a symmetric positive definite linear equation, that is,

[0055]

[0056] Symmetric positive definite linear equations can be solved efficiently using the well-established conjugate gradient method.

[0057] Optionally, an image processing method based on a covariance optimized diffusion model is characterized in that, for an untrained unconditional diffusion model, a maximum likelihood method is used to train the unconditional diffusion model based on the acquired training image, and the output of the unconditional diffusion model is used as the mean and covariance of the Gaussian approximation in the bootstrapping method, including:

[0058] -Parameterizing time-dependent Gaussian distributions using differentiable neural networks where x t is a noisy image during the diffusion process, and the mean μ of the Gaussian distribution is t (x t ) and covariance Σ t (x t ) is the output of the differentiable neural network, and the input of the differentiable neural network is the noisy image x t .

[0059] - by minimizing q t (x0|x t ) and p t (x0|x t ) is used as the loss function to train the differentiable neural network:

[0060]

[0061] Among them, p t (x0|x t ) is the real given noisy image x t Theoretically, this is equivalent to training with the maximum likelihood method.

[0062] -In order to use a single network to output the mean and covariance of the Gaussian distribution at all times t, the final loss function is defined using integration:

[0063]

[0064] Here, θ is the parameter of the neural network.

[0065] -Use Monte Carlo integration method to calculate the objective function value.

[0066] Optionally, the image processing method based on the covariance optimization diffusion model is characterized in that, for the pre-trained unconditional diffusion model, the pre-trained unconditional diffusion model provides a mean prediction, and when an inverse diffusion covariance prediction branch is included, the optimal value of the inverse diffusion covariance is converted as the optimal covariance of the Gaussian approximation; when the inverse diffusion covariance prediction branch is not included, the optimal covariance of the Gaussian approximation is estimated by the Monte Carlo method, including:

[0067] If the pre-trained unconditional diffusion model includes the inverse diffusion covariance prediction branch, the optimal value of the inverse diffusion covariance is converted under the constraint of the diagonal covariance to obtain the optimal value of the Gaussian approximate covariance, specifically:

[0068]

[0069] Among them, v t is the inverse diffusion variance (elements on the covariance diagonal), r t is the variance of the required Gaussian approximation. Therefore, the following transformation can be made

[0070]

[0071] If the pre-trained unconditional diffusion model does not contain the inverse diffusion covariance prediction branch, under the isotropic and input-independent covariance constraint Next, using the Monte Carlo method

[0072]

[0073] Approximate r t The best solution of the analysis:

[0074]

[0075] Optionally, the unconditional diffusion model is guided by a bootstrapping method using the Gaussian approximation, where the bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for sampling a conditional inverse diffusion process. The final samples of the conditional inverse diffusion process obey the conditional distribution of the original image x0 given the measurement value, and the final samples are used as the reconstructed image, including:

[0076] -Define the time-dependent Gaussian perturbation kernel of the original image x0 Get x0,x t ,y time-dependent joint distribution

[0077] - Ordinary differential equations (ODEs) defining stochastic boundary conditions according to the joint distribution:

[0078]

[0079] in, is the true conditional posterior mean, estimated based on the bootstrap method. The ordinary differential equation defined by x t For all t, distribution p t (x t |y). In particular, at t=0, it is the conditional distribution p(x0|y) of the original image x0 given the measurement value.

[0080] -By drawing from a Gaussian distribution Medium sampling x T , and simulate the ODE using an ODE solver from t=T to t=0, with x0 obeying p(x0|y).

[0081] According to a second aspect of the present invention, there is provided an image processing system for image reconstruction, image denoising, image restoration, image deblurring or image super-resolution, comprising:

[0082] The measurement guidance module includes a measurement model establishment submodule and a Gaussian approximation guidance submodule. The measurement model establishment submodule obtains measurement values, establishes an image linear degradation model, and establishes an image degradation mathematical model according to different image degradation scenarios. The Gaussian approximation guidance submodule guides the image linear degradation model to derive a guidance method based on Gaussian approximation.

[0083] Unconditional diffusion model establishment module: using convolutional layers, nonlinear layers, and up- and down-sampling layers to build a U-Net architecture network as an unconditional diffusion model, and using the unconditional diffusion model to provide the mean and covariance of the Gaussian approximation in the guidance method; the unconditional diffusion model is an untrained unconditional diffusion model or a pre-trained unconditional diffusion model;

[0084] Gaussian approximation mean and covariance prediction module: For an untrained unconditional diffusion model, the maximum likelihood method is used to train the unconditional diffusion model based on the training image, and the output of the unconditional diffusion model is used as the mean and covariance of the Gaussian approximation in the guidance method; for a pre-trained unconditional diffusion model, the pre-trained unconditional diffusion model provides a mean prediction, and when an inverse diffusion covariance prediction branch is included, the optimal value of the inverse diffusion covariance is converted as the optimal covariance of the Gaussian approximation; when the inverse diffusion covariance prediction branch is not included, the optimal covariance of the Gaussian approximation is estimated by the Monte Carlo method;

[0085] Reconstruction module: Use the Gaussian approximation bootstrapping method to guide the unconditional diffusion model, the bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for conditional inverse diffusion process sampling, the final sample of the conditional inverse diffusion process obeys the conditional distribution of the original image x0 after the measurement value is given, and the final sample is used as the reconstructed image.

[0086] According to a third aspect of the present invention, there is provided an image processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor may be used to execute the above-described image processing method or system when executing the computer program.

[0087] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0088] The image processing method, device, and storage medium provided by the present invention are used for image reconstruction, image denoising, image restoration, image deblurring, or image super-resolution. Compared with standard supervised learning-based deep neural network methods, they ensure reconstruction accuracy while being applicable to various reconstruction problems, greatly saving resources.

[0089] Compared with the existing method of guiding unconditional diffusion model, the image processing method, device and storage medium provided by the present invention use maximum likelihood to obtain optimal covariance, improve reconstruction accuracy and reduce sensitivity to hyperparameters, and improve image processing accuracy such as image reconstruction, image denoising, image restoration, image deblurring or image super-resolution.

[0090] The image processing method, device and storage medium provided by the present invention improve the computing speed and reconstruction accuracy compared with traditional iterative optimization algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0092] Figure 1 A flowchart of an image processing method provided by one embodiment of the present invention; DETAILED DESCRIPTION

[0093] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that those skilled in the art may make several variations and improvements without departing from the scope of the present invention. These all fall within the scope of protection of the present invention. Parts not described in detail below may be referred to in the summary of the invention or may be implemented using existing technologies.

[0094] Reference Figure 1 As shown, the present invention provides an image processing method for image reconstruction, image denoising, image restoration, image deblurring or image super-resolution, comprising:

[0095] S100: Obtain a measurement value y, establish an image linear degradation model, and derive a guidance method based on Gaussian approximation according to the image linear degradation model.

[0096] The S200 uses convolutional layers, nonlinear layers, and upsampling and downsampling layers to build a U-Net architecture as an unconditional diffusion model. For the specific architecture, refer to "Improved Denoising Diffusion Probabilistic Models" presented at the 2021 International Conference on Machine Learning (ICML). The unconditional diffusion model provides the mean and covariance of the Gaussian approximation in the guidance method. The unconditional diffusion model can be either an unconditional diffusion model without pretraining or a pretrained unconditional diffusion model.

[0097] In a scenario where a pre-trained unconditional diffusion model is not provided, a training image is obtained, and the training image is used to pre-train the unconditional diffusion model using the maximum likelihood method of the present invention. The output of the unconditional diffusion model is used as the mean and covariance of the Gaussian approximation in the guidance method.

[0098] In a scenario where a pre-trained unconditional diffusion model is provided, the pre-trained unconditional diffusion model provides a mean prediction. In an embodiment, a conversion method and a Monte Carlo method are used to estimate the optimal covariance of the Gaussian approximation.

[0099] S300: Using a Gaussian approximation bootstrapping method to bootstrap the unconditional diffusion model. This bootstrapping method can transform the unconditional posterior mean into a conditional posterior mean for sampling in a conditional inverse diffusion process. The final samples of the conditional inverse diffusion process obey the conditional distribution of the original image x0 after a given measurement value, and the final samples are used as the reconstructed image.

[0100] In the numerical experiments, this embodiment selected 100 images from the ImageNet natural image dataset and the FFHQ face dataset for testing, and the test images were all unified to 256x256 pixels. The test scenarios for image degradation and reconstruction selected image restoration, de-Gaussian blur, de-motion blur, and super-resolution. The unconditional diffusion model selected the ImageNet pre-trained unconditional diffusion model provided by "Improved Denoising Diffusion Probabilistic Models" published at the 2021 International Conference on Machine Learning (ICML), and the FFHQ pre-trained unconditional diffusion model provided by "Diffusion posterior sampling for general noisy inverse problems" published at the 2023 International Conference on Learning Representations (ICLR). The inverse diffusion process sampling method uses the Heun second-order ordinary differential equation (ODE) sampler, and uniformly uses 50-step sampling.

[0101] In this specific application embodiment, the first type of guided method is used to achieve image reconstruction in the selected image degradation scene. Specifically, in this embodiment, the image linear degradation model includes:

[0102] For image restoration, the original image is x0. Multiply x0 by a 0-1 diagonal sampling matrix diag(m) to obtain a measurement value y = diag(m)x0 + n. This measurement value represents the image to be restored with missing pixels. The 0-1 diagonal sampling matrix diag(m) is a diagonal matrix, with elements in m being either 0 or 1.

[0103] When the image degradation scenario is image deblurring: the original image is denoted as x0, and the original image x0 is convolved with a convolution kernel k to obtain a measurement value y=x0*k+n.

[0104] - When the image degradation scenario is image super-resolution: let the original image be x0, downsample the original image x0 to obtain a measurement value y=down_sample(x0)+n.

[0105] Specifically, under the isotropic Gaussian approximation condition, including DPS, PGDM, and Monte Carlo method, this specific application embodiment includes the following three main steps:

[0106] Step 1: Based on the image linear degradation model, the specific closed-form solution of the likelihood score function in the first-class bootstrapping method is derived:

[0107] -The closed-form solution in general form is

[0108]

[0109] -For image restoration, the closed-form solution of v is

[0110]

[0111] -For image deblurring, the closed-form solution of v is

[0112]

[0113] -For image super-resolution, the closed-form solution of v is

[0114]

[0115] Step 2: Use the property of the likelihood score function where the conditional posterior mean is equal to the posterior mean plus a constant times:

[0116]

[0117] Get an approximation to the conditional posterior mean.

[0118] Step three: using the approximate value of the conditional posterior mean to replace the posterior mean in the unconditional inverse diffusion process sampling.

[0119] Specifically, under the condition of not being an isotropic Gaussian approximation, including the conversion method, this specific application embodiment includes the following three main steps:

[0120] Step 1: Get the approximate value of the likelihood score function based on the conjugate gradient method

[0121] Step 2: Use the property of the likelihood score function where the conditional posterior mean is equal to the posterior mean plus a constant times:

[0122]

[0123] Get an approximation to the conditional posterior mean.

[0124] Step three: using the approximate value of the conditional posterior mean to replace the posterior mean in the unconditional inverse diffusion process sampling.

[0125] This example compares two other reconstruction methods for guided unconditional diffusion models, both of which were published at the 2023 International Conference on Learning Representations (ICLR) conference. They are the PGDM method proposed in "Pseudoinverse-guided diffusion models for inverse problems" and the DPS method proposed in "Diffusion posterior sampling for general noisy inverse problems."

[0126] In early experiments, it was found that using the optimal posterior covariance for all sampling steps resulted in poor performance. This may be due to the fact that the use of the Gaussian approximation is only reasonable for small noise levels. To address this issue, the optimal posterior covariance is only used in the last few sampling steps (when the diffuse noise intensity is less than 0.2), and for high noise levels we use the PGDM covariance respectively. The specific comparative test results are shown in Table 1. The evaluation criteria are PSNR (in dB), where the higher the value, the better the effect. For LPIPS, the lower the value, the better the effect.

[0127] The experimental results in Table 1 show that the images reconstructed by the system of this embodiment are significantly superior to those obtained by the other two methods in terms of reconstruction quality in most cases. The other three methods are all existing guided unconditional diffusion models, but their performance is limited because the covariance of the Gaussian approximation is not theoretically optimal. This embodiment uses the Monte Carlo method and the transformation method to achieve the optimal covariance without additional training, thereby improving reconstruction accuracy.

[0128] Table 1: Comparison of image reconstruction performance (PSNR, unit: dB, LPIPS) of DPS, PGDM and the method proposed in this invention (Monte Carlo method, transformation method)

[0129]

[0130] In another embodiment, the second type of guided method is used to achieve image reconstruction in the selected image degradation scene. Specifically, the image linear degradation model includes:

[0131] For image restoration, the original image is x0. Multiply x0 by a 0-1 diagonal sampling matrix diag(m) to obtain a measurement value y = diag(m)x0 + n. This measurement value represents the image to be restored with missing pixels. The 0-1 diagonal sampling matrix diag(m) is a diagonal matrix, with elements in m being either 0 or 1.

[0132] In the numerical experiment, this embodiment selected 100 images from the FFHQ face dataset for testing, and the test images were all uniformly 256x256 pixels. The test scenario for image degradation and reconstruction selected image restoration, and the unconditional diffusion model selected the ImageNet pre-trained unconditional diffusion model provided by "Improved Denoising Diffusion Probabilistic Models" published at the 2021 International Conference on Machine Learning (ICML), and the FFHQ pre-trained unconditional diffusion model provided by "Diffusion posterior sampling for general noisy inverse problems" published at the 2023 International Conference on Learning Representations (ICLR). The inverse diffusion process sampling method uses the Heun second-order stochastic differential equation (SDE) sampler, and uniformly uses 50-step sampling.

[0133] Specifically, under the isotropic Gaussian approximation condition, including DiffPIR and Monte Carlo method, this specific application embodiment includes the following two main steps:

[0134] Step 1: Based on the image linear degradation model, the closed-form solution of the conditional posterior mean in the second-type bootstrap method is derived:

[0135] For image restoration, the closed-form solution is

[0136]

[0137] -For image deblurring, the closed-form solution is

[0138]

[0139] -For image super-resolution, the closed-form solution is

[0140]

[0141] For image restoration, the closed-form solution is

[0142]

[0143] -For image deblurring, the closed-form solution is

[0144]

[0145] -For image super-resolution, the closed-form solution is

[0146]

[0147] Step 2: Using the approximate value of the conditional posterior mean to replace the posterior mean in the unconditional inverse diffusion process sampling.

[0148] Specifically, under the condition of not being an isotropic Gaussian approximation, including the conversion method, this specific application embodiment includes the following two main steps:

[0149] Step 1: obtain an approximate value of the conditional posterior mean in the second-kind bootstrap method based on the conjugate gradient method;

[0150] Step 2: using the approximate value of the conditional posterior mean to replace the posterior mean in the unconditional inverse diffusion process sampling;

[0151] This example compares another reconstruction method based on the second-class guided unconditional diffusion model, the PGDM method proposed in "Denoising diffusion models for plug-and-play image restoration" published at the 2023 Conference on Computer Vision and Pattern Recognition (CVPR) conference.

[0152] In early experiments, we found that using the best a posteriori covariance for all sampling steps led to poor performance. This is probably because the use of the Gaussian approximation is only reasonable for small noise levels. To address this issue, we only use the best a posteriori covariance in the last few sampling steps (when the diffuse noise intensity is less than 0.2), and for high noise levels we use the DiffPIR covariance separately. The specific comparative test results are shown in Table 2. The evaluation standard is LPIPS, and the lower the value, the better the effect.

[0153] Table 2: Comparison of image reconstruction performance (LPIPS) of DPS, DiffPIR and the method proposed in this invention (Monte Carlo method, transformation method)

[0154] Covariance \λ 1e-2 1e-1 1e0 1e1 1e2 DiffPIR 0.3830 0.3514 0.2504 0.2491 0.3073 Monte Carlo method 0.2834 0.2739 0.2421 0.2482 0.3019 Conversion method 0.2835 0.2745 0.2409 0.2478 0.3007

[0155] The experimental results in Table 2 show that the images reconstructed by the system of this embodiment significantly outperform those obtained by DiffPIR in terms of reconstruction quality under all hyperparameter settings. The covariance of the Gaussian approximation used by DiffPIR is not theoretically optimal, thus limiting performance. This embodiment uses the Monte Carlo and transformation methods to achieve the optimal covariance without additional training, thereby improving reconstruction accuracy.

[0156] Based on the same technical concept, an embodiment of the present invention further provides an image processing system, comprising:

[0157] Measurement guidance module: includes measurement model establishment submodule and Gaussian approximation guidance submodule, where:

[0158] Measurement model establishment submodule: obtains measurement values, establishes an image linear degradation model, and establishes an image degradation mathematical model based on different image degradation scenarios; specifically, it includes any of the following:

[0159] The general form of the image linear degradation model is: y = Dx0 + n, where the original image is x0, the measurement value is y, D is the linear degradation matrix, and n is the additive Gaussian white noise with standard deviation σ;

[0160] -When the image processing is image sparse reconstruction: let the original image be x0, sample the original image using the sampling matrix A, and obtain the measurement value y=Ax0+n;

[0161] When the image processing is image denoising: the original image is denoted as x0, the original image x0 is added to the noise matrix to obtain a measurement value y, which is a noisy image;

[0162] When the image processing is compressed sensing image reconstruction: the original image is denoted as x0, and the original image x0 is multiplied by a random sampling matrix A to obtain a measurement value y=Ax0+n;

[0163] When the image processing is image restoration: let the original image be x0, multiply the original image x0 by a 0-1 diagonal sampling matrix diag(m) to obtain a measurement value y=diag(m)x0+n, where the measurement value is the image to be restored with the missing pixels. The 0-1 diagonal sampling matrix diag(m) is a diagonal matrix, and the elements in m are either 0 or 1;

[0164] - When the image processing is image deblurring: let the original image be x0, convolve the original image x0 with a convolution kernel k, and obtain the measurement value y=x0*k+n, which can be expressed as Where F is the discrete Fourier transform matrix, is the Fourier transform of convolution kernel k;

[0165] - When the image processing is image super-resolution: let the original image be x0, downsample the original image x0, and obtain the measurement value y=downsample(x0)+n, which can be approximately expressed as Where F is the discrete Fourier transform matrix, D ↓ is the standard downsampling matrix, which selects the upper left corner pixel of each block after dividing the image into blocks for downsampling. is the Fourier transform of the convolution kernel k, used to approximate nonlinear bicubic interpolation

[0166] Gaussian approximation guidance submodule: A guidance method based on Gaussian approximation is derived based on the image linear degradation model. According to the different usages of Gaussian approximation in guidance, guidance is divided into two categories, including:

[0167] - First-class bootstrapping method: The conditional posterior mean is described using the posterior mean and the likelihood score function, and the posterior expectation can be obtained using the output of the unconditional diffusion model. Based on the conditions of the isotropic Gaussian approximation, a closed-form solution to the score function approximation is derived, or a numerical solution to the score function approximation is obtained based on the conjugate gradient.

[0168] - Second type of bootstrapping: using q t (x0|x t ,y) as an approximation of the conditional posterior mean, deriving a closed-form solution to the mean of the Gaussian distribution based on the isotropic Gaussian approximation condition, or obtaining a numerical solution to the mean of the Gaussian distribution based on the conjugate gradient;

[0169] Unconditional diffusion model establishment module: using convolutional layers, nonlinear layers, and up- and down-sampling layers to build a U-Net architecture network as an unconditional diffusion model, and using the unconditional diffusion model to provide the mean and covariance of the Gaussian approximation in the guidance method; the unconditional diffusion model is an untrained unconditional diffusion model or a pre-trained unconditional diffusion model;

[0170] Gaussian approximation mean and covariance prediction module: For an untrained unconditional diffusion model, the maximum likelihood method is used to train the unconditional diffusion model based on the training image, and the output of the unconditional diffusion model is used as the mean and covariance of the Gaussian approximation in the guidance method; for a pre-trained unconditional diffusion model, the pre-trained unconditional diffusion model provides a mean prediction, and when an inverse diffusion covariance prediction branch is included, the optimal value of the inverse diffusion covariance is converted as the optimal covariance of the Gaussian approximation; when the inverse diffusion covariance prediction branch is not included, the optimal covariance of the Gaussian approximation is estimated by the Monte Carlo method;

[0171] Reconstruction module: Use the Gaussian approximation bootstrapping method to guide the unconditional diffusion model, the bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for conditional inverse diffusion process sampling, the final sample of the conditional inverse diffusion process obeys the conditional distribution of the original image x0 after the measurement value is given, and the final sample is used as the reconstructed image.

[0172] In the above embodiments of the present invention, the image processing system is used to execute the image processing method in any one of the above embodiments.

[0173] In one embodiment of the present invention, an image processing device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the image processing method or the image processing system when executing the program.

[0174] In one embodiment of the present invention, a computer device is also provided, comprising: a communication interface, a memory, and a processor, wherein: the memory is used to store a program for implementing the image processing method in any of the above embodiments; the processor is used to load and execute the program stored in the memory to implement each step of the image processing method in any of the above embodiments.

[0175] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0176] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0177] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0178] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0179] The number of communication interfaces, memory, and processors can be at least one, and the communication interfaces, memory, and processors can communicate with each other via a communication bus. The communication interface can be used to receive data sent by other devices and can include an interface for a computer device to communicate with other devices, or a communication interface used for communication between components within the computer device. The specific description of the processor and memory can refer to the above embodiments and will not be repeated here.

[0180] In practical applications, the computer device may be a server, a computer, etc. Therefore, the structure of the computer device is not limited to a communication interface, a memory, and a processor, but may also include other hardware devices, such as other storage devices, etc., which can be determined according to the functions of the computer device.

[0181] Compared with the model trained by supervised learning methods, which is limited to a single reconstruction scenario, the image processing method, system, device and readable medium provided by the present invention can be applied to solving general linear inverse problems such as image denoising, compressed sensing, image restoration, image super-resolution, image deblurring, etc. after one training.

[0182] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to implement the composition of the system, that is, the embodiments in the method can be understood as preferred examples of constructing the system, which will not be elaborated here.

[0183] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered a hardware component, and the devices included therein for implementing the various functions can also be considered as structures within the hardware component; the devices for implementing the various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference will be made to the descriptions of the devices and computer equipment disclosed in the embodiments for similarities and differences between them. Since the devices and computer equipment described in the embodiments correspond to the methods disclosed in the embodiments, their descriptions are relatively simple. For relevant details, refer to the descriptions of the methods.

[0185] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An image processing method based on a covariance optimized diffusion model for image reconstruction, image denoising, image restoration, image deblurring or image super-resolution, characterized in that: include: Obtaining a measurement value y, establishing an image linear degradation model, and deriving a guidance method based on Gaussian approximation according to the image linear degradation model; A U-Net architecture network is constructed using convolutional layers, nonlinear layers, and up- and down-sampling layers as an unconditional diffusion model, and the unconditional diffusion model is used to provide the mean and covariance of the Gaussian approximation in the guidance method; the unconditional diffusion model is an untrained unconditional diffusion model or a pre-trained unconditional diffusion model; For an untrained unconditional diffusion model, using a maximum likelihood method to train the unconditional diffusion model based on the training image, and using the output of the unconditional diffusion model as the mean and covariance of the Gaussian approximation in the bootstrapping method; For a pre-trained unconditional diffusion model, the pre-trained unconditional diffusion model provides a mean prediction, and when an inverse diffusion covariance prediction branch is included, the optimal value of the inverse diffusion covariance is converted as the optimal covariance of the Gaussian approximation; when the inverse diffusion covariance prediction branch is not included, the optimal covariance of the Gaussian approximation is estimated by a Monte Carlo method; Using the Gaussian approximation bootstrapping method to bootstrap the unconditional diffusion model, the bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for sampling a conditional inverse diffusion process, wherein the final sample of the conditional inverse diffusion process obeys the conditional distribution of the original image x0 given the measurement value, and the final sample is used as the reconstructed image; The bootstrapping method based on Gaussian approximation is derived from the image linear degradation model bootstrapping, wherein the core of the bootstrapping is the estimation of the conditional posterior mean. According to the different usages of Gaussian approximation in the bootstrapping, the bootstrapping is divided into two categories, including: - First-Type Bootstrapping: Conditional Posterior Mean Using the posterior mean and likelihood score function describe Among them, σ t is the noise standard deviation of the noisy image, and the posterior expectation The output of the unconditional diffusion model can be used to obtain the score function approximation; based on the conditions of the isotropic Gaussian approximation, a closed-form solution of the score function approximation is derived, or a numerical solution of the score function approximation is obtained based on the conjugate gradient; - Second type of bootstrapping method: According to Bayes' theorem, p t (x0|x t ,y)∝p t (y|x0)p t (x0|x t ), use q t (x0|x t ) approximate p t (x0|x t ), and get p t (x0|x t ,y)'s approximate q t (x0|x t ,y)∝p t (y|x0)q t (x0|x t ), where q t (x0|x t ,y) is Gaussian distribution; use q t (x0|x t ,y) as an approximation of the conditional posterior mean; based on the isotropic Gaussian approximation condition, a closed-form solution of the Gaussian distribution mean is derived, or a numerical solution of the Gaussian distribution mean is obtained based on the conjugate gradient.

2. The image processing method based on the covariance optimization diffusion model according to claim 1, characterized in that: The obtaining of the measurement value y and establishing the linear image degradation model refers to establishing an image degradation mathematical model according to different image degradation scenarios, which specifically includes any of the following: The general form of the image linear degradation model is: y = Dx0 + n, where the original image is x0, the measurement value is y, D is the linear degradation matrix, and n is the additive Gaussian white noise with standard deviation σ; - When the image processing is image sparse reconstruction: let the original image be x0, sample the original image by the sampling matrix n, and obtain the measurement value y=Ax0+n; When the image processing is image denoising: the original image is denoted as x0, the original image x0 is added to the noise matrix to obtain a measurement value y, which is a noisy image; When the image processing is compressed sensing image reconstruction: the original image is denoted as x0, and the original image x0 is multiplied by a random sampling matrix A to obtain a measurement value y=Ax0+n; When the image processing is image restoration: let the original image be x0, multiply the original image x0 by a 0-1 diagonal sampling matrix diag(m) to obtain a measurement value y=diag(m)x0+n, where the measurement value is the image to be restored with the missing pixels. The 0-1 diagonal sampling matrix diag(m) is a diagonal matrix, and the elements in m are either 0 or 1; - When the image processing is image deblurring: let the original image be x0, convolve the original image x0 with a convolution kernel k, and obtain the measurement value y=x0*k+n, which can be expressed as Where F is the discrete Fourier transform matrix, is the Fourier transform of convolution kernel k; - When the image processing is image super-resolution: let the original image be x0, downsample the original image x0, and obtain the measurement value y=downsample(x0)+n, which can be approximately expressed as Where F is the discrete Fourier transform matrix, D ↓ is the standard downsampling matrix, which selects the upper left corner pixel of each block after dividing the image into blocks for downsampling. is the Fourier transform of the convolution kernel k, which is used to approximate nonlinear bicubic interpolation.

3. The image processing method based on the covariance optimization diffusion model according to claim 1, characterized in that: The method of deriving a closed-form solution to the score function approximation based on the isotropic Gaussian approximation condition, or obtaining a numerical solution to the score function approximation based on the conjugate gradient, includes: Under the condition of isotropic Gaussian approximation, that is, According to the image degradation linear model, an approximate formula of the score function based on Gaussian approximation is derived, including any of the following: -The closed-form solution in general form is Avoid high-dimensional matrix inversion operations and obtain efficient closed-form solutions, including any of the following: Define σ as the standard deviation of additive white Gaussian noise, symbol F is the discrete Fourier transform matrix, ⊙ is the Hadamard product, ⊙ s block-wise multiplication operator, is the Fourier transform coefficient of k, for The conjugate of F ↓s is the transformation matrix for discrete Fourier transform of the s-fold downsampled signal It is the average downsampling of s times the block, that is, the average value of s×s different blocks is calculated,↓ s It is s times the standard downsampling, that is, retaining the upper left corner pixel of each different s×s block and discarding other pixels, ↑ s It is s times standard upsampling, that is, upsampling by padding with zeros; -For image restoration, the closed-form solution of v is in, -For image deblurring, the closed-form solution of v is Where A is the defuzzified measurement matrix F -1 diag(k)F -For image super-resolution, the closed-form solution of v is Where A is the measurement matrix of image super-resolution According to the closed-form solution under the general form, if the temporary variable u is defined to satisfy v=A T u, then u is the form of the inverse matrix multiplied by the vector, and the inverse matrix is ​​a symmetric positive definite matrix, which expresses u as the solution of the symmetric positive definite linear equation, that is, Symmetric positive definite linear equations are efficiently solved using the well-established conjugate gradient method.

4. The image processing method based on the covariance optimization diffusion model according to claim 1, characterized in that: Based on the isotropic Gaussian approximation condition, q is derived t (x0|x t ,y) by closed-form solution, or by obtaining q based on conjugate gradients t (x0|x t ,y) means, including: Under the condition of isotropic Gaussian approximation, that is, According to the image linear degradation model, q is derived t (x0|x t ,y) includes any of the following: -The closed-form solution in general form is Avoid high-dimensional matrix inversion operations and obtain efficient closed-form solutions, including any of the following: -For image restoration, the closed-form solution is -For image deblurring, the closed-form solution is -For image super-resolution, the closed-form solution is in, According to the closed-form solution under the general form, it is in the form of an inverse matrix multiplied by a vector, and the inverse matrix is ​​a symmetric positive definite matrix. t (x0|x t ,y)mean It is expressed as the solution of a symmetric positive definite linear equation, that is, Symmetric positive definite linear equations are efficiently solved using the well-established conjugate gradient method.

5. The image processing method based on the covariance optimization diffusion model according to claim 1, characterized in that: For an untrained unconditional diffusion model, the maximum likelihood method is used to train the unconditional diffusion model based on the training image, and the output of the unconditional diffusion model is used as the mean and covariance of the Gaussian approximation in the bootstrapping method, including: Parameterizing Time-Dependent Gaussian Distributions Using Differentiable Neural Networks where x t is a noisy image during the diffusion process, and the mean μ of the Gaussian distribution is t (x t ) and covariance Σ t (x t ) is the output of the differentiable neural network, and the input of the differentiable neural network is the noisy image x t ; By minimizing q t (x0|x t ) and p t (x0|x t ) is used as the loss function to train the differentiable neural network: Among them, p t (x0|x t ) is the real given noisy image x t The posterior distribution of the original image x0; In order to use a single network to output the mean and covariance of the Gaussian distribution at all times t, the final loss function is defined using integration: Among them, θ is the parameter of the neural network; The objective function value is calculated using the Monte Carlo integration method.

6. The image processing method based on the covariance optimized diffusion model according to claim 1, characterized in that: If the pre-trained unconditional diffusion model includes the inverse diffusion covariance prediction branch, the optimal value of the inverse diffusion covariance is converted under the constraint of the diagonal covariance to obtain the optimal value of the Gaussian approximate covariance. Specifically, according to the following relationship: in, is the theoretical optimal value of the inverse diffusion variance, that is, the elements on the covariance diagonal. is the theoretical optimal value of the variance of the required Gaussian approximation, the scalar in the formula β t Determined by the training settings of the pre-trained unconditional diffusion model; make the following conversion in, are the predictions of the inverse diffusion variance and the required Gaussian approximation variance, respectively; If the pre-trained unconditional diffusion model does not contain the inverse diffusion covariance prediction branch, under the isotropic and input-independent covariance constraint Next, use the Monte Carlo method: Approximate r t The best solution of the analysis:

7. The image processing method based on the covariance optimization diffusion model according to claim 1, characterized in that: The unconditional diffusion model is guided by the Gaussian approximation bootstrapping method, wherein the bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for sampling a conditional inverse diffusion process, wherein the final sample of the conditional inverse diffusion process obeys the conditional distribution of the original image x0 after the measurement value is given, including: Define the time-dependent Gaussian perturbation kernel of the original image x0 Get x0,x t ,y time-dependent joint distribution The ordinary differential equation (ODE) with random boundary conditions defined according to the joint distribution is used for conditional sampling of the inverse diffusion process: in, is the true conditional posterior mean, estimated based on the bootstrap method; x defined by the ordinary differential equation t For all t, the distribution p t (x t |y), which is the conditional distribution p(x0|y) of the original image x0 given the measurement value at t=0; By Medium sampling x T , and simulate the ODE from t=T to t=0 using an ODE solver, with x0 obeying p(x0|y) using x0 as the reconstructed image.

8. An image processing system for image reconstruction, image denoising, image restoration, image deblurring or image super-resolution, characterized in that: include: The measurement guidance module includes a measurement model establishment submodule and a Gaussian approximation guidance submodule. The measurement model establishment submodule obtains measurement values, establishes an image linear degradation model, and establishes an image degradation mathematical model according to different image degradation scenarios. The Gaussian approximation guidance submodule guides the image linear degradation model to derive a guidance method based on Gaussian approximation. Unconditional diffusion model establishment module: using convolutional layers, nonlinear layers, and up- and down-sampling layers to build a U-Net architecture network as an unconditional diffusion model, and using the unconditional diffusion model to provide the mean and covariance of the Gaussian approximation in the guidance method; the unconditional diffusion model is an untrained unconditional diffusion model or a pre-trained unconditional diffusion model; Gaussian approximation mean and covariance prediction module: For an untrained unconditional diffusion model, the maximum likelihood method is used to train the unconditional diffusion model based on the training image, and the output of the unconditional diffusion model is used as the mean and covariance of the Gaussian approximation in the guidance method; for a pre-trained unconditional diffusion model, the pre-trained unconditional diffusion model provides a mean prediction, and when an inverse diffusion covariance prediction branch is included, the optimal value of the inverse diffusion covariance is converted as the optimal covariance of the Gaussian approximation; when the inverse diffusion covariance prediction branch is not included, the optimal covariance of the Gaussian approximation is estimated by the Monte Carlo method; Reconstruction module: using the Gaussian approximation bootstrapping method to guide the unconditional diffusion model, the bootstrapping method converts the unconditional posterior mean into a conditional posterior mean for sampling the conditional inverse diffusion process, the final sample of the conditional inverse diffusion process obeys the conditional distribution of the original image x0 after the measurement value is given, and the final sample is used as the reconstructed image; The Gaussian approximation guidance submodule obtains a guidance method based on Gaussian approximation according to the image linear degradation model guidance, wherein: the core of the guidance is the estimation of the conditional posterior mean. According to the different usages of Gaussian approximation in the guidance, the guidance is divided into two categories, including: - First-Type Bootstrapping: Conditional Posterior Mean Using the posterior mean and likelihood score function describe Among them, σ t is the noise standard deviation of the noisy image, and the posterior expectation The output of the unconditional diffusion model can be used to obtain the score function approximation; based on the conditions of the isotropic Gaussian approximation, a closed-form solution of the score function approximation is derived, or a numerical solution of the score function approximation is obtained based on the conjugate gradient; - Second type of bootstrapping method: According to Bayes' theorem, p t (x0|x t ,y)∝p t (y|x0)p t (x0|x t ), use q t (x0|x t ) approximate p t (x0|x t ), and get p t (x0|x t ,y)'s approximate q t (x0|x t ,y)∝p t (y|x0)q t (x0|x t ), where q t (x0|x t ,y) is Gaussian distribution; use q t (x0|x t ,y) as an approximation of the conditional posterior mean; based on the isotropic Gaussian approximation condition, a closed-form solution of the Gaussian distribution mean is derived, or a numerical solution of the Gaussian distribution mean is obtained based on the conjugate gradient.

9. An image processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it can be used to execute the image processing method according to any one of claims 1 to 7 or the image processing system according to claim 8.

Images