Plug-and-play diffusion sampling method for restoration of image with kernel error
By constructing an image non-blind restoration optimization problem model with fuzzy kernel error and using the diffusion model as a denoiser, the problem of poor fuzzy kernel error processing in the prior art is solved, and better image restoration effect and usability are achieved.
Patent Information
- Application Number
- CN202411082928.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-03
AI Technical Summary
The existing non-blind restoration methods of images are not effective when dealing with fuzzy kernel errors, which makes it difficult to achieve complete accuracy of the blurred kernel of the blurred image. Any small error is amplified during restoration, and the existence of fuzzy kernel errors is ignored.
A plug-and-play diffusion sampling method for image restoration with core error is proposed. By constructing an image non-blind restoration optimization problem model with fuzzy kernel error, combining clear image estimation and fuzzy kernel error estimation problems, the diffusion model is used as a decomposition model Noise device to realize plug-and-play image restoration.
Considering fuzzy kernel errors during the non-blind restoration process improves the effect and usability of image restoration, improves the visual fidelity and quality of the image, and is better than traditional methods.
Smart Images

Figure BDA0004984302290000051 
Figure BDA0004984302290000053 
Figure BDA0004984302290000062
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image processing, and in particular to a plug-and-play diffusion sampling method for restoring images with kernel errors. Background Art
[0002] Image non-blind restoration refers to solving a clear image from a blurred image when the blur kernel is known. Existing image non-blind restoration methods can be divided into two categories. One is the model-based optimization method. This type of method requires manual priors, but is more flexible in dealing with image restoration problems. The other is the deep learning method based on a large amount of data training. Although this type of method can achieve better results on a wide range of images, it is heavily dependent on a large amount of data training and is not conducive to generalization under different blur kernels. In order to combine the advantages of both, the traditional optimization problem is decomposed into a denoising sub-problem and a fidelity term sub-problem by using variable separation. The denoising sub-problem is solved by using the deep generative prior provided by the deep learning method based on a large amount of data training, and the fidelity term sub-problem is solved by using a model-based optimization method. This can combine the advantages of the two methods and better achieve image non-blind restoration.
[0003] However, in practical applications, the blur kernel of blurred images is estimated by blind image restoration methods, which is difficult to achieve a completely accurate level. In the pathological problem of non-blind restoration, any small error will be magnified during restoration, and there is still a gap between the forward process of existing blur and the forward process of actual blur, while the existing non-blind image restoration method ignores the existence of blur kernel error.
[0004] Therefore, it is very necessary to explore how to take blur kernel error into consideration in the process of non-blind image restoration. Summary of the invention
[0005] The purpose of this application is to provide a plug-and-play diffusion sampling method for image restoration with kernel errors, which can take into account the blur kernel error in the process of non-blind image restoration, further improve the effect of non-blind image restoration, and improve the usability of the image.
[0006] To achieve the above objectives, this application provides the following solutions:
[0007] In a first aspect, the present application provides a plug-and-play diffusion sampling method for image restoration with kernel errors, the plug-and-play diffusion sampling method for image restoration with kernel errors comprising:
[0008] Constructing an image non-blind restoration optimization problem model with a blur kernel error; the image non-blind restoration optimization problem model includes a clear image estimation problem model and a blur kernel error estimation problem model;
[0009] With a blurred image and a known blur kernel as input, the image non-blind restoration optimization problem model is solved to obtain a restored clear image.
[0010] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0011] The present application provides a plug-and-play diffusion sampling method for image restoration with kernel errors. First, a non-blind restoration optimization problem model of an image with a blur kernel error is constructed. The non-blind restoration optimization problem model of an image includes a clear image estimation problem model and a blur kernel error estimation problem model. Then, the blur image and a known blur kernel are used as inputs to solve the non-blind restoration optimization problem model of the image to obtain a restored clear image. The present application can construct a non-blind restoration optimization problem model of an image with a blur kernel error. Subsequently, the non-blind restoration optimization problem model of the image is solved to obtain a restored clear image. Thus, the blur kernel error is considered in the process of non-blind restoration of the image, further improving the effect of non-blind restoration of the image and improving the usability of the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0013] Figure 1 This is a diagram of the application environment of a plug-and-play diffusion sampling method for image restoration with kernel errors provided in Example 1 of the present application.
[0014] Figure 2 A flowchart of a plug-and-play diffusion sampling method for image restoration with kernel errors provided in Example 1 of the present application.
[0015] Figure 3 A technical roadmap for a plug-and-play diffusion sampling method for image restoration with kernel errors provided in Example 1 of the present application.
[0016] Figure 4 This is a schematic diagram of an ideal clear image provided in Example 1 of the present application; wherein, Figure 4 (a) in the figure is the ideal clear image of image A; Figure 4 (b) in the figure is the ideal clear image of the B image.
[0017] Figure 5 A schematic diagram of a blurred image provided in Example 1 of the present application; wherein, Figure 5 (a) in the figure is the blurred image of image A; Figure 5(b) in the figure is a blurred image of the B image.
[0018] Figure 6 This is a schematic diagram of restoring a clear image provided in Example 1 of the present application; wherein, Figure 6 (a) is the restored clear image of image A; Figure 6 (b) in the figure is the restored clear image of the B image.
[0019] Figure 7 A schematic diagram showing the comparison of the restoration results of image A provided in Example 1 of the present application; wherein: Figure 7 (a) in the figure is the blurred image of image A; Figure 7 (b) is the restored clear image of image A obtained by using IRCNN; Figure 7 (c) is the restored clear image of image A obtained by using DiffPIR; Figure 7 (d) is a restored clear image of image A obtained by using the method of this embodiment; Figure 7 (e) in the figure is the ideal clear image of image A.
[0020] Figure 8 A schematic diagram showing the comparison of the B image restoration results provided in Example 1 of the present application; wherein: Figure 8 (a) in the figure is the blurred image of the B image; Figure 8 (b) is the restored clear image of the B image obtained by using IRCNN; Figure 8 (c) is the restored clear image of the B image obtained by using DiffPIR; Figure 8 (d) is a restored clear image of the B image obtained by using the method of this embodiment; Figure 8 (e) in the figure is an ideal clear image of the B image. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0022] Example 1
[0023] The plug-and-play diffusion sampling method for image restoration with kernel errors provided in the embodiments of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal communicates with the server through the network. The data storage system can store the data that the server needs to process. The data storage system can be set up separately, integrated on the server, or placed on the cloud or other servers. The terminal can send the blurred image to be processed to the server. After the server receives the blurred image to be processed, for the blurred image to be processed, the server constructs an image non-blind restoration optimization problem model with a blur kernel error, takes the blurred image and the known blur kernel as input, solves the image non-blind restoration optimization problem model, and obtains a restored clear image. The server can feed back the restored clear image of the blurred image to the terminal.
[0024] In addition, in some embodiments, the plug-and-play diffusion sampling method for image restoration with kernel errors can also be implemented independently by a server or a terminal. For example, the terminal can directly perform non-blind image restoration processing on the blurred image to be processed, or the server can obtain the blurred image to be processed from the data storage system and perform non-blind image restoration processing on the blurred image to be processed.
[0025] The terminal may be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The server may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.
[0026] like Figure 2 and Figure 3 As shown, a plug-and-play diffusion sampling method for image restoration with kernel errors is provided. The method is executed by a computer device, and specifically can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 Taking the server in as an example, the plug-and-play diffusion sampling method for image restoration with kernel error includes the following steps:
[0027] Step S1, constructing an image non-blind restoration optimization problem model with blur kernel error; the image non-blind restoration optimization problem model includes a clear image estimation problem model and a blur kernel error estimation problem model.
[0028] Step S2, taking the blurred image and the known blur kernel as input, solving the image non-blind restoration optimization problem model to obtain a restored clear image.
[0029] By implementing the above-mentioned steps S1 to S2, this embodiment can construct an image non-blind restoration optimization problem model with a blur kernel error. Subsequently, the blurred image and the known blur kernel are used as input to solve the image non-blind restoration optimization problem model to obtain a restored clear image. Therefore, the blur kernel error can be considered in the process of image non-blind restoration, thereby further improving the effect of image non-blind restoration and improving the usability of the image.
[0030] Before constructing the image non-blind restoration optimization problem model with blur kernel error, this embodiment first constructs the image non-blind restoration optimization problem model without blur kernel error when blur kernel error is not considered. The construction process specifically includes:
[0031] The image blurring process can be written as:
[0032] y=Kx+n(1)
[0033] In formula (1), y is the blurred image; K is the blur kernel, which is used to simulate the blur effect of the image during the imaging process; x is the ideal clear image; and n is the noise.
[0034] The maximum a posteriori estimation is used to solve the non-blind image restoration problem, that is, to maximize the probability of restoring a clear image when the blurred image is known. The maximum a posteriori estimation of the clear image is:
[0035]
[0036] In formula (2), is the desired clear image; p(x|y) is the distribution that the clear image obeys under the condition of known blurred image.
[0037] Using the Bayesian formula to process formula (2), we can get:
[0038]
[0039] In formula (3), p(y|x) is the likelihood probability of the blurred image y; p(x) is the prior probability of the clear image.
[0040] Combined with the image blurring process, assuming that the noise n has a mean of 0 and a variance of Gaussian noise, then Then formula (3) can be further written as:
[0041]
[0042] In formula (4), is the image blur noise variance; λ is the regularization term parameter, which is set based on experience and experimental comparison; Φ(x) is the regularization term, which represents the unknown logp(x), representing the image prior. Formula (4) is the optimization model of non-blind restoration of images without blur kernel error. Solving formula (4) can obtain a clear image without considering the blur kernel error.
[0043] Referring to the construction process of the above-mentioned image non-blind restoration optimization problem model without blur kernel error, this embodiment further constructs an image non-blind restoration optimization problem model with blur kernel error, and the construction process specifically includes:
[0044] Consider the problem with blur kernel error, that is, the blur kernel is known There is a blur kernel error h between the real blur kernel K. The image blurring process considering the blur kernel error can be written as:
[0045]
[0046] Similar to the process of obtaining equation (4), when considering the blur kernel error, the maximum a posteriori estimate of the clear image is:
[0047]
[0048] In formula (6), is a known blur kernel with blur kernel error h; ν h is the regularization parameter, which is set based on experience and experimental comparison. Solving equation (6) can get the final clear image.
[0049] This embodiment further decomposes formula (6) into:
[0050]
[0051] In formula (7), (a) is the clear image estimation problem model; x is the ideal clear image; is the image blur noise variance; is the known blur kernel; h is the blur kernel error; y is the blurred image; λ is the regularization parameter; Φ(x) is the prior probability of the clear image; (b) is the blur kernel error estimation problem model; ν h is the regularization parameter. By alternately solving (a) and (b), we can obtain the estimation of the clear image and blur kernel error. Formula (7) is the optimization model of the image non-blind restoration with blur kernel error.
[0052] By using variable separation, the optimization model of non-blind image restoration with blur kernel error is decomposed into denoising sub-problems and fidelity term sub-problems. The denoising sub-problem can be solved by inserting a denoiser, and there is no need to train the denoising network, so it can achieve a plug-and-play effect. It can be called a plug-and-play image restoration framework. However, this plug-and-play image restoration framework is limited by the existing denoisers that mainly focus on Gaussian denoising and cannot handle various types of noise well. In recent years, the diffusion model has flourished in the field of image processing. With its pre-training framework that can be used for various tasks, powerful deep generative priors, interpretable generative process and stable training process, it has rapidly generalized a large number of new results. As an emerging generative model, the diffusion model shows great potential in image denoising tasks, which just makes up for the shortcomings of existing denoisers. Therefore, it is very necessary to explore how to use the generative ability of the diffusion model to learn different types of noise and use it in the plug-and-play image restoration framework.
[0053] In this embodiment, the blurred image and the known blur kernel are used as input to solve the image non-blind restoration optimization problem model to obtain a restored clear image, which specifically includes:
[0054] (1) Auxiliary variables are introduced into the clear image estimation problem model, and the clear image estimation problem model with the auxiliary variables is subjected to semi-quadratic splitting to obtain an alternating optimization problem model. The alternating optimization problem model includes a denoising problem model and a fidelity term problem model.
[0055] For the clear image estimation problem model, that is, (a) in equation (7), it only considers the optimization of x, and h is a fixed value relative to this equation. Introducing the auxiliary variable z in (a) in equation (7), we get:
[0056]
[0057] In formula (8), Φ(·) is the prior distribution of the clear image; μ is the penalty parameter.
[0058] Performing semi-quadratic splitting on equation (8) yields the alternating optimization problem model, which is:
[0059]
[0060] In formula (9), (a) is the denoising problem model; z k is the auxiliary variable updated in the kth iteration step; z is the auxiliary variable; is the noise level, λ is the regularization parameter, μ is the penalty parameter; x k is the update of the clear image in the kth iteration; Φ(·) is the prior distribution of the clear image; (b) is the fidelity problem model; x k+1 is the update of the clear image in the k+1th iteration step; x is the ideal clear image; is the image blur noise variance; is the known blur kernel; h is the blur kernel error; y is the blurred image.
[0061] Solving (a) in equation (9) is equivalent to solving for the noise level Clear image of x k For denoising, it can be expressed as Denoiser is an arbitrary denoiser. In this case, the denoiser can be inserted as a module into the solution of non-blind image restoration, without the need to train the denoising network for the non-blind image restoration problem, thereby achieving a plug-and-play effect. Formula (9) is a plug-and-play image restoration framework for solving clear images, which is a plug-and-play image restoration framework with blur kernel error.
[0062] (2) The alternating optimization problem model is processed using a diffusion model to obtain a processed problem model. The diffusion model is used as a denoiser. The processed problem model includes a clear image estimation sub-problem model, a clear image guidance sub-problem model and a noise image estimation formula.
[0063] This embodiment further introduces the plug-and-play diffusion prior into the plug-and-play image restoration framework, assuming that the target is to restore the image from a noise level of The noise-free image is solved from the noisy image, where
[0064] For the diffusion noise process from x0 to x t The variance of the applied noise, is the diffusion model at the t diffusion time step from x t-1 to x t The multiplication notation for the variance of the imposed noise is, Intermediate variable α t =1-β t , β t ∈(0,1) is the diffusion model noise adding process from x t-1 to x t The noise variance imposed, here the denoising subproblem is regarded as denoising the noise-free image from a certain time step of the inverse diffusion process, so let The diffusion model score function is defined as s θ (x t ,t) is a network designed to approximate the score function, ε θ It is a pre-trained unconditional diffusion model denoising network, which can be pre-trained on image datasets according to actual applications.
[0065] The diffusion model is used to process the alternating optimization problem model to obtain the processed problem model, which is:
[0066]
[0067] In formula (10), (a) is the clear image estimation sub-problem model; is the clear image of the tth diffusion time step; z is the auxiliary variable; For the diffusion noise process from x0 to x t The variance of the applied noise; x0 is the noise image at the 0th diffusion time step; x t is the noise image at the t-th diffusion time step; Φ(·) is the prior distribution of the clear image; (b) is the clear image guided sub-problem model; is the estimation of the clear image after the t-th diffusion time step; x is the ideal clear image; y is the blurred image; is the known blur kernel; h is the blur kernel error; μ is the penalty parameter; is the image blur noise variance; (c) is the noise image estimation formula; x t-1 is the noise image at the t-1th diffusion time step; is the diffusion model at the t-1 diffusion time step from x t-2 to x t-1 The symbol for the multiplication of the applied noise variance; x t-2 is the noise image at the t-2 diffusion time step; ζ is the weight parameter, which is set based on experience and experimental comparison; is the prediction noise; t is the noise at the tth diffusion time step.
[0068] In formula (10), (a) indicates that the clear image of the current diffusion time step is estimated by the noise image of the current diffusion time step (also called the inverse diffusion process time step), (b) indicates that the forward model of image blur is used to guide the clear image to be solved, and (c) indicates that the noise transmitted to the next diffusion time step is introduced by introducing the weight parameter ζ. and control ε t and prediction noise , generates a noise image for the next diffusion time step.
[0069] (3) The diffusion model is used to perform a first-order approximate solution to the clear image estimation subproblem model and obtain the clear image estimation formula.
[0070] Using the diffusion model to perform a first-order approximate solution to (a) in equation (9), we obtain:
[0071]
[0072] In formula (11), z k is x k The estimated clear image under the diffusion model in VE form s θ (x k ,k) is a network designed to approximate the score function.
[0073] Then the clear image estimation formula obtained by using the diffusion model to perform a first-order approximate solution to the clear image estimation sub-problem model is:
[0074]
[0075] In formula (12), is a clear image of the tth diffusion time step; is the diffusion model at the t diffusion time step from x t-1 to x t The symbol for the multiplication of the applied noise variance; x t-1 is the noise image at the t-1 diffusion time step; x t is the noise image at the tth diffusion time step; ε θ (x t ,t) is the pre-trained unconditional diffusion model denoising network, and t is the tth diffusion time step.
[0076] (4) The clear image guidance sub-problem model is solved using fast Fourier transform to obtain the clear image guidance formula.
[0077] The analytical solution of the clear image guidance sub-problem model obtained by fast Fourier transform is:
[0078]
[0079] In formula (13), F -1 is the inverse fast Fourier transform; F is the fast Fourier transform; for conjugate of; noise variance ratio Formula (13) is the clear image guidance formula.
[0080] Formula (12), Formula (13) and Formula (c) in Formula (10) are the solutions after the plug-and-play diffusion prior is introduced into the plug-and-play image restoration framework. They are the solutions of the plug-and-play image restoration framework for solving the clear image, realizing the clear image solution based on the plug-and-play diffusion prior.
[0081] (5) The fuzzy kernel error estimation problem model is solved using fast Fourier transform to obtain the fuzzy kernel error estimation formula.
[0082] The analytical solution of the fuzzy kernel error estimation problem model (i.e. (b) in equation (7)) obtained by fast Fourier transform is:
[0083]
[0084] Formula (14) is the solution formula of the fuzzy kernel error estimation problem model for solving the fuzzy kernel error, that is, the fuzzy kernel error estimation formula.
[0085] After solving equations (12), (13) and (c) in equation (10) alternately and iteratively to obtain a clear image, equation (14) is solved to obtain a blur kernel error. At this time, the blur kernel error does not participate in the alternate iterative solution process of the clear image. In order to integrate the blur kernel error into the alternate iterative solution process of the clear image, this embodiment forms a solution step under the approximate relationship on the basis of establishing an approximate relationship.
[0086] In order to simplify the calculation and improve the calculation efficiency, this embodiment introduces the iterative process of blur kernel error into the inverse process of solving the clear image, and uses the sample x at the current diffusion time step. t ~p(x t |y,h) establish an approximate relationship:
[0087]
[0088] When t→0, q t (x0|y,h) converges to p(x0|h,y), then the kernel error optimization process at the tth diffusion time step is:
[0089]
[0090] Therefore, by integrating formula (16) into formula (10), we can obtain:
[0091]
[0092] The last term in equation (17) is (c) in equation (10), which is abbreviated here.
[0093] At this time, the fuzzy kernel error estimation formula of equation (14) is rewritten as:
[0094]
[0095] In formula (18), h (t) is the fuzzy kernel error at the tth diffusion time step; F -1 is the inverse fast Fourier transform; F is the fast Fourier transform; is the clear image at the tth diffusion time step; y is the blurred image; is the known blur kernel; ν his the regularization parameter; is the image blur noise variance.
[0096] At this time, the clear image guidance formula of equation (13) is rewritten as:
[0097]
[0098] In formula (19), is the estimate of the clear image after the t-th diffusion time step; F -1 is the inverse fast Fourier transform; F is the fast Fourier transform; is a known blur kernel; h (t) is the blur kernel error at the tth diffusion time step; y is the blurred image; ρ t is the noise variance ratio; is a clear image of the t-th diffusion time step.
[0099] (6) Taking the blurred image and the known blur kernel as input, the clear image estimation formula, blur kernel error estimation formula, clear image guidance formula and noise image estimation formula are used to perform alternating iterations to obtain the restored clear image.
[0100] initialization From t = T to t = 0, the clear image estimation formula (Equation (12)), the blur kernel error estimation formula (Equation (18)), the clear image guidance formula (Equation (19)) and the noise image estimation formula (Equation (10) (c)) are iteratively solved, and the final obtained x0 and h are the clear image and blur kernel error, thereby completing the solution of the clear image and blur kernel error. The total diffusion time step T can be adjusted according to the actual application settings.
[0101] Taking the blurred image and the known blur kernel as input, the clear image estimation formula, blur kernel error estimation formula, clear image guidance formula and noise image estimation formula are used to perform alternating iterations to obtain the restored clear image, including:
[0102] 1) Taking the noisy image of the current diffusion time step as input, the clear image of the current diffusion time step is calculated using the clear image estimation formula.
[0103] 2) Taking the blurred image, known blur kernel and the clear image of the current diffusion time step as input, the blur kernel error of the current diffusion time step is calculated using the blur kernel error estimation formula.
[0104] 3) Taking the blurred image, the known blur kernel, the clear image of the current diffusion time step and the blur kernel error of the current diffusion time step as input, the clear image guidance formula is used to calculate the estimate of the clear image after the current diffusion time step guidance.
[0105] 4) Taking the estimate of the clear image after the current diffusion time step as input, the noise image of the next diffusion time step is calculated using the noise image estimation formula.
[0106] 5) Determine whether the maximum number of iterations has been reached; if so, the noise image of the next diffusion time step is used as the restored clear image; if not, the number of iterations is increased by 1, and the noise image of the next diffusion time step is used as the noise image of the current diffusion time step.
[0107] This embodiment is mainly used to restore a clear image from a blurred image under the condition that the blur kernel of the blurred image is known to be inaccurate, thereby improving the usability of the image. This embodiment designs a method of inserting a diffusion model as a denoiser into the plug-and-play image restoration framework for the problem of blur kernel error, that is, the known inaccurate blur kernel. This method can be called a non-blind image restoration method based on plug-and-play diffusion sampling. This method starts from the maximum a posteriori estimation of non-blind image restoration with blur kernel error constraints, and uses variable separation to split the optimization problem into sub-problems related to fidelity terms and sub-problems related to denoising. Then, the diffusion model with strong generation and denoising capabilities is inserted as a denoiser into the plug-and-play image restoration framework, which not only improves the utilization rate of effective information in the image, but also improves the visual fidelity of the restored image.
[0108] The plug-and-play image restoration framework that combines the advantages of traditional optimization models and deep learning methods is limited by the fact that existing denoisers cannot handle multiple types of noise well, and there is a blur kernel error between the known blur kernel and the real blur kernel in practical problems. Starting from the fuzzy forward process, this embodiment corrects the existing image non-blind restoration optimization problem model while considering the blur kernel error, and proposes an image non-blind restoration optimization problem model with blur kernel error. Starting from the image non-blind restoration optimization problem model with blur kernel error, it is further split into a plug-and-play image reuse framework, and the diffusion model is used as a denoiser under the plug-and-play image restoration framework after being separated by variables, and inserted into the plug-and-play image reuse framework. Finally, by alternately iterating to solve the clear image and blur kernel error, blur image restoration and blur kernel error estimation are realized. Compared with the existing methods, this embodiment not only has a better restoration effect in terms of vision, but also performs well in improving image quality and fidelity.
[0109] In this embodiment, the ideal clear image is as follows Figure 4 As shown, the blurred image is Figure 5 As shown, the method of this embodiment is used to Figure 5 The blurred image shown in the figure is restored by non-blind image restoration to obtain a clear image. Figure 6 As shown, specifically Figure 4The ideal clear image shown in (a) is added with a mixed blur with a motion length of 20 pixels, a motion angle of 45 degrees, a defocus radius of 1 pixel, and a Gaussian noise with a noise level of 0.1, and the result is Figure 5 The blurred image shown in (a) in FIG. 1 is corrected by the method of this embodiment. Figure 5 The blurred image shown in (a) is restored to obtain Figure 6 The restored clear image shown in (a) is Figure 4 The ideal clear image shown in (b) is added with a mixed blur with a motion length of 10 pixels, a motion angle of 45 degrees, a defocus radius of 1 pixel, and a Gaussian noise with a noise level of 0.1, and the result is Figure 5 The blurred image shown in (b) in FIG. 1 is corrected by using the method of this embodiment. Figure 5 Restore the blurred image shown in (b) to get Figure 6 From the restored clear image shown in (b) in FIG. 1 , it is obvious that the restored clear image obtained by the method of this embodiment has a good effect.
[0110] Using IRCNN (Learning deep CNN denoiser prior for image restoration, image restoration method based on deep denoising prior of convolutional neural network) and DiffPIR (Denoising diffusion models for plug-and-play image restoration, plug-and-play image restoration method based on denoising diffusion model) as comparison methods, the restoration results of image A are compared. Figure 7 As shown in the figure, the B image restoration result is compared with Figure 8 In order to quantitatively evaluate the performance of the above methods, the standard metric of image restoration and the perceptual score of the generated image are combined, and three indicators are selected: Peak Signal to Noise Ratio (PSNR), Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS). The results are shown in Table 1. Figure 7 and Figure 8 The results shown in the figure and the results shown in Table 1 indicate that the method proposed in this embodiment is superior to the comparison method in terms of image quality improvement and image fidelity.
[0111] Table 1 Quantitative results of image restoration
[0112] method PSNR(dB) SSIM LPIPS IRCNN 21.6116 0.7600 0.6064 DiffPIR 21.3032 0.7146 0.5271 This embodiment 22.1876 0.7815 0.5250
[0113] The embodiment of the present application also provides an application scenario, which applies the above-mentioned plug-and-play diffusion sampling method for image restoration with kernel errors. Specifically, the plug-and-play diffusion sampling method for image restoration with kernel errors provided in this embodiment can be applied in the non-blind image restoration scenario. The non-blind image restoration scenario includes an image acquisition link, an image processing link, and an image display link. The image acquisition link is used to acquire a blurred image, the image processing link is used to perform non-blind image restoration on the blurred image to obtain a restored clear image, and the image display link is used to display the restored clear image. The plug-and-play diffusion sampling method for image restoration with kernel errors provided in this embodiment belongs to the image processing link.
[0114] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0115] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A plug-and-play diffusion sampling method for image restoration with kernel errors, characterized in that: The plug-and-play diffusion sampling method for image restoration with kernel errors includes: Constructing an image non-blind restoration optimization problem model with a blur kernel error; the image non-blind restoration optimization problem model includes a clear image estimation problem model and a blur kernel error estimation problem model; With the blurred image and the known blur kernel as input, the image non-blind restoration optimization problem model is solved to obtain a restored clear image.
2. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 1, characterized in that: The image non-blind restoration optimization problem model is: Where (a) is the clear image estimation problem model; x is the ideal clear image; is the image blur noise variance; is the known blur kernel; h is the blur kernel error; y is the blurred image; λ is the regularization parameter; Φ(x) is the prior probability of the clear image; (b) is the blur kernel error estimation problem model; ν h is the regularization parameter.
3. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 1, characterized in that: Taking the blurred image and the known blur kernel as input, solving the image non-blind restoration optimization problem model to obtain a restored clear image specifically includes: Introducing auxiliary variables into the clear image estimation problem model, and performing semi-quadratic splitting on the clear image estimation problem model with the auxiliary variables introduced, to obtain an alternating optimization problem model; the alternating optimization problem model includes a denoising problem model and a fidelity term problem model; The alternating optimization problem model is processed by using a diffusion model to obtain a processed problem model; the diffusion model is used as a denoiser; the processed problem model includes a clear image estimation sub-problem model, a clear image guidance sub-problem model and a noise image estimation formula; Using the diffusion model to perform a first-order approximate solution on the clear image estimation sub-problem model to obtain a clear image estimation formula; The clear image guidance sub-problem model is solved by using fast Fourier transform to obtain a clear image guidance formula; The blur kernel error estimation problem model is solved by using fast Fourier transform to obtain a blur kernel error estimation formula; With a blurred image and a known blur kernel as input, the clear image estimation formula, the blur kernel error estimation formula, the clear image guidance formula and the noise image estimation formula are used to perform alternating iterative solutions to obtain a restored clear image.
4. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 3, characterized in that: The alternating optimization problem model is: Among them, (a) is the denoising problem model; z k is the auxiliary variable updated in the kth iteration step; z is the auxiliary variable; is the noise level, λ is the regularization parameter, μ is the penalty parameter; x k is the update of the clear image in the kth iteration; Φ(·) is the prior distribution of the clear image; (b) is the fidelity problem model; x k+1 is the update of the clear image in the k+1th iteration step; x is the ideal clear image; is the image blur noise variance; is the known blur kernel; h is the blur kernel error; y is the blurred image.
5. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 3, characterized in that: The processed problem model is: Among them, (a) is the clear image estimation sub-problem model; is the clear image of the tth diffusion time step; z is the auxiliary variable; For the diffusion noise process from x0 to x t The variance of the applied noise; x0 is the noise image at the 0th diffusion time step; x t is the noise image at the t-th diffusion time step; Φ(·) is the prior distribution of the clear image; (b) is the clear image guided sub-problem model; is the estimation of the clear image after the t-th diffusion time step; x is the ideal clear image; y is the blurred image; is the known blur kernel; h is the blur kernel error; μ is the penalty parameter; is the image blur noise variance; (c) is the noise image estimation formula; x t-1 is the noise image at the t-1th diffusion time step; is the diffusion model at the t-1 diffusion time step from x t-2 to x t-1 The symbol for the multiplication of the applied noise variance; x t-2 is the noise image of the t-2th diffusion time step; ζ is the weight parameter; is the prediction noise; t is the noise at the tth diffusion time step.
6. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 3, characterized in that: The clear image estimation formula is: in, is a clear image of the tth diffusion time step; is the diffusion model at the t diffusion time step from x t-1 to x t The symbol for the multiplication of the applied noise variance; x t-1 is the noise image at the t-1 diffusion time step; x t is the noise image at the tth diffusion time step; ε θ (x t ,t) is the pre-trained unconditional diffusion model denoising network.
7. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 3, characterized in that: The clear image guidance formula is: in, is the estimate of the clear image after the t-th diffusion time step; F -1 is the inverse fast Fourier transform; F is the fast Fourier transform; is a known blur kernel; h (t) is the blur kernel error at the tth diffusion time step; y is the blurred image; ρ t is the noise variance ratio; is a clear image of the t-th diffusion time step.
8. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 3, characterized in that: The fuzzy kernel error estimation formula is: Among them, h (t) is the fuzzy kernel error at the tth diffusion time step; F -1 is the inverse fast Fourier transform; F is the fast Fourier transform; is the clear image at the tth diffusion time step; y is the blurred image; is the known blur kernel; ν h is the regularization parameter; is the image blur noise variance.
9. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 3, characterized in that: Taking the blurred image and the known blur kernel as input, the clear image estimation formula, the blur kernel error estimation formula, the clear image guidance formula and the noise image estimation formula are alternately iterated to obtain a restored clear image, specifically including: Taking the noise image of the current diffusion time step as input, the clear image of the current diffusion time step is calculated using the clear image estimation formula; Taking the blurred image, the known blur kernel and the clear image of the current diffusion time step as input, the blur kernel error of the current diffusion time step is calculated using the blur kernel error estimation formula; Taking the blurred image, the known blur kernel, the clear image at the current diffusion time step and the blur kernel error at the current diffusion time step as input, the clear image guidance formula is used to calculate the estimation of the clear image after the current diffusion time step guidance; Taking the estimation of the clear image after the current diffusion time step as input, the noise image of the next diffusion time step is calculated using the noise image estimation formula; Determine whether the maximum number of iterations has been reached; if so, the noise image of the next diffusion time step is used as the restored clear image; if not, the number of iterations is increased by 1, and the noise image of the next diffusion time step is used as the noise image of the current diffusion time step.
10. The plug-and-play diffusion sampling method for kernel error image restoration according to claim 5, characterized in that: The calculation formula for the predicted noise is: in, To predict noise; is the diffusion model at the t diffusion time step from x t-1 to x t The symbol for the multiplication of the applied noise variance; x t-1 is the noise image at the t-1 diffusion time step; x t is the noise image at the tth diffusion time step; is the estimate of the clear image after bootstrapping at the t-th diffusion time step.
