Plug-and-play splitting algorithm for blind image restoration

Abstract

The invention belongs to the field of image restoration, namely, an unknown nonlinear or blind restoration task for a degradation model is realized by utilizing a diffusion model based on a plug-and-play splitting algorithm, and particularly relates to a plug-and-play splitting algorithm for blind image restoration. The invention relates to a plug-and-play splitting algorithm for blind image restoration. The application field of the plug-and-play image restoration method is expanded while the deviation between the restoration result of the existing plug-and-play image restoration method and a real image is reduced and the restoration quality is improved by designing an indirect guide strategy. In addition, the invention further provides an unknown degradation model prediction module which is used for predicting an unknown degradation model in a nonlinear or blind task, and therefore it is ensured that the indirect guide strategy can be executed smoothly.

Description

Technical Field

[0001] The invention belongs to the field of image restoration, that is, a diffusion model based on a plug-and-play splitting algorithm is used to realize nonlinear or blind restoration tasks with unknown degradation models, and is specifically a plug-and-play splitting algorithm for blind image restoration. Background Art

[0002] Image restoration is a classic problem in computer vision, which aims to restore the damaged or missing parts of the image content. Among them, blind image restoration has become one of the core challenges in this field because it needs to complete high-quality reconstruction tasks without reference information. The main difficulties are: 1) Unlike common neural network-based training and learning methods, it is often impossible to obtain good image references for the area to be restored during blind image restoration. This makes it difficult to directly apply traditional supervised learning methods, which puts higher requirements on the design of algorithms; 2) There are many types of pictures in the real world, and each type has its own unique structure and characteristics. How to design a versatile, efficient and flexible solution for blind image restoration has become a major problem; 3) As an inherently ill-posed problem, blind image restoration must not only ensure that the repaired area looks natural and harmonious, but also reflect the information of the real scene as accurately as possible to achieve visual consistency and authenticity. Therefore, blind image restoration has always been a very challenging task.

[0003] At present, the research methods on image restoration can be roughly divided into three categories: traditional model-driven methods, data-driven deep learning methods, and plug-and-play image restoration methods. Traditional model-driven methods mostly rely on specific degradation models and regularization terms to constrain the solution process to obtain reasonable restoration results. For example, in traditional machine learning algorithms, L1 norm, L2 norm, total variation, etc. are usually selected as regularization terms. However, the design of regularization terms usually depends on prior knowledge, and the parameters that control the strength of regularization terms are also very sensitive and difficult to select. In addition, the robustness of such methods is poor and difficult to apply to practical engineering. Data-driven deep learning methods often design appropriate network models and learn data distribution from a large amount of paired image data. By designing and training networks separately for different restoration tasks, better restoration effects can be achieved. However, this method is highly dependent on large-scale and high-quality training data sets, which makes it unsuitable for blind image restoration tasks that lack reference information. The plug-and-play image restoration method allows users to use trained neural networks including diffusion models instead of embedding prior knowledge into prior terms, so that the restoration process does not require additional training. For different restoration tasks, we only need to use different degradation models. The plug-and-play image restoration method can use variable separation optimization algorithms such as the semi-quadratic splitting method to alternately optimize the degradation model and the regularization term split, thereby efficiently and flexibly handling a variety of restoration tasks.

[0004] From the perspective of processing ideas, the research focus of existing plug-and-play image restoration methods is mainly on the optimization of prior terms, while the improvement of data terms is relatively small. However, the data term is a key factor in determining which restoration tasks this algorithm can handle. In addition, the guidance strategy of the image restoration algorithm has an important impact on whether the final restoration effect conforms to the real image. Therefore, in order to effectively complete blind image restoration tasks such as low-light image enhancement, comprehensive consideration and optimization should be carried out from the perspectives of constraints and guidance strategies. Summary of the invention

[0005] Aiming at the problem that the current plug-and-play image restoration method cannot handle the problem that the image degradation model is difficult to accurately model in blind image restoration tasks such as low-light image enhancement, the present invention proposes a plug-and-play splitting algorithm for blind image restoration, which is used to expand the application field of the plug-and-play image restoration method and improve the restoration quality.

[0006] The present invention is implemented by adopting the following technical scheme: a plug-and-play splitting algorithm for blind image restoration, comprising the following steps:

[0007] Step 1: Input the real degraded image y;

[0008] Step 2: Randomly sample a noise image X from a standard normal distribution T;

[0009] Step 3: Start the iterative process of repair, the iterative process is from T to 1, and the number of intermediate iterations is represented by t;

[0010] Step 4: Transform the noisy image X T Input into the denoising diffusion probability model to perform a denoising process and obtain the solution x of the prior sub-problem T ;

[0011] Step 5: Using the formula Solve the intermediate auxiliary variable m of the solution to the priori subproblem obtained in step 4 T ; In the formula, is the parameter in the denoising diffusion probability model, ε θ (x T ,T) is the function approximator in the denoising diffusion probability model, and the input is x T ,T, the output is Gaussian noise;

[0012] Step 6: Use the degradation model m predicted by the degradation model prediction module lr =fm T +B will be the intermediate auxiliary variable m T Degenerates into a degraded image m lr , f and B represent the parameters of the predicted degradation model;

[0013] Step 7: Calculate the real degraded image y and the degraded image m of the intermediate auxiliary variable lr The MSE loss between lr ));

[0014] Step 8: Using the loss obtained in step 7, calculate the guidance condition Where s is the guiding scale;

[0015] Step 9: Get the conditional distribution based on the bootstrap condition obtained in step 8 Here, μ and ∑ represent the mean and variance respectively, and their specific definitions refer to Formula 2. Then, a pair of images X is randomly sampled from the conditional distribution using the denoising diffusion probability model. T-1 ; This round of iteration ends, return to step 4 to start the next round of iteration, and X T-1 As the input of the next round of iteration, it is input into the denoising diffusion probability model for a denoising process to obtain the solution x of the next round of prior sub-problem T-1 , by the solution x of the prior subproblem T-1 Then calculate the intermediate auxiliary variable m T-1 , and then by the intermediate auxiliary variable m T-1 Calculate the degraded image m lr , calculate the real degraded image y and the degraded image m of the intermediate auxiliary variablelr After the MSE loss between the two, we get the conditional distribution Use the denoising diffusion probability model to randomly sample an image X in the conditional distribution T-2 , this round of iteration ends, return to step 4 to start the next round of iteration, and X T-2 As input for the next iteration;

[0016] Step 10: End the iterative process and output the repaired image x0.

[0017] As shown in steps 5 and 6 above, the traditional numerical optimization method that relies on closed numerical solutions is abandoned, and the real degraded image is used to solve the data subproblem and the solution of the data subproblem is used to indirectly guide the generation of the solution of the prior subproblem. This indirect guidance strategy can not only effectively complete the image restoration task, but also effectively reduce the negative impact of the deviation generated in the calculation process of the closed numerical solution on the final result.

[0018] In nonlinear or blind tasks, the degradation model is usually unknown. Therefore, in order to ensure that the indirect guidance strategy can proceed smoothly, the present invention uses y=fx+B to simulate the complex degradation process in order to solve the problem of unknown degradation model in nonlinear or blind tasks.

[0019] In order to solve the problem that the current plug-and-play image restoration method cannot handle the image degradation model in blind image restoration tasks such as low-light image enhancement, the present invention proposes a plug-and-play splitting algorithm for blind image restoration. By designing an indirect guidance strategy, the deviation between the restoration result of the current plug-and-play image restoration method and the real image is reduced, the restoration quality is improved, and the application field of the plug-and-play image restoration method is expanded. In addition, the present invention also proposes an unknown degradation model prediction module for predicting unknown degradation models in nonlinear or blind tasks, thereby ensuring that the indirect guidance strategy proposed by the present invention can be smoothly executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a diagram of the network model architecture of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments:

[0022] Reference Figure 1In this embodiment, a plug-and-play splitting algorithm for blind image restoration combines the plug-and-play image restoration algorithm with a denoising diffusion probability model to handle nonlinear or blind image restoration tasks such as low-light image enhancement. The steps are as follows: construct an indirect guidance module and an unknown degradation model prediction module. The current plug-and-play image restoration method often solves the data subproblem by using a closed numerical solution that depends on the degradation model, and guides the generation of the final solution through a numerical guidance strategy to complete the image restoration task. However, such methods are often also constrained by closed numerical solutions, that is, they are only applicable to situations where there are true or approximate closed numerical solutions. For nonlinear or blind image restoration tasks where closed numerical solutions do not exist, these methods are powerless. In addition, the closed numerical solution used to solve the data subproblem is derived based on the solution of the priori subproblem. However, the solution of the priori subproblem has a certain randomness under the influence of the diffusion model, which leads to an inevitable deviation between the closed numerical solution and the true solution. This deviation will continue to accumulate with the transmission of the numerical solution, and eventually affect the restoration result, making it significantly different from the original image.

[0023] Therefore, the indirect guidance module proposed in the present invention abandons the traditional numerical optimization method that relies on closed numerical solutions, and instead adopts real degraded images to solve data subproblems and uses the solutions of data subproblems to indirectly guide the generation of solutions to prior subproblems, thereby effectively reducing the negative impact of deviations generated in the calculation process of closed numerical solutions on the final results while completing the image restoration task.

[0024] In order to guide and reconstruct the corresponding clean image x through the real degraded image y, thereby solving the data sub-problem, the original inverse denoising distribution p in the Denoising Diffusion Probabilistic Model (DDPM) is transformed into θ (x t-1 |x t ) is transformed into a conditional distribution p θ (x t-1 |x t ,y), the specific derivation process is as follows:

[0025]

[0026] In the formula, for simplicity, u is used to represent the distribution p θ (x t-1 |x t ) and the distribution p(y|x t ) is the result of the product. The constant generated in the product process is added to K1 to obtain K2. K1 and K2 are both constants. The distribution of u is as follows:

[0027] u~N(μ θ (x t ,t)+∑g,Σ) (2)

[0028] in, Σ=Σ θ (x t ),Σ θ (x t ) is the variance estimator, with input x t , the output is the corresponding variance, μ θ (x t ,t) is the mean estimator, the input is x t and t, the output is the corresponding mean, θ represents the model parameter, p θ (x t-1 |x t ) represents the inverse denoising distribution in DDPM, p(y|x t ) is to convert x t The probability that the high-quality image obtained after denoising is consistent with the true degraded image y.

[0029] p(y|x t An approximate solution of ) is as follows:

[0030]

[0031] Where E is an image distance metric, Z is a normalization factor, and s is a scaling factor that controls the magnitude of the guidance.

[0032] That is, (3) encourages x after the degradation model T(·) t It is consistent with the real degraded image y. Calculate the gradient on both sides of (3):

[0033] logp(y|x t )=-logZ-sE(T(x t ),y) (4)

[0034]

[0035] The distance metric E is measured using the mean square error (MSE).

[0036] In summary, by inverse denoising distribution p θ (x t-1 |x t ) is added to the mean It can be transformed into the conditional distribution p θ (x t-1 |x t,y), thereby achieving the purpose of solving the data sub-problem with the help of the real degraded image y.

[0037] In fact, in the specific solution process of DDPM, the inverse denoising process usually starts from x t An auxiliary variable is predicted, and then this auxiliary variable m and x are used t Let's sample the next image together. The solution formula for the auxiliary variable is as follows:

[0038]

[0039] Therefore, in order to achieve a better repair effect, the present invention splits the data sub-problem into equation (5) and equation (6), and processes the two parts separately, that is, firstly, the corresponding intermediate variable m is solved using the input of the data sub-problem, and then the degradation model T(·) is used to degenerate m into mlr , and finally use the real degraded image y and mlr Calculate the distance loss to guide the generation of the solution to the next prior sub-problem.

[0040] In nonlinear or blind tasks, the degradation model is usually unknown. Therefore, in order to ensure that the above guidance strategy can be carried out smoothly, the present invention adopts a simple and effective degradation model to simulate the complex degradation process in nonlinear or blind tasks. The specific definition of the introduced model is:

[0041] y=fx+B (7)

[0042] Among them, f represents the light factor, which is a scalar; B represents the light mask, which is a vector with the same dimension as x.

[0043] When x and y have the same size, the transformation between any damaged image and its corresponding high-quality image can be captured by f and B. Therefore, the present invention can use this model to simulate complex unknown degradation models.

[0044] The input of the above indirect guidance module is the solution x of the prior subproblem t , the output is an MSE loss. The process is as follows: Solve for the intermediate auxiliary variables The intermediate variables are degraded by m using the degradation model predicted by the degradation model prediction module. lr =fm t +B, and finally use the real degraded image and the degraded intermediate auxiliary variable to calculate the MSE loss l = E(y,m lr )), where f and B are randomly initialized at the beginning of the repair process and then continuously optimized during the repair process.

[0045] The input of the above degradation model prediction module is the intermediate auxiliary variable m t , the output is the degraded image m of the intermediate auxiliary variable lr Since f and B are unknown parameters of the degradation model, for each individual corrupted image, f and B are first randomly initialized and then optimized synchronously during the reverse sampling process of DDPM.

[0046] In the above-mentioned blind image restoration method based on the plug-and-play splitting algorithm, the input degraded images for testing are randomly selected from the public datasets ImageNet, FFHQ, LOL, and VE_LOL_L, where the image size in the ImageNet and FFHQ datasets is 256×256, and the image size in the LOL and VE_LOL_L datasets is 600×400.

Claims

1. A plug-and-play splitting algorithm for blind image restoration, characterized by: The following steps are involved: Step 1: Input the real degraded image y; Step 2: Randomly sample a noise image X from a standard normal distribution T ; Step 3: Start the iterative process of repair, the iterative process is from T to 1, and the number of intermediate iterations is represented by t; Step 4: Transform the noisy image X T Input into the denoising diffusion probability model to perform a denoising process and obtain the solution x of the prior sub-problem T ; Step 5: Using the formula Solve the intermediate auxiliary variable m of the solution to the prior subproblem obtained in step 4 T ; In the formula, is the parameter in the denoising diffusion probability model, ε θ (x T ,T) is the function approximator in the denoising diffusion probability model, and the input is x T ,T, the output is Gaussian noise; Step 6: Use the degradation model m predicted by the degradation model prediction module lr =fm T +B will be the intermediate auxiliary variable m T Degenerates into a degraded image m lr , f and B represent the parameters of the predicted degradation model; Step 7: Calculate the real degraded image y and the degraded image m of the intermediate auxiliary variable lr The MSE loss between lr )); Step 8: Using the loss obtained in step 7, calculate the guidance condition Where s is the guiding scale; Step 9: Get the conditional distribution based on the bootstrap condition obtained in step 8 Here, μ and ∑ represent mean and variance respectively; Afterwards, a pair of images X is randomly sampled from the conditional distribution using the denoising diffusion probability model T-1 ; This round of iteration ends, return to step 4 to start the next round of iteration, and X T-1 As the input of the next round of iteration, it is input into the denoising diffusion probability model for a denoising process to obtain the solution x of the next round of prior sub-problem T-1 , by the solution x of the prior subproblem T-1 Then calculate the intermediate auxiliary variable m T-1 , and then by the intermediate auxiliary variable m T-1 Calculate the degraded image m lr , calculate the real degraded image y and the degraded image m of the intermediate auxiliary variable lr After the MSE loss between the two, we get the conditional distribution Use the denoising diffusion probability model to randomly sample an image X in the conditional distribution T-2 , this round of iteration ends, return to step 4 to start the next round of iteration, and X T-2 As input for the next iteration; Step 10: End the iterative process and output the repaired image x0.

2. The plug-and-play splitting algorithm for blind image restoration according to claim 1, characterized in that: The input real degraded image y for testing is randomly selected from the public datasets ImageNet, FFHQ, LOL, and VE_LOL_L, where the image size in the ImageNet and FFHQ datasets is 256×256, and the image size in the LOL and VE_LOL_L datasets is 600×400.

Images