A plug-and-play splitting algorithm for blind image inpainting

Abstract

The present invention belongs to the field of image restoration, namely, utilizing a diffusion model based on a plug-and-play splitting algorithm to implement nonlinear or blind restoration tasks for unknown degradation models. Specifically, it is a plug-and-play splitting algorithm for blind image restoration. A plug-and-play splitting algorithm for blind image restoration, by designing an indirect guidance strategy, reduces the deviation between the restoration results of the current plug-and-play image restoration method and the real image, improves the restoration quality, and expands the application field of the plug-and-play image restoration method. In addition, the present invention also proposes an unknown degradation model prediction module for predicting unknown degradation models in nonlinear or blind tasks, thereby ensuring that the indirect guidance strategy proposed by the present invention can be smoothly executed.

Description

Technical Field

[0001] The present invention belongs to the field of image restoration, that is, using a diffusion model based on a plug-and-play splitting algorithm to achieve nonlinear or blind restoration tasks with unknown degradation models, specifically a plug-and-play splitting algorithm for blind image restoration. Background Art

[0002] Image restoration is a classic problem in computer vision, which aims to restore the image content of damaged or missing parts. Among them, blind image restoration has become one of the core challenges in this field because it needs to complete high-quality reconstruction tasks without reference information. The main difficulties are: 1) Unlike common neural network-based training and learning methods, it is often impossible to obtain good image references for the area to be restored during blind image restoration. This makes it difficult to directly apply traditional supervised learning methods, which puts higher requirements on the design of algorithms; 2) There are many types of pictures in the real world, and each type has its own unique structure and characteristics. How to design a versatile, efficient and flexible solution for blind image restoration has become a major problem; 3) As an inherently ill-posed problem, blind image restoration must not only ensure that the repaired area looks natural and harmonious, but also reflect the information of the real scene as accurately as possible to achieve visual consistency and authenticity. Therefore, blind image restoration has always been an extremely challenging task.

[0003] Currently, research methods for image inpainting can be broadly categorized into three categories: traditional model-driven methods, data-driven deep learning methods, and plug-and-play image inpainting methods. Traditional model-driven methods often rely on specific degradation models and regularization terms to constrain the solution process and achieve reasonable inpainting results. For example, in traditional machine learning algorithms, the L1 norm, L2 norm, total variation, and other regularization terms are commonly chosen. However, the design of regularization terms often relies on prior knowledge, and the parameters controlling the strength of the regularization term are sensitive and difficult to select. Furthermore, these methods suffer from poor robustness, making them difficult to apply in practical engineering. Data-driven deep learning methods often design appropriate network models and learn the data distribution from a large amount of paired image data. Network design and training are performed separately for different inpainting tasks, achieving good inpainting results. However, this approach relies heavily on large, high-quality training datasets, making it unsuitable for blind image inpainting tasks that lack reference information. Plug-and-play image inpainting methods, on the other hand, allow users to embed trained neural networks, including diffusion models, into the prior terms instead of prior knowledge, eliminating the need for additional training. For different restoration tasks, only different degradation models need to be used. The plug-and-play image restoration method can use variable separation optimization algorithms such as the semi-quadratic splitting method to alternately optimize the degradation model and the regularization term split, thereby efficiently and flexibly handling various restoration tasks.

[0004] Existing plug-and-play image inpainting methods primarily focus on optimizing the prior, while relatively little effort has been devoted to improving the data term. However, the data term is a key factor in determining which inpainting tasks such algorithms can handle. Furthermore, the guidance strategy of an image inpainting algorithm significantly influences whether the final inpainting result matches the true image. Therefore, to effectively complete blind image inpainting tasks such as low-light image enhancement, a comprehensive consideration and optimization approach should be considered from both the perspectives of constraints and guidance strategies. Summary of the Invention

[0005] Aiming at the problem that current plug-and-play image restoration methods are unable to handle the problem that image degradation models are difficult to accurately model in blind image restoration tasks such as low-light image enhancement, this paper proposes a plug-and-play splitting algorithm for blind image restoration, which is used to expand the application field of plug-and-play image restoration methods and improve the restoration quality.

[0006] The present invention is implemented by adopting the following technical solution: a plug-and-play splitting algorithm for blind image restoration, comprising the following steps:

[0007] Step 1: Input the real degraded image y;

[0008] Step 2: Randomly sample a noise image X from a standard normal distribution T;

[0009] Step 3: Start the iterative process of repair, the iterative process is from T to 1, and the number of intermediate iterations is represented by t;

[0010] Step 4: Convert the noisy image X T Input into the denoising diffusion probability model to perform a one-step denoising process and obtain the solution x of the prior sub-problem T ;

[0011] Step 5: Using the formula Solve the intermediate auxiliary variable m of the prior subproblem solution obtained in step 4 T Where, is the parameter in the denoising diffusion probability model, ε θ (x T ,T) is the function approximator in the denoising diffusion probability model, and the input is x T ,T, the output is Gaussian noise;

[0012] Step 6: Use the degradation model m predicted by the degradation model prediction module lr =fm T +B will be the intermediate auxiliary variable m T Degenerates into a degraded image m lr , f and B represent the parameters of the predicted degradation model;

[0013] Step 7: Calculate the true degraded image y and the degraded image m of the intermediate auxiliary variable lr The MSE loss between lr ));

[0014] Step 8: Using the loss obtained in step 7, calculate the guidance condition Where s is the guiding scale;

[0015] Step 9: Get the conditional distribution based on the bootstrap condition obtained in step 8 Here μ and ∑ represent the mean and variance respectively. The specific definition is given in Formula 2. Then, a random image X is sampled from the conditional distribution using the denoising diffusion probability model. T-1 ; This round of iteration ends, return to step 4 to start the next round of iteration, and X T-1 As the input of the next round of iteration, it is input into the denoising diffusion probability model for a one-step denoising process to obtain the solution x of the next round of prior sub-problem T-1 , by the solution x of the prior subproblem T-1 Then calculate the intermediate auxiliary variable m T-1 , and then by the intermediate auxiliary variable m T-1 Calculate the degraded image m lr , calculate the real degraded image y and the degraded image m of the intermediate auxiliary variablelr After the MSE loss between the two, the conditional distribution is obtained. Use the denoising diffusion probability model to randomly sample an image X in the conditional distribution T-2 , this round of iteration ends, return to step 4 to start the next round of iteration, and X T-2 As input for the next iteration;

[0016] Step 10: End the iterative process and output the repaired image x0.

[0017] As shown in steps 5 and 6 above, traditional numerical optimization methods that rely on closed-form numerical solutions are abandoned. Instead, real degraded images are used to solve the data subproblem, and the solution to the data subproblem is used to indirectly guide the generation of the solution to the prior subproblem. This indirect guidance strategy not only effectively completes the image restoration task, but also effectively reduces the negative impact of deviations generated during the calculation of the closed-form numerical solution on the final result.

[0018] In nonlinear or blind tasks, the degradation model is usually unknown. Therefore, to ensure that the indirect guidance strategy can proceed smoothly, this paper uses y=fx+B to simulate the complex degradation process to address the problem of unknown degradation model in nonlinear or blind tasks.

[0019] To address the inability of existing plug-and-play image restoration methods to accurately model image degradation in blind image restoration tasks, such as low-light image enhancement, this paper proposes a plug-and-play splitting algorithm for blind image restoration. By designing an indirect guidance strategy, this algorithm reduces the deviation between the restoration results of existing plug-and-play image restoration methods and the true image, improving restoration quality while expanding the application of plug-and-play image restoration methods. Furthermore, this paper proposes an unknown degradation model prediction module for predicting unknown degradation models in nonlinear or blind tasks, thereby ensuring the smooth implementation of the proposed indirect guidance strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a diagram of the network model architecture of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments:

[0022] Reference Figure 1In this embodiment, a plug-and-play splitting algorithm for blind image restoration combines a plug-and-play image restoration algorithm with a denoising diffusion probability model to address nonlinear or blind image restoration tasks such as low-light image enhancement. The steps are as follows: construct an indirect guidance module and an unknown degradation model prediction module. Existing plug-and-play image restoration methods often solve the data subproblem using closed numerical solutions that rely on the degradation model and guide the generation of the final solution through numerical guidance strategies to complete the image restoration task. However, such methods are often constrained by the requirement for closed numerical solutions, meaning they are only applicable when true or approximate closed numerical solutions exist. These methods are incapable of solving nonlinear or blind image restoration tasks where closed numerical solutions do not exist. Furthermore, the closed numerical solutions used to solve the data subproblem are derived from the solutions of the prior subproblems. However, the solutions of the prior subproblems have a certain degree of randomness due to the influence of the diffusion model, which inevitably leads to deviations between the closed numerical solutions and the true solutions. This deviation accumulates as the numerical solutions are transferred, ultimately affecting the restoration results and causing significant differences from the original image.

[0023] Therefore, the indirect guidance module proposed in the present invention abandons the traditional numerical optimization method that relies on closed numerical solutions, and instead adopts real degraded images to solve the data subproblem and uses the solution of the data subproblem to indirectly guide the generation of the solution of the prior subproblem, thereby completing the image restoration task while effectively reducing the negative impact of the deviation generated in the calculation process of the closed numerical solution on the final result.

[0024] In order to guide and reconstruct the corresponding clean image x through the real degraded image y, thereby solving the data sub-problem, the original inverse denoising distribution p in the Denoising Diffusion Probabilistic Model (DDPM) is converted into θ (x t-1 |x t ) is transformed into a conditional distribution p θ (x t-1 |x t ,y), the specific derivation process is as follows:

[0025]

[0026] In the formula, for simplicity, u is used to represent the distribution p θ (x t-1 |x t ) and the distribution p(y|x t ) is the product of the constant generated during the multiplication and K1, thus obtaining K2, where both K1 and K2 are constants. The distribution of u is as follows:

[0027] u~N(μ θ (x t ,t)+∑g,Σ) (2)

[0028] in, Σ=Σ θ (x t ),Σ θ (x t ) is the variance estimator, the input is x t , the output is the corresponding variance, μ θ (x t ,t) is the mean estimator, the input is x t and t, the output is the corresponding mean, θ represents the model parameter, p θ (x t-1 |x t ) represents the inverse denoising distribution in DDPM, p(y|x t ) is x t The probability that the high-quality image obtained after denoising is consistent with the true degraded image y.

[0029] p(y|x t An approximate solution of ) is as follows:

[0030]

[0031] Where E is an image distance metric, Z is a normalization factor, and s is a scaling factor that controls the magnitude of the guidance.

[0032] That is, (3) encourages x after the degradation model T(·) t is consistent with the real degraded image y. Calculate the gradient on both sides of (3):

[0033] logp(y|x t )=-logZ-sE(T(x t ),y) (4)

[0034]

[0035] The distance metric E is measured using the mean square error (MSE).

[0036] In summary, by inverse denoising distribution p θ (x t-1 |x t ) is added to the mean It can be transformed into the conditional distribution p θ (x t-1 |x t,y), thereby achieving the purpose of solving the data sub-problem with the help of the real degraded image y.

[0037] In fact, in the specific solution process of DDPM, the inverse denoising process usually starts from x t Predict an auxiliary variable, and then use this auxiliary variable m and x t Let's sample the next image together. The solution formula for the auxiliary variable is as follows:

[0038]

[0039] Therefore, in order to achieve a better repair effect, the present invention splits the data sub-problem into equations (5) and (6), and processes these two parts separately. First, the corresponding intermediate variable m is solved using the input of the data sub-problem, and then the degradation model T(·) is used to degenerate m into mlr , and finally use the real degraded image y and mlr Calculate the distance loss to guide the generation of the solution to the next prior sub-problem.

[0040] In nonlinear or blind tasks, the degradation model is usually unknown. Therefore, in order to ensure that the above guidance strategy can be carried out smoothly, this paper adopts a simple and effective degradation model to simulate the complex degradation process. The specific definition of the introduced model is:

[0041] y=fx+B (7)

[0042] Among them, f represents the light factor, which is a scalar; B represents the light mask, which is a vector with the same dimension as x.

[0043] When x and y have the same size, the transformation between any damaged image and its corresponding high-quality image can be captured by f and B. Therefore, the present invention can use this model to simulate complex unknown degradation models.

[0044] The input of the above indirect guidance module is the solution x of the prior sub-problem t , the output is an MSE loss. The process is as follows: solve the intermediate auxiliary variables Use the degradation model predicted by the degradation model prediction module to degrade the intermediate variables by m lr =fm t +B, and finally use the real degraded image and the degraded intermediate auxiliary variable to calculate the MSE loss l=E(y,m lr )), where f and B are randomly initialized at the beginning of the repair process and then continuously optimized during the repair process.

[0045] The input of the above degradation model prediction module is the intermediate auxiliary variable m t , the output is the degraded image m of the intermediate auxiliary variable lr Since f and B are unknown parameters of the degradation model, for each individual corrupted image, f and B are first randomly initialized and then optimized synchronously during the reverse sampling process of DDPM.

[0046] For the blind image restoration method based on the plug-and-play splitting algorithm, the degraded images used for testing are randomly selected from the public datasets ImageNet, FFHQ, LOL, and VE_LOL_L. The image size in the ImageNet and FFHQ datasets is 256×256, and the image size in the LOL and VE_LOL_L datasets is 600×400.

Claims

1. A plug-and-play splitting algorithm for blind image restoration, characterized by: The following steps are involved: Step 1: Input the real degraded image y; Step 2: Randomly sample a noise image X from a standard normal distribution T ; Step 3: Start the iterative process of repair, the iterative process is from T to 1, and the number of intermediate iterations is represented by t; Step 4: Convert the noisy image X T Input into the denoising diffusion probability model to perform a one-step denoising process and obtain the solution x of the prior sub-problem T ; Step 5: Using the formula Solve the intermediate auxiliary variable m of the prior subproblem solution obtained in step 4 T Where, is the parameter in the denoising diffusion probability model, ε θ (x T ,T) is the function approximator in the denoising diffusion probability model, and the input is x T ,T, the output is Gaussian noise; Step 6: Use the degradation model m predicted by the degradation model prediction module lr =fm T +B will be the intermediate auxiliary variable m T Degenerates into a degraded image m lr , f and B represent the parameters of the predicted degradation model; Step 7: Calculate the true degraded image y and the degraded image m of the intermediate auxiliary variable lr The MSE loss between lr )); Step 8: Using the loss obtained in step 7, calculate the guidance condition Where s is the guiding scale; Step 9: Get the conditional distribution based on the bootstrap condition obtained in step 8 Here μ and ∑ represent mean and variance respectively; Afterwards, a pair of images X is randomly sampled from the conditional distribution using the denoising diffusion probability model. T-1 ; This round of iteration ends, return to step 4 to start the next round of iteration, and X T-1 As the input of the next round of iteration, it is input into the denoising diffusion probability model for a one-step denoising process to obtain the solution x of the next round of prior sub-problem T-1 , by the solution x of the prior subproblem T-1 Then calculate the intermediate auxiliary variable m T-1 , and then by the intermediate auxiliary variable m T-1 Calculate the degraded image m lr , calculate the real degraded image y and the degraded image m of the intermediate auxiliary variable lr After the MSE loss between the two, the conditional distribution is obtained. Use the denoising diffusion probability model to randomly sample an image X in the conditional distribution T-2 , this round of iteration ends, return to step 4 to start the next round of iteration, and X T-2 As input for the next iteration; Step 10: End the iterative process and output the repaired image x0.

2. The plug-and-play splitting algorithm for blind image restoration according to claim 1, characterized in that: The input real degraded image y for testing is randomly selected from the public datasets ImageNet, FFHQ, LOL, and VE_LOL_L. The image size in the ImageNet and FFHQ datasets is 256×256, and the image size in the LOL and VE_LOL_L datasets is 600×400.

Images