Plug-and-play image restoration method, device, equipment and medium based on large model

Abstract

The present application relates to a plug-and-play image restoration method, device, equipment and medium based on a large model. The method comprises: after the target image is brought into the objective function of the blind image restoration problem, it is decomposed into two sub-problems of solving the high-quality image and the degradation matrix, and the two sub-problems are solved by alternating iterative solution. In each iterative solution process, the degradation matrix solution sub-problem is first solved by a neural network, and then the diffusion large model and the intermediate degradation matrix are used to solve the high-quality image solution sub-problem, and the intermediate high-quality image of the current iteration calculation is obtained, until the number of iterations meets the preset number, then the intermediate high-quality image obtained by the current iteration calculation is the high-quality image after the target image is restored. The use of this method can effectively improve the image restoration performance and efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a plug-and-play image restoration method, device, equipment and medium based on a large model. Background Art

[0002] Image restoration methods play a vital role in various tasks such as target detection, recognition and classification. Due to the limitations of objective imaging conditions such as lighting environment, equipment accuracy, and the reality that the target is tiny, small, dark, and weak, most of the acquired optical images often have problems such as low resolution and image degradation. At present, when the degradation process is known, that is, under the condition of non-blind image super-resolution, the existing image restoration algorithms are relatively mature. However, in practical applications, the degradation model is often unknown, that is, facing the problem of blind image restoration. As a result, the problem is highly ill-conditioned and non-convex, making it difficult to solve directly.

[0003] Existing image restoration methods are mainly divided into two categories: model-based methods and learning-based methods. Model-based methods have clear and explicit mathematical expressions and use alternating iterative methods to optimize and solve image restoration problems. Their advantages are strong algorithm interpretability, good algorithm generalization performance, plug-and-play in multiple scenarios, and good use of prior information of data. However, since the problem itself is highly ill-conditioned and non-convex, model-based methods are prone to fall into local optimality. In practical applications, they have the following main shortcomings: (a) They are prone to fall into local optimality, and convergence is not guaranteed under highly non-convex and ill-conditioned conditions; (b) The hyperparameters of the algorithm need to be manually adjusted; (c) The solution process involves matrix inversion operations, which has high computational complexity.

[0004] Learning-based methods often model the degradation model by carefully designing an end-to-end black box network. After the network is fully trained with a large amount of paired data, it can achieve excellent image restoration performance on test data similar to the training data. Among them, methods based on large models usually design larger models, use more model parameters and more training data to obtain better image restoration performance. However, learning-based methods have the following four defects: (a) the algorithm has poor interpretability; (b) the algorithm requires a large amount of paired data for training, resulting in strong data dependence; (c) the algorithm requires a carefully designed network structure; (d) the model requires a lot of time to train. These shortcomings limit the practical application of learning-based methods, especially those based on large models, in image restoration problems. Summary of the invention

[0005] Based on this, it is necessary to provide a large model-based plug-and-play image restoration method, device, equipment and medium that can improve restoration performance and efficiency in response to the above technical problems.

[0006] A plug-and-play image restoration method based on a large model, the method comprising:

[0007] Acquire a target image to be restored;

[0008] After the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix;

[0009] The two sub-problems are solved by an alternating iterative solution method. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0010] The machine learning model and the intermediate degradation matrix are used to solve the high-quality image solving subproblem to obtain an intermediate high-quality image of the current iterative calculation;

[0011] Until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

[0012] In one embodiment, when a neural network is used to solve the degradation matrix solution subproblem, the optimization problem of the degradation matrix is ​​converted into a parameter optimization problem of the neural network. The converted degradation matrix solution subproblem is expressed as:

[0013] ;

[0014] In the above formula, represents the target image to be restored, represents a neural network, represents the network parameters, represents the F-norm calculation, represents the weight, represents the high-quality image to be solved, represents the degradation matrix predicted by the neural network, Degenerate matrix The prior term of .

[0015] In one embodiment, in each iterative calculation, when optimizing the parameters of the neural network, the update of the neural network is guided by the random prior of the degradation matrix, including:

[0016] The random degradation matrix generated by Monte Carlo random sampling is used as the label;

[0017] A loss function is calculated according to the label and the predicted degradation matrix output by the neural network, and the network parameters of the neural network are updated according to the calculation results.

[0018] In one embodiment, in each iterative calculation, when optimizing the parameters of the neural network, the neural network is also updated according to the data fidelity loss function, including:

[0019] Performing image degradation on the intermediate high-quality image obtained by the previous iterative calculation according to the predicted degradation matrix output by the neural network to obtain a degraded image;

[0020] The data fidelity loss function is obtained by calculating according to the degraded image and the target image to be restored;

[0021] The network parameters of the neural network are updated according to the data fidelity loss function.

[0022] In one embodiment, when using a machine learning model and the intermediate degradation matrix to solve the high-quality image solution sub-problem, a standard normal distribution with a mean of 0 and a variance of 1 is used as the high-quality image distribution in the K-step reverse process of recovering the required high-quality data from noise, and the machine learning model uses a diffusion large model.

[0023] In one embodiment, the diffusion model is used to perform reasoning and solving the high-quality image solving sub-problem according to the intermediate degradation matrix, which is expressed as:

[0024] ;

[0025] In the above formula, Indicated in The intermediate high-quality image predicted in the iterative calculation is represents the hyperparameter, Represented as the diffusion model, represents the network parameters of the diffusion model, represents the number of iterative calculations, represents the image to be noisy, represents the update step size, Represents the intermediate degradation matrix obtained in the current iterative calculation.

[0026] The present application also provides a plug-and-play image restoration device based on a large model, the device comprising:

[0027] The image acquisition module to be restored is used to acquire the target image to be restored;

[0028] An objective function decomposition module is used to bring the target image into the objective function of the blind image restoration problem, and then decompose the objective function into two sub-problems for solving a high-quality image and a degradation matrix;

[0029] A degradation matrix subproblem solving module is used to solve the two subproblems in an alternating iterative solution manner. In each iterative solution process, a neural network is used to solve the degradation matrix solution subproblem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0030] A high-quality image sub-problem solving module, used to solve the high-quality image solving sub-problem by using a machine learning model and the intermediate degradation matrix to obtain an intermediate high-quality image of the current iterative calculation;

[0031] The image iterative restoration module is used to calculate an intermediate high-quality image obtained by the current iteration until the number of iterations meets a preset number, and the intermediate high-quality image is the high-quality image after the target image is restored.

[0032] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Acquire a target image to be restored;

[0034] After the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix;

[0035] The two sub-problems are solved by an alternating iterative solution method. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0036] The machine learning model and the intermediate degradation matrix are used to solve the high-quality image solving subproblem to obtain an intermediate high-quality image of the current iterative calculation;

[0037] Until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0039] Acquire a target image to be restored;

[0040] After the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix;

[0041] The two sub-problems are solved by an alternating iterative solution method. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0042] The machine learning model and the intermediate degradation matrix are used to solve the high-quality image solving subproblem to obtain an intermediate high-quality image of the current iterative calculation;

[0043] Until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

[0044] The above-mentioned plug-and-play image restoration method, device, equipment and medium based on a large model, after bringing the target image into the objective function of the blind image restoration problem, decomposes the objective function into two sub-problems for solving the high-quality image and the degradation matrix, and solves the two sub-problems in an alternating iterative solution manner. In each iterative solution process, a neural network is first used to solve the degradation matrix solution sub-problem, and the random prior of the degradation matrix is ​​used to guide the update of the neural network. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iteration calculation, and the diffusion large model and the intermediate degradation matrix are used to solve the high-quality image solution sub-problem to obtain the intermediate high-quality image of the current iteration calculation, until the number of iterations meets the preset number, then the intermediate high-quality image obtained by the current iteration calculation is the high-quality image after the target image is restored. The use of this method can improve the restoration performance and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a plug-and-play image restoration method based on a large model in one embodiment;

[0046] Figure 2 This is a schematic diagram of the experimental results of removing Gaussian blur in an experiment;

[0047] Figure 3 A schematic diagram of the experimental results of de-motion blur in an experiment;

[0048] Figure 4A schematic diagram of the image super-resolution experimental results in an experiment;

[0049] Figure 5 A schematic diagram of the image completion experimental results in an experiment;

[0050] Figure 6 A schematic diagram of the low light enhancement experimental results in an experiment;

[0051] Figure 7 A schematic diagram of the experimental results of real image restoration in an experiment;

[0052] Figure 8 is a structural block diagram of a plug-and-play image restoration device based on a large model in one embodiment;

[0053] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In the prior art, when restoring an image, a model-based or neural training-based approach is usually adopted, but both approaches have shortcomings. Figure 1 As shown, a plug-and-play image restoration method based on a large model is provided, which specifically includes the following steps:

[0056] Step S100: acquiring a target image to be restored.

[0057] Step S110, after the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix.

[0058] Step S120, the two sub-problems are solved by alternating iterative solution. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the random prior of the degradation matrix is ​​used to guide the update of the neural network. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation.

[0059] Step S130, using a machine learning model and an intermediate degradation matrix to solve the high-quality image solving subproblem, and obtaining an intermediate high-quality image of the current iterative calculation.

[0060] Step S140, until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

[0061] In step S100 , the target image to be restored is an optical image.

[0062] For the image super-resolution problem, the degradation process of the low-resolution image is generally shown in the following formula:

[0063] (1)

[0064] In formula (1), represents the degraded image, Represents the corresponding high-quality image, that is, the image after image restoration, represents the degradation matrix, represents additive noise.

[0065] In the non-blind image restoration problem, high-resolution images Unknown, and the degenerate matrix is known. For example, in the image super-resolution problem, the degradation matrix is the downsampling matrix. In the image completion problem, the degradation matrix is the mask matrix. In the image deblurring problem, the degradation matrix is a known fuzzy matrix.

[0066] In the blind image restoration problem, high-resolution images Unknown, and the degenerate matrix The relevant prior information of is also unknown. For example, in the blind image deblurring problem, the degradation matrix is an unknown fuzzy matrix. In the problems of image deraining, defogging, and deshadowing, the degradation matrix is the corresponding rain, fog and shadow matrix. In practical applications, the real image degradation may not only be a single image degradation type, but also a combination of multiple different degradation matrices. This method is aimed at image restoration under the blind image restoration problem.

[0067] Furthermore, the non-blind image restoration problem can be regarded as a special form of the blind image restoration problem. is known, only high quality images is something that needs to be solved. Therefore, the objective function of the non-blind image restoration problem can be written as:

[0068] (2)

[0069] The objective function of the blind image restoration problem is as follows:

[0070] (3)

[0071] In formula (3), and Indicates high quality image and the degradation matrix The prior term of and represents its corresponding weight. The non-blind image restoration problem can be regarded as a special form of the blind image restoration problem. Since the degradation matrix It is known that, for the convenience of expression, the subsequent image restoration problem is introduced in the form of the blind image restoration problem in formula (3).

[0072] In step S110, since there are two unknown variables in formula (3) that need to be solved, the problem, formula (3), is further decomposed into two sub-problems to solve the high-quality image. and the degradation matrix Solve them separately:

[0073] (4)

[0074] In this embodiment, for the two unknown variables in formula (4), a plug-and-play method based on random priors is proposed to solve , that is, for the degenerate matrix To solve the problem, we propose a plug-and-play approach using a pre-trained, ready-made diffusion model. , that is, for high-quality images To solve.

[0075] In other embodiments, in addition to the diffusion large model, the machine learning model may also adopt machine learning models such as transformer convolutional neural network, fully connected neural network, etc.

[0076] In step S120, when solving formula (4), an alternating iteration method is used to solve it. In each iterative calculation process, the degradation matrix is ​​first solved, and then the degradation matrix obtained by the current iterative solution is used to solve the high-quality image. Before the number of iterations reaches the preset number, the next iterative calculation is performed based on the intermediate degradation matrix and the intermediate high-quality image obtained by the current iterative calculation. When the number of iterations meets the predicted number of iterations, the calculation is terminated and the currently obtained high-quality image is output.

[0077] Furthermore, in this embodiment, for the degradation matrix solving subproblem, a lightweight network without pre-training is proposed. To assist in solving this problem, Prediction network for the degradation matrix Network parameters. Network The initial input of is a fixed random noise, which can be but not limited to zero mean, standard / unit variance Gaussian distribution noise, etc. The output of is the predicted degradation matrix .

[0078] Specifically, when a neural network is used to solve the degradation matrix solution subproblem, the optimization problem of the degradation matrix is ​​converted into a parameter optimization problem of the neural network. The converted degradation matrix solution subproblem is expressed as:

[0079] (5)

[0080] In formula (5), represents the target image to be restored, represents a neural network, represents the network parameters, represents the F-norm calculation, represents the weight, represents the high-quality image to be solved, represents the degradation matrix predicted by the neural network, Degenerate matrix The prior term of . Among them, is the first item, the data fidelity item, is the second term, the degenerate matrix The prior term of .

[0081] In this embodiment, the neural network for solving the degradation matrix solution subproblem is a fully connected neural network.

[0082] Since, in this method, the network It has not undergone any pre-training, and it is difficult to achieve the degradation matrix only by relying on data fidelity terms and traditional model priors Based on the neural network-assisted solution of problem (5), a random prior based on random sampling is proposed to assist the solution of problem (5). Assuming the degradation matrix Follow the distribution ,in Degenerate matrix By parameters decide or control.

[0083] In one embodiment, in the problem of removing Gaussian blur from an image, the degradation matrix is the Gaussian blur matrix, and the parameters is the variance of Gaussian blur. Therefore, the degradation matrix can be The prior distribution of is written in the following integral form:

[0084] (6)

[0085] However, since formula (6) requires all possible parameters It is usually difficult to solve the integral by traversal in practical applications.

[0086] It should be noted here that in addition to solving the Gaussian blur problem, this method can also solve the image restoration problems of motion blur, rain removal, fog removal, and shadow removal.

[0087] Therefore, in this method, it is proposed to use Monte Carlo random sampling to approximate the integral transformation by summation:

[0088] (7)

[0089] In formula (7), Indicates Random sampling.

[0090] The random prior constructed above For guiding the network Updates.

[0091] Specifically, in each iterative calculation, when optimizing the parameters of the neural network, the random prior of the degradation matrix is ​​used to guide the update of the neural network, including: using the random degradation matrix generated by Monte Carlo random sampling as a label, calculating the degradation matrix predicted by the neural network and the random degradation matrix Calculated by the difference between:

[0092] (8)

[0093] The loss function is calculated based on the labels and the predicted degradation matrix output by the neural network, and the network parameters of the neural network are updated based on the calculation results.

[0094] Specifically, the above loss (Formula (8)) is accumulated and calculated times, and then use The sum of the loss functions calculated is used to update the network:

[0095] (9)

[0096] Assumptions Representing a neural network The number of network parameter updates is In the iteration calculation, the network parameters Through random prior loss The expression to do the update is as follows:

[0097] (10)

[0098] In formula (10), represents the update iteration step size, Indicates common Optimization algorithm.

[0099] Furthermore, in each iterative calculation, when the parameters of the neural network are optimized, the neural network is also updated according to the data fidelity loss function, including: degrading the intermediate high-quality image obtained in the previous iterative calculation according to the predicted degradation matrix output by the neural network to obtain a degraded image, calculating based on the degraded image and the target image to be restored to obtain the data fidelity loss function, and finally updating the network parameters of the neural network according to the data fidelity loss function.

[0100] In each iteration, the network parameters An update is also performed based on the data fidelity term, where the data fidelity loss It can be expressed as:

[0101] (11)

[0102] In the In the iteration, the network parameters Through data fidelity loss The expression to do the update is as follows:

[0103] (12)

[0104] Finally, the two update terms represented by formula (10) and formula (12) are combined to get the network parameters in The complete update expression at iteration:

[0105] (13)

[0106] Therefore, in each iteration, the network The process of predicting the degradation matrix is ​​as follows:

[0107] (14)

[0108] Next, when solving the subproblem of high-quality image resolution, this method proposes to use a pre-trained, ready-made diffusion model to solve it. Assume that the diffusion model is ,in Large model for diffusion Network parameters. Diffusion large model The initial input is a noisy image and the number of iterations .

[0109] The diffusion model generally defines a K-step forward process of adding noise to high-quality data and a K-step reverse process of recovering the required high-quality data from the noise. When using the diffusion model and the intermediate degradation matrix to solve the high-quality image solution subproblem, define For high quality image distribution, is a standard normal distribution with mean 0 and variance 1.

[0110] When using a ready-made, pre-trained diffusion model To solve the image restoration problem, the reverse process is used as the inference process of image restoration to estimate the high-quality image as follows:

[0111] (15)

[0112] In formula (15), Indicated in The first iteration is the predicted high-quality image, Represents a hyperparameter.

[0113] However, simply restoring the image using formula (15) cannot guarantee the fidelity of the restored high-quality image. However, the diffusion model is retrained using paired high-quality and low-quality image training data. This will lead to high training and data costs.

[0114] In order to solve the above problem, in this embodiment, the data consistency expert knowledge unique to image restoration is introduced into the original diffusion large model image restoration process, that is, an update step is added to formula (15), as shown below:

[0115] (16)

[0116] Furthermore, It can be further written as follows:

[0117] (17)

[0118] Formula (17) is obtained based on the image degradation model and maintains and consistency.

[0119] Therefore, in each iteration process, the diffusion model is used to infer and solve the high-quality image solving sub-problem according to the intermediate degradation matrix, which can be expressed as:

[0120] (18)

[0121] In formula (18), Indicated in The intermediate high-quality image predicted in the iterative calculation is represents the hyperparameter, Represented as the diffusion model, represents the network parameters of the diffusion model, represents the number of iterative calculations, represents the image to be noisy, represents the update step size, Represents the intermediate degradation matrix obtained in the current iterative calculation.

[0122] In one embodiment, a pseudo code for implementing the above-mentioned large model-based plug-and-play image restoration method is provided, as shown below:

[0123] Algorithm 1

[0124] Input: low quality image ;

[0125] 1 Initialization: ;

[0126] 2 for to1do

[0127] 3 Obtained by Monte Carlo random sampling ;

[0128] 4 ;

[0129] 5 ;

[0130] 6 ;

[0131] 7 ;

[0132] 8 ;

[0133] 9 ;

[0134] 10 ;

[0135] 11 ;

[0136] Output: High quality image , the degradation matrix ;

[0137] In this paper, the effect of the large model plug-and-play blind image restoration method (hereinafter referred to as this method) proposed in this paper is also verified through experiments, which include experiments on different image restoration task scenarios.

[0138] 1. Experimental conditions:

[0139] 1.1) This method builds a deep learning model based on the pytorch framework.

[0140] 1.2) Test on a public optical image dataset. This dataset contains four different open source datasets.

[0141] 1.3) The performance of the proposed method is verified in seven degradation scenarios: Gaussian blur removal, motion blur removal, image super-resolution, image denoising, image completion, low-light enhancement, and true image restoration.

[0142] 2. Experimental content:

[0143] The experimental results of removing Gaussian blur are as follows Figure 2 shown.

[0144] The experimental results of de-motion blur are as follows Figure 3 shown.

[0145] The image super-resolution experimental results are as follows Figure 4 shown.

[0146] The image denoising experimental results are shown in the following table:

[0147] Table 1 Experimental results of removing Poisson noise

[0148]

[0149] Table 2 Experimental results of removing Rayleigh noise

[0150]

[0151] Table 3 Experimental results of removing salt and pepper noise

[0152]

[0153] The image completion experiment results are as follows Figure 5 shown.

[0154] The low light enhancement experiment results are as follows Figure 6 shown.

[0155] The experimental results of real image restoration are as follows Figure 7 shown.

[0156] From the above experimental results, it can be seen that the performance of this method is significantly better than other cutting-edge learning-based methods (FKP-DIP, BSRDM) in different image degradation scenarios. The test results are quantitatively compared using qualitative visual comparison results or quantitative peak signal-to-noise ratio (PSNR), and it is found that the performance of this method is the best.

[0157] In the above-mentioned plug-and-play image restoration method based on a large model, the plug-and-play image restoration method, device, equipment and medium based on a large model, after bringing the target image into the objective function of the blind image restoration problem, decompose the objective function into two sub-problems for solving the high-quality image and the degradation matrix, and solve the two sub-problems in an alternating iterative solution manner. In each iterative solution process, a neural network is first used to solve the degradation matrix solution sub-problem, and the random prior of the degradation matrix is ​​used to guide the update of the neural network. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation, and the diffusion large model and the intermediate degradation matrix are used to solve the high-quality image solution sub-problem to obtain the intermediate high-quality image of the current iterative calculation, until the number of iterations meets the preset number of times, then the intermediate high-quality image obtained by the current iterative calculation is the high-quality image after the target image is restored. This method can ensure that without model fine-tuning, model retraining and additional training data, the existing, pre-trained diffusion large model can be used to plug and play to effectively solve various image restoration tasks, while having the advantages of strong interpretability and good generalization ability.

[0158] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0159] In one embodiment, Figure 8As shown, a plug-and-play image restoration device based on a large model is provided, comprising: an image acquisition module 200 to be restored, an objective function decomposition module 210, a degradation matrix sub-problem solving module 220, a high-quality image sub-problem solving module 230 and an image iterative restoration module 240, wherein:

[0160] The image to be restored acquisition module 200 is used to acquire the target image to be restored;

[0161] An objective function decomposition module 210 is used to bring the target image into the objective function of the blind image restoration problem, and then decompose the objective function into two sub-problems for solving a high-quality image and a degradation matrix;

[0162] The degradation matrix sub-problem solving module 220 is used to solve the two sub-problems in an alternating iterative solution manner. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0163] A high-quality image sub-problem solving module 230 is used to solve the high-quality image solving sub-problem by using a machine learning model and the intermediate degradation matrix to obtain an intermediate high-quality image of the current iterative calculation;

[0164] The image iterative restoration module 240 is used to calculate an intermediate high-quality image obtained by the current iteration until the number of iterations meets a preset number, and then the intermediate high-quality image obtained by the current iteration calculation is the high-quality image after the target image is restored.

[0165] For the specific definition of the plug-and-play image restoration device based on the large model, please refer to the definition of the plug-and-play image restoration method based on the large model above, which will not be repeated here. Each module in the above-mentioned plug-and-play image restoration device based on the large model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0166] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a plug-and-play image restoration method based on a large model is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0167] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0169] Acquire a target image to be restored;

[0170] After the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix;

[0171] The two sub-problems are solved by an alternating iterative solution method. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0172] The machine learning model and the intermediate degradation matrix are used to solve the high-quality image solving subproblem to obtain an intermediate high-quality image of the current iterative calculation;

[0173] Until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

[0174] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0175] Acquire a target image to be restored;

[0176] After the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix;

[0177] The two sub-problems are solved by an alternating iterative solution method. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation;

[0178] The machine learning model and the intermediate degradation matrix are used to solve the high-quality image solving subproblem to obtain an intermediate high-quality image of the current iterative calculation;

[0179] Until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

[0180] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0181] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0182] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A plug-and-play image restoration method based on a large model, characterized in that: The method comprises: Acquire a target image to be restored; After the target image is introduced into the objective function of the blind image restoration problem, the objective function is decomposed into two sub-problems of solving the high-quality image and the degradation matrix; The two sub-problems are solved by an alternating iterative solution method. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation, wherein the degradation matrix solution sub-problem is solved by a neural network, and the update of the neural network is guided by the random prior of the degradation matrix, including: The random degradation matrix generated by Monte Carlo random sampling is used as the label, which is expressed as: In the above formula, represents a random prior, Indicates Random sampling, represents the random degradation matrix, represents the variance of Gaussian blur; The loss function is calculated based on the label and the predicted degradation matrix output by the neural network: First, the degradation matrix predicted by the neural network and the random degradation matrix are calculated. The difference between the two is calculated using the following formula: The cumulative loss calculated by the above formula is calculated times, using The sum of the loss functions calculated is expressed as: Then update the network parameters of the neural network according to the calculation results. Assume Representing a neural network The number of network parameter updates is In the iteration calculation, the network parameters Through random prior loss The expression to do the update is as follows: In the above formula, represents the update iteration step size, express Optimization algorithms; The machine learning model and the intermediate degradation matrix are used to solve the high-quality image solving subproblem to obtain an intermediate high-quality image of the current iterative calculation; Until the number of iterations meets the preset number, the intermediate high-quality image calculated in the current iteration is the high-quality image after the target image is restored.

2. The plug-and-play image restoration method according to claim 1, characterized in that: When using a neural network to solve the degradation matrix solution subproblem, the optimization problem of the degradation matrix is ​​converted into a parameter optimization problem of the neural network. The converted degradation matrix solution subproblem is expressed as: In the above formula, represents the target image to be restored, represents a neural network, Network parameters, represents the F-norm calculation, represents the weight, represents the high-quality image to be solved, represents the degradation matrix predicted by the neural network, Degenerate matrix The prior term of .

3. The plug-and-play image restoration method according to claim 2, characterized in that: In each iterative calculation, when optimizing the parameters of the neural network, the neural network is also updated according to the data fidelity loss function, including: Performing image degradation on the intermediate high-quality image obtained by the previous iterative calculation according to the predicted degradation matrix output by the neural network to obtain a degraded image; The data fidelity loss function is obtained by calculating according to the degraded image and the target image to be restored; The network parameters of the neural network are updated according to the data fidelity loss function.

4. The plug-and-play image restoration method according to claim 1, characterized in that: When using the machine learning model and the intermediate degradation matrix to solve the high-quality image solution sub-problem, a standard normal distribution with a mean of 0 and a variance of 1 is used as the high-quality image distribution in the K-step reverse process of recovering the required high-quality data from noise. The machine learning model uses a diffusion large model.

5. The plug-and-play image restoration method according to claim 4, characterized in that: The diffusion model is used to infer and solve the high-quality image solving sub-problem according to the intermediate degradation matrix, which is expressed as: In the above formula, Indicated in The intermediate high-quality image predicted in the iterative calculation is represents the hyperparameter, Represented as the diffusion model, represents the network parameters of the diffusion model, represents the number of iterative calculations, represents the image to be denoised, represents the update step size, Represents the intermediate degradation matrix obtained in the current iterative calculation.

6. A plug-and-play image restoration device based on a large model, characterized in that: The device comprises: The image acquisition module to be restored is used to acquire the target image to be restored; An objective function decomposition module is used to bring the target image into the objective function of the blind image restoration problem, and then decompose the objective function into two sub-problems for solving a high-quality image and a degradation matrix; The degradation matrix sub-problem solving module is used to solve the two sub-problems in an alternating iterative solution manner. In each iterative solution process, a neural network is used to solve the degradation matrix solution sub-problem, and the update of the neural network is guided by the random prior of the degradation matrix. At the same time, the neural network is also updated according to the data fidelity loss function to obtain the intermediate degradation matrix of the current iterative calculation, wherein the degradation matrix solution sub-problem is solved by using a neural network, and the update of the neural network is guided by the random prior of the degradation matrix, including: The random degradation matrix generated by Monte Carlo random sampling is used as the label, which is expressed as: In the above formula, represents a random prior, Indicates Random sampling, represents the random degradation matrix, represents the variance of Gaussian blur; The loss function is calculated based on the label and the predicted degradation matrix output by the neural network: First, the degradation matrix predicted by the neural network and the random degradation matrix are calculated. The difference between the two is calculated using the following formula: The cumulative loss calculated by the above formula is calculated times, using The sum of the loss functions calculated is expressed as: Then update the network parameters of the neural network according to the calculation results. Assume Representing a neural network The number of network parameter updates is In the iteration calculation, the network parameters Through random prior loss The expression to do the update is as follows: In the above formula, represents the update iteration step size, express Optimization algorithms; A high-quality image sub-problem solving module, used to solve the high-quality image solving sub-problem by using a machine learning model and the intermediate degradation matrix to obtain an intermediate high-quality image of the current iterative calculation; The image iterative restoration module is used to calculate an intermediate high-quality image obtained by the current iteration until the number of iterations meets a preset number, and the intermediate high-quality image is the high-quality image after the target image is restored.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images