Image restoration method, system, medium and electronic equipment based on visual generation large model

Abstract

The application provides an image restoration method, system, medium and equipment based on visual generation of a large model, the method comprising: obtaining a to-be-restored image and a candidate forward operator set of the to-be-restored image; randomly initializing parameters of all candidate forward operators in the candidate forward operator set; initializing and inputting the to-be-restored image into a visual generation large model, and making the visual generation large model perform a restoration operation in parallel based on the all candidate forward operators to obtain a plurality of corresponding restored images; and determining the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images. The application avoids dependence on degradation information, has better generalization ability, improves stability and applicability, has higher explainability, and is suitable for image restoration tasks in actual complex scenes.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image restoration, and particularly relates to an image restoration method and system based on a visual generation large model, a medium and an electronic device. BACKGROUND

[0002] Image restoration refers to a process of recovering an original image as clearly and completely as possible from a damaged or incomplete image through certain algorithms or techniques. Common image damage types include noise, blur, compression distortion, etc. Image restoration is widely used in medical image processing, satellite image recovery, and surveillance video sharpening.

[0003] Existing technologies in the image restoration task mainly include two types of methods: the first type is a training-based method, which trains a neural network to achieve image restoration by constructing a data set containing pairs of damaged images and clear images. However, this type of method performs better in the training data covered scenarios, but has poor generalization ability. The second type of method uses a generative model to achieve restoration by injecting the information of the image to be restored in the image generation process, without additional training. Although this method eliminates the training restriction, its dependence on degradation information limits its applicability in real scenarios. Therefore, existing technologies still have significant limitations in computational efficiency, generalization ability, and adaptability to actual complex scenarios, and need to be further improved. SUMMARY

[0004] The purpose of the present application is to provide an image restoration method and system based on a visual generation large model, which can improve computational efficiency, generalization ability, and adaptability to actual complex scenarios.

[0005] In a first aspect, the present application provides an image restoration method based on a visual generation large model, which comprises:

[0006] obtaining a to-be-restored image and a candidate forward operator set of the to-be-restored image;

[0007] randomly initializing the parameters of all candidate forward operators in the candidate forward operator set;

[0008] initializing and inputting the to-be-restored image into a visual generation large model, and making the visual generation large model perform a restoration operation in parallel based on all candidate forward operators to obtain a plurality of restored images corresponding to the to-be-restored image;

[0009] determining the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images.

[0010] In an implementation form of the first aspect, the method further comprises:

[0011] constructing a joint distribution model comprising the image to be restored, parameters of the candidate forward operator, the candidate forward operator, and a restored image; wherein,

[0012] constructing the joint distribution model based on a likelihood of the image to be restored, a conditional distribution of the parameters of the candidate forward operator, a prior distribution of the candidate forward operator, and a prior distribution of a restored image.

[0013] In an implementation form of the first aspect, the restoring operation comprises:

[0014] sampling a current restored image based on an input image of a current iteration, a candidate forward operator, and parameters of the candidate forward operator;

[0015] optimizing the parameters of the candidate forward operator based on a maximum a posteriori estimation;

[0016] re-sampling based on the optimized parameters of the candidate forward operator to update the current restored image, and taking the updated current restored image as an input image of a next iteration.

[0017] In an implementation form of the first aspect, sampling a current restored image based on an input image of a current iteration, a candidate forward operator, and parameters of the candidate forward operator comprises:

[0018] obtaining a scoring function of the input image of the current iteration based on the candidate forward operator, the parameters of the candidate forward operator, and the image to be restored;

[0019] sampling the current restored image based on the scoring function.

[0020] In an implementation form of the first aspect, sampling the current restored image based on the scoring function comprises:

[0021] obtaining a guiding term and gradient information of a posterior distribution of the input image of the current iteration based on the scoring function;

[0022] obtaining the current restored image based on the guiding term and the gradient information of the posterior distribution of the input image of the current iteration; wherein the guiding term is used to keep the input image of the current iteration consistent with the image to be restored.

[0023] In an implementation form of the first aspect, optimizing the parameters of the candidate forward operator based on a maximum a posteriori estimation comprises:

[0024] obtaining a loss function of image restoration based on the maximum a posteriori estimation;

[0025] perform gradient calculation on parameters of the candidate forward operator based on the loss function;

[0026] perform optimization on parameters of the candidate forward operator based on a gradient descent algorithm to minimize the loss function.

[0027] In an implementation form of the first aspect, determining the most matching candidate forward operator and the corresponding restored image based on the image to be restored and the plurality of restored images comprises:

[0028] respectively obtaining a structural similarity index of the image to be restored and each restored image;

[0029] determining a maximum structural similarity index based on all the structural similarity indexes;

[0030] taking the candidate forward operator corresponding to the maximum structural similarity index as the most matching candidate forward operator, and outputting the corresponding restored image.

[0031] In a second aspect, the present application provides an image restoration system based on a visual generative large model, comprising:

[0032] an obtaining module configured to obtain an image to be restored and a candidate forward operator set of the image to be restored;

[0033] an initializing module configured to randomly initialize parameters of all candidate forward operators in the candidate forward operator set;

[0034] a restoring module configured to initialize the image to be restored and input it into a visual generative large model, so that the visual generative large model performs a restoration operation in parallel based on all the candidate forward operators to obtain a plurality of corresponding restored images;

[0035] a matching module configured to determine the most matching candidate forward operator and the corresponding restored image based on the image to be restored and the plurality of restored images.

[0036] In a third aspect, the present application provides an electronic device, comprising a processor and a memory; the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory, so that the electronic device performs the image restoration method based on a visual generative large model as described above.

[0037] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, which is executed by an electronic device to implement the image restoration method based on a visual generative large model as described above.

[0038] The image restoration method, system, device and storage medium based on a visual generation large model have the following beneficial effects:

[0039] 1. The image restoration method, system, device and storage medium based on a visual generation large model can realize image restoration without additional training, greatly reducing the dependence on large-scale training data and computing resources.

[0040] 2. The image restoration method, system, device and storage medium based on a visual generation large model do not rely on pre-known degradation information, and can realize accurate modeling of complex image degradation mechanisms by optimizing forward operator parameters.

[0041] 3. The image restoration method, system, device and storage medium based on a visual generation large model have good generalization ability, can adapt to various image damage forms, and are suitable for various image restoration occasions.

[0042] 4. The image restoration method, system, device and storage medium based on a visual generation large model significantly improve the efficiency and resource adaptability of the method through parallel sampling algorithm design.

[0043] 5. The image restoration method, system, device and storage medium based on a visual generation large model gradually restore image details using a diffusion inverse process, and output forward operators and their parameters that best match the degradation mechanism, further enhancing the interpretability of the method, and being suitable for image restoration tasks in actual industrial complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1A An application scenario of the image restoration method based on a visual generation large model is shown.

[0045] Figure 1A The structure of an end-cloud interaction scenario in the implementation modes is shown.

[0046] Figure 2 A flowchart of the image restoration method based on a visual generation large model in an embodiment is shown.

[0047] Figure 3 A flowchart of the image restoration method based on a visual generation large model in an embodiment is shown.

[0048] Figure 4 A flowchart of the image restoration method based on a visual generation large model in an embodiment is shown.

[0049] Figure 5 An effect diagram of the image restoration method based on a visual generation large model is shown.

[0050] Figure 6 A structure diagram of the image restoration system based on a visual generation large model in an embodiment is shown.

[0051] Figure 7Fig. 1 shows a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] The present application is herein described, by way of example only, with reference to embodiments thereof. It is to be understood that variations and modifications will be apparent to those skilled in the art and that the scope of the present application encompasses all such obvious variations and modifications. The embodiments described herein are merely given by way of example and the scope of the present application is not limited thereto. It is to be understood that the phraseology and terminology used herein is for the purpose of description and not of limitation. Any use of the terms "comprising", "comprises", "including", "includes" or "having", "has" is to be construed as specifying the presence of stated features or components, but does not preclude the presence or addition of one or more other features, components or steps. The use of the terms "first", "second" and "third" does not limit the quantity or order of those terms. These terms are used to distinguish between two or more components or steps. The use of the term "about" in relation to a numerical value shall mean that the value is within a range of plus or minus 10% from the value.

[0053] It is to be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reading and understanding the above description. The scope of the application should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the appended claims, along with their full scope of equivalents.

[0054] In addition, throughout this document, the description and drawings are provided to illustrate potential implementations of the application and are not intended to limit the scope of the application. The various embodiments can be implemented in any number of ways, using hardware, software or a combination thereof. The embodiments described use terminology and nomenclature that is purely exemplary based on the specific applications discussed. It is intended that the terminology and nomenclature be broadly interpreted, consistent with the broadest possible interpretation of the principles described herein. The use of any specific terminology or nomenclature is merely to first exemplify the principles described herein and is non-limiting. It is also intended that the various embodiments can be interchanged or combined, where possible, without departing from the scope of the application.

[0055] Currently, existing techniques in image restoration tasks mainly fall into two categories of methods: the first category is a training-based method, which trains a neural network to achieve image restoration by constructing a data set containing pairs of damaged images and clear images. This kind of method relies on high-quality large-scale data sets, and these data sets are often difficult to obtain in practical applications. At the same time, due to the limited generalization ability of neural networks, when the degradation form of the input image is inconsistent with the pattern in the training data, the restoration effect will decrease significantly. In addition, the training process itself is complex and has high computational cost, especially in terms of the demand for computing resources.

[0056] The second type of method restores the image by injecting information of the image to be restored into the image generation process with the help of a generative model, without additional training. However, this type of method usually needs to know the forward operator and its parameters that cause the image degradation in advance, which is a big difficulty in practical applications. In real scenarios, the mechanism of image degradation is often unknown and complex, so this restoration method that relies on explicit degradation information is difficult to meet the actual needs. In contrast, blind image restoration is closer to practical applications, because it does not need prior degradation information, but directly infers the degradation mechanism from the damaged image and restores it.

[0057] In general, the first type of method performs better in scenarios covered by the training data, but has poor generalization ability; the second type of method is free from the limitation of training, but its applicability in real scenarios is limited due to its dependence on degradation information. Therefore, the existing technology still has significant limitations in computational efficiency, generalization ability, and adaptability to actual complex scenarios, and needs to be further improved.

[0058] To at least solve the above problems, the embodiments of the present application provide an image restoration method based on a visual generation large model, which restores the image step by step in the diffusion inverse process by constructing a set of forward operators covering multiple image degradation forms, and optimizing the parameters of the candidate forward operators. Finally, the forward operator and its parameters that best match the image degradation are selected according to the optimization results, and a high-quality clear image is output. The present application avoids the dependence on degradation information, has good generalization ability, improves stability and applicability, and has higher interpretability, and is suitable for image restoration tasks in actual complex scenarios.

[0059] In some other embodiments, the image restoration method based on the visual generation large model provided by the embodiments of the present application can be applied to an image restoration device based on the visual generation large model as shown in Figure 1A As shown in Figure 1A The image restoration device based on the visual generation large model 1 includes a processor 11 and a database 12. The image restoration method based on the visual generation large model provided by the embodiments of the present application can be applied to the processor 11.

[0060] The processor 11 in Figure 1A The processor 11 in Figure 1A Although only one processor 11 and one database 12 are shown in Figure 1A The examples in

[0061] In some other embodiments, the image restoration device 1 based on the visual generation large model can also not contain the database 12, but only contain the processor 11 with storage function. The image restoration method based on the visual generation large model provided in the embodiments of the present application can be applied to the processor 11. The processor 11 with storage function can include a computer, a mobile phone, and the like, which are not limited herein.

[0062] In yet some other embodiments, the image restoration method based on the visual generation large model described in the present application can be applied to an end-cloud interaction scenario. Figure 1B The structure schematic diagram of the end-cloud interaction scenario in these implementations is shown. As Figure 1B shown, the end-cloud interaction system 2 includes a terminal 20 and a cloud server 21, and the terminal 20 and the cloud server 21 can communicate with each other, and the communication mode is not limited to a wired or wireless mode.

[0063] The terminal 20 can be mobile or fixed, for example, the terminal 20 can be a wireless terminal or a wired terminal, and the wireless terminal can refer to a device with wireless transceiver function, which can be deployed in an image restoration scenario. The terminal 20 can be a mobile phone, a notebook computer, and the like, which are not limited herein. The cloud server 21 can include one or more servers, or include one or more processing nodes, or include one or more virtual machines running on the server, and the cloud server 21 can also be referred to as a server cluster, a management platform, an image restoration center based on the visual generation large model, and the like, which are not limited herein.

[0064] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application.

[0065] The image restoration method based on the visual generation large model provided in the following embodiments of the present application can be implemented, for example, by the processor 11 shown in Figure 1A or the cloud server 21 shown in Figure 1B . Figure 2 The flowchart of the image restoration method based on the visual generation large model described in the embodiments of the present application is shown. As Figure 2 shown, the image restoration method based on the visual generation large model includes steps S1 to S4.

[0066] S1, obtaining a to-be-restored image and a candidate forward operator set of the to-be-restored image.

[0067] Regarding image restoration, it is usually necessary to restore the original clear image from a known damaged image and a known forward operator, which is defined as wherein y represents the to-be-restored image after the action of the forward operator, ​is the parameter in the forward operator, x is the clear image we need to restore (i.e. the restored image), z is independent and identically distributed Gaussian noise with variance

[0068] In practical applications, the damage form of the image is various, which may be caused by noise, blur, compression distortion and other factors. In order to effectively restore the clear image, a candidate forward operator set suitable for the actual scene is constructed through step S1, that is, a set of potential image damage factors of the image to be restored, denoted as The set contains all candidate forward operators, which are used to simulate different types of image damage. Specifically, the candidate forward operator set covers a variety of possible image damage types, such as motion blur, JPEG compression, low resolution, etc., which can effectively simulate various damage forms in practical applications. Among them, each candidate forward operator corresponds to a type of image damage.

[0069] S2, randomly initialize the parameters of all candidate forward operators in the candidate forward operator set.

[0070] By randomly initializing the parameters of all candidate forward operators, the visual generation large model can have a good starting point.

[0071] S3, initialize the image to be restored and input it into the visual generation large model, so that the visual generation large model performs restoration operations in parallel based on all candidate forward operators to obtain corresponding multiple restored images.

[0072] In the process of image restoration, as the visual generation large model gradually reasons, not only the clear image needs to be restored, but also the parameters of each candidate forward operator need to be optimized at the same time. Therefore, in order to improve the efficiency and resource adaptability of the method, a parallel sampling method is adopted, and step S3 adopts a parallel sampling design, which can simultaneously integrate all forward operators from the candidate forward operator set , and then construct a single batch for processing, that is, perform restoration operations in parallel based on all candidate forward operators.

[0073] The following will take one candidate forward operator as an example to explain the specific content of step S3. However, it should be noted that the model actually performs step S3 on all candidate forward operators at the same time, which greatly improves the calculation efficiency and saves the calculation resources.

[0074] First, a joint distribution model is constructed to accurately represent the relationship between the variables in the image restoration process. Specifically, the joint distribution model includes the to-be-restored image, the parameters of the candidate forward operator, the candidate forward operator, and the restored image; wherein the joint distribution model is constructed based on the likelihood of the to-be-restored image, the conditional distribution of the parameters of the candidate forward operator, the prior distribution of the candidate forward operator, and the prior distribution of the restored image.

[0075] That is, the joint distribution model includes the restored image x0, the to-be-restored image y, a candidate forward operator and its corresponding parameters The joint distribution model can be represented as:

[0076]

[0077] wherein, is the conditional probability density function, representing the distribution of the to-be-restored image y given x0, and the parameter The conditional distribution of y is a Gaussian distribution with mean and covariance matrix

[0078] wherein p(x0) is the prior distribution of the image x0, which is modeled using a pre-trained visual generative large model. The prior distribution is the probability distribution of x0 without any observed data. It reflects the selection based on domain knowledge, historical data, or other assumptions before the data is observed.

[0079] wherein, is the conditional distribution of the forward operator parameter , and is the prior distribution of the forward operator which is defined as a uniform distribution in this embodiment.

[0080] After determining the joint distribution model, the to-be-restored image y is initialized to serve as the initialization input of the visual generative large model. Specifically, noise disturbance is added to the to-be-restored image y to serve as the initial state (noisy image) x T input into the visual generative large model, thereby being able to provide meaningful semantic information to assist the model to converge to a more accurate clear image restoration.

[0081] Then, the visual generative large model will perform a restoration operation based on the initial state x T and a candidate forward operator and its corresponding parameters Figure 3 ​A flowchart of an image restoration method based on a visual generation large model is shown in the embodiments of the present application, as shown in Figure 3 The restoration operation includes steps S31 to S33.

[0082] S31, based on the input image of the current iteration, a candidate forward operator and the parameter sampling of the candidate forward operator, the current restoration image is obtained.

[0083] Specifically, based on the candidate forward operator, the parameters of the candidate forward operator and the image to be restored, the scoring function of the input image of the current iteration is obtained, and the current restoration image is sampled based on the scoring function. In fact, the visual generation large model can generate a random image without any constraints as a diffusion model. However, in this embodiment, the inference of the visual generation large model needs to be guided to complete the image restoration task. That is, the constraint information needs to be added in the inverse diffusion process of the visual generation large model, so that the generated image is the restoration image required by the task, rather than irrelevant random image.

[0084] Specifically, the visual generation large model generates a restoration image by gradually denoising in an iterative manner, so that the visual generation large model learns a scoring function that can give a noisy image x t The score (gradient) of its original image x0 is predicted. This scoring function can also be considered as modeling the noise at each time, so as to estimate the direction of the noise and help the reverse denoising process. That is, by guiding the model to gradually denoise in an iterative manner through a specific scoring function, the generated restoration image x0 needs to meet the given candidate forward operator, that is, it needs to meet , and also needs to be as close as possible to the image to be restored y.

[0085] Specifically, the posterior distribution of the input image x t of the current iteration is: The diffusion process is guided by the scoring function of x t to sample from this distribution. The scoring function of x t is:

[0086]

[0087] Where the first term is a data prior term provided by the pre-trained visual generation large model, indicating the input image x t of the current iteration. The generation process under a candidate forward operator and parameters , that is, the gradient information of the posterior distribution of the input image of the current iteration. The second term is a guiding term that forces the input image x tconsistency with the image to be restored y, thereby constraining the similarity between the current restored image and the actual degraded image.

[0088] Further, the sampling based on the scoring function to obtain the current restored image comprises: obtaining a guide term and gradient information of a posterior distribution of the input image of the current iteration based on the scoring function; and obtaining the current restored image based on the guide term and the gradient information of the posterior distribution of the input image of the current iteration; wherein the guide term is used to keep the input image of the current iteration consistent with the image to be restored y.

[0089] As described above, the gradient information of the posterior distribution of the input image of the current iteration is which can guide the sampling, and the guide term is which, as a constraint condition, can guide the input image x t consistent with the image to be restored y, so that the current restored image generated by the model meets the given candidate forward operator and is as close as possible to the image to be restored y.

[0090] Further, the noise prediction network of the visual generation large model approximates the scoring function through visual generation, and the scoring function is defined as follows:

[0091]

[0092] wherein μ θ (x t ) is a predicted value (de-noising predicted value) generated by the model, σ θ (x t ) is a noise estimate under the current state x t , is a time step related scalar, is a guide term, and the image is adjusted by making the input image of the current iteration consistent with the image to be restored y.

[0093] In fact, the scoring function is approximated by the noise prediction network of the visual generation large model, so that the noise term of the predicted image at each time t can be learned, and the noise can be effectively removed to obtain the current restored image x t-1 .

[0094] By iteratively performing the above restoration operation, the visual generation large model can gradually guide the input image x t of the current iteration to approach the correct restored image x0 through the synergistic effect of the guide term and the noise prediction network.

[0095] S32, optimizing the parameters of the candidate forward operator based on maximum posterior estimation.

[0096] Specifically, while performing the restoration operation to restore the image each time, the parameters of the candidate forward operator also need to be optimized simultaneously. In some embodiments, for each candidate forward operator its parameters are optimized using maximum a posteriori estimation, which aims to maximize the posterior distribution of the given data (the image to be restored y and the input image x t of the current iteration) to estimate the parameters of the candidate forward operator that is, to find the optimal parameters such that the posterior probability is maximized.

[0097] In some embodiments, optimizing the parameters of the one candidate forward operator based on the maximum a posteriori estimation comprises: obtaining a loss function of image restoration based on the maximum a posteriori estimation; performing gradient calculation on the parameters of the one candidate forward operator based on the loss function; and minimizing the loss function based on a gradient descent algorithm to optimize the parameters of the one candidate forward operator.

[0098] According to the maximum a posteriori estimation, the optimal parameters are obtained by optimizing the following objective function:

[0099]

[0100] where the first term in the right side optimization problem corresponds to the likelihood of y in the joint distribution, i.e., the loss function of image restoration, which measures the error of converting the denoised image to the image to be restored y through the candidate forward operator ; and the second term corresponds to the prior of in the joint distribution, which can suppress overfitting and guide the parameters to converge to reasonable values. Here, since the current iteration input image x t obtained by the model in the iteration process is actually an image with noise, in order to avoid the influence of noise on the estimation of the parameters of the candidate forward operator, the denoised image is obtained based on the estimation of x t to optimize the parameters.

[0101] In the parameter optimization process, in order to solve the optimal parameters the gradient descent method is used to solve the optimization problem. By minimizing the above objective function, we can obtain a recursive optimization process, the specific steps are as follows:

[0102] 1. Calculate the gradient: for a given loss function: calculate the gradient with respect to the parameters :

[0103]

[0104] 2. Using the calculated gradient information, update the parameters of the candidate forward operator by the update rule of gradient descent

[0105] In some cases, the candidate forward operator is not differentiable, which makes the standard gradient descent method cannot be directly applied. Then we can use zero-order optimization method to solve it. Through this method, we can estimate the parameters of the forward operator according to the error between the current restored image and the image to be restored So as to realize the image restoration more accurately.

[0106] S33, resample based on the optimized parameters of the candidate forward operator to update the current restored image, and take the updated current restored image as the input image of the next iteration.

[0107] Specifically, based on the optimized parameters of the candidate forward operator S31 is executed again to obtain an updated current restored image. The updated image is taken as the input image of the next iteration, and steps S31 to S33 are repeatedly executed. That is, in each iteration, the restoration operation (steps S31 to S33) is performed, and after step S33 is executed, step S31 is repeated to obtain an updated current restored image. At this time, one iteration is completed, and the updated current restored image is taken as the input of the next iteration.

[0108] In each round of iteration sampling, the model can utilize the constraints of the degraded image and the known forward operator to continuously optimize the restored image until it converges to a clear image.

[0109] S4, determining the most matched candidate forward operator and the corresponding restored image based on the image to be restored and the plurality of restored images.

[0110] Figure 4 The flowchart of the image restoration method based on the visual generation large model according to the embodiments of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, step S4 includes steps S41 to S43.

[0111] S41, respectively obtaining the structural similarity indexes of the image to be restored and each restored image.

[0112] S42, determining the maximum structural similarity index based on all the structural similarity indexes.

[0113] S43, taking the candidate forward operator corresponding to the maximum structural similarity index as the most matched candidate forward operator, and outputting the corresponding restored image.

[0114] In some embodiments, a structural similarity index (SSIM) is used to approximate The SSIM is an image quality assessment index that can effectively measure the perceptual similarity between two images, which takes into account brightness, contrast and structural information. Using SSIM can effectively capture the differences in perception between the restored image and the image to be restored. By comparing the SSIM values of all restored images, the maximum structural similarity index can be determined. At this time, the candidate forward operator with the maximum SSIM value and its corresponding restored image is selected as the final output restored image. That is

[0115]

[0116] where i represents the i-th candidate forward operator.

[0117] Therefore, the present application can realize image restoration without additional training, greatly reducing the dependence on large-scale training data and computing resources. Secondly, the present application does not rely on pre-known degradation information, and realizes accurate modeling of complex image degradation mechanism by optimizing the parameters of the forward operator. Thirdly, the present application has good generalization ability and can adapt to various image damage forms. Finally, the present application significantly improves the efficiency and resource adaptability of the method through the parallel sampling algorithm design. The present application uses the diffusion inverse process to gradually restore image details, and outputs the forward operator and its parameters that best match the degradation mechanism, further enhancing the interpretability of the method, and is suitable for image restoration tasks in actual industrial complex scenarios. The specific image restoration effect is shown in Figure 5 .

[0118] The protection scope of the image restoration method based on the visual generation large model according to the embodiments of the present application is not limited to the execution order of the steps listed in the embodiments. Any scheme realized by adding, replacing or modifying the steps of the prior art according to the principles of the present application is included in the protection scope of the present application.

[0119] The embodiments of the present application also provide an image restoration system based on a visual generation large model, which can implement the image restoration method based on a visual generation large model according to the present application. However, the implementation device of the image restoration system based on a visual generation large model according to the present application includes but is not limited to the structure of the image restoration system based on a visual generation large model according to the embodiments. Any structure modification or replacement of the prior art according to the principles of the present application is included in the protection scope of the present application.

[0120] Figure 6 The structure of the image restoration system based on a visual generation large model according to the embodiments of the present application is shown in the structure schematic diagram of the image restoration system based on a visual generation large model according to the embodiments of the present application, as Figure 6As shown, the image restoration system 4 based on the visual generation large model comprises an acquisition module 41, an initialization module 42, a restoration module 43 and a matching module 44.

[0121] The acquisition module 41 is configured to acquire a to-be-restored image and a candidate forward operator set of the to-be-restored image.

[0122] The initialization module 42 is configured to randomly initialize parameters of all candidate forward operators in the candidate forward operator set.

[0123] The restoration module 43 is configured to initialize the to-be-restored image and input the to-be-restored image into the visual generation large model, so that the visual generation large model performs a restoration operation in parallel based on all the candidate forward operators to obtain a plurality of restored images corresponding to the candidate forward operators.

[0124] The matching module 44 is configured to determine the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images.

[0125] It should be noted that the structures and principles of the acquisition module 41, the initialization module 42, the restoration module 43 and the matching module 44 correspond to the steps in the image restoration method based on the visual generation large model one by one, and therefore will not be described in detail here.

[0126] In several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules / units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or modules or units, and can be electrical, mechanical or other forms.

[0127] The modules / units described as separate components can or can not be physically separated, and the components displayed as modules / units can or can not be physical modules, that is, they can be located in one place or distributed on a plurality of network units. According to actual needs, some or all of the modules / units can be selected to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in each embodiment of the present application can be integrated in one processing module, or each module / unit can be physically separated, or two or more modules / units can be integrated in one module / unit.

[0128] Those skilled in the art should further appreciate that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be embodied in electronic hardware, computer software, or a combination of both, and that the described examples can generally be implemented as code means or instructions on a processor of a general-purpose computer, a special-purpose computer, or a dedicated computer. The disclosure is not limited by the particular type of computer or computer system that is used to implement the described examples. For clarity, the aforementioned description sets forth aspects of the described examples in connection with the illustrated embodiments. It should be appreciated that various modifications can be made to the described examples, and implementations other than those described herein can be possible.

[0129] The embodiments of the present application further provide a computer readable storage medium. Those skilled in the art can understand that all or part of the steps of the method described above can be completed by a processor instructed by a program, and the program can be stored in a computer readable storage medium. The storage medium is a non-transitory medium, such as a random access memory, a read only memory, a flash memory, a hard disk, a solid state disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. integrated with one or more available medium sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital video disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0130] The embodiments of the present application further provide an electronic device. The electronic device includes a processor and a memory.

[0131] The memory is used to store a computer program.

[0132] The memory includes a ROM, a RAM, a disk, a U disk, a memory card, or various media that can store program codes such as an optical disc.

[0133] The processor is connected with the memory, and is used to execute the computer program stored in the memory, so that the electronic device executes the image restoration method based on the visual generation large model described above.

[0134] Preferably, the processor can be a general processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; also can be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0135] As shown in Figure 7 , the electronic device of the present application is in the form of a general computing device. The components of the electronic device can include, but are not limited to, one or more processors or processing units 51, a memory 52, a bus 53 that connects different system components, including the memory 52 and the processing unit 51.

[0136] The bus 53 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0137] The electronic device typically includes a variety of computer system readable media. These media can be any available media that is accessible by the electronic device and includes both volatile and non-volatile media, removable and non-removable media.

[0138] The memory 52 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 523 can be used for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 7 (not shown), commonly referred to as a "hard disk drive", for reading from and writing to non-removable, non-volatile magnetic media (e.g., a platter). Although not specifically shown, such computer system typically can further include a removable / non-removable, volatile / non-volatile memory system reader (e.g., a floppy disk drive), a CD-ROM, DVD, Blu-ray, or other disk reader, or other suitable device. Figure 7A disk drive, a floppy disk drive, a CD-ROM drive, a DVD-ROM drive, or other removable media drive, a flash memory card drive, a digital video disc (DVD) drive, or other removable media drive, or any other suitable device for reading from and / or writing to a removable nonvolatile media (for example, a "floppy disk", a CD-ROM, a DVD-ROM, etc.) can also be provided, as can be seen in Fig. 1. In such cases, each drive can be connected to the system bus 53 by one or more data media interfaces. The memory 52 can include, for example, one or more program products 524 having a set (e.g., at least one) of program modules 5241 configured to carry out the functions of embodiments of the application.

[0139] The program / utility 524 having a set (at least one) of program modules 5241 can be stored in memory 52 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data, and a user interface program, etc. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, can include implementation of a networking environment.

[0140] The electronic device can also communicate with one or more external devices 56 such as a keyboard or a pointing device, displays, etc.; one or more devices that enable a user to interact with the electronic device; and / or one or more devices (e.g., a network card, a modem, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can occur via an input / output (I / O) interface 54. Still yet, the electronic device can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) through a network adapter 55. As Figure 7 illustrated, the network adapter 55 can communicate with the other components of the electronic device through the bus 53. It should be understood that, although not shown explicitly, other hardware and / or software components could be used in conjunction with the electronic device. Such components include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0141] Embodiments of the present application can also provide a computer program product including one or more computer instructions. When loaded and executed by a computing device, the computer instructions can generate, in whole or in part, the processes or functions described in embodiments of the present application. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, or data center to another website, computer, or data center through wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0142] The computer program product is executed by a computer, and the computer executes the method of the foregoing method embodiments. The computer program product can be a software installation package, and in a case where the foregoing method needs to be used, the computer program product can be downloaded and executed on the computer.

[0143] The descriptions of the corresponding flows or structures of the various drawings are each focused on, and parts not described in detail in a certain flow or structure can be referred to the related descriptions of other flows or structures.

[0144] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and category of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought of the present application should be covered by the claims of the present application.

Claims

1. A method for generating an image restoration of a large model based on vision, characterized by, The method comprises: obtaining a to-be-restored image and a candidate forward operator set of the to-be-restored image; randomly initializing parameters of all candidate forward operators in the candidate forward operator set; initializing the to-be-restored image and inputting it into a visual generation large model, so that the visual generation large model performs a restoration operation in parallel based on all candidate forward operators to obtain a plurality of corresponding restored images; wherein the restoration operation comprises: based on the input image of the current iteration, a candidate forward operator and parameter sampling of the candidate forward operator, a current restored image is obtained; based on maximum a posteriori estimation, the parameters of the candidate forward operator are optimized; based on the optimized parameters of the candidate forward operator, resampling is performed to update the current restored image, and the updated current restored image is taken as the input image of the next iteration; determining the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images; wherein optimizing the parameters of the candidate forward operator based on maximum a posteriori estimation comprises: obtaining a loss function of image restoration based on the maximum a posteriori estimation; performing gradient calculation on the parameters of the candidate forward operator based on the loss function; minimizing the loss function based on a gradient descent algorithm to optimize the parameters of the candidate forward operator; and The method further comprises: constructing a joint distribution model, the joint distribution model comprising the to-be-restored image, the parameters of the candidate forward operator, the candidate forward operator and the restored image; wherein the joint distribution model is constructed based on the likelihood of the to-be-restored image, the conditional distribution of the parameters of the candidate forward operator, the prior distribution of the candidate forward operator and the prior distribution of the restored image. 2.The vision-based large model generated image restoration method of claim 1, wherein, Based on the input image of the current iteration, a candidate forward operator and parameter sampling of the candidate forward operator, a current restored image is obtained, comprising: obtaining a scoring function of the input image of the current iteration based on the candidate forward operator, the parameters of the candidate forward operator and the to-be-restored image; sampling to obtain the current restored image based on the scoring function. 3.The vision-based large model generated image restoration method of claim 2, wherein, Sampling to obtain the current restored image based on the scoring function comprises: obtaining a guide term and gradient information of the posterior distribution of the input image of the current iteration based on the scoring function; obtaining the current restored image based on the guide term and the gradient information of the posterior distribution of the input image of the current iteration; wherein the guide term is used to keep the input image of the current iteration consistent with the to-be-restored image. 4.The method of claim 1, wherein, Optimizing the parameters of the candidate forward operator based on maximum a posteriori estimation comprises: obtaining a loss function of image restoration based on the maximum a posteriori estimation; performing gradient calculation on the parameters of the candidate forward operator based on the loss function; minimizing the loss function based on a gradient descent algorithm to optimize the parameters of the candidate forward operator. 5.The method of claim 1, wherein, Determining the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images comprises: respectively acquire structure similarity indexes of the to-be-restored image and each restored image; determine a maximum structure similarity index based on all the structure similarity indexes; determine the candidate forward operator corresponding to the maximum structure similarity index as the most matched candidate forward operator, and output the corresponding restored image.

6. A large model-based image restoration system based on vision, characterized by, The system comprises: an acquisition module configured to acquire a to-be-restored image and a set of candidate forward operators of the to-be-restored image; an initialization module configured to randomly initialize parameters of all candidate forward operators in the set of candidate forward operators; a restoration module configured to initialize the to-be-restored image and input the to-be-restored image into a visual generation large model, so that the visual generation large model performs a restoration operation in parallel based on all the candidate forward operators to acquire a plurality of restored images; wherein the restoration operation comprises: acquiring a current restored image based on an input image of a current iteration, a candidate forward operator and parameter sampling of the candidate forward operator; optimizing parameters of the candidate forward operator based on maximum a posteriori estimation; resampling based on the optimized parameters of the candidate forward operator to update the current restored image, and taking the updated current restored image as an input image of the next iteration; a matching module configured to determine the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images; wherein optimizing the parameters of the candidate forward operator based on maximum a posteriori estimation comprises: acquiring a loss function of image restoration based on the maximum a posteriori estimation; performing gradient calculation on the parameters of the candidate forward operator based on the loss function; and minimizing the loss function based on a gradient descent algorithm to optimize the parameters of the candidate forward operator; and constructing a joint distribution model, the joint distribution model comprising the to-be-restored image, parameters of the candidate forward operator, the candidate forward operator and a restored image; wherein the joint distribution model is constructed based on a likelihood of the to-be-restored image, a conditional distribution of the parameters of the candidate forward operator, a prior distribution of the candidate forward operator and a prior distribution of a restored image.

7. An electronic device, comprising: comprise: a processor and a memory; the memory is configured to store a computer program; the processor is configured to execute the computer program stored in the memory, so that the electronic device executes the image restoration method based on the visual generation large model according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the electronic device to implement the image restoration method based on the visual generation large model according to any one of claims 1-5. The program is executed by the electronic device to implement the image restoration method based on the visual generation large model according to any one of claims 1-5.

Images