Image restoration method and system based on visual generation large model, medium and electronic equipment

Abstract

The invention provides an image restoration method and system based on a visual generation large model, a medium and equipment. The method comprises the following steps: acquiring a to-be-restored image and a candidate forward operator set of the to-be-restored image; parameters of all candidate forward operators in the candidate forward operator set are randomly initialized; initializing the to-be-restored image and inputting the to-be-restored image into a visual generation large model, and enabling the visual generation large model to execute a restoration operation in parallel and iteratively based on all the candidate forward operators so as to obtain a plurality of corresponding restored images; and determining the most matched candidate forward operator and the corresponding restored image based on the to-be-restored image and the plurality of restored images. According to the method, dependence on degradation information is avoided, good generalization ability is achieved, stability and applicability are improved, meanwhile, higher interpretability is achieved, and the method is suitable for image restoration tasks of actual complex scenes.

Description

Technical Field

[0001] The present application belongs to the field of image restoration technology, and in particular, relates to an image restoration method, system, medium and electronic device based on visual generation of a large model. Background Art

[0002] Image restoration refers to the process of restoring a damaged or incomplete image to a clearer and more complete original image through certain algorithms or technologies. Common types of image damage include noise, blur, compression distortion, etc. Image restoration is widely used in medical image processing, satellite image restoration, surveillance video clarity, and other fields.

[0003] Existing technologies in image restoration tasks are mainly divided into two categories: the first category is a training-based method, which trains a neural network to achieve image restoration by constructing a data set containing damaged images and clear image pairs. However, this type of method performs well in scenes covered by training data, but has poor generalization ability. The second type of method uses a generative model to achieve restoration by injecting information of the image to be restored during the image generation process without the need for additional training. Although it has gotten rid of the limitations of training, its dependence on degraded information limits its applicability in real scenes. Therefore, existing technologies still have significant limitations in terms of computational efficiency, generalization ability, and adaptability to actual complex scenes, and are in urgent need of further improvement. Summary of the invention

[0004] The purpose of this application is to provide an image restoration method, system, device and storage medium based on a visually generated large model, which can improve computing efficiency, generalization ability and adaptability to actual complex scenes.

[0005] In a first aspect, the present application provides an image restoration method based on visual generation of a large model, the method comprising:

[0006] Acquire an image to be restored and a set of candidate forward operators for the image to be restored;

[0007] Randomly initialize parameters of all candidate forward operators in the candidate forward operator set;

[0008] Initializing the image to be restored and inputting it into the visual generation model, so that the visual generation model performs a restoration operation based on all the candidate forward operators in parallel and iteratively to obtain a corresponding plurality of restored images;

[0009] The best matching candidate forward operator and the corresponding restored image are determined based on the image to be restored and the multiple restored images.

[0010] In an implementation of the first aspect, the method further includes:

[0011] Constructing a joint distribution model, the joint distribution model includes the image to be restored, the parameters of the candidate forward operator, the candidate forward operator and the restored image; wherein,

[0012] The joint distribution model is constructed based on the likelihood of the image to be restored, the conditional distribution of the parameters of the candidate forward operators, the prior distribution of the candidate forward operators and the prior distribution of the restored image.

[0013] In an implementation of the first aspect, the restoration operation includes:

[0014] Acquire a current restored image based on an input image of a current iteration, a candidate forward operator, and parameter sampling of the candidate forward operator;

[0015] Optimizing parameters of the candidate forward operator based on maximum a posteriori estimation;

[0016] The current restored image is updated by resampling based on the optimized parameters of the candidate forward operator, and the updated current restored image is used as an input image for the next iteration.

[0017] In an implementation of the first aspect, acquiring a current restored image based on an input image of a current iteration, a candidate forward operator, and parameter sampling of the candidate forward operator includes:

[0018] Acquire a scoring function of the input image of the current iteration based on the candidate forward operator, the parameter of the candidate forward operator and the image to be restored;

[0019] The current restored image is acquired based on sampling of the scoring function.

[0020] In an implementation of the first aspect, acquiring the current restored image by sampling based on the scoring function includes:

[0021] Acquire the gradient information of the posterior distribution of the guide item and the input image of the current iteration based on the scoring function;

[0022] The current restored image is obtained based on the guide item and the gradient information of the posterior distribution of the input image of the current iteration; wherein the guide item is used to make the input image of the current iteration and the image to be restored consistent.

[0023] In an implementation manner of the first aspect, optimizing the parameters of the candidate forward operator based on maximum a posteriori estimation includes:

[0024] Acquire a loss function for image restoration based on the maximum a posteriori estimation;

[0025] Performing gradient calculation on parameters of the candidate forward operator based on the loss function;

[0026] The loss function is minimized based on a gradient descent algorithm to optimize the parameters of the candidate forward operator.

[0027] In an implementation of the first aspect, determining the best-matching candidate forward operator and the corresponding restored image based on the image to be restored and the multiple restored images includes:

[0028] Respectively obtaining a structural similarity index of the image to be restored and each restored image;

[0029] Determining the maximum structural similarity index based on all the structural similarity indexes;

[0030] The candidate forward operator corresponding to the largest structural similarity index is used as the best matching candidate forward operator, and the corresponding restored image is output.

[0031] In a second aspect, the present application provides an image restoration system based on visual generation of a large model, the system comprising:

[0032] An acquisition module, used for acquiring an image to be restored and a set of candidate forward operators for the image to be restored;

[0033] An initialization module, used for randomly initializing parameters of all candidate forward operators in the candidate forward operator set;

[0034] A restoration module, used for initializing the image to be restored and inputting it into the visual generation model, so that the visual generation model performs a restoration operation based on all the candidate forward operators in parallel and iteratively to obtain a corresponding plurality of restored images;

[0035] A matching module is used to determine the best matching candidate forward operator and the corresponding restored image based on the image to be restored and the multiple restored images.

[0036] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-mentioned image restoration method based on visual generation of a large model.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned image restoration method based on visual generation of a large model.

[0038] As described above, the image restoration method, system, device and storage medium based on visual generation of large models described in this application have the following beneficial effects:

[0039] 1. This application can achieve image restoration without additional training, greatly reducing the dependence on large-scale training data and computing resources.

[0040] 2. This application does not rely on pre-known degradation information, and achieves accurate modeling of complex image degradation mechanisms by optimizing forward operator parameters.

[0041] 3. This application has good generalization ability, can adapt to various forms of image damage, and is suitable for various image restoration occasions.

[0042] 4. This application significantly improves the efficiency and resource adaptability of the method through the design of parallel sampling algorithm.

[0043] 5. This application uses the diffusion inverse process to gradually restore image details and outputs the forward operator and its parameters that best match the degradation mechanism, further enhancing the interpretability of the method and making it suitable for image restoration tasks in actual complex industrial scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1A Shown is a schematic diagram of an application scenario of the image restoration method based on visual generation of a large model described in this application.

[0045] Figure 1B Shown are structural diagrams of the client-cloud interaction scenarios in these implementations.

[0046] Figure 2 Shown is a flow chart of an image restoration method based on visual generation of a large model in an embodiment of the present application.

[0047] Figure 3 Shown is a flow chart of an image restoration method based on visual generation of a large model in an embodiment of the present application.

[0048] Figure 4 Shown is a flow chart of an image restoration method based on visual generation of a large model in an embodiment of the present application.

[0049] Figure 5 Shown is a schematic diagram of the effect of the image restoration method based on visual generation of a large model described in an embodiment of the present application.

[0050] Figure 6 Shown is a structural schematic diagram of an image restoration system based on visual generation of a large model in an embodiment of the present application.

[0051] Figure 7Shown is a schematic structural diagram of an electronic device of the present application in one embodiment. DETAILED DESCRIPTION

[0052] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0053] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0054] In addition, in this application, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0055] At present, the existing technologies in image restoration tasks are mainly divided into two categories: the first category is training-based methods, which train neural networks to achieve image restoration by constructing a dataset containing damaged images and clear image pairs. This type of method relies on high-quality large-scale datasets, which are often difficult to obtain in practical applications. At the same time, due to the limited generalization ability of neural networks, when the degraded form of the input image is inconsistent with the pattern in the training data, the restoration effect will be significantly reduced. In addition, the training process itself is complex and computationally expensive, and the demand for computing resources is particularly significant.

[0056] The second type of method uses a generative model to achieve restoration by injecting information about the image to be restored during the image generation process without the need for additional training. However, this type of method usually requires prior knowledge of the forward operator and its parameters that cause image degradation, which is a major difficulty in practical applications. In real scenes, the mechanism of image degradation is often unknown and complex, so this restoration method that relies on clear degradation information is difficult to meet actual needs. In contrast, blind image restoration is closer to practical applications because it does not require prior degradation information, but directly infers the degradation mechanism from the damaged image and restores it.

[0057] In general, the first type of method performs well in scenarios covered by training data, but has poor generalization ability; although the second type of method is free from the limitation of training, its dependence on degraded information limits its applicability in real scenarios. Therefore, the existing technology still has significant limitations in terms of computational efficiency, generalization ability, and adaptability to actual complex scenarios, and further improvement is urgently needed.

[0058] In order to at least solve the above problems, the embodiment of the present application provides an image restoration method based on a large model of visual generation, by constructing a set of forward operators covering various forms of image degradation, and gradually restoring the image in the diffusion inverse process, while optimizing the parameters of the candidate forward operators. Finally, the forward operator and its parameters that best match the image degradation are selected according to the optimization results, and a high-quality clear image is output. The present application avoids dependence on degradation information, has good generalization ability, improves stability and applicability, and has higher interpretability, and is suitable for image restoration tasks in actual complex scenes.

[0059] In some other embodiments, the image restoration method based on visual generation of a large model provided in the embodiments of the present application can be applied to Figure 1A The image restoration device based on visual generation of large models is shown. Figure 1A As shown, the image restoration device 1 based on visual generation of a large model includes a processor 11 and a database 12. The image restoration method based on visual generation of a large model provided in the embodiment of the present application can be applied to the processor 11.

[0060] in, Figure 1A The processor 11 in the embodiment may be a single processor or a processor cluster or a cloud computing center composed of multiple processors, and the specific details are not limited here. Figure 1A Only one processor 11 and one database 12 are shown, but it should be understood that Figure 1A The examples are only used to understand the present solution, and the specific numbers of processors 11 and databases 12 should be flexibly determined based on actual conditions.

[0061] In some other embodiments, the image restoration device 1 based on visual generation of a large model may not include the database 12, but only include a processor 11 with a storage function. The image restoration method based on visual generation of a large model provided in the embodiment of the present application can be applied to the processor 11. The processor 11 with a storage function may include a computer, a mobile phone, etc., which is not limited here.

[0062] In still other embodiments, the image restoration method based on visual generation of a large model described in the present application can be applied to end-cloud interaction scenarios. Figure 1B The following is a schematic diagram of the structure of the client-cloud interaction scenario in these implementations. Figure 1B As shown, the terminal-cloud interaction system 2 includes a terminal 20 and a cloud server 21. The terminal 20 and the cloud server 21 can communicate with each other, and the communication method is not limited to wired or wireless.

[0063] Among them, the terminal 20 can be mobile or fixed, for example, the terminal 20 can be a wireless terminal or a wired terminal. The wireless terminal can refer to a device with wireless transceiver function, which can be deployed in the image restoration scene. The terminal 20 can be a mobile phone, a laptop computer, etc., which is not limited here. The cloud server 21 can include one or more servers, or one or more processing nodes, or one or more virtual machines running on the server. The cloud server 21 can also be called a server cluster, a management platform, an image restoration center based on visual generation of large models, etc., which is not limited in the embodiments of the present application.

[0064] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0065] The following embodiments of the present application provide an image restoration method based on visual generation of a large model. For example, the method can be Figure 1A The processor 11 or Figure 1B It is implemented by the cloud server 21 shown. Figure 2 The flowchart of the image restoration method based on visual generation of a large model described in the embodiment of the present application is shown as follows: Figure 2 As shown, the image restoration method based on visual generation of a large model includes steps S1 to S4.

[0066] S1. Obtain an image to be restored and a set of candidate forward operators for the image to be restored.

[0067] Regarding image restoration, it is usually necessary to recover the original clear image from a known damaged image and a known forward operator, which is defined as Among them, y represents the forward operator The image to be restored after the action, is the parameter in the forward operator, x is the clear image we need to restore (that is, the restored image), z is the independent and identically distributed Gaussian noise, and the variance is

[0068] In practical applications, images may be damaged in various forms, which may be caused by noise, blur, compression distortion and other factors. In order to effectively restore a clear image, a candidate forward operator set suitable for the actual scene is constructed through step S1, that is, a set of potential image damage factors of the image to be restored, denoted as This set includes all candidate forward operators for simulating different types of image damage. Specifically, the candidate forward operator set covers a variety of possible image damage types, such as motion blur, JPEG compression, low resolution, etc., and can effectively simulate various forms of damage in practical applications. Each candidate forward operator corresponds to one image damage type.

[0069] S2. Randomly initialize the parameters of all candidate forward operators in the candidate forward operator set.

[0070] By randomly initializing the parameters of all candidate forward operators, we can ensure that large visual generation models have a good starting point.

[0071] S3. Initialize the image to be restored and input it into the visual generation model, so that the visual generation model performs restoration operations based on all the candidate forward operators in parallel and iteratively to obtain multiple corresponding restored images.

[0072] In the process of image restoration, as the visual generation model is gradually inferred, it is not only necessary to restore the clear image, but also to optimize the parameters of each candidate forward operator at the same time. Therefore, in order to improve the efficiency and resource adaptability of the method, a parallel sampling method is adopted. Step S3 adopts a parallel sampling design, which can simultaneously select from the candidate forward operator set All forward operators are integrated in , and then a single batch is constructed for processing, that is, the reduction operation is performed iteratively in parallel based on all candidate forward operators.

[0073] The following will take a candidate forward operator as an example to illustrate the specific content of step S3. However, it should be noted that in fact, the model simultaneously executes step S3 for all candidate forward operators, thereby greatly improving the computing efficiency and saving computing resources.

[0074] First, a joint distribution model is constructed to accurately represent the relationship between the variables in the image restoration process. Specifically, the joint distribution model includes the image to be restored, the parameters of the candidate forward operators, the candidate forward operators, and the restored image; wherein the joint distribution model is constructed based on the likelihood of the image to be restored, the conditional distribution of the parameters of the candidate forward operators, the prior distribution of the candidate forward operators, and the prior distribution of the restored image.

[0075] That is, the joint distribution model includes the restored image x0, the image to be restored y, a candidate forward operator and its corresponding parameters Then the joint distribution model can be expressed as:

[0076]

[0077] in, is the conditional probability density function, which means that given x0, and parameters After that, the distribution of the image y to be restored, and the conditional distribution of y is Gaussian distribution The mean is The covariance matrix is

[0078] Where p(x0) is the prior distribution of image x0, which is modeled using a pre-trained visual generative model. The prior distribution is the probability distribution of x0 in the absence of any observed data. It reflects the choice based on domain knowledge, historical data, or other assumptions before data is observed.

[0079] in, is the forward operator parameter The conditional distribution of Forward operator The prior distribution of is defined as a uniform distribution in this embodiment.

[0080] After the joint distribution model is determined, the image to be restored y is initialized to serve as the initialization input of the visual generation model. Specifically, noise disturbance is added to the image to be restored y to serve as the initial state (noisy image) x T The input is then fed into a large visual generation model, which can provide meaningful semantic information and assist the model in converging to more accurate clear image restoration.

[0081] Afterwards, the visual generation model will be based on the initial state x T and a candidate forward operator and its corresponding parameters Perform a restore operation. Figure 3The flowchart of the image restoration method based on visual generation of a large model described in the embodiment of the present application is shown as follows: Figure 3 As shown, the restoration operation includes steps S31 to S33.

[0082] S31. Acquire a current restored image based on an input image of a current iteration, a candidate forward operator, and parameter sampling of the candidate forward operator.

[0083] Specifically, a scoring function of the input image of the current iteration is obtained based on the candidate forward operator, the parameters of the candidate forward operator and the image to be restored, and the current restored image is obtained based on sampling of the scoring function. In fact, the visual generation model is a diffusion model, which can generate a random image without any constraints. However, in this embodiment, it is necessary to guide the reasoning of the visual generation model to complete the image restoration task. That is, it is necessary to add constraint information in the inverse diffusion process of the visual generation model so that the image it generates is the restored image required by the task, rather than an irrelevant random image.

[0084] Specifically, the visual generative model generates a restored image by gradually denoising in an iterative manner. Therefore, the visual generative model learns a scoring function that can give a noisy image x t Predict the score (gradient) of its original image x0. This scoring function can also be considered as modeling the noise at each moment, thereby estimating the direction of the noise and helping the reverse denoising process. That is, through a specific scoring function, the model is guided to iterate and gradually denoise, so that the generated restored image x0 satisfies a given candidate forward operator, that is, satisfies In the case of , it is also necessary to satisfy the requirement that it is as close as possible to the image y to be restored.

[0085] Specifically, the input image x of the current iteration t The posterior distribution of is: By x t The scoring function of x guides the diffusion process to sample from this distribution. t The scoring function is:

[0086]

[0087] The first term is the data prior provided by the pre-trained visual generation model, which represents the input image x of the current iteration. t In a candidate forward operator and parameters The generation process under is the gradient information of the posterior distribution of the input image of the current iteration. The second guiding term forces the input image x of the current iteration to tThe consistency with the image to be restored y constrains the similarity between the current restored image and the actual degraded image.

[0088] Furthermore, acquiring the current restored image based on sampling of the scoring function includes: acquiring a guide item and gradient information of the posterior distribution of the input image of the current iteration based on the scoring function; acquiring the current restored image based on the guide item and gradient information of the posterior distribution of the input image of the current iteration; wherein the guide item is used to make the input image of the current iteration and the image to be restored consistent.

[0089] As mentioned above, the gradient information of the posterior distribution of the input image of the current iteration is It can guide sampling, and the guide item is As a constraint, it can guide the input image x of the current iteration t The consistency with the image y to be restored is to achieve that the current restored image generated by the model satisfies a given candidate forward operator and is as close as possible to the image y to be restored.

[0090] Furthermore, the scoring function is approximated by the noise prediction network of the visual generation large model, which is defined as follows:

[0091]

[0092] Among them, μ θ (x t ) is the predicted value generated by the model (denoised predicted value), σ θ (x t ) is the current state x t The noise estimation under is a time-step-dependent scalar, That is, it is a guiding term, which adjusts the image by making the input image of the current iteration consistent with the image y to be restored.

[0093] In fact, the scoring function is approximated by the noise prediction network of the visual generative model, so that the noise term of the predicted image at each time t can be learned, and the noise can be effectively removed to obtain the current restored image x t-1 .

[0094] By iteratively performing the above restoration operation, the visual generation model can gradually guide the input image x of the current iteration through the synergy of the guidance term and the noise prediction network. t Approximate the correct restored image x0.

[0095] S32. Optimize the parameters of the candidate forward operator based on maximum a posteriori estimation.

[0096] Specifically, each time the restoration operation is performed to restore the image, the parameters of the candidate forward operators need to be optimized at the same time. In some embodiments, for each candidate forward operator The maximum a posteriori estimation is used to optimize its parameters, and its goal is to maximize the given data (the image to be restored y and the input image x of the current iteration) t ) to estimate the parameters of the candidate forward operator That is, find the optimal parameters So that the posterior probability maximum.

[0097] In some embodiments, optimizing the parameters of the candidate forward operator based on maximum a posteriori estimation includes: obtaining a loss function for image restoration based on the maximum a posteriori estimation; performing gradient calculation on the parameters of the candidate forward operator based on the loss function; and minimizing the loss function based on a gradient descent algorithm to optimize the parameters of the candidate forward operator.

[0098] According to the maximum a posteriori estimation, the optimal parameters It is obtained by optimizing the following objective function:

[0099]

[0100] Among them, the first term in the optimization problem on the right side corresponds to the likelihood of y in the joint distribution, that is, the loss function of image restoration, which measures the probability of passing the candidate forward operator Denoised image The error converted to the image y to be restored; the second term corresponds to the joint distribution The prior can suppress overfitting and guide the parameters to converge to reasonable values. Among them, since the model obtains the input image x of the current iteration during the iteration process t is actually a noisy image. Therefore, in order to avoid the influence of noise on the parameter estimation of the candidate forward operator, based on x t Get the denoised image by estimating To optimize the parameters.

[0101] In the process of parameter optimization, in order to solve the optimal parameters The gradient descent method is used to solve the optimization problem. By minimizing the above objective function, we can get a recursive optimization process. The specific steps are as follows:

[0102] 1. Calculate the gradient: For a given loss function: Calculate relative to parameter The gradient is:

[0103]

[0104] 2. Use the calculated gradient information to update the parameters using the gradient descent update rule

[0105] In some cases, the candidate forward operator is not differentiable, which makes the standard gradient descent method inapplicable directly. In this case, the zero-order optimization method can be used to solve it. In this way, we can estimate the parameters of the forward operator based on the error between the current restored image and the image to be restored. This enables more accurate image restoration.

[0106] S33, resampling based on the optimized parameters of the candidate forward operator to update the current restored image, and using the updated current restored image as an input image for the next iteration.

[0107] Specifically, based on the candidate forward operator The optimized parameters are used to execute step S31 again to obtain an updated current restored image. The updated image is used as the input image for the next iteration, and steps S31 to S33 are repeated. That is, in each iteration, the restoration operation (steps S31 to S33) is performed. When step S33 is completed, step S31 is repeated to obtain an updated current restored image. At this point, one iteration ends, and the updated current restored image is used as the input for the next iteration.

[0108] In each round of iterative sampling, the model can use the constraints of the degraded image and the known forward operator to continuously optimize the restored image until it converges to a clear image.

[0109] S4. Determine the best-matching candidate forward operator and the corresponding restored image based on the image to be restored and the multiple restored images.

[0110] Figure 4 The flowchart of the image restoration method based on visual generation of a large model described in the embodiment of the present application is shown as follows: Figure 4 As shown, step S4 includes steps S41 to S43.

[0111] S41, respectively obtaining a structural similarity index of the image to be restored and each restored image.

[0112] S42: Determine the maximum structural similarity index based on all the structural similarity indexes.

[0113] S43: taking the candidate forward operator corresponding to the largest structural similarity index as the best matching candidate forward operator, and outputting the corresponding restored image.

[0114] In some embodiments, the structural similarity index (SSIM) is used to approximate SSIM is an image quality assessment metric that can effectively measure the perceptual similarity between two images. It takes into account brightness, contrast, and structural information. Using SSIM can effectively capture the difference between the restored image and the image to be restored at the perceptual level. By comparing the SSIM values ​​of all restored images, the largest structural similarity index can be determined. At this point, the candidate forward operator with the largest SSIM value is selected. and its corresponding restored image As the final output restored image.

[0115]

[0116] Where i represents the i-th candidate forward operator.

[0117] Therefore, the present application can achieve image restoration without additional training, greatly reducing the dependence on large-scale training data and computing resources. Secondly, the present application does not rely on pre-known degradation information, and achieves accurate modeling of complex image degradation mechanisms by optimizing forward operator parameters. Thirdly, the present application has good generalization capabilities and can adapt to various forms of image damage. Finally, the present application significantly improves the efficiency and resource adaptability of the method through the design of parallel sampling algorithms. The present application uses the inverse process of diffusion to gradually restore image details, and outputs the forward operator and its parameters that best match the degradation mechanism, further enhancing the interpretability of the method, which is suitable for image restoration tasks in actual complex industrial scenes. The specific image restoration effects are as follows: Figure 5 shown.

[0118] The protection scope of the image restoration method based on visual generation of a large model described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0119] An embodiment of the present application also provides an image restoration system based on a visually generated large model. The image restoration system based on a visually generated large model can implement the image restoration method based on a visually generated large model described in the present application. However, the implementation device of the image restoration system based on a visually generated large model described in the present application includes but is not limited to the structure of the image restoration system based on a visually generated large model listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0120] Figure 6 The structure diagram of the image restoration system based on visual generation of a large model according to an embodiment of the present application is shown as follows: Figure 6As shown, the image restoration system 4 based on visual generation of large models includes an acquisition module 41, an initialization module 42, a restoration module 43 and a matching module 44.

[0121] An acquisition module 41 is used to acquire an image to be restored and a set of candidate forward operators of the image to be restored;

[0122] An initialization module 42, used for randomly initializing parameters of all candidate forward operators in the candidate forward operator set;

[0123] A restoration module 43, used for initializing the image to be restored and inputting it into the visual generation model, so that the visual generation model performs a restoration operation based on all the candidate forward operators in parallel and iteratively to obtain a corresponding plurality of restored images;

[0124] The matching module 44 is used to determine the best matching candidate forward operator and the corresponding restored image based on the image to be restored and the multiple restored images.

[0125] It should be noted that the structures and principles of the acquisition module 41, initialization module 42, restoration module 43 and matching module 44 correspond one to one with the steps in the above-mentioned image restoration method based on visual generation of a large model, so they will not be described in detail here.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0127] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0128] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0129] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0130] The embodiment of the present application further provides an electronic device, wherein the electronic device includes a processor and a memory.

[0131] The memory is used to store computer programs.

[0132] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0133] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned image restoration method based on visual generation of a large model.

[0134] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0135] like Figure 7 As shown, the electronic device of the present application is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 51, a memory 52, and a bus 53 connecting different system components (including the memory 52 and the processing unit 51).

[0136] Bus 53 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0137] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0138] The memory 52 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 523 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 53 via one or more data medium interfaces. The memory 52 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0139] A program / utility 524 having a set (at least one) of program modules 5241 may be stored, for example, in the memory 52, such program modules 5241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 5241 generally perform the functions and / or methods of the embodiments described herein.

[0140] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 54. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 55. Figure 7 As shown, the network adapter 55 communicates with other modules of the electronic device via the bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0141] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0142] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0143] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0144] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An image restoration method based on visual generation of a large model, characterized in that: The method comprises: Acquire an image to be restored and a set of candidate forward operators for the image to be restored; Randomly initialize parameters of all candidate forward operators in the candidate forward operator set; Initializing the image to be restored and inputting it into the visual generation model, so that the visual generation model performs a restoration operation based on all the candidate forward operators in parallel and iteratively to obtain a corresponding plurality of restored images; The best matching candidate forward operator and the corresponding restored image are determined based on the image to be restored and the multiple restored images.

2. The image restoration method based on visual generation of large models according to claim 1 is characterized in that: The method further comprises: Constructing a joint distribution model, the joint distribution model includes the image to be restored, the parameters of the candidate forward operator, the candidate forward operator and the restored image; wherein, The joint distribution model is constructed based on the likelihood of the image to be restored, the conditional distribution of the parameters of the candidate forward operators, the prior distribution of the candidate forward operators and the prior distribution of the restored image.

3. The image restoration method based on visual generation of large models according to claim 1 is characterized in that: The restoration operation includes: Acquire a current restored image based on an input image of a current iteration, a candidate forward operator, and parameter sampling of the candidate forward operator; Optimizing parameters of the candidate forward operator based on maximum a posteriori estimation; The current restored image is updated by resampling based on the optimized parameters of the candidate forward operator, and the updated current restored image is used as an input image for the next iteration.

4. The image restoration method based on visual generation of large models according to claim 3 is characterized in that: Acquiring a current restored image based on an input image of a current iteration, a candidate forward operator, and parameter sampling of the candidate forward operator includes: Acquire a scoring function of the input image of the current iteration based on the candidate forward operator, the parameter of the candidate forward operator and the image to be restored; The current restored image is acquired based on sampling of the scoring function.

5. The image restoration method based on visual generation of large models according to claim 4 is characterized in that: Sampling and acquiring the current restored image based on the scoring function includes: Acquire the gradient information of the posterior distribution of the guide item and the input image of the current iteration based on the scoring function; The current restored image is obtained based on the guide item and the gradient information of the posterior distribution of the input image of the current iteration; wherein the guide item is used to make the input image of the current iteration and the image to be restored consistent.

6. The image restoration method based on visual generation of large models according to claim 3 is characterized in that: Optimizing the parameters of the candidate forward operator based on maximum a posteriori estimation includes: Acquire a loss function for image restoration based on the maximum a posteriori estimation; Performing gradient calculation on parameters of the candidate forward operator based on the loss function; The loss function is minimized based on a gradient descent algorithm to optimize the parameters of the candidate forward operator.

7. The image restoration method based on visual generation of large models according to claim 1 is characterized in that: Determining the best-matching candidate forward operator and the corresponding restored image based on the image to be restored and the multiple restored images includes: Respectively obtaining a structural similarity index of the image to be restored and each restored image; Determining the maximum structural similarity index based on all the structural similarity indexes; The candidate forward operator corresponding to the largest structural similarity index is used as the best matching candidate forward operator, and the corresponding restored image is output.

8. An image restoration system based on visual generation of large models, characterized in that: The system comprises: An acquisition module, used for acquiring an image to be restored and a set of candidate forward operators for the image to be restored; An initialization module, used for randomly initializing parameters of all candidate forward operators in the candidate forward operator set; A restoration module, used for initializing the image to be restored and inputting it into the visual generation model, so that the visual generation model performs a restoration operation based on all the candidate forward operators in parallel and iteratively to obtain a corresponding plurality of restored images; A matching module is used to determine the best matching candidate forward operator and the corresponding restored image based on the image to be restored and the multiple restored images.

9. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory so that the electronic device executes the image restoration method based on visual generation of a large model according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, the image restoration method based on visual generation of a large model described in any one of claims 1 to 7 is implemented.

Images