Image restoration method, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image restoration method, electronic equipment and a computer readable storage medium, and relates to the technical field of large model technology and image restoration. The method comprises the following steps: acquiring a to-be-restored degraded image; sampling inversion processing is carried out on the degraded image, a noise image is obtained, the noise image is used for reserving potential image features of the degraded image, and the potential image features are used for reproducing the degraded image according to the sampling process; and performing image restoration processing on the noise image to generate a restored image. The image restoration method and device solve the technical problems that the cost is high and a known degradation model needs to be used when an image restoration technology based on a bottom degradation model is used for training the bottom degradation model in the related technology.

Description

Technical Field

[0001] This application relates to the fields of large model technology and image restoration technology. Specifically, it relates to an image restoration method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image restoration is a digital image processing technology, which refers to the process of restoring a clean and high-quality image from a degraded image.

[0003] Currently, image restoration technology based on deep learning has become the mainstream, and its paradigm can be roughly divided into a supervised paradigm and an unsupervised paradigm. Supervised image restoration relies on a large-scale paired dataset collected in advance to train the model. However, when the underlying degradation model changes, a new dataset needs to be collected to train a new model, resulting in high training costs and long training time. Unsupervised image restoration uses the degradation model to generate clean images through maximum likelihood or posterior sampling problems. However, a known degradation model still needs to be used during the inference process, and the degradation model is usually too complex, resulting in a large amount of computation and being unable to be estimated.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of this application provide an image restoration method, an electronic device, and a computer-readable storage medium to at least solve the technical problems in the related art that the image restoration technology based on the underlying degradation model has high costs when training the underlying degradation model and needs to use a known degradation model.

[0006] According to one aspect of the embodiments of this application, an image restoration method is provided, including: obtaining a degraded image to be restored; performing sampling inversion processing on the degraded image to obtain a noise image, where the noise image is used to retain the potential image features of the degraded image, and the potential image features are used to reproduce the degraded image according to the sampling process; performing image restoration processing on the noise image to generate a restored image.

[0007] According to another aspect of the embodiments of this application, an image restoration method is further provided, including: obtaining a target role degraded image to be restored; performing sampling inversion processing on the target role degraded image to obtain a target role noise image, where the target role noise image is used to retain the potential target role image features of the target role degraded image, and the target role potential image features are used to reproduce the target role degraded image according to the sampling process; performing image restoration processing on the target role noise image to generate a target role restored image.

[0008] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; a processor for running the program, wherein when the program runs, it executes any one of the above image restoration methods.

[0009] According to another aspect of the embodiments of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program runs, it controls the device where the computer-readable storage medium is located to execute any one of the above image restoration methods.

[0010] In the embodiments of the present application, by obtaining a degraded image to be restored, then performing sampling inversion processing on the degraded image to obtain a noisy image retaining the potential image features of the degraded image, and finally performing image restoration processing on the noisy image, a clean and high-quality restored image is generated to complete the repair of the degraded image, thereby achieving the purpose of obtaining a clean and high-quality restored image. Since the present application can be directly applied to the degraded image to be restored without assuming a degradation model, that is, without training or fine-tuning for the degradation model, it has stronger versatility, can be inferred in the encoding space of the VAE, and reduces the model training cost, thus achieving the technical effect of repairing the degraded image at low cost and obtaining a clean and high-quality restored image, and further solving the technical problem that the image restoration technology based on the underlying degradation model in the related art has a high cost in training the underlying degradation model and requires the use of a known degradation model.

[0011] It is easy to notice that the above general description and the following detailed description are only for exemplifying and explaining the present application and do not constitute a limitation to the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0013] Figure 1 is a schematic diagram of an application scenario of an image restoration method according to Embodiment 1 of the present application;

[0014] Figure 2 is a flowchart of an image restoration method according to Embodiment 1 of the present application;

[0015] Figure 3 is a schematic diagram of DDIM inversion according to Embodiment 1 of the present application;

[0016] Figure 4 is a schematic diagram of the sampling process according to Embodiment 1 of the present application;

[0017] Figure 5 It is a flowchart of an image restoration method according to Embodiment 2 of the present application;

[0018] Figure 6 It is a schematic structural diagram of an image restoration device according to Embodiment 3 of the present application;

[0019] Figure 7 It is a schematic structural diagram of another image restoration device according to Embodiment 3 of the present application;

[0020] Figure 8 It is a structural block diagram of a computer terminal according to an embodiment of the present application. Detailed implementation manners

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0023] The technical solution provided by the present application is mainly implemented by large model technology. Here, the large model refers to a deep learning model with a large number of model parameters, which usually can include hundreds of millions, tens of billions, hundreds of billions, trillions or even more than one quadrillion model parameters. The large model can also be called a foundation model. Through pre-training of the large model with a large amount of unlabeled corpus, a pre-trained model with more than one hundred million parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization ability. For example, large language models (LLMs), multi-modal pre-training models, etc.

[0024] It should be noted that in actual applications, the large model can be fine-tuned with a small number of samples on the pre-trained model, so that the large model can be applied to different tasks. For example, the large model can be widely applied in the fields of natural language processing (NLP for short), computer vision, speech processing, etc. Specifically, it can be applied to tasks in the field of computer vision such as visual question answering (VQA for short), image captioning (IC for short), image generation, etc., and can also be widely applied to tasks in the field of natural language processing such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0025] First, some nouns or terms that appear in the process of describing the embodiments of the present application are applicable to the following explanations:

[0026] Degraded image: An image that has lost the quality and details of the original image due to some reasons. Degradation may be caused by factors such as image damage, noise, blur, or compression. Degraded images usually lose sharpness, color, and details, affecting the viewer's understanding and recognition of the image content.

[0027] Diffusion Model: A mathematical model used to describe the diffusion process and is a commonly used generative model.

[0028] Denoising Diffusion Probabilistic Model (DDIM): A sampling algorithm based on a probabilistic model, used to infer the potential real data given the observed data, and can be understood as a diffusion model with deterministic sampling.

[0029] Currently, image inpainting techniques based on deep learning have shown good performance and dominated this field, and their paradigms can be roughly divided into supervised paradigms and unsupervised paradigms. Supervised image inpainting solutions usually rely on pre-collecting large-scale paired datasets to train the model, and this solution implicitly assumes that the training and test data should be of the same distribution. Therefore, when the test cases deviate from the pre-assumed distribution, the performance of this solution will deteriorate severely. In addition, once the underlying degradation model changes, it is necessary to re-collect a new dataset and re-train a new model, and this process is both time-consuming and expensive, resulting in a high training cost.

[0030] Unsupervised image inpainting methods explicitly utilize a degradation model to generate clean images through maximum likelihood or posterior sampling problems. For example, assuming a linear degradation model and relying on the ideal properties of linear formulas to sample from the posterior distribution, and implementing image inpainting by means of a diffusion model without learning. However, in the inference process, it is still required to use a known degradation model, and in practice, the underlying degradation model may be too complex to estimate or the computational cost may be too high to apply. In addition, due to the projection of the Variational Autoencoder (VAE), the entanglement between degradation information and clean information may become more complex, so the above methods may not be able to train the diffusion in the encoding space of the VAE.

[0031] It can be seen that the image inpainting technology of the underlying degradation model in the related art has the following defects.

[0032] Defect 1: It is necessary to pre-collect a paired data set to train the underlying degradation model. The data set collection is difficult and the training cost of the underlying degradation model is high.

[0033] Defect 2: It does not allow an unknown degradation model.

[0034] Defect 3: The underlying degradation model is usually too complex to estimate, and the underlying degradation model usually has a large computational cost and cannot be applied.

[0035] Defect 4: It cannot be equipped with the diffusion trained in the encoding space of the VAE.

[0036] In view of the above defects, no effective solution has been proposed before this application.

[0037] Embodiment 1

[0038] According to an embodiment of the present application, an image restoration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0039] Considering that the number of model parameters of the large model is huge and the computing resources of the mobile terminal are limited, the above image restoration method provided by the embodiment of the present application can be applied to, for example, Figure 1 as shown in the application scenario, but not limited thereto. In such as Figure 1In the application scenario shown, the large model is deployed in server 10, and server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. Here, the client devices 20 can include, but are not limited to: smart phones, tablets, laptops, handheld computers, personal computers, smart home devices, in-vehicle devices, etc. The client device 20 can interact with the user through a graphical user interface to call the large model, thereby implementing the method provided in the embodiments of the present application.

[0040] In the embodiments of the present application, the system composed of the client device and the server can perform the following steps: The client device performs steps such as obtaining the degraded image to be restored and sending the degraded image to be restored to the server. The server performs steps such as performing sampling inversion processing on the obtained degraded image to be restored to obtain a noise image, performing image restoration processing on the noise image to generate a restored image, and returning the restored image to the client device. It should be noted that in the case where the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be performed in the client device.

[0041] In the above operating environment, the present application provides an Figure 2 image restoration method as shown. Figure 2 It is a flowchart of an image restoration method according to Embodiment 1 of the present application. As Figure 2 shown, the method can include the following steps:

[0042] Step S21, obtaining the degraded image to be restored;

[0043] Step S22, performing sampling inversion processing on the degraded image to obtain a noise image, where the noise image is used to retain the potential image features of the degraded image, and the potential image features are used to reproduce the degraded image according to the sampling process;

[0044] Step S23, performing image restoration processing on the noise image to generate a restored image.

[0045] The degraded image to be restored can be understood as a low-quality image that needs to be image repaired. Generally, the clarity, color, and details of the degraded image are poor. Exemplarily, the degraded image to be restored can be an input image input by the user that wants to be repaired.

[0046] Sampling inversion processing of a degraded image can be understood as the inversion of sampling processing of the degraded image. The inversion of sampling processing is usually used to restore the original image or extract specific information, that is, to restore the original image of the degraded image or extract specific information in the degraded image, and can restore the details and information of the original image, or extract specific image features or information in some way, which is not limited here.

[0047] Exemplarily, sampling processing of a degraded image can adopt a Denoising Diffusion Probabilistic Model (DDIM) sampling algorithm, that is, perform DDIM inversion on the degraded image, which is not limited here.

[0048] The potential image features of the degraded image can be the details and information of the original image restored by inverting the sampling processing of the degraded image, and can be understood as the latent variables in the low-probability region of the degraded image. In the embodiments of the present application, the potential image features can reproduce the degraded image according to the sampling process, that is, any latent variable contains the required information about the degraded image.

[0049] The noise image is the image obtained after sampling inversion processing of the degraded image. The noise image retains the potential image features of the degraded image, that is, the noise image also retains the required information about the degraded image.

[0050] The restored image is the image obtained after image restoration processing of the noise image. The restored image can be understood as a complete, clean and high-quality image corresponding to the degraded image, that is, the image obtained after image repair of the degraded image. Exemplarily, the image restoration processing performed on the noise image includes but is not limited to image denoising, image repair, deblurring, super-resolution, etc., which is not limited here.

[0051] In the embodiments of the present application, by obtaining the degraded image y to be restored, and then performing sampling inversion processing on the degraded image y, a noise image y is obtained that retains the potential image features of the degraded image y τ , and finally by performing image restoration processing on the noise image y τ , a clean and high-quality restored image x is generated to complete the repair of the degraded image.

[0052] It can be seen that in the inference stage of the present application, a high-quality image can be obtained only through the inversion of the sampling algorithm and the image restoration algorithm. Both the inversion of the sampling algorithm and the image restoration algorithm are general and can be directly applied to the degraded image to be restored. Therefore, there is no need to assume a degradation model, making the generality stronger. It can overcome the technical problems in the related art of reasoning in the encoding space of VAE and requiring a special degradation model to complete the reasoning, and allows an unknown degradation model. Moreover, the present application does not require paired data for supervised training, solving the problem of difficult data collection. In addition, when processing highly JPEG-compressed images, compared with the supervised schemes in the related art, such as the Fully Convolutional Belief Neural Network (FBCNN), and the unsupervised schemes, such as the Deep Distributed Representation Model for JPEG Image Compression (DDRM-JPEG), the present application can obtain clearer and higher-quality images.

[0053] The above image restoration method provided by the embodiments of the present application can be, but is not limited to, applied to application scenarios involving image restoration in fields such as game services, e-commerce services, education services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services. For example: image restoration in games, image restoration in e-commerce, image restoration of medical images, etc., which are not limited here.

[0054] By adopting the embodiments of the present application, by obtaining the degraded image to be restored, then performing sampling inversion processing on the degraded image to obtain a noise image retaining the potential image features of the degraded image, and finally performing image restoration processing on the noise image, a clean and high-quality restored image is generated, completing the repair of the degraded image, thereby achieving the purpose of obtaining a clean and high-quality restored image. Since the present application can be directly applied to the degraded image to be restored without assuming a degradation model, that is, without training or fine-tuning for the degradation model, the generality is stronger, it can reason in the encoding space of VAE, and the model training cost is reduced, thus achieving the technical effect of repairing the degraded image at low cost and obtaining a clean and high-quality restored image, and further solving the technical problems in the related art that the image restoration technology based on the underlying degradation model has a high cost in training the underlying degradation model and requires the use of a known degradation model.

[0055] In an optional embodiment, in step S22, the sampling inversion processing of the degraded image to obtain a noise image includes the following method steps:

[0056] Step S221: Obtain a sampling trajectory, where the sampling trajectory is the trajectory generated during the process of deterministically sampling the degraded image.

[0057] Step S222: Perform noise-added sampling on the sampling trajectory to obtain a noisy image.

[0058] In the embodiments of the present application, the sampling inversion process can select the inversion of the deterministic sampling process, that is, the DDIM inversion. It can be understood that DDIM is a diffusion model of deterministic sampling, and the inversion of the sampling process of the degraded image is performed through this diffusion model of deterministic sampling.

[0059] Exemplarily, DDIM can be a pre-trained diffusion model. Considering that the inversion of the DDIM sampling algorithm of the diffusion model can obtain the required information of the degraded image, therefore, by reconstructing the degraded image, a series of latent image features of the degraded image are generated using the DDIM inversion algorithm. Even if all the information about the clean image is lost, the diffusion model can still generate a clean image, ensuring the fidelity of the degraded image and also solving the image restoration problem in extreme cases.

[0060] The sampling trajectory is the DDIM sampling trajectory. The sampling trajectory is the trajectory generated during the process of performing DDIM sampling on the degraded image, and is used to represent the distribution law of the sampling points in time or space when performing DDIM sampling processing on the degraded image.

[0061] In the embodiments of the present application, when performing sampling inversion processing on the degraded image, the sampling trajectory for performing DDIM inversion on the degraded image can be obtained first, and then noise-added sampling is performed on the sampling trajectory, so as to obtain the noisy image corresponding to the degraded image.

[0062] Exemplarily, the latent image feature y of the degraded image y can be determined through DDIM inversion τ , y τ which can be represented by formula (1).

[0063] y τ = DDIM -1 (y) Formula (1)

[0064] where DDIM -1 (·) represents the inversion of DDIM; τ represents the degree of DDIM inversion, 0 < τ ≤ T.

[0065] Figure 3 is the schematic diagram of the DDIM inversion according to Embodiment 1 of the present application. As Figure 3 shown, the degraded image can be Figure 3 the low-quality picture P 0 (y 0 ), Figure 3Three low-quality images are exemplified, and each low-quality image is represented by a symbol (i.e., Figure 3 the circles, rectangles, and triangles in Figure 3 ), and the corresponding sampling trajectories are as shown by the lines in t (y t ). Perform DDIM inversion processing on the three low-quality images P τ (y τ ), so as to obtain the corresponding noise image P

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] In an alternative embodiment, the latent image feature is a plurality of latent variables corresponding to the sampling trajectory, that is, the latent image feature can be understood as a plurality of sampling points located in the low-probability region in the sampling trajectory.

[0067] In an alternative embodiment, in step S23, perform image restoration processing on the noise image to generate a restored image, including the following method steps:

[0068] Step S231, perform image correction processing on the noise image within a continuous plurality of time steps to obtain a correction result;

[0069]

[0070] Step S232, perform denoising processing on the correction result to generate a restored image.

[0071] Since high-quality images are generated during sampling in the pre-trained diffusion model, indicating that the latent image features are located in the low-probability region, that is, the latent variables are located in the low-probability region, this application proposes to use the latent variables as an initialization and guide the latent variables to move to the nearby high-probability region through a designed correction algorithm, so as to obtain high-quality images.

[0072] It can be understood that the low-probability region and the high-probability region are evaluated by the diffusion model.

[0073] In the embodiments of this application, the image correction processing performed on the noise image can be variance-preserving sampling processing, which is not limited herein.

[0074] In the embodiments of this application, within a continuous plurality of time steps, by performing image correction processing on the noise image, a correction result is obtained, and then denoising processing is performed on the correction result, that is, denoising sampling is performed, so as to be able to generate a restored image.

[0075] Exemplarily, after obtaining the informative latent variable y τ , within each time step, the latent variable can be corrected by performing variance-preserving sampling on the latent variable, and gradually denoised through denoising processing to obtain a clean high-quality restored image.

[0075] In an alternative embodiment, in step S231, within a continuous plurality of time steps, an image correction process is performed on the noisy image to obtain a correction result, including the following method steps:

[0076] Step S2311, at a first moment, obtain the probability distributions of the Gaussian noise corresponding to the first sampling step index and the first sampling result corresponding to the second sampling step index within a continuous plurality of time steps, where the second sampling step index is the previous index adjacent to the first sampling step index;

[0077] Step S2312, calculate a second sampling result based on a preset coefficient, Gaussian noise, and probability distribution, and repeat the calculation until a preset number of sampling steps is reached to obtain a correction result, where the preset coefficient satisfies a preset constraint condition.

[0078] The sampling step index can be understood as the position of each sampling point marked by a digital index during sampling. Exemplarily, taking the sampling running for M steps as an example, that is, the preset number of sampling steps can be M steps, and m can be the index of the number of steps in this sampling process.

[0079] Exemplarily, taking the first sampling step index as m and the second sampling step index as m - 1 as an example, that is, the first sampling step index is the next index adjacent to the second sampling step index, and also the second sampling step index is the previous index adjacent to the first sampling step index.

[0080] The preset coefficient can be η l and η g , which are used to represent the parameters of the sampling function adopted when performing image correction processing on the noisy image and need to satisfy a preset constraint condition. Exemplarily, the preset coefficient can be the coefficient of variance-preserving sampling.

[0081] In the embodiment of the present application, when performing image correction processing on the noisy image within a continuous plurality of time steps, at a first moment, for example, at time t, the probability distributions of the Gaussian noise corresponding to the first sampling step index and the first sampling result corresponding to the second sampling step index within a continuous plurality of time steps can be obtained, that is, the Gaussian noise corresponding to the m-th step index within a continuous plurality of time steps is obtained and the probability distribution of the first sampling result corresponding to the (m - 1)-th step index After that, based on the preset coefficients η l and η g , Gaussian noise and probability distribution calculate a second sampling result Then repeat the foregoing image correction processing steps until the preset number of sampling steps is completed, that is, until M steps are completed, so as to obtain a correction result

[0082] Exemplarily, taking the image correction process of a noisy image using variance-preserving sampling as an example, the second sampling result obtained within each time step of the variance-preserving sampling can be represented by formula (2).

[0083]

[0084] Wherein, pt() represents a probability distribution; m takes values from 1 to M; It means that at time t, the initialization of the variance-preserving sampling is

[0085] In an alternative embodiment, the Gaussian noise is the noise with a mean of a first preset value, a variance of a second preset value, and the same dimension as the second sampling result added during the sampling iteration corresponding to the first sampling step index.

[0086] In the embodiments of the present application, the first preset value can be 0, and the second preset value can be denoted as l.

[0087] That is, the Gaussian noise is the noise with a mean of 0, a variance of l, and the same dimension as the second sampling result added during the sampling iteration corresponding to the first sampling step index m, which can be understood as the Gaussian noise with a mean of 0, a variance of l, and the same dimension as the added during the m-th step of the variance-preserving sampling iteration, and the same dimension as the mean.

[0088] In an alternative embodiment, the preset constraint condition is determined by a preset scalar and a preset noise addition method. The preset scalar is a scalar for determining the step size corresponding to a continuous plurality of time steps, and the preset noise addition method is the noise addition method of the diffusion noise corresponding to the pre-trained diffusion model.

[0089] The preset constraint condition can be determined by a preset scalar and a preset noise addition method. Among them, the preset scalar is a scalar for determining the step size corresponding to a continuous plurality of time steps, denoted as γ. Exemplarily, γ is specified to be between 0 and 1, satisfying 0 < γ < 1, and is a scalar for determining the step size. The preset noise addition method is the noise addition method of the diffusion noise corresponding to the pre-trained diffusion model, that is, the noise addition method of the diffusion noise corresponding to the pre-trained DDIM diffusion model, denoted as α t .

[0090] In the embodiments of the present application, the preset coefficient needs to satisfy the preset constraint condition, that is, η l needs to satisfy η l = γ(1 - α t ), and η g needs to satisfy

[0091] In an alternative embodiment, the image restoration method further includes the following method steps:

[0092] Step S2313: Obtain the noise predicted by the pre-trained diffusion model at the first moment and the preset noise addition method;

[0093] Step S2314: Calculate the probability distribution based on the noise predicted by the pre-trained diffusion model and the preset noise addition method.

[0094] Probability distribution can be determined based on the pre-trained diffusion model, and the noise predicted by the pre-trained diffusion model at the first moment can be expressed as

[0095] Probability distribution of the first sampling result can be obtained by acquiring the noise predicted by the pre-trained diffusion model at the first moment and the preset noise addition method α t , and calculated based on and α t .

[0096] Exemplarily, the probability distribution can be represented by formula (3).

[0097]

[0098] In an alternative embodiment, in step S232, denoising processing is performed on the correction result to generate a restored image, including the following method steps:

[0099] Step S2321: Perform deterministic sampling on the correction result to obtain a third sampling result at the second moment, and repeat sampling until the target moment is reached to generate a restored image, where the second moment is the previous moment adjacent to the first moment.

[0100] The second moment is the previous moment adjacent to the first moment, which can be denoted as the t - 1 moment.

[0101] In the embodiment of the present application, within each time step, after obtaining the correction result, denoising processing still needs to be performed on the correction result. The third sampling result y at the t - 1 moment can be obtained by performing DDIM sampling on the correction result t-1 , and then repeat sampling until the target moment, that is, the moment when t = 0, that is, the restoration of the degraded image is completed, and the restored image y is obtained 0 .

[0102] Exemplarily, at the t moment, running the variance-preserving sampling for M steps will obtain the correction result Then perform denoising sampling on the correction result to obtain y t-1 , and then enter the loop at the t - 1 moment and finally obtain the restored image y 0 .

[0103] Exemplarily, the third sampling result y t-1 can be represented by formula (4).

[0104]

[0105] Figure 4 is a schematic diagram of the sampling process according to Embodiment 1 of the present application. The present application uses a latent variable as an initialization, that is, Figure 3 the noise image P obtained by DDIM inversion of the low-quality image in τ (y τ ) is used as the starting state, and a correction algorithm is designed to guide the latent variable to move from a low-probability region to a nearby high-probability region, that is, the noise image P is subjected to image restoration processing P τ (y τ ) through DDIM and variance-preserving sampling t (x t ) to obtain the restored image P 0 (x 0 ).

[0106] It can be seen that the restored image P 0 (x 0 ) has significantly improved clarity, cleanliness, and quality compared to the input low-quality image P 0 (y 0 ).

[0107] In an alternative embodiment, a graphical user interface is provided by a terminal device, and the content displayed on the graphical user interface at least partially includes an image restoration scenario. The image restoration method further includes the following method steps:

[0108] Step S24, in response to a first control operation performed on the graphical user interface, input a degraded image;

[0109] Step S25, in response to a second control operation performed on the graphical user interface, select a first sampling method and a second sampling method from multiple candidate sampling methods, where the first sampling method is a deterministic sampling method, and the second sampling method is used to control the variance of the sampling result probability distribution corresponding to a preset moment during the sampling iteration process to remain unchanged;

[0110] Step S26, in response to a third control operation performed on the graphical user interface, perform sampling inversion processing using the first sampling method to obtain a noise image, and perform image restoration processing on the noise image using the second sampling method to generate a restored image;

[0111] Step S27, display the restored image within the graphical user interface.

[0112] In the graphical user interface in the embodiments of the present application, at least an image restoration scene is displayed. The user can input a degraded image in this image restoration scene by performing control operations, select a first sampling method and a second sampling method from multiple candidate sampling methods, and perform steps such as a submission operation for restoring the image. It can be understood that the above problem-solving scene can be, but is not limited to, application scenarios involving image restoration in fields such as games, e-commerce, education, medical care, conferences, social networks, financial products, logistics, and navigation.

[0113] The above graphical user interface further includes a first control (or a first touch area). When a first touch operation acting on the first control (or the first touch area) is detected, the degraded image input by the user can be obtained. The above degraded image can be uploaded by the user from the image input area in the graphical user interface through the first touch operation. The above first touch operation can be operations such as point selection, box selection, tick selection, conditional filtering, etc., which are not limited here.

[0114] The above graphical user interface further includes a second control (or a second touch area). When a second touch operation acting on the second control (or the second touch area) is detected, the user can be enabled to select a first sampling method and a second sampling method from multiple candidate sampling methods. For example, the DDIM sampling method and the variance-preserving sampling method selected by the user can be obtained. The above second touch operation can be operations such as point selection, box selection, tick selection, conditional filtering, etc., which are not limited here.

[0115] The second sampling method in the present application is used to control the variance of the probability distribution of the sampling result corresponding to a preset moment during the sampling iteration process to remain unchanged. That is, the variance-preserving sampling method is used to control the variance of the probability distribution of the sampling result corresponding to the t moment during the sampling iteration process to remain unchanged.

[0116] The above graphical user interface further includes a third control (or a third touch area). When a third touch operation acting on the third control (or the third touch area) is detected, according to the submission operation of the user, sampling inversion processing can be performed using the first sampling method to obtain a noise image, and image restoration processing can be performed on the noise image using the second sampling method to generate a restored image. The above third touch operation can be operations such as point selection, box selection, tick selection, conditional filtering, etc., which are not limited here.

[0117] After the restored image is obtained, the restored image can be displayed in the graphical user interface.

[0118] It should be noted that the above first touch operation, second touch operation, and third touch operation can all be operations where the user touches the display screen of the above terminal device with a finger and touches the terminal device. This touch operation can include single-point touch and multi-point touch. Among them, the touch operation of each touch point can include clicking, long pressing, hard pressing, swiping, etc. The above first touch operation, second touch operation, and third touch operation can also be touch operations implemented through input devices such as a mouse and a keyboard, which are not restricted here.

[0119] It can be seen that the purpose of this application is to recover a clean image x from the degraded image y. The key idea of this application is to search for the clean image distribution (represented by the diffusion prior, that is, the image prior captured by a diffusion model pre-trained on a large number of images with different distributions) to find the clean image while being faithful to the input degraded image. It should be noted that the large number of images with different distributions used in this application need to simultaneously satisfy the fidelity of the input degraded image and conform to the model distribution of the pre-trained diffusion model.

[0120] This application reconstructs the degraded image and uses the DDIM inversion algorithm to generate latent image features that retain the information of the input image. These latent variables are located in low-probability regions because sampling from the diffusion model usually produces clean images rather than degraded images. Although these latent variables cannot directly recover the clean image, they can inherit the information of the input image and provide a good initialization for subsequent sampling. Inspired by this, this application proposes isotropic sampling to guide these low-probability latent variables to move to nearby high-probability regions, thereby generating clean and high-quality samples. Thus, even without knowing the specific degradation model, isotropic sampling can be used as a general solution to ensure fidelity.

[0121] It is easy to understand that the beneficial effects of the image restoration method provided by this application include the following points.

[0122] Beneficial effect (1): It can avoid paired data supervised training and solve the problem of difficult collection of paired data.

[0123] Beneficial effect (2): It can be directly applied to the degraded image to be processed without the need for special training or fine-tuning for a certain type of degradation model;

[0124] Beneficial effect (3): It allows an unknown degradation model;

[0125] Beneficial effect (4): It can improve the quality of the images generated by the diffusion model.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0127] In addition, it should also be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0128] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

[0129] Embodiment 2

[0130] In the operating environment as in Embodiment 1, this application provides Figure 5 an image restoration method as shown in Figure 5 FIG. which is a flowchart of an image restoration method according to Embodiment 2 of this application. As shown in Figure 5 FIG., the method includes:

[0131] Step S51, obtaining a target role degraded image to be restored;

[0132] Step S52, performing sampling inversion processing on the target role degraded image to obtain a target role noise image, where the target role noise image is used to retain the potential target role image features of the target role degraded image, and the target role potential image features are used to reproduce the target role degraded image according to the sampling process;

[0133] Step S53, performing image restoration processing on the target role noise image to generate a target role restored image.

[0134] The target character can be a human character, an animal character, a virtual character, etc., without limitation here.

[0135] The degraded image of the target character to be restored can be understood as a low-quality image of the target character that needs to be image-restored. Generally, the clarity, color, and details of the degraded image of the target character are poor. Exemplarily, the degraded image of the target character to be restored can be the input image of the target character that the user inputs and wants to restore.

[0136] Performing sampling inversion processing on the degraded image of the target character can be understood as the inversion of the sampling processing of the degraded image of the target character. The inversion of the sampling processing is usually used to restore the original image or extract specific information, that is, to restore the original image of the degraded image of the target character or extract specific information in the degraded image of the target character, which can restore the details and information of the original image, or extract specific image features or information in a certain way, without limitation here.

[0137] Exemplarily, the sampling processing of the degraded image of the target character can adopt the Denoising Diffusion Probabilistic Model (DDIM) sampling algorithm, that is, perform DDIM inversion on the degraded image of the target character, without limitation here.

[0138] The potential image features of the degraded image of the target character can be the details and information of the original image restored by inverting the sampling processing of the degraded image of the target character, and can be understood as the latent variables in the low-probability region of the degraded image of the target character. In the embodiments of the present application, the potential image features can reproduce the degraded image of the target character according to the sampling process, that is, any latent variable contains the required information about the degraded image of the target character.

[0139] The noise image is the image obtained after performing sampling inversion processing on the degraded image of the target character. The noise image retains the potential image features of the degraded image of the target character, that is, the noise image also retains the required information about the degraded image of the target character.

[0140] The restored image is the image obtained after performing image restoration processing on the noise image. The restored image can be understood as a complete, clean, and high-quality image corresponding to the degraded image of the target character, that is, the image obtained after performing image repair on the degraded image of the target character. Exemplarily, the image restoration processing performed on the noise image includes but is not limited to image denoising, image repair, deblurring, super-resolution, etc., without limitation here.

[0141] In an embodiment of the present application, by obtaining a target character degraded image to be restored, then performing sampling inversion processing on the target character degraded image to obtain a noise image retaining the potential image features of the target character degraded image, and finally performing image restoration processing on the noise image, a clean and high-quality restored image is generated, completing the repair of the target character degraded image.

[0142] It can be seen that in the inference stage of the present application, a high-quality image can be obtained only through the inversion of the sampling algorithm and the image restoration algorithm. Both the inversion of the sampling algorithm and the image restoration algorithm are general and can be directly applied to the target character degraded image to be restored. Therefore, there is no need to assume a degradation model, making the generality stronger. It can overcome the technical problems in the related art of reasoning in the encoding space of VAE and requiring the assistance of a special degradation model to complete reasoning, and allows an unknown degradation model. Moreover, the present application does not require paired data for supervised training, solving the problem of difficult data collection. In addition, when processing highly JPEG-compressed pictures, compared with the supervised schemes in the related art, such as the Fully Convolutional Belief Neural Network (FBCNN), and the unsupervised schemes, such as the Deep Distributed Representation Model for JPEG Image Compression (DDRM-JPEG), the present application can obtain a clearer and higher-quality image.

[0143] The above image restoration method provided by the embodiments of the present application can be but is not limited to being applied to application scenarios involving image restoration in fields such as game services, e-commerce services, education services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services. For example: image restoration in games, image restoration in e-commerce, image restoration of medical images, etc., which are not limited here.

[0144] By adopting the embodiment of the present application, a target character degraded image to be restored is obtained, then sampling inversion processing is performed on the target character degraded image to obtain a noise image retaining the latent image features of the target character degraded image, and finally image restoration processing is performed on the noise image to generate a clean and high-quality restored image, completing the restoration of the target character degraded image, thereby achieving the purpose of obtaining a clean and high-quality restored image. Since the present application can be directly applied to the target character degraded image to be restored without assuming a degradation model, that is, without training or fine-tuning for the degradation model, it has stronger versatility, can be inferred in the encoding space of the VAE, and reduces the model training cost, thus achieving the technical effect of restoring the target character degraded image at low cost and obtaining a clean and high-quality restored image, and further solving the technical problem that the image restoration technology based on the underlying degradation model in the related art has a high cost in training the underlying degradation model and requires the use of a known degradation model.

[0145] It should be noted that the preferred implementation manner of this embodiment can refer to the relevant description in Embodiment 1, and will not be elaborated here.

[0146] Embodiment 3

[0147] According to the embodiment of the present application, an apparatus embodiment for implementing the above image restoration is also provided. Figure 6 It is a schematic structural diagram of an image restoration apparatus according to Embodiment 3 of the present application. As Figure 6 shown, the apparatus includes:

[0148] An acquisition module 601, configured to acquire a degraded image to be restored;

[0149] A first processing module 602, configured to perform sampling inversion processing on the degraded image to obtain a noise image, where the noise image is used to retain the latent image features of the degraded image, and the latent image features are used to reproduce the degraded image according to the sampling process;

[0150] A second processing module 603, configured to perform image restoration processing on the noise image to generate a restored image.

[0151] Optionally, the first processing module 602 is further configured to: acquire a sampling trajectory, where the sampling trajectory is a trajectory generated during the process of deterministic sampling of the degraded image; perform noise-added sampling on the sampling trajectory to obtain a noise image.

[0152] Optionally, the latent image features are multiple latent variables corresponding to the sampling trajectory.

[0153] Optionally, the second processing module 603 is further configured to: perform image correction processing on the noise image within a continuous plurality of time steps to obtain a correction result; perform denoising processing on the correction result to generate a restored image.

[0154] Optionally, the second processing module 603 is further configured to: at a first moment, obtain Gaussian noise corresponding to the first sampling step index and the probability distribution of the first sampling result corresponding to the second sampling step index within a continuous plurality of time steps, where the second sampling step index is the previous index adjacent to the first sampling step index; calculate a second sampling result based on a preset coefficient, the Gaussian noise, and the probability distribution, and repeat the calculation until a preset number of sampling steps is reached to obtain a correction result, where the preset coefficient satisfies a preset constraint condition.

[0155] Optionally, the Gaussian noise is noise added during the sampling iteration corresponding to the first sampling step index, with a mean of a first preset value, a variance of a second preset value, and the same dimension as the second sampling result.

[0156] Optionally, the preset constraint condition is determined by a preset scalar and a preset noise addition method, the preset scalar is a scalar for determining the step sizes corresponding to a continuous plurality of time steps, and the preset noise addition method is the noise addition method of the diffusion noise corresponding to the pre-trained diffusion model.

[0157] Optionally, it further includes: a calculation module, configured to obtain the noise predicted by the pre-trained diffusion model at the first moment and the preset noise addition method; calculate the probability distribution based on the noise predicted by the pre-trained diffusion model and the preset noise addition method.

[0158] Optionally, the second processing module 603 is further configured to: perform deterministic sampling on the correction result to obtain a third sampling result at a second moment, and repeat the sampling until a target moment is reached to generate a restored image, where the second moment is the previous moment adjacent to the first moment.

[0159] Optionally, a graphical user interface is provided by a terminal device, and the content displayed on the graphical user interface at least partially includes an image restoration scenario. It further includes: an interaction module, configured to input a degraded image in response to a first control operation performed on the graphical user interface; select a first sampling method and a second sampling method from multiple candidate sampling methods in response to a second control operation performed on the graphical user interface, where the first sampling method is a deterministic sampling method, and the second sampling method is used to control the variance of the probability distribution of the sampling result corresponding to a preset moment during the sampling iteration to remain unchanged; perform sampling inversion processing using the first sampling method to obtain a noise image, and perform image restoration processing on the noise image using the second sampling method to generate a restored image; display the restored image within the graphical user interface.

[0160] By adopting the embodiment of the present application, a degraded image to be restored is obtained, then the degraded image is subjected to sampling inversion processing to obtain a noise image retaining the potential image features of the degraded image, and finally the noise image is subjected to image restoration processing to generate a clean and high-quality restored image, thus completing the restoration of the degraded image, and thereby achieving the purpose of obtaining a clean and high-quality restored image. Since the present application can be directly applied to the degraded image to be restored without assuming a degradation model, that is, without training or fine-tuning for the degradation model, it has stronger versatility, can be inferred in the encoding space of the VAE, and reduces the model training cost, thus realizing the technical effect of restoring the degraded image at low cost and obtaining a clean and high-quality restored image, and further solving the technical problem that the image restoration technology based on the underlying degradation model in the related art has a high cost in training the underlying degradation model and needs to use a known degradation model.

[0161] It should be noted here that the above acquisition module 601, the first processing module 602, and the second processing module 603 correspond to steps S21 to S23 in Embodiment 1. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the content disclosed in the above Embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also run in the server 10 provided in Embodiment 1.

[0162] According to an embodiment of the present application, another device embodiment for implementing the above image restoration is also provided. Figure 7 is a schematic structural diagram of another image restoration device according to Embodiment 3 of the present application, as Figure 7 shown, the device includes:

[0163] An acquisition module 701, which acquires a target character degraded image to be restored;

[0164] A first processing module 702, which performs sampling inversion processing on the target character degraded image to obtain a target character noise image, wherein the target character noise image is used to retain the potential target character image features of the target character degraded image, and the target character potential image features are used to reproduce the target character degraded image according to the sampling process;

[0165] A second processing module 703, which performs image restoration processing on the target character noise image to generate a target character restored image.

[0166] By adopting the embodiments of the present application, a target role degraded image to be restored is obtained, and then sampling inversion processing is performed on the target role degraded image to obtain a noise image that retains the latent image features of the target role degraded image. Finally, image restoration processing is performed on the noise image to generate a clean and high-quality restored image, completing the restoration of the target role degraded image, thereby achieving the purpose of obtaining a clean and high-quality restored image. Since the present application can be directly applied to the target role degraded image to be restored without assuming a degradation model, that is, without training or fine-tuning for the degradation model, it has stronger versatility, can be inferred in the encoding space of the VAE, and reduces the model training cost. Thus, the technical effect of restoring the target role degraded image at low cost to obtain a clean and high-quality restored image is achieved, and further solves the technical problem that the image restoration technology based on the underlying degradation model in the related art has a high cost in training the underlying degradation model and needs to use a known degradation model.

[0167] It should be noted here that the above-mentioned acquisition module 701, the first processing module 702, and the second processing module 703 correspond to steps S51 to S53 in Embodiment 3. The instances and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 3. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in the memory and processed by one or more processors, and the above-mentioned modules can also run in the server 10 provided in Embodiment 1.

[0168] It should be noted that the preferred implementation schemes involved in the above-mentioned embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0169] Embodiment 4

[0170] The embodiments of the present application can provide a computer terminal, and the computer terminal can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above-mentioned computer terminal can also be replaced with a terminal device such as a mobile terminal.

[0171] Optionally, in this embodiment, the above-mentioned computer terminal can be located in at least one of multiple network devices in a computer network.

[0172] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image restoration method: obtaining a degraded image to be restored; performing sampling inversion processing on the degraded image to obtain a noise image, where the noise image is used to retain the latent image features of the degraded image, and the latent image features are used to reproduce the degraded image according to the sampling process; performing image restoration processing on the noise image to generate a restored image.

[0173] Optionally, Figure 8 is a structural block diagram of a computer terminal according to an embodiment of the present application. As Figure 8 shown, the computer terminal 8 may include: one or more (only one is shown in the figure) processors 802, a memory 804, a storage controller, and a peripheral interface. Among them, the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0174] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image restoration method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored therein, that is, implements the above-mentioned image restoration method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely disposed relative to the processor, and these remote memories may be connected to the computer terminal 8 through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The processor may call the information and application programs stored in the memory through a transmission device to execute the following steps: obtaining a degraded image to be restored; performing sampling inversion processing on the degraded image to obtain a noise image, where the noise image is used to retain the potential image features of the degraded image, and the potential image features are used to reproduce the degraded image according to the sampling process; performing image restoration processing on the noise image to generate a restored image.

[0176] Optionally, the above processor may further execute the program code of the following steps: obtaining a sampling trajectory, where the sampling trajectory is a trajectory generated during the process of deterministic sampling of the degraded image; performing noise-added sampling on the sampling trajectory to obtain a noise image.

[0177] Optionally, the potential image features are a plurality of latent variables corresponding to the sampling trajectory.

[0178] Optionally, the above processor may further execute the program code of the following steps: performing image correction processing on the noise image within a plurality of consecutive time steps to obtain a correction result; performing denoising processing on the correction result to generate a restored image.

[0179] Optionally, the above-mentioned processor can also execute the program code of the following steps: At a first moment, obtain the Gaussian noise corresponding to the first sampling step index and the probability distribution of the first sampling result corresponding to the second sampling step index within a continuous plurality of time steps, where the second sampling step index is the previous index adjacent to the first sampling step index; calculate the second sampling result based on a preset coefficient, Gaussian noise, and probability distribution, and repeat the calculation until a preset number of sampling steps is reached to obtain a corrected result, where the preset coefficient satisfies a preset constraint condition.

[0180] Optionally, the Gaussian noise is the noise with a mean of a first preset value, a variance of a second preset value, and the same dimension as the second sampling result added during the sampling iteration corresponding to the first sampling step index.

[0181] Optionally, the preset constraint condition is determined by a preset scalar and a preset noise addition method. The preset scalar is the scalar that determines the step size corresponding to a continuous plurality of time steps, and the preset noise addition method is the noise addition method of the diffusion noise corresponding to the pre-trained diffusion model.

[0182] Optionally, the above-mentioned processor can also execute the program code of the following steps: Obtain the noise predicted by the pre-trained diffusion model at the first moment and the preset noise addition method; calculate the probability distribution based on the noise predicted by the pre-trained diffusion model and the preset noise addition method.

[0183] Optionally, the above-mentioned processor can also execute the program code of the following steps: Perform deterministic sampling on the corrected result to obtain a third sampling result at a second moment, and repeat the sampling until the target moment is reached to generate a restored image, where the second moment is the previous moment adjacent to the first moment.

[0184] Optionally, a graphical user interface is provided through a terminal device. The content displayed on the graphical user interface at least partially includes an image restoration scenario. The above-mentioned processor can also execute the program code of the following steps: Respond to a first control operation performed on the graphical user interface and input a degraded image; respond to a second control operation performed on the graphical user interface and select a first sampling method and a second sampling method from multiple candidate sampling methods, where the first sampling method is a deterministic sampling method, and the second sampling method is used to control the variance of the probability distribution of the sampling result corresponding to a preset moment during the sampling iteration to remain unchanged; respond to a third control operation performed on the graphical user interface, perform sampling inversion processing using the first sampling method to obtain a noise image, and perform image restoration processing on the noise image using the second sampling method to generate a restored image; display the restored image within the graphical user interface.

[0185] By adopting the embodiment of the present application, a degraded image to be restored is obtained, then the degraded image is subjected to sampling inversion processing to obtain a noise image retaining the potential image features of the degraded image, and finally the noise image is subjected to image restoration processing, thereby generating a clean and high-quality restored image, completing the restoration of the degraded image, and thus achieving the purpose of obtaining a clean and high-quality restored image. Since the present application can be directly applied to the degraded image to be restored without assuming a degradation model, that is, without training or fine-tuning for the degradation model, it has stronger versatility, can be inferred in the encoding space of the VAE, and reduces the model training cost, thereby achieving the technical effect of restoring the degraded image at low cost and obtaining a clean and high-quality restored image, and further solving the technical problem that the image restoration technology based on the underlying degradation model in the related art has a high cost in training the underlying degradation model and needs to use a known degradation model.

[0186] Those of ordinary skill in the art can understand that Figure 8 The structure shown is only illustrative, and the computer terminal 8 can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a mobile Internet device (Mobile Internet Devices, MID), a PAD and other terminal devices. Figure 8 It does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 8 may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 8 or have a different configuration from that shown in Figure 8 shown.

[0187] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware of the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disc, etc.

[0188] Embodiment 5

[0189] The embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the above computer-readable storage medium can be used to store the program code executed by the image restoration method provided in the first embodiment above.

[0190] Optionally, in this embodiment, the above computer-readable storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0191] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a degraded image to be restored; performing sampling inversion processing on the degraded image to obtain a noise image, where the noise image is used to retain the latent image features of the degraded image, and the latent image features are used to reproduce the degraded image according to the sampling process; performing image restoration processing on the noise image to generate a restored image.

[0192] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a sampling trajectory, where the sampling trajectory is a trajectory generated during the process of deterministically sampling the degraded image; performing noise-added sampling on the sampling trajectory to obtain a noise image.

[0193] Optionally, the latent image features are multiple latent variables corresponding to the sampling trajectory.

[0194] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: performing image correction processing on the noise image within a continuous plurality of time steps to obtain a correction result; performing denoising processing on the correction result to generate a restored image.

[0195] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: at a first moment, obtaining Gaussian noise corresponding to the first sampling step index and the probability distribution of the first sampling result corresponding to the second sampling step index within a continuous plurality of time steps, where the second sampling step index is the previous index adjacent to the first sampling step index; calculating a second sampling result based on a preset coefficient, Gaussian noise, and the probability distribution, and repeating the calculation until a preset number of sampling steps is reached to obtain a correction result, where the preset coefficient satisfies a preset constraint condition.

[0196] Optionally, the Gaussian noise is noise added during the sampling iteration corresponding to the first sampling step index, with a mean of a first preset value, a variance of a second preset value, and the same dimension as the second sampling result.

[0197] Optionally, the preset constraint condition is determined by a preset scalar and a preset noise-adding method. The preset scalar is a scalar for determining the step size corresponding to a continuous plurality of time steps, and the preset noise-adding method is the noise-adding method of the diffusion noise corresponding to the pre-trained diffusion model.

[0198] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining the noise predicted by the pre-trained diffusion model at a first moment and the preset noise-adding method; calculating a probability distribution based on the noise predicted by the pre-trained diffusion model and the preset noise-adding method.

[0199] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: performing deterministic sampling on the correction result to obtain a third sampling result at a second moment, repeating the sampling until a target moment is reached, and generating a restored image, where the second moment is the previous moment adjacent to the first moment.

[0200] Optionally, a graphical user interface is provided through a terminal device. The content displayed on the graphical user interface at least partially includes an image restoration scenario. In this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: in response to a first control operation performed on the graphical user interface, inputting a degraded image; in response to a second control operation performed on the graphical user interface, selecting a first sampling method and a second sampling method from multiple candidate sampling methods, where the first sampling method is a deterministic sampling method, and the second sampling method is used to control the variance of the probability distribution of the sampling result corresponding to a preset moment during the sampling iteration process to remain unchanged; in response to a third control operation performed on the graphical user interface, performing sampling inversion processing using the first sampling method to obtain a noise image, and performing image restoration processing on the noise image using the second sampling method to generate a restored image; and displaying the restored image within the graphical user interface.

[0201] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments.

[0202] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0203] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0204] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] In addition, in each embodiment of the present application, each functional unit can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0206] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0207] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An image restoration method, It is characterized in that include: Acquire a degraded image to be restored; Performing sampling inversion processing on the degraded image to obtain a noise image, wherein the noise image is used to retain the potential image features of the degraded image, and the potential image features are used to reproduce the degraded image according to the sampling process; Performing image restoration processing on the noise image to generate a restored image.

2. The image restoration method according to claim 1, It is characterized in that Performing sampling inversion processing on the degraded image to obtain the noise image includes: Acquire a sampling trajectory, wherein the sampling trajectory is a trajectory generated in a process of deterministically sampling the degraded image; Noise sampling is performed on the sampling trajectory to obtain the noise image.

3. The image restoration method according to claim 1, It is characterized in that The potential image features are multiple latent variables corresponding to the sampling trajectory.

4. The image restoration method according to claim 1, It is characterized in that Performing image restoration processing on the noisy image to generate the restored image includes: Performing image correction processing on the noisy image in a plurality of consecutive time steps to obtain a correction result; The correction result is subjected to denoising processing to generate the restored image.

5. The image restoration method according to claim 4, It is characterized in that Performing image correction processing on the noise image within the plurality of consecutive time steps to obtain the correction result includes: At a first moment, obtaining a probability distribution of Gaussian noise corresponding to a first sampling step index and a first sampling result corresponding to a second sampling step index in the plurality of consecutive time steps, wherein the second sampling step index is a previous index adjacent to the first sampling step index; A second sampling result is calculated based on a preset coefficient, the Gaussian noise and the probability distribution, and the calculation is repeated until a preset number of sampling steps is reached to obtain the correction result, wherein the preset coefficient satisfies a preset constraint condition.

6. The image restoration method according to claim 5, It is characterized in that The Gaussian noise is noise added during the sampling iteration corresponding to the first sampling step index, which has a mean of a first preset value, a variance of a second preset value, and the same dimension as the second sampling result.

7. The image restoration method according to claim 5, It is characterized in that The preset constraint condition is determined by a preset scalar and a preset noise adding method, the preset scalar is a scalar that determines the step length corresponding to the plurality of consecutive time steps, and the preset noise adding method is a noise adding method of the diffusion noise corresponding to the pre-trained diffusion model.

8. The image restoration method according to claim 7, It is characterized in that The image restoration method further comprises: Obtaining the noise predicted by the pre-trained diffusion model at the first moment and the preset noise adding method; The probability distribution is calculated based on the noise predicted by the pre-trained diffusion model and the preset noise adding method.

9. The image restoration method according to claim 8, It is characterized in that Performing denoising on the correction result to generate the restored image includes: Deterministic sampling is performed on the correction result to obtain a third sampling result at a second moment, and the sampling is repeated until a target moment is reached to generate the restored image, wherein the second moment is a previous moment adjacent to the first moment.

10. The image restoration method according to claim 1, It is characterized in that A graphical user interface is provided through a terminal device, wherein the content displayed by the graphical user interface at least partially includes an image restoration scene, and the image restoration includes: In response to a first control operation performed on the graphical user interface, inputting the degraded image; In response to a second control operation performed on the graphical user interface, a first sampling method and a second sampling method are selected from a plurality of candidate sampling methods, wherein the first sampling method is a deterministic sampling method, and the second sampling method is used to control the variance of the probability distribution of the sampling result corresponding to a preset time in the sampling iteration process to remain unchanged; In response to a third control operation performed on the graphical user interface, performing sampling inversion processing using the first sampling method to obtain the noise image, and performing image restoration processing on the noise image using the second sampling method to generate the restored image; The restored image is displayed in the graphical user interface.

11. An image restoration method, It is characterized in that include: Obtaining a degraded image of a target character to be restored; Performing sampling inversion processing on the target character degraded image to obtain a target character noise image, wherein the target character noise image is used to retain the latent target character image features of the target character degraded image, and the target character latent image features are used to reproduce the target character degraded image according to the sampling process; Perform image restoration processing on the target character noise image to generate a target character restoration image.

12. An electronic device, It is characterized in that include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the image restoration method described in any one of claims 1 to 11 when running.

13. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the image restoration method according to any one of claims 1 to 11.

Images