Blind image restoration method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a blind image restoration method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-restored image; inputting the to-be-restored image into a pre-established blind image restoration model, and outputting to obtain a restored image; wherein the blind image restoration model is pre-established based on a vector quantization variational auto-encoder, a multi-layer perceptron and a continuous diffusion generation model. According to the embodiment of the invention, by utilizing the superiority of the continuous diffusion generation model in learning data manifold distribution, the vector quantization variational automatic encoder is used as a codec between the image space and the submerged space, and the generation process in the submerged space of image encoding is constructed, so that a low-quality image under complex degradation is repaired, and the image quality is improved. And a good image restoration effect is achieved. A continuous diffusion generation mode is introduced, and a preprocessed guide embedding mode is constructed, so that guide conditions in a high-quality image prediction process are more reasonable and interpretable.

Description

Technical Field

[0001] The embodiments of the present disclosure belong to the technical field of blind image restoration, and specifically relate to a blind image restoration method, system, electronic device, and storage medium. Background Art

[0002] The blind image restoration problem is one of the most complex problems in the field of image restoration. The image restoration problem is to restore low-quality images with noise, blur, artifacts, etc. into clear high-quality images. Based on the degradation form of low-quality images, it can be divided into known degraded image restoration and unknown image degradation restoration, of which the latter is called blind image restoration.

[0003] Blind image restoration requires finding key content in the input low-quality image based on the prior information of natural images, and predicting the relevant texture information to obtain a clear and natural high-quality image. However, this process of directly predicting the corresponding high-quality image from a low-quality image is an ill-conditioned inverse problem, which requires sufficient and reasonable prior information to generate accurate and natural results.

[0004] Existing technical means use discrete diffusion generative model methods to repair low-quality images, but do not strike a good balance between low-quality content preservation and high-quality texture prediction, resulting in the inability to achieve good repair results in overly degraded low-quality images. Summary of the invention

[0005] The embodiments of the present disclosure aim to solve at least one of the technical problems existing in the prior art and provide a blind image restoration method, system, electronic device and storage medium.

[0006] One aspect of the present disclosure provides a blind image restoration method, the method comprising:

[0007] Obtain the image to be repaired;

[0008] The image to be repaired is input into a pre-established blind image repair model, and a repaired image is output; wherein,

[0009] The blind image restoration model is pre-established based on a vector quantized variational autoencoder, a multi-layer perceptron and a continuous diffusion generation model.

[0010] Furthermore, the blind image restoration model is pre-established through the following steps:

[0011] Randomly degrade the high-quality image to obtain a low-quality image;

[0012] Inputting the high-quality image and the low-quality image into a vector quantization variational autoencoder respectively to obtain a high-quality potential feature vector and a low-quality potential feature vector;

[0013] Mapping the low-quality latent feature vector to a high-quality space;

[0014] The scale coefficient is obtained by decoupling using a multi-layer perceptron;

[0015] A continuous diffusion generation model is trained according to the high-quality potential feature vector, the low-quality potential feature vector and the scale coefficient to obtain a blind image restoration model.

[0016] Furthermore, the randomly degrading the high-quality image to obtain the low-quality image includes:

[0017] The high-quality image is input into a random degradation model, and a low-quality image is output; wherein the random degradation model includes a random degradation type and a random degradation parameter.

[0018] Further, mapping the low-quality potential feature vector to a high-quality space includes:

[0019] The low-quality potential feature vector is input into a Transformer basic transformation model; wherein the objective function of the Transformer basic transformation model is:

[0020]

[0021] In the formula, x represents a low-quality image, y represents a high-quality image, and p data represents the sample distribution of high-quality images, G(·) represents the random degradation model, Represents the basic transformation model of Transformer, represents the current embedding condition and ε(·) represents the latent feature vector.

[0022] Furthermore, the objective function of the continuous diffusion generation model is:

[0023]

[0024] In the formula, represents the continuous diffusion generation model, α i , β i represents the scale factor, n i represents the current random noise, σ i =t i represents the current diffusion time step, Represents the variable decoded back to pixel space, represents the pixel space reconstruction loss The hyperparameter weights of .

[0025] Furthermore, the method further comprises:

[0026] The encoder / decoder and codebook of the vector quantized variational autoencoder are trained; wherein the objective function of the vector quantized variational autoencoder is:

[0027]

[0028] In the formula, z i =ε(y)+n i represents the noisy random variable at the current time step of the continuous diffusion generative model, sg[·] represents the current partial parameters frozen during training, and e represents the vector in the codebook.

[0029] Another aspect of the present disclosure provides a blind image restoration system, the system comprising:

[0030] An acquisition module, used for acquiring an image to be repaired;

[0031] A restoration module is used to input the image to be restored into a pre-established blind image restoration model and output a restored image; wherein,

[0032] The blind image restoration model is pre-established based on a vector quantized variational autoencoder, a multi-layer perceptron and a continuous diffusion generation model.

[0033] Furthermore, the system also includes a model module for pre-establishing the blind image restoration model;

[0034] The model module is specifically used for:

[0035] Randomly degrade the high-quality image to obtain a low-quality image;

[0036] Inputting the high-quality image and the low-quality image into a vector quantization variational autoencoder respectively to obtain a high-quality potential feature vector and a low-quality potential feature vector;

[0037] Mapping the low-quality latent feature vector to a high-quality space;

[0038] The scale coefficient is obtained by decoupling using a multi-layer perceptron;

[0039] A continuous diffusion generation model is trained according to the high-quality potential feature vector, the low-quality potential feature vector and the scale coefficient to obtain a blind image restoration model.

[0040] Another aspect of the present disclosure provides an electronic device, comprising:

[0041] At least one processor; and a memory in communication with the at least one processor, for storing one or more programs, which, when executed by the at least one processor, enables the at least one processor to implement the blind image restoration method described above.

[0042] Another aspect of the present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program implements the blind image restoration method described above when executed by a processor.

[0043] The blind image restoration method, system, electronic device and storage medium of the disclosed embodiment utilize the superiority of the continuous diffusion generation model in learning the data manifold distribution, use the vector quantization variational autoencoder as the codec between the image space and the latent space, and construct the generation process under the latent space of image coding, so as to repair the low-quality image under complex degradation and achieve a good image restoration effect. The continuous diffusion generation method is introduced, and the pre-processed guided embedding method is constructed to make the guidance conditions in the high-quality image prediction process more reasonable and explainable. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a blind image restoration method according to an embodiment of the present disclosure;

[0045] Figure 2 A schematic diagram of a blind image restoration model establishment process according to another embodiment of the present disclosure;

[0046] Figure 3 It is a structural schematic diagram of a blind image restoration system according to another embodiment of the present disclosure;

[0047] Figure 4 The figure is a schematic diagram of the structure of an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] For the problem of blind image restoration, the feature latent spaces constructed by different solutions are completely different. Their essence is to construct a reasonable mapping relationship to map a low-quality image into its corresponding high-quality image. The embodiment of the present disclosure uses a diffusion generation model to construct a reasonable conditional guidance method and proposes a continuous diffusion model restoration method (LEDM-IR) in latent space. The following mainly introduces the relevant key technologies: vector quantization variational autoencoder and continuous diffusion generation model.

[0049] Vector Quantised Variational AutoEncoder (VQVAE) comes from the paper Neural

[0050] Discrete Representation Learning is a generative model that uses vectorization technology to train a discrete codebook to achieve discrete representation of images. VQVAE can achieve discrete compression and restoration of images, and has important applications in visual tasks such as image autoregressive generation. For a given image sample x, first encode the latent space vector z e (x) = ε(x), where ε is the encoder model. Then, using the discrete codebook, a list of vectors associated with corresponding indices, we get the vectorized latent space vector:

[0051]

[0052] where e i is a discrete vector in the codebook. The entire encoder can be recorded as ε(x) = z q (x), and finally the decoder is used to reconstruct the vectorized result and obtain

[0053] The Elucidating Design Model (EDM) comes from the paper "Elucidating the Design Space of Diffusion-Based Generative Models". It is a generative model that links a random variable ε with a known distribution to the actual distribution of the real image sample y through multi-step diffusion and sampling in the form of a Markov chain. The distribution of the real image sample is denoted by p data , the optimization goal of EDM is:

[0054]

[0055] in represents a Gaussian random distribution with variance σ, Represents the EDM model. EDM needs to construct forward diffusion and reverse sampling processes in a continuous time step space, and use the ordinary differential equation (ODE) form to obtain approximate numerical results. In the reverse sampling process, the parameter σ needs to be discretized. Generally, it is discretized according to the time step, taking σ(t) = t, and then performing step-by-step sampling. Finally, EDM links a known pure Gaussian distribution with an unknown complex real data distribution to obtain an unconditional prediction generation model, which can simulate the generation of real data distribution, sample any sample in it, and generate image samples with similar distribution.

[0056] Distribution learning of high-quality images is an effective way to obtain prior information. The embodiments of the present disclosure utilize the superiority of the diffusion generative model in learning the distribution of data manifolds. For problems such as blind image restoration, a generation process is constructed in the latent space of image coding, in which VQVAE (Vector Quantised Variational AutoEncoder) is used as the codec between the image space and the latent space, and EDM (Elucidating Design Model) is used as the diffusion generation process. A suitable training objective function is designed to obtain the final solution. This solution can repair low-quality images under complex degradation and can also achieve good restoration effects in many real-world images.

[0057] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0058] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0059] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0060] It should be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present disclosure. As used in this disclosure, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0061] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present disclosure, and therefore cannot be used to limit the protection scope of the present disclosure.

[0062] like Figure 1 As shown, an embodiment of the present disclosure provides a blind image restoration method, the method comprising:

[0063] Step S1, obtaining an image to be repaired.

[0064] Specifically, a low-quality image that needs to be repaired is obtained.

[0065] Step S2, inputting the image to be repaired into a pre-established blind image repair model, and outputting a repaired image; wherein the blind image repair model is pre-established based on a vector quantized variational autoencoder, a multi-layer perceptron and a continuous diffusion generation model.

[0066] Specifically, the low-quality image to be repaired obtained in the previous step S1 is input into a blind image repair model established in this embodiment, so as to output a repaired high-quality image. The blind image repair model is pre-established through the following steps:

[0067] Randomly degrade the high-quality image to obtain a low-quality image;

[0068] Inputting the high-quality image and the low-quality image into a vector quantization variational autoencoder respectively to obtain a high-quality potential feature vector and a low-quality potential feature vector;

[0069] Mapping the low-quality latent feature vector to a high-quality space;

[0070] The scale coefficient is obtained by decoupling using a multi-layer perceptron;

[0071] A continuous diffusion generation model is trained according to the high-quality potential feature vector, the low-quality potential feature vector and the scale coefficient to obtain a blind image restoration model.

[0072] Specifically, the blind image inpainting problem hopes to exploit the content guidance of low-quality images x and the general high-quality image distribution p data The difficulty in image restoration is how to maintain the content information of the original image and supplement and predict sufficient texture information. The embodiment of the present disclosure is based on the continuous diffusion generation model under the coding latent space, constructs a condition-guided restoration process, and realizes the restoration of low-quality images through multi-step training in high-quality image data sets. Figure 2The figure shows the schematic diagram of the establishment process of the blind image restoration model, where x represents a low-quality image, y represents a high-quality image, ε represents an encoder, Represents a decoder, Represents the variable decoded back to pixel space, t i represents the current diffusion time step, α i ,β i Represents the scale factor.

[0073] First, the high-quality image y is input into the encoder ε to obtain the latent feature vector ε(y), and then the image random degradation formula is used to obtain the simulated low-quality image x=G(y), where G is the random degradation model, including the random degradation type and random degradation parameters. Similarly, the latent feature vector ε(x) is obtained from the encoder ε, and the low-quality feature vector is preprocessed using the basic transformation model Transformer and mapped to the adjacent high-quality space. The Transformer model The optimization objective function is:

[0074]

[0075] Among them, p data represents the distribution of high-quality image samples, Represents the current embedded condition.

[0076] Secondly, we construct the conditional embedding method in the continuous diffusion model and train the continuous diffusion model. i , first use the multilayer perceptron (MLP) to decouple the two scale coefficients α i ,β i ,Right now in Represents the MLP model. Then according to the current parameter σ i =t i Under the setting of Then use the current random diffusion form to train the diffusion generation model According to the image restoration task, a new loss objective function is constructed:

[0077]

[0078] in, is the variable decoded back to pixel space, is the pixel space reconstruction loss The hyperparameter weights of .

[0079] Specifically, the conditional embedding method constructed is given to guide the current latent space. and the decoupled scale factor α i ,β i , and the noise random variable z at the current time step of the continuous diffusion model i =ε(y)+n i The conditional embedding method is cross_attention(α i z i ,β i c), where cross_attention is an attention mechanism used in the connection layer of the neural network.

[0080] Finally, according to the encoder and decoder of the vector quantized variational autoencoder, the encoder-decoder and codebook need to be trained, and the optimization objective function is:

[0081]

[0082] Among them, sg[] means freezing the current part of the parameters during training, and e represents the vector in the codebook.

[0083] The specific reasoning algorithm process can be shown as follows:

[0084]

[0085] Where S noise represents a slight noise perturbation coefficient and can be displayed as a hyperparameter.

[0086] A blind image restoration method of the disclosed embodiment utilizes the superiority of the continuous diffusion generation model in learning data manifold distribution, uses the vector quantization variational autoencoder as the codec between the image space and the latent space, and constructs a generation process under the latent space of image coding, thereby repairing low-quality images under complex degradation and achieving a good image restoration effect. The continuous diffusion generation method is introduced, and the pre-processed guided embedding method is constructed to make the guidance conditions in the high-quality image prediction process more reasonable and explainable.

[0087] like Figure 3 As shown, another embodiment of the present disclosure provides a blind image restoration system, the system comprising:

[0088] An acquisition module 310 is used to acquire an image to be restored;

[0089] The restoration module 320 is used to input the image to be restored into a pre-established blind image restoration model and output a restored image; wherein,

[0090] The blind image restoration model is pre-established based on a vector quantized variational autoencoder, a multi-layer perceptron and a continuous diffusion generation model.

[0091] For example, Figure 3 As shown, the system further includes a model module 330 for pre-establishing the blind image restoration model;

[0092] The model module 330 is specifically used for:

[0093] Randomly degrade high-quality images to obtain low-quality images;

[0094] Inputting the high-quality image and the low-quality image into a vector quantization variational autoencoder respectively to obtain a high-quality potential feature vector and a low-quality potential feature vector;

[0095] Mapping the low-quality latent feature vector to a high-quality space;

[0096] The scale coefficient is obtained by decoupling using a multi-layer perceptron;

[0097] A continuous diffusion generation model is trained according to the high-quality potential feature vector, the low-quality potential feature vector and the scale coefficient to obtain a blind image restoration model.

[0098] Specifically, a blind image restoration system of an embodiment of the present disclosure is used to implement the blind image restoration method described in the above embodiment. The specific implementation process has been described in detail in the above embodiment and will not be repeated herein.

[0099] A blind image restoration system of the disclosed embodiment utilizes the superiority of the continuous diffusion generation model in learning data manifold distribution, uses the vector quantization variational autoencoder as the codec between the image space and the latent space, and constructs a generation process under the latent space of image coding, thereby repairing low-quality images under complex degradation and achieving a good image restoration effect. The continuous diffusion generation method is introduced, and the pre-processed guided embedding method is constructed to make the guidance conditions in the high-quality image prediction process more reasonable and explainable.

[0100] like Figure 4 As shown, another embodiment of the present disclosure provides an electronic device, including:

[0101] At least one processor 401; and a memory 402 communicatively connected to the at least one processor 401, for storing one or more programs, which, when executed by the at least one processor 401, enable the at least one processor 401 to implement the blind image restoration method described above.

[0102] The memory 402 and the processor 401 are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors 401 and the memory 402 together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor 401 is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor 401.

[0103] The processor 401 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management and other control functions. The memory 402 can be used to store data used by the processor 401 when performing operations.

[0104] Yet another embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program implements the blind image restoration method described above when executed by a processor.

[0105] The computer-readable storage medium may be included in the system or electronic device of the present disclosure, or may exist independently.

[0106] Computer-readable storage media may be any tangible media that contains or stores a program, which may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, optical fiber, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0107] The computer-readable storage medium may also include a data signal propagated in baseband or as part of a carrier wave, in which the computer-readable program code is carried. Specific examples include but are not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0108] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A blind image restoration method, characterized in that: The method comprises: Obtain the image to be repaired; The image to be repaired is input into a pre-established blind image repair model, and a repaired image is output; wherein, The blind image restoration model is pre-established based on a vector quantized variational autoencoder, a multi-layer perceptron and a continuous diffusion generation model.

2. The method according to claim 1, characterized in that The blind image restoration model is pre-established through the following steps: Randomly degrade high-quality images to obtain low-quality images; Inputting the high-quality image and the low-quality image into a vector quantization variational autoencoder respectively to obtain a high-quality potential feature vector and a low-quality potential feature vector; Mapping the low-quality latent feature vector to a high-quality space; The scale coefficient is obtained by decoupling using a multi-layer perceptron; A continuous diffusion generation model is trained according to the high-quality potential feature vector, the low-quality potential feature vector and the scale coefficient to obtain a blind image restoration model.

3. The method according to claim 2, characterized in that The randomly degrading the high-quality image to obtain the low-quality image includes: The high-quality image is input into a random degradation model, and a low-quality image is output; wherein the random degradation model includes a random degradation type and a random degradation parameter.

4. The method according to claim 2, characterized in that: Mapping the low-quality potential feature vector to a high-quality space includes: The low-quality potential feature vector is input into a Transformer basic transformation model; wherein the objective function of the Transformer basic transformation model is: In the formula, x represents a low-quality image, y represents a high-quality image, and p data represents the sample distribution of high-quality images, G(·) represents the random degradation model, Represents the basic transformation model of Transformer, represents the current embedding condition and ε(·) represents the latent feature vector.

5. The method according to claim 4, characterized in that The objective function of the continuous diffusion generation model is: In the formula, represents the continuous diffusion generation model, α i , β i represents the scale factor, n i represents the current random noise, σ i =t i represents the current diffusion time step, Represents the variable decoded back to pixel space, represents the pixel space reconstruction loss The hyperparameter weights of .

6. The method according to claim 5, characterized in that The method further comprises: The encoder / decoder and codebook of the vector quantized variational autoencoder are trained; wherein the objective function of the vector quantized variational autoencoder is: In the formula, z i =ε(y)+n i represents the noisy random variable at the current time step of the continuous diffusion generative model, sg[·] represents the current partial parameters frozen during training, and e represents the vector in the codebook.

7. A blind image restoration system, characterized in that: The system comprises: An acquisition module, used for acquiring an image to be repaired; A restoration module is used to input the image to be restored into a pre-established blind image restoration model and output a restored image; wherein, The blind image restoration model is pre-established based on a vector quantized variational autoencoder, a multi-layer perceptron and a continuous diffusion generation model.

8. The system according to claim 7, characterized in that The system also includes a model module, which is used to pre-establish the blind image restoration model; The model module is specifically used for: Randomly degrade high-quality images to obtain low-quality images; Inputting the high-quality image and the low-quality image into a vector quantization variational autoencoder respectively to obtain a high-quality potential feature vector and a low-quality potential feature vector; Mapping the low-quality latent feature vector to a high-quality space; The scale coefficient is obtained by decoupling using a multi-layer perceptron; A continuous diffusion generation model is trained according to the high-quality potential feature vector, the low-quality potential feature vector and the scale coefficient to obtain a blind image restoration model.

9. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor, for storing one or more programs, which, when executed by the at least one processor, enables the at least one processor to implement the blind image restoration method described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the blind image restoration method according to any one of claims 1 to 6 is implemented.

Images