Image inpainting method and apparatus

Abstract

Embodiments of the present application provide an image inpainting method, which comprises: obtaining input data to be input according to an image to be inpainted. The input data to be input is input into a diffusion model to fuse the input data to be input and a randomly generated Gaussian noise map through the diffusion model to obtain fused data; wherein the diffusion model comprises a noise prediction model. Noise parameters of the fused data are estimated through the noise prediction model. Output data corresponding to the inpainted image is generated according to the input data to be input and the noise parameters. The output data is output through the diffusion model, and the output data is the inpainted image corresponding to the image to be inpainted. The technical solution of the embodiments of the present application can inpaint a low-quality image through a diffusion model to generate a corresponding high-quality image, thereby improving the effect of image processing.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and in particular to an image repairing method and device, computer equipment and a computer readable storage medium. BACKGROUND

[0002] With the development of computer technology, people have higher quality requirements for images and videos, and thus there are demands for image repairing and resolution improvement. In the prior art, if too much noise and detail is lost in an image, the repairing effect may be poor.

[0003] It should be noted that the above content is not necessarily prior art, and is not used to limit the patent protection scope of the present application. SUMMARY

[0004] Embodiments of the present application provide an image repairing method and device, computer equipment and a computer readable storage medium to solve or alleviate one or more technical problems mentioned above.

[0005] One aspect of embodiments of the present application provides an image repairing method, which comprises:

[0006] acquiring input data to be input according to an image to be repaired;

[0007] inputting the input data to be input into a diffusion model to fuse the input data to be input and randomly generated Gaussian noise through the diffusion model to obtain fused data, wherein the diffusion model comprises a noise prediction model;

[0008] estimating a noise parameter of the fused data through the noise prediction model;

[0009] reconstructing output data corresponding to the repaired image according to the input data to be input and the noise parameter;

[0010] outputting the output data through the diffusion model, the output data being a repaired image corresponding to the image to be repaired.

[0011] Optionally, the acquiring of the input data to be input according to the image to be repaired comprises:

[0012] inputting the image to be repaired into a variational encoder-decoder, the variational encoder-decoder being used for compression, dimension reduction and data reconstruction;

[0013] acquiring reconstruction data of the variational encoder-decoder, wherein the reconstruction data is the input data to be input.

[0014] Optionally, the variational encoder-decoder comprises an encoder, a decoder and a sampler.

[0015] Correspondingly, the variational codec is trained by inputting different sample images and performing the following multiple rounds of operations:

[0016] inputting a sample image into the encoder to obtain an encoding vector corresponding to the sample image;

[0017] inputting the encoding vector into the decoder to obtain a first sample reconstruction object;

[0018] randomly generating a sampling vector by the sampler;

[0019] inputting the sampling vector into the decoder to obtain a second sample reconstruction object;

[0020] obtaining a first loss value according to the first sample reconstruction object and the second sample reconstruction object;

[0021] adjusting parameters of the variational codec according to the first loss value.

[0022] Optionally, the diffusion model is obtained by the following training operations:

[0023] obtaining sample reconstruction data of a sample image by the trained variational codec;

[0024] inputting the sample reconstruction data as sample input data of the diffusion model to perform model training on the diffusion model.

[0025] Optionally, the diffusion model is trained by multiple sample image pairs, each sample image pair including a first sample image and a second sample image corresponding to the same picture but having different qualities, and each sample image corresponds to one round of training operation of the diffusion model, wherein each round of training operation is as follows:

[0026] randomly generating Gaussian noise;

[0027] adding the Gaussian noise to the first sample image to obtain a first noise image;

[0028] adding the second sample image to the first noise image by a conditional likelihood function to obtain a second noise image;

[0029] obtaining a predicted noise parameter of the second noise image by the noise prediction model;

[0030] obtaining a second loss value according to the predicted noise parameter and a real noise parameter, wherein the real noise parameter is a noise parameter corresponding to the Gaussian noise;

[0031] adjusting parameters of the noise prediction model according to the second loss value.

[0032] Optionally, the picture quality of the first sample image is higher than the picture quality of the second sample image.

[0033] Optionally, the definition of the first sample image is higher than the definition of the second sample image; and / or

[0034] The resolution of the first sample image is higher than the resolution of the second sample image.

[0035] Another aspect of the embodiments of the present application provides an image repairing device, the device comprising:

[0036] An acquisition module is configured to acquire input data according to an image to be repaired;

[0037] An input module is configured to input the input data into a diffusion model, so as to fuse the input data and a randomly generated Gaussian noise map through the diffusion model, and obtain fused data; wherein the diffusion model comprises a noise prediction model.

[0038] An estimation module is configured to estimate a noise parameter of the fused data through the noise prediction model.

[0039] A reconstruction module is configured to reconstruct output data corresponding to the repaired image according to the input data and the noise parameter.

[0040] An output module is configured to output the output data through the diffusion model, and the output data is the repaired image corresponding to the image to be repaired.

[0041] Another aspect of the embodiments of the present application provides a computer device, comprising:

[0042] At least one processor; and

[0043] A memory connected in communication with the at least one processor;

[0044] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0045] Another aspect of the embodiments of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer instructions, and the computer instructions are executed by a processor to implement the method described above.

[0046] The embodiments of the present application can include the following advantages by adopting the above technical solutions:

[0047] Image generation capabilities based on diffusion models can restore low-quality images to high-quality ones, thus improving image processing results. Furthermore, because diffusion models achieve efficient computation through the diffusion process, they can improve the computational efficiency of image restoration and overall image processing efficiency. Attached Figure Description

[0048] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0049] Figure 1 This diagram schematically illustrates the operating environment of the image restoration method according to Embodiment 1 of this application;

[0050] Figure 2 A flowchart illustrating an image restoration method according to Embodiment 1 of this application is shown schematically.

[0051] Figure 3 Schematic illustration Figure 1 Flowchart of the sub-steps in step S200;

[0052] Figure 4 The diagram illustrates a new addition to the image restoration method according to Embodiment 1 of this application;

[0053] Figure 5 This schematically illustrates another additional flowchart of the image restoration method according to Embodiment 1 of this application;

[0054] Figure 6 This schematically illustrates another additional flowchart of the image restoration method according to Embodiment 1 of this application;

[0055] Figure 7 The flowchart illustrating an exemplary application of training a variational codec is shown in the diagram.

[0056] Figure 8 The flowchart illustrates an exemplary application of training a noise prediction model;

[0057] Figure 9 The illustration shows a set of images showing the restoration results of the images to be restored and the restored images;

[0058] Figure 10 The illustration shows another set of images to be repaired and the repaired images.

[0059] Figure 11A block diagram of an image inpainting apparatus according to Embodiment Two of the present application is schematically shown; and

[0060] Figure 12 A hardware architecture schematic diagram of a computer device according to Embodiment Three of the present application is schematically shown. DETAILED DESCRIPTION

[0061] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0062] It should be noted that the description involving "first", "second" and the like in the embodiments of the present application is only for the purpose of description, and should not be understood as indicating or implying the relative importance of the technical features indicated or implicitly indicating the number of technical features. Therefore, the features with "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of those of ordinary skill in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection claimed by the present application.

[0063] In the description of the present application, it should be understood that the numerical reference before the step does not indicate the order of execution of the steps before and after, but is only used to facilitate the description of the present application and to distinguish each step, therefore it should not be understood as a limitation of the present application.

[0064] First, the terms involved in the present application are explained:

[0065] Diffusion Models: A class of latent variable models. The goal of diffusion models is to learn the underlying structure of a dataset by modeling the way data points diffuse in latent space. Diffusion models work by learning the decay of information due to noise and then using the learned patterns to generate images. Diffusion models can generate realistic images by iteratively diffusing and mixing pixel values. This model uses a set of initial pixel values and then gradually diffuses and propagates information by calculating and updating the neighboring pixels around each pixel. In each iteration, the model updates the value of each pixel by considering the weights and interactions of the surrounding pixels, allowing the image to gradually form. Since diffusion models perform local calculations on each pixel and can leverage pixel interactions, they can capture both microscopic details and global structures in images. Through step-by-step iteration and information propagation, diffusion models can fill in blank areas, restore lost details, and generate images that are natural, coherent, and realistic. Additionally, diffusion models can generate diverse images based on input initial conditions and parameter settings. By changing initial pixel values, iteration counts, diffusion rules, and weights, different image generation results can be explored. This flexibility makes diffusion models have broad application potential in generating artistic images, image restoration, and enhancement.

[0066] Likelihood function: A function of parameters in a statistical model that represents the likelihood of the unknown parameters given the known output of a random variable.

[0067] Super-Resolution (SR): A technique for reconstructing high-resolution images from low-resolution images. It can improve image clarity and detail, making images clearer and more textured, reducing noise interference, and enhancing image contrast and color.

[0068] Variational Autoencoder (VAE) is a kind of artificial neural network structure, VAE belongs to probabilistic generative model (Probabilistic Generative Model). According to the different functions, VAE can be divided into encoder and decoder. The encoder can map the input variable to the latent space corresponding to the parameter of the variational distribution, and the decoder is basically the opposite of the encoder. The decoder is to convert the vector in the latent space back to the reconstructed data.

[0069] Image noise: refers to the problems in digital images caused by sensor sampling, digital signal processing, transmission and other links, resulting in unnecessary interference and noise in the image. Image noise can manifest as image blur, line disappearance, color distortion and other problems, which can reduce the quality and clarity of the image. The sources of image noise include sensor noise, digital signal noise, transmission noise, etc.

[0070] Blur repair: refers to using digital image processing technology to remove noise and details in low-quality images to generate high-quality images. Blur denoising is used to reduce the clarity and accuracy of the image, reduce noise interference, and at the same time enhance the contrast and color of the image. Common picture blur repair techniques include image enhancement, image repair, etc. Among them, image enhancement techniques include histogram equalization, color enhancement, etc., and image repair techniques include removing borders, removing backgrounds, etc.

[0071] U-Net model (Convolutional Networks for Biomedical Image Segmentation): an improved FCN (Fully Convolutional Networks) structure. The U-Net network structure is symmetrical, and because its structure is drawn to resemble the letter U, it is called the U-Net model. The U-Net model is composed of a left half of the compression channel and a right half of the expansion channel.

[0072] Training: learning from existing data to achieve a certain function.

[0073] Inference: using a certain function to quickly and efficiently operate on unknown data to obtain the expected result.

[0074] Secondly, in order to facilitate the understanding of the technical solutions provided by the person skilled in the art, the related technologies are described as follows:

[0075] The image repair and super-resolution technology known to the inventor is realized based on the algorithm of surrounding pixels. However, in the case of very blurred images or videos, or excessive loss of noise and details, if the image repair and super-resolution are based on the algorithm of surrounding pixels, it may result in poor repair effect of the image or video, and the super-resolution effect may not meet the expectation.

[0076] To this end, the embodiment of the present application provides an image restoration technical solution. In this technical solution, (1) based on the generation ability of the diffusion model, the low-quality photos and old videos are restored and super-resolved, so that the processing of details and textures in this solution is more accurate. (2) The variational codec is used to compress and reduce the dimension of the image to be restored, reducing the consumption of computing resources in the training and inference process of the diffusion model, and improving the processing efficiency. (3) The diffusion model is used to process the image to be restored, so that this solution can not only be used in image super-resolution technology, but also be used in image deblurring, noise reduction and other image restoration tasks. And this solution can effectively process different types, sizes and resolutions of images to be restored. (4) By using the conditional likelihood function and super-resolution technology, this solution can generate high-quality images with higher clarity and more realistic, thereby improving the visual effect of the image. (5) During the training process of the diffusion model, the conditional likelihood function is introduced, so that the trained diffusion model can better capture the subtle details and textures in the image to be restored, thereby more accurately performing super-resolution processing. (6) When the diffusion model processes the image to be restored, the diffusion model can achieve efficient calculation through the diffusion process, thereby improving the calculation speed in the image restoration process and improving the efficiency of image processing. And the method of restoring and super-resolving the image through the diffusion model is more efficient than the traditional method based on optimization or machine learning. See the following.

[0077] Finally, for the convenience of understanding, an exemplary operating environment is provided below.

[0078] As shown in Figure 1 , the operating environment diagram includes: as shown in Figure 1 , the environment diagram includes a service platform 2, a network 4, and a client 6, wherein:

[0079] The service platform 2 can be composed of a single or multiple computing devices. The multiple computing devices can include virtualized computing instances. The virtualized computing instances can include virtual machines, such as emulations of computer systems, operating systems, servers, and the like. The computing devices can load the virtual machines based on virtual images and / or other data defining specific software (e.g., operating systems, specialized applications, servers) for emulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device.

[0080] The service platform 2 can be configured to communicate with the client 6 and the like through a network 4. The network 4 includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 can include physical links such as coaxial cable links, twisted-pair cable links, fiber-optic links, combinations thereof, and / or the like, or wireless links such as cellular links, satellite links, Wi-Fi links, and / or the like.

[0081] The service platform 2 can provide image / video inpainting, image / video super-resolution, variational codec training, diffusion model training, and / or the like services, such as inpainting an image uploaded by the client or improving the resolution of the image.

[0082] The client 6 can be an electronic device running an operating system such as Windows, Android™, or iOS, such as a smartphone, a tablet device, a laptop computer, a virtual reality device, a gaming device, a set-top box, an in-vehicle terminal, a smart television. Based on the operating system, various applications can be run, such as uploading an image to be inpainted to the service platform 2. Of course, the client 6 can also provide a local image inpainting function.

[0083] The client 6 can provide / configure a user access page, which can be used to upload an image to be inpainted and the like.

[0084] It should be noted that the above devices are exemplary, and the number and types of devices can be adjusted in different scenarios or according to different needs.

[0085] The following describes the technical solutions of the present application through a plurality of embodiments with the service platform 2 as the execution subject. It should be understood that these embodiments can be implemented in a variety of different forms, and should not be interpreted as being limited to the embodiments set forth herein.

[0086] Embodiment One

[0087] Figure 2 An image inpainting method according to the present application is schematically shown in the flowchart of Embodiment One.

[0088] As shown in the flowchart, the image inpainting method can include steps S200-S208, wherein: Figure 2

[0089] Step S200: obtaining input data to be input according to an image to be inpainted.

[0090] The image to be inpainted can be a picture or video with poor quality, such as an old photo, a blurred video, and the like.

[0091] Poor quality can be understood as poor clarity and / or resolution, or serious edge distortion, color distortion, and the like. ​

[0092] The image to be repaired can be a locally stored image or a user-uploaded image.

[0093] The image to be repaired can be various types of pictures or videos, such as pictures in JPG (Joint Photographic Experts Group) format or PNG (Portable Network Graphics), etc.

[0094] The input data to be input is data obtained based on the image to be repaired, as input of a subsequent diffusion model.

[0095] The input data to be input exists in the form of a tensor, that is, each number corresponds to one or more pixels in the image to be repaired.

[0096] The input data to be input is corresponding data obtained after the image to be repaired is compressed or reduced in dimension.

[0097] In the case of obtaining the image to be repaired, the input data to be input can be obtained in various ways, such as variational autoencoder, PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), LLE (Locally Linear Embedding), etc.

[0098] In step S202, the input data to be input is input into the diffusion model to fuse the input data to be input and the randomly generated Gaussian noise map through the diffusion model to obtain fused data; wherein the diffusion model includes a noise prediction model.

[0099] Diffusion model: (1) The change of pixel value can be calculated based on the surrounding neighborhood of each pixel. (2) The change rule of each pixel value is defined by a diffusion function, such as: how to update its own value according to the pixel values of its surrounding neighborhood. The diffusion function can be a linear function, a nonlinear function or a probability model (such as a noise prediction model). (3) The pixel value can be updated continuously through iteration until the stopping condition or convergence is reached. The specific repair process is: gradually repairing the image through multiple iterations. In each iteration, the diffusion model calculates the new value of each pixel to repair and restore the information of the image. The similarity of pixels within the neighborhood, gradient, etc. can also be introduced through the diffusion function to reconstruct the lost pixel information. In the case where the quality of the repaired image meets the preset requirements, the iteration is stopped.

[0100] The fusion method can include data / feature splicing, data / feature combination, etc., which can be selected according to actual needs and is not limited herein.

[0101] In step S204, the noise parameter of the fused data is predicted by the noise prediction model.

[0102] The noise prediction model can include a U-Net model, a GRU (Gate Recurrent Unit) neural network model, etc.

[0103] The noise parameter is obtained according to the noise distribution of the to-be-input data.

[0104] In step S206, the output data corresponding to the repaired image is reconstructed and generated according to the to-be-input data and the noise parameter.

[0105] The corresponding output data is continuously reconstructed and generated according to the noise parameter in an iterative manner until a stop condition or convergence is reached.

[0106] When repairing the to-be-repaired image, the noise parameter is predicted by the noise prediction model. Since the noise parameter predicted each time is sampled from a noise distribution, it has certain randomness and diversity, which makes the reconstructed output data (i.e., the repaired image) more detailed and diverse in details. Therefore, the generated repaired image can restore more high-definition and delicate detail information and high-frequency information in the detail part, thereby improving the image processing effect.

[0107] In step S208, the output data is output by the diffusion model, and the output data is the repaired image corresponding to the to-be-repaired image.

[0108] The output data is the reconstructed data obtained after multiple iterations, that is, the corresponding repaired image.

[0109] The repaired image can include restored pixel values to fill in the damaged area and restore the integrity of the image, achieving the image repair effect.

[0110] The diffusion model can perform local calculation on each pixel and can utilize the interaction between pixels, which can capture the micro details and global structure in the image. Through step-by-step iteration and information propagation, the diffusion model can fill in the blank area of the image, restore the lost details, and generate an image with natural, coherent, and realistic feeling.

[0111] Based on the image generation capability of the diffusion model, the diffusion model is used to repair low-quality photos (to-be-repaired images) and old videos to generate corresponding high-quality images, thereby improving the image processing effect. In addition, when processing the to-be-repaired image, the diffusion model can realize efficient calculation through the diffusion process, thereby improving the calculation speed in the image repair process and improving the efficiency of image processing.

[0112] The diffusion model consumes large computing resources in the image inpainting process. To reduce the consumption of resources, the following optional embodiments are provided.

[0113] In an optional embodiment, as shown in Figure 3 Step S202 can include:

[0114] Step S300, inputting the image to be inpainted into a variational autoencoder, the variational autoencoder being used for compression, dimensionality reduction, and data reconstruction.

[0115] Step S302, obtaining the reconstructed data of the variational autoencoder, wherein the reconstructed data is the input data.

[0116] The variational autoencoder is a generative model used for data reconstruction. In the variational autoencoder, the image to be inpainted is input into an encoder, which converts the image to be inpainted into two vectors of mean and variance, and the two vectors form a Gaussian distribution.

[0117] Compression: Reduce the amount of data to reduce storage space without losing useful information, thereby improving its transmission, storage, and processing efficiency.

[0118] Dimensionality reduction: By retaining some important features and removing some redundant features, the dimensionality of the data features is reduced.

[0119] The reconstructed data is the data after the image to be inpainted is sequentially compressed, dimensionally reduced, and reconstructed.

[0120] The specific process of the variational autoencoder processing the image to be inpainted is as follows: the encoder maps the image to be inpainted to a latent space to obtain an encoding vector, and the decoder converts the encoding vector in the latent space back to reconstructed data (as input data of the diffusion model). The latent space is a continuous space subject to Gaussian distribution.

[0121] In this embodiment, the variational autoencoder is used to process the image to be inpainted, i.e., to compress and reduce the dimensionality of the image to be inpainted. On the one hand, by processing through the variational autoencoder, redundant information can be effectively removed and the amount of data can be reduced. By inputting the reconstructed data processed by the variational autoencoder into the diffusion model, the consumption of computing resources of the diffusion model in the image inpainting process can be further reduced, and the efficiency of the diffusion model in image processing can be improved. On the other hand, since the variational autoencoder can remove redundant information, the subsequent diffusion model can effectively capture key features when processing the image to be inpainted, reduce the interference of invalid image information, and thus make the processing of details and textures more accurate and improve the effect of image processing.

[0122] The variational codec can be pre-trained in various ways. The following provides a training method for reference:

[0123] In optional embodiments, as shown in Figure 4 and Figure 7 , the variational codec includes an encoder, a decoder, and a sampler; correspondingly, the variational codec is trained by inputting different sample images as inputs and performing the following multiple rounds of operations:

[0124] Step S400: inputting a sample image into the encoder to obtain an encoding vector corresponding to the sample image.

[0125] Step S402: inputting the encoding vector into the decoder to obtain a first sample reconstruction object.

[0126] Step S404: randomly generating a sampling vector by the sampler.

[0127] Step S406: inputting the sampling vector into the decoder to obtain a second sample reconstruction object.

[0128] Step S408: obtaining a first loss value according to the first sample reconstruction object and the second sample reconstruction object.

[0129] The first loss value (loss) is the error between the sampling vector and the encoding vector.

[0130] Step S410: adjusting the parameters of the variational codec according to the first loss value.

[0131] The parameters of the variational codec are adjusted by the first loss value, which increases the similarity of the distribution of the sampling vector and the encoding vector and reduces the occurrence of the case that the image corresponding to the input data and the output data of the variational codec is not the same picture.

[0132] In this embodiment, the sampling vector is randomly generated by the sampler, compared with the encoding vector to obtain the first loss value, and then the parameters of the variational codec are gradually adjusted by the loss value to train the variational codec. By training the variational codec, the reconstruction accuracy of the encoding vector is improved, and the trained variational codec can compress and reduce the dimension of the image to be repaired without losing key image information as much as possible.

[0133] The diffusion model consumes a large amount of computing resources during training. To reduce resource consumption, the following optional embodiments are provided.

[0134] In optional embodiments, as shown in Figure 5 , the following can be included:

[0135] Step S500, obtaining sample reconstruction data of the sample image through the trained variational autoencoder.

[0136] Step S502, performing model training on the diffusion model by taking the sample reconstruction data as sample input data of the diffusion model.

[0137] The sample reconstruction data is compressed data obtained based on the sample image, which is used as input for subsequent training of the diffusion model, and the sample reconstruction data is data compressed and reduced in dimension by the trained variational autoencoder.

[0138] In this embodiment, the variational autoencoder is trained before the diffusion model is trained, so that the sample input data of the diffusion model is processed by the variational autoencoder. By processing the sample input data through the variational autoencoder, redundant information can be effectively removed. In addition, by compressing and reducing the sample image, the data amount can be effectively reduced, thereby improving the calculation efficiency and saving the calculation resources during training of the diffusion model.

[0139] An exemplary training scheme of the diffusion model is provided below.

[0140] In an optional embodiment, as shown in Figure 6 and Figure 8 The diffusion model is trained by a plurality of sample image pairs, each sample image pair including a first sample image and a second sample image corresponding to the same picture but having different image qualities, and each sample image corresponds to one round of training operation of the diffusion model, wherein each round of training operation is as follows:

[0141] Step S600, randomly generating Gaussian noise.

[0142] The Gaussian noise is a kind of noise with a normal distribution (also known as Gaussian distribution) probability density function.

[0143] Step S602, adding the Gaussian noise to the first sample image to obtain a first noise image.

[0144] The Gaussian noise can be added step by step, and one first noise image is formed each time the Gaussian noise is added until a pure noise image is obtained.

[0145] The Gaussian noise can also be added at one time to directly obtain a pure noise image.

[0146] The first sample image x_0 is a sample image with high image quality.

[0147] The first noise image is a picture x_t.

[0148] Step S604, the second sample image is added to the first noise map by a conditional likelihood function to obtain a second noise map.

[0149] The second sample image can be a sample image with the same picture as the first sample image but with lower quality.

[0150] Step S606, the predicted noise parameters of the second noise map are obtained by the noise prediction model.

[0151] Step S608, the second loss value is obtained according to the predicted noise parameters and the real noise parameters, wherein the real noise parameters are the noise parameters corresponding to the Gaussian noise.

[0152] The second loss value is the difference between the predicted noise parameters and the real noise parameters.

[0153] The real noise parameters are the noise recorded in the process of adding noise to the first sample image.

[0154] Step S610, the parameters of the noise prediction model are adjusted according to the second loss value.

[0155] The specific training operation process of the diffusion model will be introduced below:

[0156] The training principle of the diffusion model: by continuously adding Gaussian noise to destroy the first sample image, and then by reversing this noise process, learning how to restore, that is, adding Gaussian noise and then removing Gaussian noise. When training the diffusion model, it can be divided into forward process and reverse process. The forward process and reverse process will be introduced below.

[0157] ① The forward process is also called diffusion process. The forward process is to change the first sample image x_0 into a pure Gaussian noise first noise map x_t by gradually adding Gaussian noise, so as to achieve the purpose of destroying the first sample image. Specifically:

[0158] The first sample image with higher quality is input into the diffusion model, and the randomly generated Gaussian noise ∈ is added to the first sample image, and then the first sample image is destroyed, that is, a noise adding process is performed on the first sample image.

[0159] The noise adding process can be represented by the following formula:

[0160]

[0161] wherein is a pre-set hyperparameter, called Noise schedule (noise schedule), which is a small value list, ∈ t-1 ~ N(0, 1) is Gaussian noise, and t is the number of noise adding steps.

[0162] From the iterative derivation of formula (1), the formula of x_0 to x_t can be obtained as follows:

[0163]

[0164] wherein, is a hyperparameter set according to the Noise schedule, ∈ ~ N(0, 1) is also a Gaussian noise, and t is the number of steps of noise addition.

[0165] Formula (1) or (2) can be used to describe the forward process. Formula (1) is used for the process of gradually destroying a picture, and formula (2) is used for the process of destroying a picture at one step.

[0166] After obtaining the first noise map, the training of the reverse process can be performed.

[0167] ②Reverse process: by estimating the real Gaussian noise, the first noise map x_t that is destroyed is gradually recovered into the first sample image x_0 through multiple iterations.

[0168] The reverse process is represented by the formula:

[0169]

[0170] In the training of the reverse process, in order to accurately estimate the real Gaussian noise, a model for estimating the real noise from x_t and t, i.e., the noise prediction model ∈ θ (x t , t), needs to be trained. By training the noise prediction model ∈ θ (x t , t), the noise predicted by the noise prediction model is similar to the real Gaussian noise ∈ used to destroy the first sample image. σ t is a fixed constant, and its specific value and selection will be determined according to specific applications and requirements.

[0171] The training process of the noise prediction model is as follows: the second sample image is added to the first noise map x_0 through the conditional likelihood function to obtain the second noise map. The second sample image with lower quality is added to the first noise map through the conditional likelihood function, so that the generated second noise map is the same as the first and second sample images. For example, the first and second sample images are a photo of the same cat, and the generated second noise map is also a photo of the same cat.

[0172] The second noise map is input into the noise prediction model to estimate the noise parameter of the second noise map, and the estimated predicted noise parameter ∈ The real noise parameter of the real noise ∈ is compared, and a second loss value Loss is obtained. The second loss value Loss can be represented by the following formula:

[0173]

[0174] The parameters of the noise prediction model are adjusted through the second loss value.

[0175] In this embodiment, the conditional likelihood function is introduced when the diffusion model is trained, so that the trained diffusion model can better capture the subtle details and textures in the image for subsequent image inpainting and super-resolution processing. In addition, the noise prediction model is trained so that the noise prediction model can more accurately predict the noise distribution of each detail part of the image to be repaired. The advantages of doing so are: subsequent reconstruction of output data (i.e. repaired image) can be performed according to the predicted noise parameters of each detail part. And because the predicted noise parameters are closer to the real Gaussian noise parameters, the diffusion model will process details and textures more accurately, and can generate images that are more realistic and have higher clarity, thereby improving the effect of image processing.

[0176] The diffusion model described above can be used to improve image quality, and the picture quality of the first sample image is higher than that of the second sample image.

[0177] The diffusion model described above can be used for image inpainting and super-resolution technology.

[0178] In optional embodiments, the clarity of the first sample image is higher than that of the second sample image. In other embodiments, the resolution of the first sample image is higher than that of the second sample image.

[0179] The clarity refers to the clarity of each detail and its boundary on the image. In this embodiment, a large number of contrast pictures with different clarity can be used as training samples, which can enable the trained diffusion model to perform image deblurring and other image inpainting tasks, so that the clarity of the repaired image is increased, thereby improving the visual effect of the image.

[0180] Image resolution is a set of performance parameters for evaluating the richness of details contained in an image, including time resolution, spatial resolution, and color scale resolution. Compared with low-resolution images, high-resolution images usually contain greater pixel density, richer texture details, and higher reliability. In this embodiment, a large number of different resolution reference images can be used as training samples, so that the trained diffusion model can be used in super-resolution technology to generate high-resolution images. Moreover, since high-resolution images have more abundant texture details, the repaired images processed by the diffusion model have clearer and more delicate details.

[0181] In order to make the present application easier to understand, an exemplary application is provided below.

[0182] In this exemplary application, the service platform 2 is connected with the client through the network and provides image repair services to the client.

[0183] First, practical application:

[0184] The blurred picture is input into the VAE to obtain the reconstructed data. The reconstructed data is input into the diffusion model, and the diffusion model fuses (such as splices) the reconstructed data and the randomly generated Gaussian noise image to obtain the fusion data. Then, the fusion data is input into the U-net model to output the estimated noise. The blurred picture is restored based on the estimated noise.

[0185] As shown in the left image input into the diffusion model, the right image can be output. Figure 9 As shown in the left image input into the diffusion model, the right image can be output. Figure 9 As shown in the left image input into the diffusion model, the right image can be output.

[0186] As shown in the left image input into the diffusion model, the right image can be output. Figure 10 As shown in the left image input into the diffusion model, the right image can be output. Figure 10 As shown in the left image input into the diffusion model, the right image can be output.

[0187] As shown in the left image input into the diffusion model, the right image can be output. Figure 9 As shown in the left image input into the diffusion model, the right image can be output. Figure 9 As shown in the left image input into the diffusion model, the right image can be output. Figure 9 As shown in the left image input into the diffusion model, the right image can be output.

[0188] Second, model training process:

[0189] ①First, the variational encoder is trained, and a specific training process is as follows:

[0190] S21: input the sample image into the encoder to obtain the corresponding encoding vector, and then input the encoding vector into the decoder to obtain the first sample reconstructed object.

[0191] S22: a sampler is used to randomly generate a sampling vector, and the sampling vector is input into the decoder to obtain a second sample reconstructed object.

[0192] S23: the first sample reconstructed object and the second sample reconstructed object are compared to obtain a first loss value.

[0193] S24: the parameters of the variational autoencoder are adjusted according to the first loss value.

[0194] ②After the variational autoencoder is trained, the diffusion model is trained. A specific training process is as follows:

[0195] S31: a high-quality sample image is input into the variational autoencoder for processing, and the processed high-quality sample image is input into the diffusion model.

[0196] S31: Gaussian noise is added to the processed high-quality sample image to obtain a first noise image.

[0197] S32: a corresponding low-quality sample image is added to the first noise image by a conditional likelihood function to obtain a second noise image.

[0198] S33: a prediction noise parameter of the second noise image is obtained by a noise prediction model.

[0199] S34: a second loss value is obtained according to the prediction noise parameter and a real noise parameter.

[0200] S35: the parameters of the noise prediction model are adjusted according to the second loss value.

[0201] Embodiment Two

[0202] Figure 11 A block diagram of an image repairing apparatus according to Embodiment Two of the present application is schematically shown. The image repairing apparatus can be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of completing a specific function. The functions of the program modules in the embodiments will be specifically described below. As shown in the figure, the image repairing apparatus 1100 can include an acquisition module 1110, an input module 1120, an estimation module 1130, a reconstruction module 1140, and an output module 1150, wherein: Figure 11

[0203] The acquisition module 1110 is configured to acquire input data according to a to-be-repaired image.

[0204] ​The input module 1120 is configured to input the to-be-input data into the diffusion model, so as to fuse the to-be-input data and the randomly generated Gaussian noise map through the diffusion model to obtain fused data; and the diffusion model comprises a noise prediction model.

[0205] The estimation module 1130 is configured to estimate a noise parameter of the fused data through the noise prediction model.

[0206] The reconstruction module 1140 is configured to reconstruct output data corresponding to the repaired image according to the to-be-input data and the noise parameter.

[0207] The output module 1150 is configured to output the output data through the diffusion model, and the output data is the repaired image corresponding to the to-be-repaired image.

[0208] In an optional embodiment, the acquisition module 1110 is further configured to:

[0209] input the to-be-repaired image into a variational codec, and the variational codec is configured to compress, reduce dimensions and reconstruct data;

[0210] acquire reconstruction data of the variational codec, and the reconstruction data is the to-be-input data.

[0211] In an optional embodiment, the variational codec comprises an encoder, a decoder and a sampler.

[0212] The image repairing apparatus can further comprise a first training module (not identified) configured to:

[0213] input a sample image into the encoder to obtain an encoding vector corresponding to the sample image;

[0214] input the encoding vector into the decoder to obtain a first sample reconstruction object;

[0215] randomly generate a sampling vector through the sampler;

[0216] input the sampling vector into the decoder to obtain a second sample reconstruction object;

[0217] acquire a first loss value according to the first sample reconstruction object and the second sample reconstruction object;

[0218] adjust parameters of the variational codec according to the first loss value.

[0219] In an optional embodiment, the apparatus can further comprise a second training module (not identified) configured to:

[0220] obtain sample reconstruction data of the sample image through the trained variational encoder-decoder;

[0221] train the diffusion model by taking the sample reconstruction data as sample input data of the diffusion model.

[0222] In an optional embodiment, the diffusion model is trained by a plurality of sample image pairs, and each sample image pair includes a first sample image and a second sample image corresponding to the same picture but having different picture qualities; and the second training module (not labeled) is further configured to:

[0223] randomly generate Gaussian noise;

[0224] add the Gaussian noise to the first sample image to obtain a first noise image;

[0225] add the second sample image to the first noise image through a conditional likelihood function to obtain a second noise image;

[0226] obtain a predicted noise parameter of the second noise image through the noise prediction model;

[0227] obtain a second loss value according to the predicted noise parameter and a real noise parameter; the real noise parameter is a noise parameter corresponding to the Gaussian noise;

[0228] adjust a parameter of the noise prediction model according to the second loss value.

[0229] In an optional embodiment, the picture quality of the first sample image is higher than that of the second sample image.

[0230] In an optional embodiment, the definition of the first sample image is higher than that of the second sample image; and / or the resolution of the first sample image is higher than that of the second sample image.

[0231] Embodiment Three

[0232] Figure 12 A hardware architecture schematic diagram of a computer device 10000 suitable for implementing the image inpainting method according to Embodiment Three of the present application is schematically shown. In some embodiments, the computer device 10000 can be a terminal device such as a smartphone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workstation, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 can be a rack-mounted server, a blade server, a tower server, or a cabinet server (including a standalone server, or a server cluster composed of multiple servers), etc. As shown in the figure, the computer device 10000 includes a processor 10010, a memory 10020, a storage 10030, a keyboard 10040, a mouse 10050, a display screen 10060, and a communication interface 10070. The processor 10010, the memory 10020, the storage 10030, the keyboard 10040, the mouse 10050, the display screen 10060, and the communication interface 10070 are communicatively connected to each other through a bus. Figure 12As shown, the computer device 10000 includes, but is not limited to, a memory 10010, a processor 10020, and a network interface 10030, which are communicatively linked via a system bus.

[0233] The memory 10010 includes at least one type of computer-readable storage media, which includes a flash memory, a hard disk, a multimedia card micro, a card-type memory (e.g., SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 10010 can be an internal memory module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 10010 can include both an internal memory module and an external storage device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application programs installed in the computer device 10000, such as program codes of the image inpainting method, and the like. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0234] The processor 10020 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips in some embodiments. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication of the computer device 10000, and the like. In this embodiment, the processor 10020 is used to run program codes or process data stored in the memory 10010.

[0235] The network interface 10030 can include a wireless network interface or a wired network interface, and is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 with an external terminal through a network, to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal, and the like. The network can be an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, and the like wireless or wired network.

[0236] It should be noted that, Figure 12 Only the computer device with the components 10010-10030 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0237] In this embodiment, the image inpainting method stored in the memory 10010 can also be divided into one or more program modules, and executed by one or more processors (such as the processor 10020) to complete the embodiments of the present application.

[0238] Embodiment Four

[0239] The embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium has a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the image inpainting method in the embodiments.

[0240] In this embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer readable storage medium is usually used to store an operating system and various application software installed on the computer device, such as program codes of the image repairing method in the embodiments, etc. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0241] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be realized by a general computer device, which can be concentrated on a single computer device or distributed on a network composed of multiple computer devices, and optionally, each module or each step can be realized by program codes executable by a computer device, so that each module or each step can be stored in a storage device and executed by a computer device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0242] It should be noted that the above is only the preferred embodiment of the present application, and does not limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image inpainting method characterized by, The method comprises: obtaining input data from the image to be repaired; inputting the input data into a diffusion model to fuse the input data and randomly generated Gaussian noise through the diffusion model to obtain fused data; wherein the diffusion model comprises a noise prediction model; estimating noise parameters of the fused data through the noise prediction model; reconstructing output data corresponding to the repaired image according to the input data and the noise parameters; outputting the output data through the diffusion model, wherein the output data is the repaired image corresponding to the image to be repaired; wherein the input data is obtained from the image to be repaired, comprising: inputting the image to be repaired into a variational encoder-decoder, wherein the variational encoder-decoder is used for compression, dimension reduction and data reconstruction; obtaining the reconstructed data of the variational encoder-decoder, wherein the reconstructed data is the input data; wherein the variational encoder-decoder comprises an encoder, a decoder and a sampler; correspondingly, the variational encoder-decoder is trained by inputting different sample images as inputs and performing the following multiple rounds of operations: inputting the sample image into the encoder to obtain an encoding vector corresponding to the sample image; inputting the encoding vector into the decoder to obtain a first sample reconstruction object; randomly generating a sampling vector through the sampler; inputting the sampling vector into the decoder to obtain a second sample reconstruction object; obtaining a first loss value according to the first sample reconstruction object and the second sample reconstruction object; adjusting the parameters of the variational encoder-decoder according to the first loss value.

2. The method of claim 1, wherein, The diffusion model is obtained through the following training operations: obtaining sample reconstruction data of the sample image through the trained variational encoder-decoder; inputting the sample reconstruction data as sample input data of the diffusion model to perform model training on the diffusion model.

3. The method of claim 1, wherein, The diffusion model is trained by multiple sample image pairs, wherein each sample image pair comprises a first sample image and a second sample image corresponding to the same picture but having different picture qualities; each sample image corresponds to one round of training operation of the diffusion model, wherein each round of training operation is as follows: randomly generating Gaussian noise; adding the Gaussian noise to the first sample image to obtain a first noise map; adding the second sample image to the first noise map through a conditional likelihood function to obtain a second noise map; obtaining a predicted noise parameter of the second noise map through the noise prediction model; obtaining a second loss value according to the predicted noise parameter and a real noise parameter; wherein the real noise parameter is a noise parameter corresponding to the Gaussian noise; adjusting the parameters of the noise prediction model according to the second loss value.

4. The method of claim 3, wherein: the picture quality of the first sample image is higher than that of the second sample image.

5. The method of claim 4, wherein: the definition of the first sample image is higher than that of the second sample image; and / or the resolution of the first sample image is higher than that of the second sample image.

6. An image inpainting apparatus characterized by comprising: The device comprises: The acquisition module is configured to acquire input data according to the image to be repaired. The input module is configured to input the input data into a diffusion model to fuse the input data and a randomly generated Gaussian noise map by the diffusion model to obtain fused data, wherein the diffusion model comprises a noise prediction model. The estimation module is configured to estimate a noise parameter of the fused data by the noise prediction model. The reconstruction module is configured to reconstruct output data corresponding to a repaired image according to the input data and the noise parameter. The output module is configured to output the output data by the diffusion model, and the output data is the repaired image corresponding to the image to be repaired. The acquisition module is further configured to input the image to be repaired into a variational autoencoder, the variational autoencoder is configured to compress, reduce dimensions, and reconstruct data, and acquire reconstruction data of the variational autoencoder, wherein the reconstruction data is the input data. The variational autoencoder comprises an encoder, a decoder, and a sampler. Correspondingly, the image repairing apparatus can further comprise a first training module configured to input a sample image into the encoder to obtain an encoding vector corresponding to the sample image, input the encoding vector into the decoder to obtain a first sample reconstruction object, randomly generate a sampling vector by the sampler, input the sampling vector into the decoder to obtain a second sample reconstruction object, acquire a first loss value according to the first sample reconstruction object and the second sample reconstruction object, and adjust parameters of the variational autoencoder according to the first loss value.

7. A computer device, comprising: comprise: at least one processor; and a memory connected with the at least one processor in communication; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method of any one of claims 1 to 5.

9. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to implement the method of any one of claims 1 to 5.

Images