Image Completion Method, System, Device, and Storage Medium Based on Model Priors

Abstract

Embodiments of the present application disclose an image completion method, system, device, and storage medium based on model prior. The technical solution provided by the embodiments of the present application obtains a missing image and information on a specified completion area of the missing image; and then inputs the missing image and the information on the specified completion area into a pre-constructed image completion model to output a completed image of the missing image. Among them, the image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges. The pre-trained model is pre-trained for image completion based on training samples, and after the pre-trained model converges, prior samples are generated based on the pre-trained model. By adopting the above technical means, more realistic and diverse image completion results can be provided, and the diversity of the images completed by the image completion model can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular, to an image completion method, system, device, and storage medium based on model prior. Background Art

[0002] Currently, image completion technology is widely used in various scenarios such as short videos and live broadcasts. Image inpainting is an important technical direction in the field of computer vision, which is used to complete the invisible or missing parts of an image, and at the same time make the whole image reasonable without a sense of incongruity, so that the completed image is as realistic as possible. Existing image completion technologies are mainly implemented based on image completion models, and the image completion models are constructed based on generative adversarial networks. By training an image completion model based on a generative adversarial network, some areas are randomly masked during training for completion, thereby realizing image completion.

[0003] However, for the existing image completion model obtained by simply masking some areas of the image for model training, the completed images are relatively single and lack diversity. The completed images obtained in this way have relatively poor completion effects and are prone to a sense of incongruity in the completed images. Summary of the Invention

[0004] The embodiments of the present application provide an image completion method, system, device, and storage medium based on model prior, which can improve the diversity of completed images, improve the image completion effect, and solve the technical problems of single and incongruous completion effects of existing image completion models.

[0005] In a first aspect, the embodiments of the present application provide an image completion method based on model prior, including:[[]]

[0006] Obtain a missing image and information on a specified completion area of the missing image;

[0007] Input the missing image and the specified completion area information into a pre-constructed image completion model, and output a completed image of the missing image. The image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges;

[0008] The pre-trained model is pre-trained for image completion based on training samples. After the pre-trained model converges, prior samples are generated based on the pre-trained model. The prior samples include a noisy image and multiple denoised images obtained by performing image completion on the corresponding noisy image.

[0009] In a second aspect, the embodiments of the present application provide an image completion system based on model prior, including:[[]]

[0010] An acquisition module, configured to acquire a missing image and information on a specified completion region of the missing image;

[0011] A completion module, configured to input the missing image and the specified completion region information into a pre-built image completion model, and output a completed image of the missing image. The image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges; the pre-trained model is pre-trained for image completion based on the training samples, and after the pre-trained model converges, prior samples are generated based on the pre-trained model. The prior samples include a noisy image and multiple denoised images obtained by performing image completion on the corresponding noisy image.

[0012] In a third aspect, an embodiment of the present application provides an image completion device based on model prior, including:

[0013] A memory and one or more processors;

[0014] The memory is configured to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image completion method based on model prior as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are configured to execute the image completion method based on model prior as described in the first aspect when executed by a computer processor.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which contains instructions. When the instructions run on a computer or a processor, the computer or the processor is caused to execute the image completion method based on model prior as described in the first aspect.

[0018] In the embodiments of the present application, a missing image and information on a specified completion area of the missing image are obtained; then, the missing image and the information on the specified completion area are input into a pre-constructed image completion model to output a completed image of the missing image. Among them, the image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges. The pre-trained model is pre-trained for image completion based on training samples. After the pre-trained model converges, prior samples are generated based on the pre-trained model. Here, the prior samples include a noisy image and multiple denoised images obtained by performing image completion on the corresponding noisy image. By adopting the above technical means, the image completion model is trained with the model parameters, training samples, and prior samples provided by the pre-trained model, and the image completion model obtained in this way can provide more realistic and diverse image completion results. Based on the pre-trained model, optimization training is further performed to further improve the diversity of the images completed by the image completion model, reduce image completion defects and a sense of incongruity, and make the completed images more realistic and stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an image completion method based on model prior provided by an embodiment of the present application;

[0020] Figure 2 is a training flowchart of the pre-trained model in the embodiments of the present application;

[0021] Figure 3 is a schematic diagram of image noise addition and denoising processing in the embodiments of the present application;

[0022] Figure 4 is a training flowchart of the image completion model in the embodiments of the present application;

[0023] Figure 5 is a schematic structural diagram of an image completion system based on model prior provided by an embodiment of the present application;

[0024] Figure 6 is a schematic structural diagram of an image completion device based on model prior provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following further describes specific embodiments of this application in conjunction with the accompanying drawings. It can be understood that the specific embodiments described herein are merely for explaining this application and not for limiting this application. Additionally, it should be noted that for ease of description, only parts related to this application are shown in the drawings rather than all the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0026] An image completion method based on model prior provided by this application aims to train an image completion model through the model parameters, training samples, and prior samples provided by a pre-trained model, so as to improve the diversity of the completed images on the basis of the pre-trained model and make the completed images generated by the image completion model more realistic and stable. For traditional image completion schemes, when performing image completion, one method is to use interpolation to fill the completion information step by step from the outside of the missing position to the missing area. This method can relatively well complete small areas, but the completion effect for large areas is not good. Another method is to use an image completion method based on a generative adversarial network. By training a generative adversarial model, some areas are randomly occluded during training for image completion, thereby achieving image completion. Although this method can complete large areas, the completed images are relatively single and lack diversity. Based on this, an image completion method based on model prior in an embodiment of this application is provided to solve the technical problems of the single and incongruous effects of the existing image completion models in completing images.

[0027] Embodiment:

[0028] Figure 1 The flowchart of an image completion method based on model prior provided by an embodiment of this application is given. The image completion method based on model prior provided in this embodiment can be executed by an image completion device based on model prior. The image completion device based on model prior can be implemented in software and / or hardware. The image completion device based on model prior can be composed of two or more physical entities or can be composed of one physical entity. Generally speaking, the image completion device based on model prior can be a computer, an image processing server, a mobile phone, a tablet, or other processing devices.

[0029] The following describes the example of using the image completion device based on model prior as the main body to execute the image completion method based on model prior. Refer to Figure 1 , the image completion method based on model prior specifically includes:

[0030] S110. Obtain the missing image and the specified completion region information of the missing image;

[0031] S120. Input the missing image and the specified completion region information into a pre-constructed image completion model, and output the completed image of the missing image. The image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges; the pre-trained model performs image completion training based on the training samples in advance. After the pre-trained model converges, prior samples are generated based on the pre-trained model. The prior samples include the noisy images and multiple denoised images obtained by performing image completion on the corresponding noisy images.

[0032] An image completion method based on model prior provided by an embodiment of the present application performs image completion processing on an image to be completed through a pre-trained image completion model during image completion, so as to complete the missing part of the image. Define the image to be completed as the missing image, and the image after image completion processing as the completed image. By inputting the missing image into the image completion model, the image completion model predicts the missing partial regions of the missing image, and then outputs the corresponding completed image.

[0033] Moreover, in order to make the completed image output by the image completion model more diverse and realistic, the present application trains the image completion model with the prior information of the pre-trained model, so that the image completion model not only has the basic image completion ability of the pre-trained model, but also has the ability to generate diverse completed images.

[0034] Prior to this, a pre-trained model is first trained, and the pre-trained model is a diffusion model. Then, the pre-trained model is used to generate a variety of prior samples. The prior samples include noisy images and multiple denoised images obtained by image completion of the corresponding noisy images. Each noisy image and the denoised image obtained by image completion processing of the corresponding image form a data pair, which is used for subsequent training of the image completion model. By using the model parameters of the pre-trained model as the initial model parameters of the image completion model, the training samples of the pre-trained model and the prior samples generated by it are input into the image completion model to train the image completion model until the model converges. Since a noisy image in the prior samples includes multiple denoised images, the diversity of the image completion processing results can be reflected. Subsequently, prior samples that can reflect the diversity of the image completion processing results are added to the basis of the training samples, making the trained image completion model more diverse and having a better image completion effect. Moreover, the image completion model is further optimized for image completion training based on the model parameters after convergence of the pre-trained model training as the initial model parameters, so that the image completion results of the model have fewer defects, the image completion is more realistic, and the model processing speed is faster.

[0035] Specifically, referring to Figure 2 , the training process of the pre-trained model includes:

[0036] S101. Perform noise addition processing on the training samples based on the pre-trained model to obtain noisy samples;

[0037] S102. Perform image completion on the noisy samples based on the pre-trained model to obtain denoised samples, calculate the first model loss function according to the denoised samples, and iteratively adjust the model parameters of the pre-trained model based on the first model loss function until the pre-trained model converges.

[0038] The training process of the pre-trained model includes the noise addition and denoising processes of the training samples. As Figure 3 shown, the noise addition process is a forward process, which is used to add image noise to the sample images of the training samples, so that a missing area is formed on the sample images for subsequent image completion training to predict and complete the missing area. The denoising process is the inverse process of the noise addition process. By performing denoising prediction on the noisy images, the original sample images are continuously inferred. Referring to Figure 3 , x0 to x T represents the processing process of the sample images from time 0 to T, that is, the noise addition process, in which noise is continuously added to the real sample images. x T to x0 represents the processing process of the sample images from time T to 0, that is, the denoising process, in which denoising inference is continuously performed to predict the initial sample images, that is, the sample images x0 at time point 0.

[0039] Among them, the noisy samples are obtained by adding noise to the training samples based on the pre-trained model, including:

[0040] At each specified moment within the set time period, Gaussian distribution noise is added to the training images of the training samples based on the pre-trained model to obtain the Gaussian noise distribution probabilities of each training image, and the Gaussian noise distribution probabilities of each training image are used as the noisy samples;

[0041] The denoised samples are obtained by image completion of the noisy samples based on the pre-trained model, including:

[0042] Based on the Gaussian noise distribution probabilities of each training image and the set estimation parameters, the real-time noise distribution probabilities at each specified moment within the set time period are predicted in sequence, and the prediction result at the first specified moment within the set time period is used as the denoised sample.

[0043] Taking a training image as shown in Figure 3 as an example, noise addition or denoising processing is performed at the specified moment within the set time period, and thus the real-time noise distribution probability at any specified moment t can be obtained:

[0044]

[0045] And the Gaussian noise distribution probabilities from the specified moment 0 to T total, that is, the Gaussian noise distribution probability of the training image:

[0046]

[0047] Among them, β represents a hyperparameter, which is the variance of the preset Gaussian distribution, and β can take a linear interpolation from 1 e-4 to 1 e-2 , t represents the specified moment, its time step is fixed, N represents the Gaussian distribution, T represents the total duration of the set time period, and x t represents the image after noise addition processing at the moment t.

[0048] By giving a sample image x0, based on the above noise addition processing flow, Gaussian distribution noise is added at each specified moment to obtain x1, x2...x T , as the time step continuously increases, finally an image x T containing the Gaussian noise distribution information of the sample image is obtained.

[0049] Among them, for the given x0, then at the moment t:

[0050]

[0051] Here, α s and α t represent the step size hyperparameters, I represents the normal Gaussian distribution, αt = 1 - β t 。

[0052] Furthermore, since the noise addition process continuously adds Gaussian distributed noise at specified times according to a set time period, the real-time noise distribution probability q(x t-1 |x t ) cannot be directly inferred theoretically during the denoising process. In an embodiment of the present application, based on the originally noise-added data after noise addition processing through a set estimation parameter θ, the real-time noise distribution probability of each noise-added image at each specified time within the set time period is predicted, that is, the real-time noise distribution probability predicted at any specified time t:

[0053] p θ (x t-1 |x t ):= N(x t-1 ; μ θ (x t , t), ∑ θ (x t , t)) (1)

[0054] Then the total predicted Gaussian noise distribution probability from specified time 0 to T is expressed as:

[0055]

[0056] Then, under the condition of a given sample image x0, it can be obtained through Bayes' formula:

[0057]

[0058] And it can be obtained:

[0059]

[0060] Among them, μ represents the predicted probability mean, and ε represents the predicted noise distribution.

[0061] During the denoising process, the Gaussian noise at each step can be used to predict ε t (x θ , t) through x t and t, and then the probability mean μ(x t , t) can be obtained according to the above formula (4). Combining formulas (1), (2), and (3), the predicted real-time noise distribution probability q(x t-1 |x t ) can be obtained, thereby predicting the image x t-1 .

[0062] Based on this principle, the pre-trained model is trained with noise addition and denoising. During training, the step size T is selected as 1000, β linearly increases from 0.0004 to 0.002, the learning rate is initially set to 0.00001, and then decays according to the number of training times. During training, the sample images are randomly occluded for noise addition processing, and image completion training is performed based on the above training process. The loss function is calculated for the denoised samples predicted during the training process. Define its loss function as the first model loss function, and iteratively adjust the model parameters of the pre-trained model based on the first model loss function until the pre-trained model converges. Among them, the first model loss function adopts the L1 loss function, and the first model loss function is expressed as:

[0063] Loss = abs(M(x) – N(u,σ))

[0064] Among them, M(x) represents the true probability distribution of the image, N represents the Gaussian distribution, u represents the mean of the noise, and σ represents the variance of the noise.

[0065] Furthermore, based on the trained pre-trained model, various data pairs, that is, prior samples, are generated using this pre-trained model. Among them, generating prior samples based on the pre-trained model includes:

[0066] Input the noisy image into the pre-trained model, and perform image completion on the noisy image for a set number of times based on the pre-trained model to obtain a corresponding number of denoised images. Use the set number of noisy images and the corresponding denoised images as prior samples.

[0067] Among them, the noisy image can be a noisy image obtained by properly adding noise to the sample image of the training sample through the pre-trained model, or an image after occlusion processing can be used as the noisy image and input into the pre-trained model. For the same noisy image, various reasonable completion effects also need to be generated to obtain multiple different denoised images, and multiple data pairs with non-unique results are constructed to increase the diversity of the data. It can be understood that for the model's image completion processing, as long as the image completion is reasonable, similar to the fact that the answer is not unique, such a data set is more in line with the actual reasoning characteristics. In addition, the position of completion, the size of completion, and the pattern shape all need to be random, so that the more similar completion data is generated, the higher the diversity of the data.

[0068] Furthermore, since it is relatively difficult to guarantee the quality of the images randomly completed by the model, there will be some denoised images with poor effects. To improve the training effect of the image completion model and make the completed images more realistic and stable, it is necessary to clean the prior fake samples. Optionally, in the embodiments of the present application, based on the prior samples generated by the pre-trained model, the completion processing scores of each denoised image are calculated based on the noisy image and the corresponding denoised image, and the prior samples are screened based on the completion processing scores.

[0069] Among them, by training a generative adversarial model for randomly generating graphs, the evaluation ability of the discriminator for the denoising effect of the generated graphs is trained. The discriminator of the trained generative adversarial model is used for data screening of prior samples. By inputting the denoised images into the discriminator, the discriminator outputs corresponding completion processing scores according to the denoising quality, and then based on the completion processing scores, a relatively high portion of the data is screened out for training the image completion model.

[0070] Optionally, based on the completion processing scores of each denoised image, the generated completion processing scores are sorted, and a completion processing score distribution curve is plotted. Then, a threshold is selected according to the distribution curve, and the denoised images higher than the threshold are saved to construct the final prior samples for training the image completion model. In practical applications, a binary classification model can also be trained for data screening of prior samples. The embodiments of the present application do not make a fixed limitation on the specific data screening method and will not elaborate here.

[0071] Furthermore, based on the screened prior samples, combined with the training samples of the pre-trained model and the model parameters when the model converges, the image completion model can be trained. Referring to Figure 4 , the training process of the image completion model includes:

[0072] S103: Using the model parameters when the pre-trained model converges as the initial model parameters, input the training samples and the prior samples into the image completion model;

[0073] S104: Based on the image completion model, use the training samples and the prior samples for image completion training, calculate the second model loss function based on the training results, and iteratively adjust the initial model parameters based on the second model loss function until the image completion model converges.

[0074] Among them, using the model parameters when the pre-trained model converges as the initial model parameters can make the image completion model converge faster and improve the model training efficiency. Moreover, since the pre-trained model has a certain image completion ability, further training the model on this basis can enable the image completion model to inherit the advantages of the original model, optimize the image completion effect of the model, and make the generated completed images more real and stable. At the same time, training the model with prior samples can enable the model to have the ability to generate diverse completed images and improve the diversity of the completed images. In addition, through training with diverse samples, the image completion model can adapt to the image completion of different missing images, and can perform image completion more efficiently than the pre-trained model, and its model time consumption is at the millisecond level.

[0075] Based on the prior information provided by the above pre-trained model, during the model training process, the loss function is calculated according to the completed images generated by the image completion model. Define its loss function as the second model loss function, and the second model loss function includes the perceptual loss function and the L1 loss function. The perceptual loss function is expressed as:

[0076] Perceptual loss=E((VGG(x)-VGG(M(x)))2)

[0077] The L1 loss function is expressed as:

[0078] L1_loss=E(x–M(x))

[0079] Where, X represents the real image, and M(x) represents the predicted image.

[0080] During training, the Adam optimizer is used, and the learning rate is set to 0.0001. The second loss function is calculated based on the training results of the image completion model, and the initial model parameters are continuously iteratively adjusted according to the value of the second loss function until the image completion model converges, completing the model training.

[0081] Furthermore, based on the trained image completion model, by obtaining the missing image to be completed and the information of the specified completion area of the missing image, the missing image and the information of the specified completion area are input into the pre-constructed image completion model, and image completion processing is performed based on the image completion model to predict the image information at the specified completion area, and the predicted completion area is fused back to the missing image to obtain the final completed image.

[0082] Exemplarily, in practical applications, the user inputs an image to be completed and specifies the area to be completed, and the image completion model performs image completion processing to output the completed image. By inputting a video and specifying the area to be completed in the video, the image completion model predicts the image information of the completion area for each frame of the image, and the image can be restored frame by frame to output the completed video.

[0083] As described above, by obtaining a missing image and information on a specified completion region of the missing image; then inputting the missing image and the specified completion region information into a pre-constructed image completion model to output a completed image of the missing image. Among them, the image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges. The pre-trained model is pre-trained for image completion based on the training samples. After the pre-trained model converges, prior samples are generated based on the pre-trained model. Here, the prior samples include a noisy image and multiple denoised images obtained by performing image completion on the corresponding noisy image. By using the above technical means, the image completion model is trained with the model parameters, training samples, and prior samples provided by the pre-trained model, and the obtained image completion model can provide more realistic and diverse image completion results. On the basis of the pre-trained model, optimization training is carried out to further improve the diversity of the images completed by the image completion model, reduce image completion defects and a sense of incongruity, and make the completed images more realistic and stable.

[0084] Based on the above embodiments, Figure 5 This is a schematic structural diagram of an image completion system based on model prior provided by this application. Refer to Figure 5 , the image completion system based on model prior provided in this embodiment specifically includes: an acquisition module 21 and a completion module 22.

[0085] Among them, the acquisition module 21 is configured to acquire a missing image and information on a specified completion region of the missing image;

[0086] The completion module 22 is configured to input the missing image and the specified completion region information into a pre-constructed image completion model to output a completed image of the missing image. The image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges; the pre-trained model is pre-trained for image completion based on the training samples. After the pre-trained model converges, prior samples are generated based on the pre-trained model. The prior samples include a noisy image and multiple denoised images obtained by performing image completion on the corresponding noisy image.

[0087] Specifically, the training process of the pre-trained model includes:

[0088] Performing noise addition processing on the training samples based on the pre-trained model to obtain noisy samples;

[0089] Performing image completion on the noisy samples based on the pre-trained model to obtain denoised samples, and calculating a first model loss function based on the denoised samples, and iteratively adjusting the model parameters of the pre-trained model based on the first model loss function until the pre-trained model converges.

[0090] Among them, adding noise to the training samples based on the pre-trained model to obtain noisy samples includes:

[0091] At each specified moment within the set time period, adding Gaussian distribution noise to the training images of the training samples based on the pre-trained model to obtain the Gaussian noise distribution probabilities of each training image, and using the Gaussian noise distribution probabilities of each training image as the noisy samples;

[0092] Performing image completion on the noisy samples based on the pre-trained model to obtain denoised samples, including:

[0093] Predicting the real-time noise distribution probabilities at each specified moment within the set time period in sequence based on the Gaussian noise distribution probabilities of each training image and the set estimation parameters, and using the prediction result at the first specified moment within the set time period as the denoised sample.

[0094] Generating prior samples based on the pre-trained model, including:

[0095] Inputting the noisy images into the pre-trained model, performing image completion on the noisy images for a set number of times based on the pre-trained model to obtain the corresponding number of denoised images, and using the set number of noisy images and the corresponding denoised images as the prior samples.

[0096] Calculating the completion processing scores of each denoised image based on the noisy images and the corresponding denoised images, and performing data screening on the prior samples based on the completion processing scores.

[0097] Specifically, the training process of the image completion model includes:

[0098] Using the model parameters when the pre-trained model converges as the initial model parameters, and inputting the training samples and the prior samples into the image completion model;

[0099] Performing image completion training on the training samples and the prior samples based on the image completion model, calculating the second model loss function based on the training results, and iteratively adjusting the initial model parameters based on the second model loss function until the image completion model converges.

[0100] Among them, the second model loss function includes a perceptual loss function and an L1 loss function.

[0101] As described above, by obtaining a missing image and information on a specified completion region of the missing image; and then inputting the missing image and the information on the specified completion region into a pre-constructed image completion model to output a completed image of the missing image. Among them, the image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges. The pre-trained model is pre-trained for image completion based on training samples. After the pre-trained model converges, prior samples are generated based on the pre-trained model. Here, the prior samples include a noisy image and multiple denoised images obtained by performing image completion on the corresponding noisy image. By adopting the above technical means, the image completion model is trained using the model parameters, training samples, and prior samples provided by the pre-trained model, and the image completion model obtained in this way can provide more realistic and diverse image completion results. Based on the pre-trained model, optimization training is further carried out to further improve the diversity of the images completed by the image completion model, reduce image completion defects and a sense of incongruity, and make the completed images more realistic and stable.

[0102] The image completion system based on model prior provided by an embodiment of the present application can be configured to execute the image completion method based on model prior provided by the above embodiment, and has corresponding functions and beneficial effects.

[0103] Based on the above actual example, an embodiment of the present application further provides an image completion device based on model prior. Referring to Figure 6 , the image completion device based on model prior includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The memory 32, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the image completion method based on model prior described in any embodiment of the present application (for example, the acquisition module and the completion module in the image completion system based on model prior). The communication module 33 is configured to perform data transmission. The processor 31 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory, that is, implements the above image completion method based on model prior. The input device 34 can be configured to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the device. The output device 35 may include a display device such as a display screen. The above-provided image completion device based on model prior can be configured to execute the image completion method based on model prior provided by the above embodiment, and has corresponding functions and beneficial effects.

[0104] Based on the above embodiments, an embodiment of the present application further provides a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are configured to execute an image completion method based on model prior when executed by a computer processor. The storage medium can be any of various types of memory devices or storage devices. Of course, for the computer-readable storage medium provided by an embodiment of the present application, the computer-executable instructions are not limited to the image completion method based on model prior as described above, and can also execute related operations in the image completion method based on model prior provided by any embodiment of the present application.

[0105] Based on the above embodiments, an embodiment of the present application further provides a computer program product. Essentially, or the part that contributes to the prior art, or all or part of the technical solution of the present application can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes several instructions for causing a computer device, a mobile terminal, or a processor therein to execute all or part of the steps of the image completion method based on model prior described in various embodiments of the present application.

Claims

1. An image completion method based on model prior, characterized in that Including: Obtaining a missing image and specified completion region information of the missing image; Inputting the missing image and the specified completion region information into a pre-constructed image completion model to output a completed image of the missing image. The image completion model uses the model parameters when the pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges; The pre-trained model is pre-trained for image completion based on the training samples. After the pre-trained model converges, prior samples are generated based on the pre-trained model. The prior samples include noisy images and multiple denoised images obtained by performing image completion on the corresponding noisy images.

2. The method for image completion based on model prior according to claim 1, wherein The training process of the pre-trained model includes: Performing noise addition processing on the training samples based on the pre-trained model to obtain noisy samples; Performing image completion on the noisy samples based on the pre-trained model to obtain denoised samples, calculating a first model loss function based on the denoised samples, and iteratively adjusting the model parameters of the pre-trained model based on the first model loss function until the pre-trained model converges.

3. The method for image completion based on model prior according to claim 2, wherein The performing noise addition processing on the training samples based on the pre-trained model to obtain noisy samples includes: At each specified moment within a set time period, adding Gaussian distribution noise to the training images of the training samples based on the pre-trained model to obtain the Gaussian noise distribution probabilities of each training image, and using the Gaussian noise distribution probabilities of each training image as the noisy samples; The performing image completion on the noisy samples based on the pre-trained model to obtain denoised samples includes: Predicting the real-time noise distribution probabilities of each specified moment within the set time period based on the Gaussian noise distribution probabilities of each training image and set estimation parameters, and using the prediction result at the first specified moment within the set time period as the denoised sample.

4. The method for image completion based on model prior according to claim 1, wherein The generating the prior samples based on the pre-trained model includes: Inputting the noisy images into the pre-trained model, performing image completion on the noisy images a set number of times based on the pre-trained model to obtain a corresponding number of the denoised images, and using the set number of the noisy images and the corresponding denoised images as the prior samples.

5. The method for image completion based on model prior according to claim 4, wherein After generating the prior samples based on the pre-trained model, it further includes: Calculating the completion processing scores of each of the denoised images based on the noisy images and the corresponding denoised images, and performing data screening on the prior samples based on the completion processing scores.

6. The method for image completion based on model prior according to claim 1, wherein The training process of the image completion model includes: Using the model parameters when the pre-trained model converges as the initial model parameters, and inputting the training samples and the prior samples into the image completion model; Performing image completion training on the image completion model using the training samples and the prior samples, calculating a second model loss function based on the training results, and iteratively adjusting the initial model parameters based on the second model loss function until the image completion model converges.

7. An image completion system based on model prior, characterized in that, Including: An acquisition module configured to acquire a missing image and specified completion region information of the missing image; A completion module configured to input the missing image and the specified completion region information into a pre-constructed image completion model, and output a completed image of the missing image. The image completion model uses the model parameters when a pre-trained model converges as the initial model parameters, and performs image completion training based on the training samples of the pre-trained model and the prior samples generated by the pre-trained model until the image completion model converges. The pre-trained model is pre-trained for image completion based on the training samples, and after the pre-trained model converges, the prior samples are generated based on the pre-trained model. The prior samples include a noise-added image and a plurality of denoised images obtained by performing image completion on the noise-added image.

8. An image completion device based on model prior, characterized in that, Comprising: A memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the model prior-based image completion method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are configured to execute the model prior-based image completion method according to any one of claims 1-6 when executed by a computer processor.

10. A computer program product, characterized in that, The computer program product contains instructions that, when the instructions run on a computer or a processor, cause the computer or the processor to execute the model prior-based image completion method according to any one of claims 1-6.

Images