An accelerated image inpainting method based on diffusion model

Abstract

The application discloses an accelerated image inpainting method based on a diffusion model, which realizes the result of final image inpainting by sampling from random Gaussian noise through a two-stage reverse diffusion process, and the two stages are divided into a coarse stage and a fine stage; firstly, a coarse result is sampled from random Gaussian noise in the coarse stage; then, the coarse result is up-sampled to obtain an up-sampled coarse result as an image prior of the fine stage, noise is added to the up-sampled coarse result as an input of the fine stage, a refined result is generated by sampling through the fine stage, and the final image is repaired; in the two stages, the conditional denoising module CDM is used for denoising, and the conditional resampling module CRM is used for making the repair effect more harmonious; experimental results show that the application keeps the repair precision of the original method based on the diffusion model, and realizes about 60 times of acceleration.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, to the field of image inpainting, and particularly to an accelerated image inpainting method based on a diffusion model. Background Technology

[0002] Image inpainting is a low-level research area in computer vision. This task aims to fill in missing or damaged areas of an image with plausible visual content. Inpainting techniques have many applications, such as restoring damaged photographs, editing images, and removing unwanted objects from images.

[0003] Currently, the publicly available technical solutions for this task mainly fall into two categories. One is traditional algorithms that do not involve deep learning; the other is deep learning methods that have been widely used recently and have also shown impressive performance and results in image inpainting tasks.

[0004] Traditional image inpainting methods can be mainly divided into two categories: diffusion-based and patch-based methods. Diffusion-based methods [3,4] render the occluded region based on the appearance information of the surrounding undamaged regions. Patch-based methods [5,6] reconstruct the occluded region by searching for and pasting the most similar patch in the undamaged regions of the image. Although these methods have achieved good performance, they have high computational costs in calculating patch similarity and are difficult to reconstruct patches with rich semantics.

[0005] Deep generation methods have achieved great success in image inpainting tasks. To produce clearer results, Pathak et al. [7] introduced adversarial loss in image inpainting and used context encoders to learn the semantics of visual structures. Nazeri et al. proposed EdgeConnect [8], which uses phantom edges to fill missing regions as a prior. To address long-distance interaction modeling in image inpainting, Li et al. proposed MAT [9] for large hole filling, which unifies the advantages of transformers and convolutions. Most existing methods are trained for specific mask distributions. Lugmayr et al. proposed a novel conditional method called RePaint [1], which conforms to the assumptions of the denoising diffusion probability model DDPM and increases the degrees of freedom of the mask for free-form inpainting.

[0006] Traditional image inpainting methods struggle to capture high-level semantic information within images, and the correlation between the pixels in the filled region and known pixels in the image is weak, making it difficult to reconstruct high-quality inpainted images. Deep learning-based methods utilize common network architectures such as Transformer, GAN, and CNN as backbone networks, but these methods lack generative diversity and are often only suitable for inpainting tasks with a single mask type, failing to address any form of image information loss in real-world scenarios. The diffusion-based image inpainting technique proposed by Lugmayr et al. effectively solves this problem, applicable to image inpainting with any mask type. However, its diffusion model's image sampling speed is too slow; inpainting a 256×256 resolution image takes over a minute, which is unacceptable for users in many application scenarios. Summary of the Invention

[0007] This invention aims to accelerate diffusion model-based image inpainting techniques, transitioning from minute-level to second-level inpainting while maintaining almost the same inpainting effect. Compared to existing methods, this invention demonstrates balanced competitiveness across multiple mask types. Furthermore, for large-size masking, the diffusion model-based image inpainting technique exhibits extremely strong capabilities in generating realistic images and improving visualization, thus solving the problem of the very time-consuming inference process in existing diffusion model-based methods.

[0008] The technical solution of the present invention is described in detail below.

[0009] This invention provides an accelerated image inpainting method based on a diffusion model. This method uses a pre-trained diffusion model to introduce an input image of the partially obscured region into a back-diffusion process, from which the inpainted image is sampled from random Gaussian noise. The back-diffusion process consists of two stages: a coarse stage and a fine stage. The specific steps are as follows:

[0010] First, a coarse result is sampled from random Gaussian noise in the coarse stage. Then, the coarse result is upsampled to obtain an upsampled coarse result, which serves as the image prior for the fine stage. Finally, noise is added to the upsampled coarse result as the input for the fine stage. The fine stage sampling generates a refined result, thus achieving the final image restoration.

[0011] In the two stages of the back diffusion process, denoising is achieved through a Conditional Denoising Module (CDM) and a Conditional Resampling Module (CRM), respectively, to make the restoration effect more harmonious. Specifically, the denoising time step T in the coarse stage... c The denoising time step T is greater than that of the fine-stage stage. f The number of iterations of the Conditional Resampling Module (CRM) in the coarse stage is less than the number of iterations of the Conditional Resampling Module (CRM) in the fine stage.

[0012] In this invention, the diffusion model is a lightweight diffusion model that reduces the number of channels and attention heads in the UNet network. During the inference phase, each denoising step utilizes a pre-trained diffusion model to accurately predict noise, gradually removing noise from random Gaussian noise to extract a clear image. Specifically, this invention eliminates the multi-head attention mechanism, replacing the four attention heads in each attention layer with a single one; increases the downsampling factor, reducing the number of parameters in subsequent layers; and reduces the number of channels in the intermediate layer feature vectors from 256 to 128. This lightweight diffusion model accelerates image inpainting.

[0013] In this invention, the loss function for training the lightweight diffusion model incorporates a perception-priority weighting, emphasizing training during the content phase to encourage the model to learn to perceive rich context. Specifically:

[0014]

[0015] in, It is the denoising score matching loss. SNR(t) is the noise image x t signal-to-noise ratio, λ t For the weighting scheme, λ t =(1-β) t )(1-α t ) / β t , where β t This refers to the predefined noise scale in the diffusion model, with values ​​ranging from (0,1).

[0016] In this invention, upsampling is performed using a bilinear interpolation method.

[0017] In this invention, denoising is performed in two stages of the back diffusion process based on denoising blocks with identical structures. In each denoising block, the input image to be repaired is used as a condition, and conditional information is added. First, m conditional denoising modules (CDM) are applied for denoising, and then n conditional resampling modules (CRM) are used to enhance the fusion of conditional information and generated content.

[0018] In this invention, the Conditional Denoising Module (CDM) incorporates denoising that supports DDIM sampling; for the input image x that has undergone the Conditional Denoising Module (CDM)... t The non-Markov inference process of DDIM is used to obtain the generated information, i.e., the denoising result. s represents the step size of DDIM sampling. Simultaneously, noise is added to the input image to be repaired, and conditional information, i.e., known information, is sampled from the occluded image to be repaired. Then the condition information With generated information Align and concatenate to generate output xt-s .

[0019] In this invention, the Conditional Resampling (CRM) module re-noises the result, and then the Conditional Denoising (CDM) module denoises it while repeatedly introducing pixel information from the known image. This fully integrates the semantic information of the given information and the generated information, making the repaired image more harmonious.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] This invention proposes an efficient image inpainting technique based on a diffusion model, which includes three acceleration strategies. Specifically, we replace the large-parameter diffusion model with a lightweight diffusion model (LWDM) adjusted in conjunction with the training objective. We accelerate the denoising process by replacing the denoising diffusion probability model DDPM sampling with skip-step DDIM sampling. Furthermore, we propose a coarse-to-fine sampling (CFS) strategy to further accelerate and improve performance. Experimental results on facial and general image inpainting tasks show that our method achieves balanced and competitive results across different mask types compared to the base model RePaint, while achieving an acceleration of approximately 60x. Attached Figure Description

[0022] Figure 1 The noise addition and denoising process of DDPMs.

[0023] Figure 2 : Sampling process of diffusion model.

[0024] Figure 3 The diffusion model sampling method in image inpainting techniques.

[0025] Figure 4 : Architecture diagram of the technical solution of this invention.

[0026] Figure 5 The above is a comparison of the existing publicly disclosed technical methods of this invention on the CelebA-HQ

[11] face dataset.

[0027] Figure 6 : Comparison of the present invention and existing publicly available technical methods on the ImageNet

[12] general dataset. Detailed Implementation

[0028] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] Image inpainting tasks are mainly based on diffusion model technology. Denoising Diffusion Probabilistic Models (DDPMs) [2] are generative models that use Markov chains to transform latent variables of simple distributions into complex data distributions. DDPMs aim to reverse the parameterized Markov image noise process. They start with Gaussian noise samples and gradually transform them into samples of the training distribution by progressively eliminating noise. The diffusion model consists of two processes: the diffusion process and the reverse process (denoising process); the noise addition and denoising processes of DDPMs are as follows: Figure 1 As shown, this figure is from reference [2].

[0030] 1. Diffusion process

[0031] The diffusion process is a Markov chain that adds Gaussian noise to the data.

[0032]

[0033] Where β t ∈(0,1),t=1,...,T.

[0034] Using reparameterization techniques, the diffusion process can be expressed in closed form for any given time step t with respect to x. t Sampling: A key characteristic of the forward noise addition process is that the image x at any number of noise addition steps can be directly sampled from x0. t :

[0035]

[0036] 2. Noise Reduction Process

[0037] The opposite denoising process is a Markov chain that transforms noise back into the data distribution:

[0038]

[0039] Where θ is the model parameter, and the mean or variance is parameterized by the model.

[0040] In inference, data can be generated directly from Gaussian noise. Sampling one... Then, according to formula (3), x t-1 ~p θ (x t-1 |x t Sampling is performed. Finally, the predicted original sharp image x0 is obtained through continuous sampling. The specific sampling algorithm is as follows: Figure 2 The figure is from reference [2].

[0041] 3. Sampling methods in image inpainting tasks

[0042] Lugmayr et al. [1] first introduced an image inpainting method based on a diffusion model, which uses a pre-trained unconditional diffusion model and introduces the input image of the occluded part into the reverse diffusion process (denoising process) to sample the inpainted image.

[0043] Specifically, in each denoising step, the denoising result can be obtained using formula (3). This contains the generated information Simultaneously, known alignment information is sampled from the masked input image using formula (2). Finally, use the join result x t-1 As input for the next noise reduction step:

[0044]

[0045] Specific sampling methods are as follows: Figure 3 The figure is from reference [1].

[0046] This invention aims to accelerate image inpainting technology based on diffusion models, transitioning from minute-level to second-level inpainting. First, it outlines the technical method proposed in this invention, such as... Figure 4 As shown. The following will introduce the three modules used in this invention: Lightweight Diffusion Model (LWDM), Conditional Denoising Module (CDM), and Conditional Resampling Module (CRM). These three modules incorporate our proposed three acceleration strategies: Lightweight Diffusion Model (LWDM), Skip Sampling Injection (DDIM), and Coarse-to-Fine Sampling Method (CFS).

[0047] I. Overview

[0048] Given the input image to be repaired as a condition, we sample the final repaired result from random Gaussian noise through a two-stage backdiffusion process. The framework is as follows: Figure 4 As shown. In the rough stage, we need T c Each denoising time step is used to extract random Gaussian noise from the Conditional Denoising Module (CDM) through a denoising module. A rough result was obtained from the sampling. The coarse upsampling result serves as the image prior for the fine-stage process, where upsampling employs a simple bilinear interpolation method. In the fine-stage process, the input... Formula (2) is used to obtain the upsampling results at a specified time step T. f The sampled data was then used to extract the final, improved result.

[0049] For both stages, we use denoising blocks with consistent structure but different parameter settings. Within the denoising block, m conditional denoising modules (CDMs) are first applied to x...t Denoising to x t-ms At the same time, conditional information is added. Then, n conditional resampling modules (CRM) are used to enhance the integration of conditional information with the generated content.

[0050] This invention is more efficient than RePaint[1], thanks to three acceleration strategies: Lightweight Diffusion Model (LWDM), Skip Sampling Model (DDIM), and Coarse-to-Fine Sampling Method (CFS). First, we replace the pre-trained diffusion model with a lightweight model to complete the noise prediction in each denoising step. Second, in the Conditional Denoising Module (CDM), denoising that supports skip sampling model (DDIM) is introduced. Finally, the specially designed coarse-to-fine sampling method (CFS) shows higher efficiency and better results. In the coarse stage, due to the large denoising time step T c We opted for a lower resolution and fewer CRM iterations. In the fine-tuning phase, due to the image prior T... f The smaller size allows us to apply more Conditional Resampling (CRM) modules to achieve more harmonious and detailed repair results. Our method significantly improves speed while maintaining competitive repair performance.

[0051] II. Lightweight Diffusion Model

[0052] The diffusion model primarily utilizes the UNet structure to predict noise for denoising. Lightweighting refers to reducing the number of parameters in the UNet, modifying the training process of the pre-trained model. Reducing the number of parameters in the pre-trained model means accelerating the denoising process at each step, since each denoising step in the inference stage requires passing through the UNet network to predict noise. In other words, reducing the number of parameters in the pre-trained diffusion model can speed up the inference process.

[0053] Specifically, the number of parameters in the UNet network in the classic diffusion model is reduced in the following ways: (1) the multi-head attention mechanism is canceled and the original four attention heads in each attention layer are replaced with a single one; (2) the downsampling factor is increased and the number of parameters in subsequent layers is reduced; (3) the number of channels in the feature vector of the intermediate layer is reduced from 256 to 128.

[0054] However, directly reducing the number of parameters may affect the performance of the model. Inspired by

[10] , we modified the loss function in the training process of the original diffusion model to compensate for the performance reduction caused by the reduction in the number of parameters. The original loss function of the diffusion model can be described in the following form:

[0055]

[0056] Wherein, the weight scheme λ t =(1-β) t )(1-αt ) / β t ,and It is the denoising score matching loss.

[0057] The modified loss function introduces a perceptual priority weighting, emphasizing training during the content phase to encourage the model to learn to perceive rich context. λ t It can be replaced with λ′ t :

[0058]

[0059] Where SNR represents the signal-to-noise ratio, and the noisy image x t The signal-to-noise ratio is The modified loss function can be defined as follows: III. Conditional Noise Reduction Module

[0060] We introduced conditional information in each denoising step. The Conditional Denoising Module (CDM) is as follows: Figure 4 As shown in (c). For the input image x that has undergone the Conditional Denoising Module (CDM). t The generated output x t-s It is a representation after denoising and fusion processing, where s represents the step size of the skip sampling DDIM.

[0061] We employ a pre-trained lightweight diffusion model, LWDM, and utilize the non-Markov inference process of skip sampling DDIM to obtain the denoising results. Its description is as follows:

[0062]

[0063] in, Is with x t Independent standard Gaussian noise. DDIM sampling is a σ t The special case of =0. Then, by adding noise to the input image according to formula (2), the known information of the input is combined with... Align. Finally, align the generated information according to formula (4). With conditional information Then, the parts are assembled.

[0064] IV. Conditional Resampling Module

[0065] The Conditional Denoising (CDM) module aims to incorporate conditional information from the input image. Additionally, a Conditional Resampling (CRM) module is needed to render the given information... With the generated information To coordinate, by adding noise to x t+ks And return to x through noise reduction t Conditional noise reduction module (CRM) such as Figure 4As shown in (d). Specifically, the conditional denoising module CRM will x t As input and to generate resampled x t As output. In each iteration, first for x t Add noise (ks step) to obtain Then, it is denoised back to x using k conditional denoising modules (CDM). t Then replace the original x with this denoised version. t .

[0066] In summary, the Conditional Resampling (CRM) module re-noises the results and then repeatedly introduces pixel information from the known image through the Conditional Denoising (CDM) module of the diffusion model, thereby fully integrating the semantic information of the given and generated information to make the repaired image more harmonious.

[0067] In summary, this invention first generates a coarse result from random noise through a conditional coarse stage. Then, the result is upsampled and noise is added as input to the fine stage to generate a refined result. In each stage, it uses a conditional denoising module (CDM) for denoising and a conditional resampling module (CRM) to make the restoration effect more harmonious. Based on this framework, this invention includes three acceleration strategies to ensure the efficiency of the inverse diffusion process. First, because the original pre-trained diffusion model has too many parameters, it is replaced with a simplified model to implement a lightweight diffusion model (LWDM). Simultaneously, to ensure the image restoration effect, the weighting scheme of the training loss is adjusted, prioritizing learning from more important noise levels and compensating for the reduction in parameters. Second, we introduce a skip sampling scheme for the denoising process, adopting the sampling method of the Denoising Diffusion Implicit Model (DDIM). Finally, we propose coarse-to-fine sampling (CFS), dividing the long denoising process into two shorter stages. In the coarse stage, due to the long denoising time step and the lengthy inverse diffusion fusion process, we reduce the image resolution and the number of CRMs. In the fine-tuning stage, with coarse prior image information already available, more CRM can be applied to enhance image details. This specially designed coarse-to-fine sampling process accelerates the denoising process while ensuring high-quality image inpainting results.

[0068] The following describes specific examples.

[0069] The comparative experiment between this invention and existing technical solutions used two datasets and one mask dataset. The CelebA-HQ

[11] dataset contains 30,000 face images with a pixel resolution of 256×256. We selected the first 27,000 images as the training set and the remaining 3,000 images for testing. To demonstrate the generalizability of our method on general datasets, we also conducted experiments on the ImageNet

[12] dataset. We used the mask test set of RePaint[1] to verify the performance of various distributed masks. The mask types are divided into 6 types: Wide, Narrow, Half, Expand, AlternatingLines and Super-Resolve 2×.

[0070] For quantitative evaluation, we used structural similarity (SSIM), relative L1, and the perceptual metric LPIPS to evaluate the performance of our method. The final time expenditure was recorded based on the inference time required for one image on a single RTX 2070 GPU.

[0071] Table 1: Comparison of the present invention and existing publicly available methods on the CelebA-HQ dataset, with SSIM as the metric.

[0072]

[0073]

[0074] Table 2: Comparison of the present invention and existing publicly available methods on the ImageNet dataset, with SSIM as the benchmark.

[0075]

[0076] Table 3: Comparison of the effects and inference time of different acceleration strategies employed in this invention, with SSIM as the metric. CFS strategy refers to the coarse-to-fine sampling method; LWDM strategy refers to the lightweight diffusion model; DDIM refers to the skip sampling method.

[0077] Strategy Reasoning time acceleration ratio Wide Narrow - 938.2s - 0.8771 0.9210 CFS 395.7s 2.4× 0.8763 0.9196 LWDM 189.4s 4.9× 0.8796 0.9186 DDIM 186.6s 5.0× 0.8778 0.9191 LWDM+CFS 76.3s 12.3× 0.8820 0.9166 DDIM+CFS 75.5s 12.4× 0.8811 0.9161 LWDM+DDIM 37.9s 24.8× 0.8814 0.9182 LWDM+DDIM+CFS 15.4s 60.9× 0.8834 0.9174

[0078] According to Table 3, the diffusion model-based inpainting technique[1] required an average of 938.2 seconds to repair an image, while this invention accelerated it to 15.4 seconds, improving the repair time from minutes to seconds. Each strategy has a certain acceleration capability, while ensuring high quality and high fidelity of image inpainting. According to the quantitative evaluation results in Tables 1 and 2, compared with existing publicly available techniques, this invention shows balanced competitiveness across multiple mask types. In addition, for large-size masking, the diffusion model-based image inpainting technique demonstrates a strong ability to generate realistic images, and the visualization effect can be seen in Table 3. Figure 5 , Figure 6 .

[0079] References

[0080] [1] A.Lugmayr, M.Danelljan, A.Romero, F.Yu, R.Timofte, and L.Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in CVPR, 2022, pp.11461–11471.

[0081] [2] J.Ho, A.Jain, and P.Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, vol.33, pp.6840–6851, 2020.

[0082] [3] C.Ballester, M.Bertalmio, V.Caselles, G.Sapiro, and J.Verdera, “Filling-in by joint interpolation of vector fields and gray levels,” IEEE Transactions on Image Processing, vol.10, no.8, pp.1200–1211, 2001.

[0083] [4] M.Bertalmio, G.Sapiro, V.Caselles, and C.Ballester, “Image inpainting,” in SIGGRAPH, 2000, pp.417–424

[0084] [5] C.Barnes, E.Shechtman, A.Finkelstein, and D.B.Goldman, “Patchmatch: A randomized correspondence algorithm for structural image editing,” ACM Trans.Graph., vol.28, no.3, pp.24, 2009.

[0085] [6]Z.Xu and J.Sun,“Image inpainting by patch propagationusing patchsparsity,”IEEETransactions on Image Processing,vol.19,no.5,pp.1153–1165,2010.

[0086] [7]D.Pathak,P.Krahenbuhl,J.Donahue,T.Darrell,and A.A.Efros,“Contextencoders:Featurelearning by inpainting,”inCVPR,2016,pp.2536–2544.

[0087] [8]K.Nazeri,E.Ng,T.Joseph,F.Z.Qureshi,and M.Ebrahimi,“Edgeconnect:Generative imageinpainting with adversarialedge learning,”arXiv:1901.00212,2019.

[0088] [9]W.Li,Z.Lin,K.Zhou,L.Qi,Y.Wang,and J.Jia,“Mat:Maskawaretransformerfor largehole image inpainting,”in CVPR,2022,pp.10758–10768.

[0089]

[10] J.Choi,J.Lee,C.Shin,S.Kim,H.Kim,and S.Yoon,“Perceptionprioritizedtraining ofdiffusion models,”in CVPR,2022,pp.11472–11481.

[0090]

[11] T.Karras,T.Aila,S.Laine,and J.Lehtinen,“Progressivegrowing ofgans for improvedquality,stability,and variation,”arXiv:1710.10196,2017.

[0091]

[12] A.Krizhevsky,I.Sutskever,and G.E.Hinton,“Imagenet classificationwith deepconvolutional neural networks,”NeurIPS,vol.25,2012.

[0092]

[13] J.Peng,D.Liu,S.Xu,and H.Li,“Generating diverse structurefor imageinpainting withhierarchical vq-vae,”in CVPR,2021,pp.10775–10784.

[0093]

[14] Z.Wan,J.Zhang,D.Chen,and J.Liao,“High-fidelity pluralisticimagecompletion withtransformers,”in ICCV,2021.

[0094]

[15] M.Zhu,D.He,X.Li,C.Li,F.Li,X.Liu,E.Ding,andZ.Zhang,“Imageinpainting byend-to-end cascaded refinement with mask awareness,”IEEETransactions onImage Processing,vol.30,pp.4855–4866,2021。

Claims

1. An accelerated image inpainting method based on a diffusion model, characterized in that, This method introduces the input image with the obscured region into the backdiffusion process to sample the repaired image from random Gaussian noise. The backdiffusion process consists of two stages: a coarse stage and a fine stage; the specific steps are as follows: First, a coarse result is sampled from random Gaussian noise in the coarse stage. Then, the coarse result is upsampled to obtain an upsampled coarse result, which serves as the image prior for the fine stage. Finally, noise is added to the upsampled coarse result as the input for the fine stage. The fine stage sampling generates a refined result, thus achieving the final image restoration. In the two stages of the back diffusion process, denoising is achieved through a Conditional Denoising Module (CDM) and a Conditional Resampling Module (CRM), respectively, to make the restoration effect more harmonious. Specifically, the denoising time step in the coarse stage... Denoising time steps greater than the fine-stage stage The number of iterations of the conditional resampling module (CRM) in the coarse stage is less than the number of iterations of the conditional resampling module (CRM) in the fine stage; where: In the two stages of the backdiffusion process, denoising is performed based on denoising blocks with identical structures. Within each denoising block, conditional information is added using the input image to be repaired as a condition. First, [the process is applied]. The conditional noise reduction module (CDM) performs noise reduction, and then uses... A conditional resampling module (CRM) is used to enhance the integration of conditional information with generated content; The Conditional Denoising Module (CDM) introduces denoising that supports DDIM sampling; for an input image sampled at any given time step t after passing through the CDM module... The non-Markov inference process of DDIM is used to obtain the generated information, i.e., the denoising result. s represents the step size of DDIM sampling. Simultaneously, noise is added to the input image to be repaired, and conditional information, i.e., known information, is sampled from the occluded image to be repaired. Then the condition information With generated information Align and splice to generate output .

2. The accelerated image inpainting method based on a diffusion model according to claim 1, characterized in that, The diffusion model is a lightweight diffusion model that reduces the number of channels and attention heads in the UNet network.

3. The accelerated image inpainting method based on a diffusion model according to claim 2, characterized in that, The loss function for training the lightweight diffusion model incorporates a perceptual priority weighting, emphasizing training during the content phase to encourage the model to learn to perceive rich contexts. Specifically: ； in, It is the denoising score matching loss. , For noisy images The signal-to-noise ratio, ; For the weighting scheme, ,in This refers to the predefined noise scale in the diffusion model, with values ​​ranging from (0, 1). .

4. The accelerated image inpainting method based on a diffusion model according to claim 1, characterized in that, Upsampling is performed using bilinear interpolation.

5. The accelerated image inpainting method based on a diffusion model according to claim 1, characterized in that, The Conditional Resampling (CRM) module adds noise to the result and then denoises it using the Conditional Denoising (CDM) module, while repeatedly introducing pixel information from the known image. This fully integrates the semantic information of the given and generated information, making the repaired image more harmonious.

Images