A method and system for image completion by modifying diffusion model trajectory

Abstract

The present invention proposes a method and system for image completion by modifying the diffusion model trajectory, which belongs to the field of image completion. The method comprises: mapping a partially masked sample image to the original image space through a trainable neural network; sampling the original image to obtain a sampled image; calculating the loss of the image completion network based on the sampled image; updating the image completion network parameters and the trainable neural network parameters based on the loss of the image completion network; and using the trained neural network and image completion network to reversely generate the completion result of the masked image. The present invention adopts a more flexible and controllable diffusion trajectory for modeling. Compared with the fixed diffusion trajectory used in the original diffusion model, it greatly improves the capacity of the diffusion model and achieves better results in image completion.

Description

Technical Field

[0001] The present invention relates to the field of image completion, and in particular to a method and system for performing image completion by modifying diffusion model trajectories. Background Art

[0002] Image completion is an important research area in computer vision. Given an image with partially obscured areas (a masked image), image completion requires predicting and filling the obscured areas based on information from the unmasked areas of the image. This ensures that the restored image appears natural, without obvious splicing or artifacts. Image completion methods fall into two main categories: texture synthesis methods, which focus on sampling similar pixel blocks from the unmasked areas of the image to fill the area to be completed; and generative models based on neural networks, which generate semantically consistent completed images based on the features of the unmasked areas of the image.

[0003] In recent years, the diffusion generative model has been proposed and has garnered significant attention in both academia and industry. It was inspired by nonequilibrium thermodynamics, also known as the thermodynamics of irreversible processes. Its primary research focuses on heat conduction and diffusion of substances. One of its key issues is evolutionary dynamics, which addresses the contradiction between microscopic reversibility and macroscopic irreversibility. Applied to generative modeling, data can be considered to be a state of equilibrium reached by various variables under the influence of external forces. When these forces disappear, the variables diffuse freely, much like the Brownian motion of particles. By simulating this diffusion process and using an iterative approach, the original data distribution is systematically and slowly destroyed, resulting in a more natural, simple, and easily computable distribution. In theory, any complex original distribution can be converted to this simple distribution through this process. For example, by continuously applying diffusion operations to the values ​​of each pixel, the image eventually becomes a Gaussian noise image. Microscopically, as long as the diffusion process is small enough and the perturbation is small enough at each step, it is reversible. The task is to quantitatively estimate this perturbation process, which is much simpler than directly and explicitly modeling the original data distribution. As long as the model learns the inverse operation of diffusion, any sample sampled from the simple distribution can be reversed back to the original distribution, generating a sample that obeys the original distribution. This process is called model generation.

[0004] Existing diffusion generative models predefine a fixed forward process and then train a neural network to fit the inverse of the forward process, resulting in a generative model. Existing conditional diffusion generative models introduce conditional modeling during neural network training, where the fitting target is still the inverse of the predefined fixed forward process. For image completion, this involves inputting a mask image as a condition into the inverse neural network. However, this approach significantly limits the model's training efficiency and performance for image completion tasks. Summary of the Invention

[0005] To overcome the above problems, the present invention proposes a method and system for image completion by modifying the diffusion model trajectory. When the mask image is introduced as a conditional model into the forward process, the fixed forward process predefined by the original diffusion model is no longer used. Instead, the diffusion trajectory of the forward process is offset, thereby improving training efficiency and performance.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] The first purpose of the present invention is to provide a method for image completion by modifying the diffusion model trajectory, comprising the following steps:

[0008] Step 1: Map the partially masked sample image to the original image space through a trainable neural network;

[0009] Step 2: Sample the original image to obtain the sampled image x t , the sampling formula is:

[0010]

[0011] Among them, t~Uniform({1,…,T}), T is the preset total number of diffusion steps, x t represents the sampled image at step t, c represents the partially masked sample image, x0 represents the original image, k t represents the coefficient that controls the diffusion trajectory offset mode, f(·) represents a trainable neural network, ∈ represents the noise that conforms to the standard Gaussian distribution added during sampling, represents the noise accumulation of the previous t steps;

[0012] Step 3: Calculate the loss of the image completion network based on the sampled image. The loss function is:

[0013]

[0014] Among them, g θ (·) represents the image completion network, whose input is x t and t;

[0015] Step 4: Update the image completion network parameters and the trainable neural network parameters in step 1 based on the loss of the image completion network.

[0016] Step 5: Use the trained neural network and image completion network to reversely generate the completion result of the masked image.

[0017] Furthermore, in step 2, the noise accumulation of the first t steps is The calculation formula is:

[0018]

[0019] α t =1-β t

[0020] Among them, β t is the preset diffusion parameter for the tth step, α t is the noise parameter at step t.

[0021] Furthermore, the diffusion trajectory offset mode includes:

[0022] Data normalization mode: An offset mode that resets the starting point of the forward process for data with different conditions, while leaving the end point unchanged;

[0023] A priori offset mode: Indicates that the end point of the forward process of data with different conditions is set to x T =f(c), while the starting point remains unchanged.

[0024] Furthermore, the image completion network adopts the UNet architecture.

[0025] Furthermore, in step 5, the formula for inversely generating the completion result of the masked image is:

[0026]

[0027] s t =k t ·f(c)

[0028] Among them, β t is the preset diffusion parameter for the tth step, α t is the noise parameter at step t, g θ (·) is the image completion network, s t is the mean shift at step t, f(·) is a trainable neural network, z represents the noise that conforms to the standard Gaussian distribution, represents Gaussian distribution, and I is the identity matrix.

[0029] Furthermore, when initializing t=T, where s T is the mean shift at step T, x T is the completed image at step T.

[0030] The second purpose of the present invention is to provide a system for image completion by modifying diffusion model trajectories, comprising:

[0031] A spatial mapping module, which is used to map the partially masked sample image to the original image space through a trainable neural network;

[0032] A pattern sampling module, which is used to sample the original image;

[0033] A loss calculation module, which is used to calculate the loss of the image completion network based on the sampled image;

[0034] A training module, which is used to update the trainable neural network parameters and the image completion network parameters according to the loss of the image completion network;

[0035] The image completion module is used to use the trained neural network and image completion network to reversely generate the completion result of the masked image.

[0036] Furthermore, the sampling formula of the pattern sampling module is:

[0037]

[0038] Among them, t~Uniform({1,…,T}), T is the preset total number of diffusion steps, x t represents the sampled image at step t, c represents the partially masked sample image, x0 represents the original image, k t represents the coefficient that controls the diffusion trajectory offset mode, f(·) represents a trainable neural network, ∈ represents the noise that conforms to the standard Gaussian distribution added during sampling, represents the noise accumulation of the previous t steps.

[0039] Furthermore, the loss function in the loss calculation module is:

[0040]

[0041] Among them, g θ (·) represents the image completion network, whose input is x t and t.

[0042] Furthermore, the formula for reversely generating the completion result of the masked image in the image completion module is:

[0043]

[0044] s t =k t ·f(c)

[0045] Among them, β t is the preset diffusion parameter for the tth step, α t is the noise parameter at step t, g θ (·) is the image completion network, s t is the mean shift at step t, f(·) is a trainable neural network, z represents the noise that conforms to the standard Gaussian distribution, represents Gaussian distribution, and I is the identity matrix.

[0046] Compared with the existing technology, the beneficial effect of the present invention is: because the present invention adopts a more flexible and controllable diffusion trajectory for modeling, compared with the fixed diffusion trajectory used in the original diffusion model, the capacity of the diffusion model is greatly improved, and better results are achieved in image completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a method for image completion by modifying diffusion model trajectories according to an embodiment of the present invention.

[0048] Figure 2 4 is a diagram showing the structure of a system for performing image completion by modifying diffusion model trajectories according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0050] The diffusion trajectory distribution of the original diffusion model is:

[0051]

[0052] Among them, q(·) represents the forward process probability distribution, t is the number of diffusion steps, and its value range is 1, 2, ..., T, T is the total number of diffusion steps preset by the diffusion model, x0 represents the original image, x t represents the noise image of x0 diffused to t steps, represents a Gaussian distribution, The diffusion parameter β of the t-th step preset by the diffusion model t Calculated, the calculation process is α t =1-β t , I is the identity matrix, α t is the noise parameter of step t, α i is the noise parameter of step i.

[0053] On this basis, in order to control the diffusion trajectory according to the mask image as a condition, the diffusion trajectory distribution of the diffusion model is modified as follows:

[0054]

[0055] Among them, c represents the mask image, that is, the image to be completed, k t represents the coefficient that controls the diffusion trajectory deviation pattern, and f(c) is the function that maps c to the x0 space.

[0056] Comparing the diffusion process to a water pipe, k t f(c) is used to change the direction of the water pipe, denoted by s t =k t f(c),s t represents the mean shift at step t, then the diffusion trajectory distribution of the modified diffusion model can be simplified as:

[0057]

[0058] Since the diffusion process is a Markov chain, according to the abbreviated diffusion trajectory distribution, the corresponding diffusion kernel can be deduced as:

[0059]

[0060] Then, according to the Bayesian formula, the posterior distribution of the diffusion kernel can be derived as:

[0061]

[0062] The diffusion trajectory distribution after abbreviation It can be seen that x0 can be expressed as:

[0063]

[0064] in, Indicates that when sampling x t The noise that conforms to the standard Gaussian distribution is added to it.

[0065] Substituting the expression of x0 into the above posterior distribution formula, we can get the mean of the posterior distribution as:

[0066]

[0067] Among them, E[·] represents the mean; in the expression of the mean of the posterior distribution, only ∈ is unknown, but due to in is known, so the present invention uses the image completion network g with parameter θ θ Defitting At this point, the predicted posterior distribution is:

[0068]

[0069] Among them, p θ (·) represents the predicted backward process probability distribution.

[0070] Image completion network g θ The loss function is expressed as:

[0071]

[0072] Among them, D KL [·] represents the KL divergence, which is a way to measure the similarity between two probability distributions.

[0073] The posterior distribution of the diffusion kernel derived above q(x t-1 |x t ,x0,c) formula and the predicted posterior distribution p θ (x t-1 |x t ,c) Substituting the formula into the loss function yields:

[0074]

[0075] Where c is a constant, Represents the coefficient of the loss function at step t;

[0076] Further simplify the loss function to obtain the final loss function:

[0077]

[0078] The coefficient k for controlling the diffusion trajectory deviation mode t , different modes can be selected according to the diffusion trajectory law of the diffusion model. There are mainly two modes:

[0079] Data normalization mode: It can be seen as a shift pattern that resets the starting point of the forward process of data under different conditions, while the end point remains unchanged;

[0080] A priori offset mode: It can be seen as setting the end point of the forward process of data with different conditions to x T =f(c), while the starting point remains unchanged.

[0081] Compared with the reverse process sampling of the original diffusion model, the method of the present invention needs to make corrections to the trajectory at each sampling step, which is reflected in the mean of the posterior distribution mentioned above. The corrected value is To ensure that the sampling is performed along the exclusive track of the mask image c. From this perspective, the difference between the two different offset modes mentioned above can be better explained:

[0082] Data normalization mode: at this time When t>1 At this time, the correction value is only valid at t = 1, which means that this mode only normalizes the data and then uses the normalized data to train the original diffusion model;

[0083] A priori offset mode: at this time Notice This is because the reverse process from Initially, the reverse process requires a correction at each step to offset f(c).

[0084] like Figure 1 As shown, the present invention realizes the conditional diffusion generation model by modifying the diffusion trajectory and performs image completion. The specific steps are as follows:

[0085] 1) Obtain training data

[0086] For a piece of training data x0,c, where x0 is the original image and c is the image after the original image is partially covered, i.e., the image to be completed, a function f(c) is established to map c to the x0 space. The function is a trainable neural network.

[0087] 2) Sample the original image x0 to obtain the sampled image x t , the sampling formula is:

[0088]

[0089] Among them, t~Uniform({1,…,T}), T is the total number of diffusion steps preset by the diffusion model, which is set to 1000, β t is the diffusion parameter of the t-th step preset by the diffusion model, and 1000 values ​​are uniformly taken between 0.0001 and 0.02 as β t ; α t =1-β t represents the noise parameter of step t, represents the accumulation of the previous t steps; k t It is the coefficient that controls the diffusion trajectory offset mode. Here, the prior offset mode is selected. I is the identity matrix; ∈ is the noise added during sampling that conforms to the standard Gaussian distribution.

[0090] For the image completion network g θ , using gradient descent to optimize the loss function

[0091] In this embodiment, the function f(c) is the same as the image completion network g θ Train together, the image completion network g θ Using UNet architecture, image completion network g θ The input is x t ,t, the output is Approximate value of

[0092] 3) Use the trained image completion network g θ , for any image c to be completed, according to the formula:

[0093]

[0094] Perform the stepwise sampling of the reverse process, starting from x T Generate x T-1 , by x T-1 Generate x T-2 , and so on until the sample x0 is generated, that is, the completed image, where:

[0095] s t =k t ·f(c)

[0096]

[0097] When executing the last step, z=0.

[0098] In order to further demonstrate the implementation effect of the present invention, the above method is applied to the following examples to reflect the effects of the present invention, and the specific steps in the examples are not repeated here.

[0099] This embodiment was experimentally validated using the Places2 dataset and an irregular mask dataset. The Places2 dataset contains over 400 unique scene categories and over 10 million images, with each scene category having 5,000 to 30,000 training images, consistent with the frequency of scenes in the real world. The irregular mask dataset is a randomly generated mask dataset that categorizes masks into different levels based on the proportion of the masked area to the entire image. The higher the mask percentage, the larger the masked area of ​​the image, making it more difficult to complete. The above implementation steps were used to evaluate the performance of different image completion methods at different mask percentages.

[0100] Table 1 shows the test results of the present invention. This example uses the FID score for evaluation. FID generates an evaluation score by comparing the generated image with the real image, calculating a "distance value." A smaller FID value indicates a closer distance between the generated image and the real image, indirectly indicating better algorithm performance. This example compares traditional image completion methods based on the generative adversarial network (GAN), namely Contextual Attention, EdgeConnect, and StructureFlow, as well as the traditional conditional diffusion model (DDPM). Compared to these image completion methods, the present invention achieves superior performance.

[0101] Table 1 Experimental results (FID)

[0102] 0％～20％ 20％～40％ 40％～60％ Contextual Attention 4.8586 18.4190 37.9432 EdgeConnect 3.0097 7.2635 19.0030 StructureFlow 2.9420 7.0354 22.3803 DDPM 2.0665 6.6129 17.3601 The present invention 1.8314 6.2915 14.9667

[0103] This embodiment also provides a system for image completion by modifying diffusion model trajectories, which is used to implement the above-mentioned embodiments. Terms such as "module," "unit," and the like used below may refer to a combination of software and / or hardware that implements a predetermined function. While the systems described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible.

[0104] like Figure 2 As shown, the system for image completion by modifying the diffusion model trajectory includes:

[0105] A spatial mapping module, which is used to map the partially masked sample image to the original image space through a trainable neural network;

[0106] A pattern sampling module, which is used to sample the original image;

[0107] A loss calculation module, which is used to calculate the loss of the image completion network based on the sampled image;

[0108] A training module, which is used to update the trainable neural network parameters and the image completion network parameters according to the loss of the image completion network;

[0109] The image completion module is used to use the trained neural network and image completion network to reversely generate the completion result of the masked image.

[0110] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0111] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0112] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or imagined by a person skilled in the art from the disclosure of the present invention should be considered to be within the scope of protection of the present invention.

Claims

1. A method for image completion by modifying diffusion model trajectories, characterized in that: The following steps are involved: Step 1: Map the partially masked sample image to the original image space through a trainable neural network; Step 2: Sample the original image to obtain the sampled image x t , the sampling formula is: Among them, t~Uniform({1,…,T}), T is the preset total number of diffusion steps, x t represents the sampled image at step t, c represents the partially masked sample image, x0 represents the original image, k t represents the coefficient that controls the diffusion trajectory offset mode, f(·) represents a trainable neural network, ∈ represents the noise that conforms to the standard Gaussian distribution added during sampling, represents the noise accumulation of the previous t steps; Step 3: Calculate the loss of the image completion network based on the sampled image. The loss function is: Among them, g θ (·) represents the image completion network, whose input is x t and t; Step 4: Update the image completion network parameters and the trainable neural network parameters in step 1 based on the loss of the image completion network. Step 5: Use the trained neural network and image completion network to reversely generate the completion result of the masked image. The formula is: s t =k t ·f(c) Among them, β t is the preset diffusion parameter for the tth step, α t is the noise parameter at step t, g θ (·) is the image completion network, s t is the mean shift at step t, f(·) is a trainable neural network, z represents the noise that conforms to the standard Gaussian distribution, represents Gaussian distribution, and I is the identity matrix.

2. The method for image completion by modifying diffusion model trajectory according to claim 1, characterized in that: In step 2, the noise accumulation of the first t steps is The calculation formula is: α t ＝1-β t Among them, β t is the preset diffusion parameter for the tth step, α t is the noise parameter at step t.

3. The method for image completion by modifying diffusion model trajectory according to claim 1, characterized in that: The diffusion trajectory offset mode includes: Data normalization mode: An offset mode that resets the starting point of the forward process for data with different conditions, while leaving the end point unchanged; A priori offset mode: Indicates that the end point of the forward process of data with different conditions is set to x T =f(c), while the starting point remains unchanged.

4. The method for image completion by modifying diffusion model trajectory according to claim 1, characterized in that: The image completion network adopts the UNet architecture.

5. The method for image completion by modifying diffusion model trajectory according to claim 1, characterized in that: When t=T is initialized, where s T is the mean shift at step T, x T is the completed image at step T.

6. A system for image completion by modifying diffusion model trajectory, used to implement the method for image completion by modifying diffusion model trajectory according to claim 1, characterized in that the system include: A spatial mapping module, which is used to map the partially masked sample image to the original image space through a trainable neural network; A pattern sampling module, which is used to sample the original image; A loss calculation module, which is used to calculate the loss of the image completion network based on the sampled image; A training module, which is used to update the trainable neural network parameters and the image completion network parameters according to the loss of the image completion network; The image completion module is used to use the trained neural network and image completion network to reversely generate the completion result of the masked image.

7. The system for image completion by modifying diffusion model trajectory according to claim 6, characterized in that: The sampling formula of the pattern sampling module is: Among them, t~Uniform({1,…,T}), T is the preset total number of diffusion steps, x t represents the sampled image at step t, c represents the partially masked sample image, x0 represents the original image, k t represents the coefficient that controls the diffusion trajectory offset mode, f(·) represents a trainable neural network, ∈ represents the noise that conforms to the standard Gaussian distribution added during sampling, represents the noise accumulation of the previous t steps.

8. The system for image completion by modifying diffusion model trajectory according to claim 7, characterized in that: The loss function in the loss calculation module is: Among them, g θ (·) represents the image completion network, whose input is x t and t.

9. The system for image completion by modifying diffusion model trajectory according to claim 6, characterized in that: The formula for reversely generating the completion result of the masked image in the image completion module is: s t =k t ·f(c) Among them, β t is the preset diffusion parameter for the tth step, α t is the noise parameter at step t, g θ (·) is the image completion network, s t is the mean shift at step t, f(·) is a trainable neural network, z represents the noise that conforms to the standard Gaussian distribution, represents Gaussian distribution, and I is the identity matrix.

Images