Training method and device of diffusion model, equipment and storage medium

Abstract

The application provides a diffusion model training method and device, equipment and a storage medium. A time step i is randomly selected from 0 to T, a time step i and a noise-added sample data xi corresponding to the time step i are input into a diffusion model to obtain a predicted noise ∈'; a time step j is randomly selected from 0 to i, a time step j and a noise-reduced sample data x j ′ The input diffusion model obtains a predicted noise ∈ j ′ ; the predicted noise ∈' and the noise-reduced sample data x j ′ and the predicted noise ∈ j ′ are calculated to obtain a first reconstruction error dist[i] from the time step i to the time step 0 and a second reconstruction error dist[j] from the time step j to the time step 0; a total loss value is obtained according to the predicted noise ∈', the real noise ∈, the first reconstruction error dist[i] and the second reconstruction error dist[j]; the model parameters are updated through back propagation according to the total loss value; the above steps are repeatedly executed until the iterative training is completed, and a trained diffusion model is obtained.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more particularly to a training method, apparatus, device, and storage medium for a diffusion model. Background Technology

[0002] In recent years, diffusion models have made significant progress in the fields of image generation and denoising. Diffusion models achieve efficient denoising and generate high-quality samples by gradually adding noise to the data and learning how to recover the data from the noise.

[0003] With the development of deep learning technology, the neural network architecture within deep learning provides a powerful foundation for image feature extraction and processing. Simultaneously, continuous advancements in hardware technology have made it possible to train large-scale diffusion models. The widespread adoption of the internet and the acceleration of digitization have generated massive amounts of image data; these abundant data resources provide ample material for training diffusion models, enabling them to be trained more effectively and thus improving the quality of generated images to produce optimal results.

[0004] In existing technologies, there are two main methods for generating original images: one is to directly predict the generated image through distillation; the other is to enhance the consistency of the model by predicting the same intermediate state at different time points. Both methods have certain limitations, making it impossible to efficiently generate accurate images. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a diffusion model to solve the technical problem that diffusion models cannot generate images faster and better.

[0006] On one hand, this application provides a training method for a diffusion model, including: setting a maximum time step T; repeating iterative training until a preset number of iterations, wherein the iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noisy sample data xi corresponding to time step i into the diffusion model to obtain predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain predicted noise ∈′; and randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain predicted noise ∈′. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ jThe first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The reconstruction error represents the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data. The total loss value is obtained based on the predicted noise ∈ ′, the real noise ∈ , the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation on the sampling path. The model parameters are updated by backpropagation based on the total loss value. The iteration training is performed until the iteration training ends, and the trained diffusion model is obtained.

[0007] In some examples, a relaxation operation is performed based on the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]'; if the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0008] In some examples, a one-way edge from i to j is established between time step i and time step j, and the edge weight edge(i, j) is defined according to the one-way edge; the edge weight edge(i, j) represents the error between time step i and time step j; the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] is calculated to obtain the third reconstruction error dist[i]'.

[0009] In some examples, if the first reconstruction error dist[i] is not greater than the third reconstruction error dist[i]', then the next iteration of training is executed directly.

[0010] In some examples, the mean squared error is calculated based on the predicted noise ∈′ and the actual noise ∈ to obtain the first loss value; the second loss value is calculated based on the first reconstruction error dist[i] and the second reconstruction error dist[j], and the sum of the first loss value and the second loss value is calculated to obtain the total loss value.

[0011] In some examples, the sum of edge weight edge(i,j) and second reconstruction error dist[j] is calculated according to the loss function; the mean square error of the sum and the first reconstruction error dist[i] is calculated to obtain the second loss value.

[0012] In some examples, a noisy image is input; the noisy image is fed into a trained diffusion model for prediction and denoising, and the image currently output by the diffusion model is used as the generated image.

[0013] In another aspect, this application provides a training apparatus for a diffusion model, comprising: a training module for setting a maximum time step T; repeated iterative training until a preset number of iterations, wherein the iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noisy sample data xi corresponding to time step i into the diffusion model to obtain predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain predicted noise ∈′; and randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain predicted noise ∈′. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The reconstruction error represents the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data. The processing module is used to obtain the total loss value based on the predicted noise ∈ ′, the real noise ∈ , the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation on the sampling path. The model parameters are updated by backpropagation based on the total loss value. The iteration training is performed until the iteration training ends, and the trained diffusion model is obtained.

[0014] In some examples, the processing module is used to: perform a relaxation operation based on the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]'; if the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0015] In some examples, the processing module is used to: establish a unidirectional edge from i to j between time step i and time step j, define the edge weight edge(i, j) according to the unidirectional edge; the edge weight edge(i, j) represents the error between time step i and time step j; calculate the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]'.

[0016] In another aspect, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the aforementioned method.

[0017] In another aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the aforementioned method.

[0018] This application provides a training method, apparatus, device, and storage medium for a diffusion model. First, a time step i is randomly selected from 0 to T. Time step i and its corresponding noisy sample data xi are input into the diffusion model to obtain the predicted noise ∈′. Then, a time step j is randomly selected from 0 to i. Time step j and its corresponding denoised sample data x are input into the diffusion model. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The total loss value is obtained based on the predicted noise ∈ ', the true noise ∈ ', the first reconstruction error dist[i], and the second reconstruction error dist[j]. Backpropagation is performed based on the total loss value to update the model parameters. The above steps are repeated until the iterative training ends, resulting in a trained diffusion model. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updating model parameters, and optimizing the model's sampling paths, thereby improving the speed and accuracy of image generation. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 The diagram above illustrates a flowchart of a training method for a diffusion model.

[0021] Figure 2 The diagram above illustrates a flowchart of a training method for a diffusion model.

[0022] Figure 3 The diagram above illustrates a flowchart of a training method for a diffusion model.

[0023] Figure 4 The diagram above illustrates a schematic of the structure of a training device for a diffusion model.

[0024] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover but not exclude inclusion. For example, a product or device that includes a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device. As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0028] In the diffusion model, the sampling path refers to the process of gradually transitioning from a high-noise state to a low-noise state; the diffusion model gradually restores the image from the noise through a reverse sampling process, and finally generates a new image.

[0029] In existing technologies, there are two main methods for generating original images: one is to directly predict the generated image through distillation; the other is to enhance the consistency of the model by predicting the same intermediate state at different time points. Both methods have limitations. In the reverse process of the diffusion model, it is impossible to determine which sampling path is optimal, thus hindering the efficient generation of accurate images.

[0030] The technical solution provided in this application aims to solve the above-mentioned technical problems of the prior art. In this embodiment, firstly, time step i is randomly selected from 0 to T, and time step i and the noisy sample data xi corresponding to time step i are input into the diffusion model to obtain the predicted noise ∈′; then, time step j is randomly selected from 0 to i, and time step j and the noisy sample data x corresponding to time step j are input into the diffusion model to obtain the predicted noise ∈′. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ jThe first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The total loss value is obtained based on the predicted noise ∈ ', the true noise ∈ ', the first reconstruction error dist[i], and the second reconstruction error dist[j]. Backpropagation is performed based on the total loss value to update the model parameters. The above steps are repeated until the iterative training ends, resulting in a trained diffusion model. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updating model parameters, and optimizing the model's sampling paths, thereby improving the speed and accuracy of image generation.

[0031] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0032] Example 1

[0033] Figure 1 The diagram above illustrates a flowchart of a training method for a diffusion model. Figure 1 As shown, the method includes:

[0034] S101, Set the maximum time step T; Repeat iterative training until the preset number of iterations is reached. Iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noisy sample data xi corresponding to time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain the predicted noise ∈′; j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The reconstruction error characterizes the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data.

[0035] S102, the total loss value is obtained based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation in the sampling path. The model parameters are updated by backpropagation based on the total loss value. The iteration training is performed until the iteration training ends, and the trained diffusion model is obtained.

[0036] Backpropagation is a technique used to train models. The backpropagation algorithm calculates the gradient of the loss function with respect to the model parameters. The model parameters are then adjusted based on the gradient of the loss function. By continuously optimizing the model parameters, the diffusion model can better learn the distribution of the data, thereby improving its ability to recover the original data from noise.

[0037] In a real-world scenario, the diffusion model is trained, with a maximum time step T set. Iterative training is repeated until a preset number of iterations is reached. The iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the corresponding noisy sample data xi into the diffusion model to obtain the predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the corresponding denoised sample data x... j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The reconstruction error represents the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data. The total loss value is obtained based on the predicted noise ∈ ′, the real noise ∈ , the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation on the sampling path. The model parameters are updated by backpropagation based on the total loss value. The iteration training is performed until the iteration training ends, and the trained diffusion model is obtained.

[0038] The maximum time step T and the number of training iterations can be set according to the actual situation. As an example, the number of training iterations can be 300. For example, the maximum time step can be set to 100. Taking the above process as an example, repeating the training iterations 300 times includes: randomly selecting time step 60 from 0-100, inputting time step 60 and its corresponding noisy sample data x60 into the diffusion model to obtain noise ∈′; randomly selecting time step 35 from 0-60, inputting time step 35 and its corresponding denoised sample data x′ 35 Input diffusion model to obtain predicted noise ∈′ 35 According to x60, ∈′, x′ 35 ,∈′ 35The first reconstruction error dist

[60] from time step 60 to time step 0 and the second reconstruction error dist

[35] from time step 35 to time step 0 are calculated. The total loss value is obtained based on the predicted noise ∈′, the real noise ∈, dist

[60] , and dist

[35] . The model parameters are updated by backpropagation based on the total loss value. The model is then returned to perform iterative training. The training is performed 300 times to obtain the trained diffusion model.

[0039] In this scheme, the total loss value is only calculated during training if the relaxation condition is met. In some examples, S102, the total loss value obtained based on the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j] includes:

[0040] The third reconstruction error dist[i]' is obtained by performing a relaxation operation based on the second reconstruction error dist[j].

[0041] If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0042] The relaxation operation typically refers to appropriately relaxing or adjusting the constraints of a problem. Specifically, in this example, the relaxation operation is dist[i] = min(edge(i,j) + dist[j], dist[i]), for all i>j>0. This operation simulates finding a better sampling path by selecting an intermediate point j as a stepping stone in the reverse process from time step i to time step 0. The relaxation condition is that the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]'.

[0043] In some examples, Figure 2 The diagram illustrates a flowchart of a diffusion model training method; a relaxation operation is performed based on the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]', specifically including:

[0044] S201, establish a one-way edge from i to j between time step i and time step j, and define the edge weight edge(i, j) according to the one-way edge; the edge weight edge(i, j) represents the error between time step i and time step j;

[0045] S202, calculate the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]'.

[0046] Creating a unidirectional edge typically involves adding a directed edge to a graph structure; this is an operation in graph theory modeling. Specifically, in a diffusion model, its reverse process is viewed as a graph, where each time step i corresponds to a node in the graph. A unidirectional edge is created between any two time steps i and j (i>j). Creating unidirectional edges in graph theory modeling within a diffusion model allows the data to be represented as nodes and edges. This enables the analysis of the data propagation path during the diffusion process using nodes and edges, thereby optimizing the path and improving the model's performance.

[0047] Furthermore, the edge weights are defined as follows: edge(i,j) = 0 if j > 0, edge(i,0) = reconstruction_error(x0',x0). In graph theory modeling, edge(i,0) = reconstruction_error(x0',x0) means defining the edge weight between node i and node 0 as the reconstruction error of data x0. This connects the graph structure with the reconstruction relationship of the data. Here, x0 represents the original data, and x0' represents the newly generated data. For example, when a time step k is randomly selected in 0-T, the time step k and the noisy sample data x corresponding to time step k are... k The input diffusion model yields the predicted noise ∈′; based on this predicted noise ∈′, the noise addition formula is used to solve the problem.

[0048] In some examples, if the first reconstruction error dist[i] is not greater than the third reconstruction error dist[i]', then the next iteration of training is executed directly. That is, if the relaxation condition is not met, the current iteration of training ends and the next iteration of training is performed directly.

[0049] In some examples, Figure 3 The example illustrates a flowchart of a training method for a diffusion model; the total loss value is obtained based on the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], specifically including:

[0050] S301, calculate the mean square error based on the predicted noise ∈′ and the actual noise ∈ to obtain the first loss value;

[0051] S302, based on the first reconstruction error dist[i] and the second reconstruction error dist[j], calculate the second loss value based on the loss function;

[0052] S303, calculate the sum of the first loss value and the second loss value to obtain the total loss value.

[0053] The loss function, based on relaxation operations, can be designed as loss = ||dist[i] - (edge(i,j) + dist[j])||2; specifically, for the loss function, when j>0, edge(i,j) = 0. When i=1, the loss function can be simplified to loss = ||dist[1]||2. During training, by calculating the gradient of the loss function, the model can update its parameters and improve its performance.

[0054] In some examples, a second loss value is calculated based on the first reconstruction error dist[i] and the second reconstruction error dist[j], according to the loss function, including:

[0055] Based on the loss function, calculate the sum of edge weight edge(i,j) and second reconstruction error dist[j].

[0056] The mean square error between the summation result and the first reconstruction error dist[i] is calculated to obtain the second loss value.

[0057] This embodiment provides a training method for a diffusion model. It simulates multiple sampling paths in the reverse process of the diffusion model and uses the shortest path algorithm in graph theory modeling to optimize the sampling paths. For example, assuming the maximum time step is 1000: before the iteration, assuming i = 10, j = 2: the two paths 10-0 and 10-2-0 are generated in 1 step and 2 steps respectively.

[0058] iter 1: Assuming relaxation is satisfied, then the reconstruction capability of path(10-0) is equivalent to that of path(10-2-0), and in this case, the generation in step 1 is equivalent to the original generation in step 2.

[0059] iter 2: Assume i = 100, j = 10, and that relaxation is satisfied, then path(100-0) is equivalent to path(100-10-0).

[0060] Based on the result of iter 1, path(100-0) is equivalent to path(100-10-2-0), so step 1 generation is equivalent to the original step 3 generation.

[0061] iter n:path(T-0) is equivalent to the combination of various paths satisfying the relaxation condition; after the iterative training ends, by continuously optimizing the path and updating the model parameters, the process of generating images by the diffusion model is changed from 1000 steps to a one-step generation of 1000-0; this greatly improves the efficiency of the model.

[0062] In practical applications, using a trained diffusion model to generate images can produce high-quality images, such as those with realistic details, diverse styles, and high resolution. In some examples, a noisy image is input; this noisy image is fed into a trained diffusion model for predictive denoising, and the model's output is used as the generated image. The input noisy image can be a randomly selected image from a standard Gaussian noise distribution. By inputting the noisy image into the model, it can directly denoise and generate images based on the pre-trained sampling path, thus achieving faster and more accurate image generation.

[0063] This application provides a training method for a diffusion model. First, a time step i is randomly selected from 0 to T. The time step i and the corresponding noisy sample data xi are input into the diffusion model to obtain the predicted noise ∈′. Then, a time step j is randomly selected from 0 to i. The time step j and the corresponding denoised sample data x are input into the diffusion model. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The total loss value is obtained based on the predicted noise ∈ ', the true noise ∈ ', the first reconstruction error dist[i], and the second reconstruction error dist[j]. Backpropagation is performed based on the total loss value to update the model parameters. The above steps are repeated until the iterative training ends, resulting in a trained diffusion model. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updating model parameters, and optimizing the model's sampling paths, thereby improving the speed and accuracy of image generation.

[0064] Example 2

[0065] Figure 4 The diagram above illustrates a schematic of a training device for a diffusion model, as shown in the image. Figure 4 As shown, the device includes:

[0066] Training module 11 is used to set the maximum time step T; iterative training is repeated until the preset number of iterations is reached. The iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noisy sample data xi corresponding to time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain the predicted noise ∈′; and randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j into the diffusion model to obtain the predicted noise ∈′. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The reconstruction error characterizes the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data.

[0067] Processing module 12 is used to obtain the total loss value based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation in the sampling path. The model parameters are updated by backpropagation based on the total loss value. The process returns to execute iterative training until the iterative training ends, and the trained diffusion model is obtained.

[0068] Backpropagation is a technique used to train models. The backpropagation algorithm calculates the gradient of the loss function with respect to the model parameters. The model parameters are then adjusted based on the gradient of the loss function. By continuously optimizing the model parameters, the diffusion model can better learn the distribution of the data, thereby improving its ability to recover the original data from noise.

[0069] In this scheme, the total loss value is only calculated when the relaxation condition is met during training. In some examples, in S102, the processing module 12 is used to calculate the total loss value based on the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], including:

[0070] The third reconstruction error dist[i]' is obtained by performing a relaxation operation based on the second reconstruction error dist[j].

[0071] If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0072] The relaxation operation typically refers to appropriately relaxing or adjusting the constraints of a problem. Specifically, in this example, the relaxation operation is dist[i] = min(edge(i,j) + dist[j], dist[i]), for all i>j>0. This operation simulates finding a better sampling path by selecting an intermediate point j as a stepping stone in the reverse process from time step i to time step 0. The relaxation condition is that the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]'.

[0073] In some examples, processing module 12 is used to perform a relaxation operation based on the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]', specifically including:

[0074] Establish a one-way edge from time step i to time step j, and define the edge weight edge(i,j) based on the one-way edge; the edge weight edge(i,j) represents the error between time step i and time step j.

[0075] The third reconstruction error dist[i]' is obtained by summing the edge weights from time step i to time step j with the second reconstruction error dist[j].

[0076] Establishing a unidirectional edge typically involves adding a directed edge to a graph structure; this is an operation in graph theory modeling. Specifically, in a diffusion model, its reverse process is viewed as a graph, where each time step i corresponds to a node in the graph. A unidirectional edge is established between any two time steps i and j (i>j). Establishing unidirectional edges in graph theory modeling within a diffusion model allows the data to be represented as nodes and edges. This enables the analysis of the data propagation path during the diffusion process using nodes and edges, thereby optimizing the path and improving the model's performance.

[0077] This embodiment provides a training device for a diffusion model. First, a time step i is randomly selected from 0 to T. The time step i and the noisy sample data xi corresponding to the time step i are input into the diffusion model to obtain the predicted noise ∈′. Then, a time step j is randomly selected from 0 to i. The time step j and the noisy sample data x corresponding to the time step j are input into the diffusion model to obtain the predicted noise ∈′. j The input diffusion model yields the predicted noise ∈ j Based on the noise-added sample data xi and the predicted noise ∈ ', and the noise-reduced sample data x j ′ and predicted noise∈ j The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated. The total loss value is obtained based on the predicted noise ∈ ', the true noise ∈ ', the first reconstruction error dist[i], and the second reconstruction error dist[j]. Backpropagation is performed based on the total loss value to update the model parameters. The above steps are repeated until the iterative training ends, resulting in a trained diffusion model. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updating model parameters, and optimizing the model's sampling paths, thereby improving the speed and accuracy of image generation.

[0078] Example 3

[0079] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0080] The electronic device includes a processor 591 and a memory 592; it may also include a communication interface 593 and a bus 594. The processor 591, memory 592, and communication interface 593 can communicate with each other via the bus 594. The communication interface 593 can be used for information transmission. The processor 591 can call logical instructions in the memory 592 to execute the methods described in the example above.

[0081] Furthermore, the logical instructions in the aforementioned memory 592 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. As a computer-readable storage medium, memory 592 can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of this application. Processor 591 executes functional applications and data processing by running the software programs, instructions, and modules stored in memory 592, thereby implementing the methods in the above method examples.

[0082] The memory 592 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 592 may include high-speed random access memory and may also include non-volatile memory.

[0083] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method in any of the embodiments.

[0084] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method in any of the embodiments.

[0085] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0086] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0087] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0088] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0089] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0090] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0091] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a diffusion model, characterized in that, include: Set the maximum time step T; The training is repeated iteratively until a preset number of iterations is reached. The iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noisy sample data xi corresponding to time step i into the diffusion model to obtain predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j... j ′ Input diffusion model to obtain predicted noise ∈ j ′ Based on the noise-added sample data xi and the predicted noise ∈′, and the noise-reduced sample data x j ′ and the predicted noise ∈ j ′ The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated; the reconstruction error characterizes the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data. The total loss value is obtained based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss caused by the relaxation operation in the sampling path. The model parameters are updated by backpropagation based on the total loss value. The iterative training is then performed until the iterative training ends, and the trained diffusion model is obtained.

2. The method according to claim 1, characterized in that, The total loss value obtained based on the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j] includes: The third reconstruction error dist[i]' is obtained by performing a relaxation operation based on the second reconstruction error dist[j]. If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained based on the predicted noise ∈', the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

3. The method according to claim 2, characterized in that, The process of obtaining the third reconstruction error dist[i]' by performing a relaxation operation based on the second reconstruction error dist[j] specifically includes: A unidirectional edge from i to j is established between time step i and time step j, and the edge weight edge(i, j) is defined according to the unidirectional edge; the edge weight edge(i, j) represents the error between time step i and time step j; The third reconstruction error dist[i]' is obtained by summing the edge weights from time step i to time step j with the second reconstruction error dist[j].

4. The method according to claim 2, characterized in that, The method further includes: If the first reconstruction error dist[i] is not greater than the third reconstruction error dist[i]', then the next iteration of training is executed directly.

5. The method according to claim 3, characterized in that, The total loss value is obtained based on the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], specifically including: The first loss value is obtained by calculating the mean square error based on the predicted noise ∈′ and the actual noise ∈′. Based on the first reconstruction error dist[i] and the second reconstruction error dist[j], the second loss value is calculated using the loss function; The total loss value is obtained by summing the first loss value and the second loss value.

6. The method according to claim 5, characterized in that, The step of calculating the second loss value based on the first reconstruction error dist[i] and the second reconstruction error dist[j] using a loss function includes: Based on the loss function, calculate the sum of the edge weight edge(i,j) and the second reconstruction error dist[j]. The mean square error between the summation result and the first reconstruction error dist[i] is calculated to obtain the second loss value.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Input a noisy image; The noisy image is input into the trained diffusion model for prediction and noise reduction, and the image currently output by the diffusion model is used as the generated image.

8. A training device for a diffusion model, characterized in that, include: The training module is used to set the maximum time step T; The training is repeated iteratively until a preset number of iterations is reached. The iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noisy sample data xi corresponding to time step i into the diffusion model to obtain predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noisy sample data x corresponding to time step j... j ′ Input diffusion model to obtain predicted noise ∈ j ′ Based on the noise-added sample data xi and the predicted noise ∈′, and the noise-reduced sample data x j ′ and the predicted noise ∈ j ′ The first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0 are calculated; the reconstruction error characterizes the degree of difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data. The processing module is used to obtain a total loss value based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]. The total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation on the sampling path. The module updates the model parameters by backpropagation based on the total loss value. The module then returns to execute the iterative training until the iterative training ends, and obtains the trained diffusion model.

9. The apparatus according to claim 8, characterized in that, The processing module uses the total loss value obtained from the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j] for: The third reconstruction error dist[i]' is obtained by performing a relaxation operation based on the second reconstruction error dist[j]. If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained based on the predicted noise ∈', the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

10. The apparatus according to claim 9, characterized in that, The processing module performs a relaxation operation on the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]', specifically used for: A unidirectional edge from i to j is established between time step i and time step j, and the edge weight edge(i, j) is defined according to the unidirectional edge; the edge weight edge(i, j) represents the error between time step i and time step j; The third reconstruction error dist[i]' is obtained by summing the edge weights from time step i to time step j with the second reconstruction error dist[j].

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

Images