Diffusion model training method and device, equipment and storage medium

Abstract

The invention provides a diffusion model training method and device, equipment and a storage medium. Randomly selecting a time step i from 0 to T, and inputting the time step i and noise-added sample data xi corresponding to the time step i into a diffusion model to obtain a predicted noise belonging to '; randomly selecting a time step j from 0 to i, and inputting the time step j and noise reduction sample data xj'corresponding to the time step j into the diffusion model to obtain predicted noise belonging to j '; calculating a first reconstruction error dist [i] from the time step i to the time step 0 and a second reconstruction error dist [j] from the time step j to the time step 0 according to the noise-added sample data xi and the predicted noise belonging to the predicted noise and the noise-reduced sample data xj'and the predicted noise belonging to the predicted noise belonging to the predicted noise; obtaining a total loss value according to the predicted noise belonging to ', the real noise belonging to', the first reconstruction error dist [i] and the second reconstruction error dist [j]; performing back propagation according to the total loss value to update model parameters; and repeatedly executing the steps until the iterative training is finished, and obtaining a trained diffusion model.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a method, apparatus, device and storage medium for training a diffusion model. Background Art

[0002] In recent years, diffusion models have made significant progress in the field of image generation and denoising; diffusion models achieve efficient denoising and generate high-quality samples by gradually adding noise to the data and learning how to recover the data from the noise.

[0003] With the development of deep learning technology, the neural network architecture in deep learning provides a strong foundation for the extraction and processing of image features. At the same time, with the continuous development of hardware technology, it is possible to train large-scale diffusion models. The popularization of the Internet and the acceleration of the digitalization process have brought a huge amount of image data; the rich data resources provide sufficient materials for the training of diffusion models, enabling them to better train diffusion models, thereby improving the quality of generated images to generate the best images.

[0004] In the prior art, there are two main methods for generating original images: one is to directly predict and generate images through distillation; the other is to enhance the consistency of the model by predicting the same intermediate state at different time points. These two methods have certain limitations, so that they cannot efficiently generate accurate images. Summary of the invention

[0005] The present application provides a diffusion model training method, apparatus, device and storage medium to solve the technical problem that the diffusion model cannot generate images faster and better.

[0006] On the one hand, the present application provides a diffusion model training method, including: setting a maximum time step T; repeating iterative training until a preset number of iterations, the iterative training including: randomly selecting a time step i from 0 to T, inputting the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain predicted noise ∈′; randomly selecting a time step j from 0 to i, inputting the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model to obtain predicted noise ∈′; j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the randomly selected sampling path from time step to 0 and the original data; the total loss value is obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], which represents the loss between the real noise and the predicted noise, as well as the loss caused by the relaxation operation on the sampling path; back propagate and update the model parameters according to the total loss value; return to perform iterative training until the iterative training is completed to obtain a trained diffusion model.

[0007] In some examples, a relaxation operation is performed on the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]'; if the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', the total loss value is obtained based on the predicted noise ∈′, the actual noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0008] In some examples, a unidirectional edge from time step i to time step j is established between time step i and time step j, and an edge weight edge(i, j) is defined based on the unidirectional edge; the edge weight edge(i, j) represents the error between time step i and time step j; the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] is calculated to obtain a third reconstruction error dist[i]'.

[0009] In some examples, if the first reconstruction error dist[i] is not greater than the third reconstruction error dist[i]', the next iterative training is directly performed.

[0010] In some examples, a first loss value is obtained by calculating the mean square error based on the predicted noise ∈′ and the actual noise ∈; a second loss value is obtained by calculating based on the loss function according to the first reconstruction error dist[i] and the second reconstruction error dist[j]; and a total loss value is obtained by calculating the sum of the first loss value and the second loss value.

[0011] In some examples, according to the loss function, the sum of the edge weight edge(i,j) and the second reconstruction error dist[j] is calculated; the mean square error between the sum and the first reconstruction error dist[i] is calculated to obtain the second loss value.

[0012] In some examples, a noisy image is input; the noisy image is input into a trained diffusion model for predictive denoising, and an image currently output by the diffusion model is used as a generated image.

[0013] In another aspect, the present application provides a training device for a diffusion model, including: a training module for setting a maximum time step T; repeating iterative training until a preset number of iterations, the iterative training comprising: randomly selecting a time step i from 0 to T, inputting the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain predicted noise ∈′; randomly selecting a time step j from 0 to i, inputting the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data; the processing module is used to obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], the total loss value represents the loss between the real noise and the predicted noise, and the loss generated under the influence of the relaxation operation on the sampling path; back propagate and update the model parameters according to the total loss value; return to perform iterative training until the iterative training is completed to obtain a trained diffusion model.

[0014] In some examples, the processing module is used to: perform a relaxation operation on the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]'; if the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then obtain a total loss value based on the predicted noise ∈′, the actual noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0015] In some examples, the processing module is used to: establish a unidirectional edge from time step i to time step j, and define an edge weight edge(i, j) based on the unidirectional edge; the edge weight edge(i, j) represents the error between time step i and time step j; calculate the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]'.

[0016] On the other hand, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the above method.

[0017] On the other hand, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0018] The present application provides a diffusion model training method, apparatus, device and storage medium. First, a time step i is randomly selected from 0 to T, and the time step i and the noise-added sample data xi corresponding to the time step i are input into the diffusion model to obtain the predicted noise ∈′; a time step j is randomly selected from 0 to i, and the time step j and the noise-reduced sample data x corresponding to the time step j are input into the diffusion model to obtain the predicted noise ∈′; j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]; update the model parameters by back propagation according to the total loss value; repeat the above steps until the iterative training is completed and a trained diffusion model is obtained. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updates model parameters and optimizes the sampling path of the model, thereby improving the speed and accuracy of the model generating images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] Figure 1 A flowchart of a method for training a diffusion model is shown in FIG.

[0021] Figure 2 A flowchart of a method for training a diffusion model is shown in FIG.

[0022] Figure 3 A flowchart of a method for training a diffusion model is shown in FIG.

[0023] Figure 4 A schematic diagram of the structure of a training device for a diffusion model is shown in FIG.

[0024] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices. The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that can perform the functions associated with the element.

[0028] In the diffusion model, the sampling path refers to the process of gradually transitioning from a high-noise state to a low-noise state; the diffusion model gradually restores the image from the noise through the reverse sampling process and finally generates a new image.

[0029] In the related technology, in the prior art, there are mainly two methods for generating original images: one is to directly predict and generate images through distillation; the other is to enhance the consistency of the model by predicting the same intermediate state at different time points. These two methods have certain limitations. In the reverse process of the diffusion model, it is impossible to confirm which sampling path is the best, so that accurate images cannot be generated efficiently.

[0030] The technical solution provided by this application is intended to solve the above technical problems of the prior art. In this embodiment, firstly, a time step i is randomly selected from 0 to T, and the time step i and the noise-added sample data xi corresponding to the time step i are input into the diffusion model to obtain the predicted noise ∈′; a time step j is randomly selected from 0 to i, and the time step j and the noise-reduced sample data x corresponding to the time step j are input into the diffusion model to obtain the predicted noise ∈′; j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]; update the model parameters by back propagation according to the total loss value; repeat the above steps until the iterative training is completed and a trained diffusion model is obtained. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updates model parameters and optimizes the sampling path of the model, thereby improving the speed and accuracy of the model generating images.

[0031] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0032] Embodiment 1

[0033] Figure 1 A flow chart of a diffusion model training method is shown in FIG. Figure 1 As shown, the method includes:

[0034] S101, set a maximum time step T; repeat iterative training until a preset number of iterations, the iterative training includes: randomly selecting a time step i from 0 to T, inputting the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting a time step j from 0 to i, inputting the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data.

[0035] S102, obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], the total loss value represents the loss between the real noise and the predicted noise, as well as the loss caused by the relaxation operation on the sampling path; perform back propagation to update the model parameters according to the total loss value; return to perform iterative training until the iterative training is completed, and obtain a trained diffusion model.

[0036] Among them, back propagation is a technology used to train models. Through the back propagation algorithm, the gradient of the loss function to the model parameters is calculated; the parameters of the model are adjusted according to the gradient of the loss function. By continuously optimizing the parameters of the model, the diffusion model can better learn the distribution of data, thereby improving the ability to recover the original data from noise.

[0037] Combined with the actual scenario, the diffusion model is trained and the maximum time step T is set; the iterative training is repeated until the preset number of iterations. The iterative training includes: randomly selecting time step i from 0 to T, inputting time step i and the noise sample data xi corresponding to time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting time step j from 0 to i, inputting time step j and the noise reduction sample data x corresponding to time step j into the diffusion model to obtain the predicted noise ∈′; j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the randomly selected sampling path from time step to 0 and the original data; the total loss value is obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], which represents the loss between the real noise and the predicted noise, as well as the loss caused by the relaxation operation on the sampling path; back propagate and update the model parameters according to the total loss value; return to perform iterative training until the iterative training is completed to obtain a trained diffusion model.

[0038] Among them, the maximum time step T and the number of iterative training can be set according to the actual situation. As an example, the number of iterative training can be 300 times. For example, the maximum time step can be set to 100. Combining the above process, for example, repeat the iterative training 300 times, the iterative training includes: randomly selecting time step 60 from 0-100, inputting time step 60 and its corresponding noise-added sample data x60 into the diffusion model to obtain noise ∈′; randomly selecting time step 35 from 0-60, inputting time step 35 and its corresponding noise-reduced sample data x′ into the diffusion model 35 Input the diffusion model to get the predicted noise ∈′ 35 ; According to x60,∈′,x′ 35 ,∈′ 35The first reconstruction error dist

[60] from time step 60 to time step 0 and the second reconstruction error dist

[35] from time step 35 to time step 0 are calculated; the total loss value is obtained according to the predicted noise ∈′, the real noise ∈, dist

[60] , and dist

[35] ; the model parameters are updated by backpropagation according to the total loss value, and the iterative training is returned. The iterative training is performed 300 times to obtain the trained diffusion model.

[0039] In the training process of this scheme, the total loss value needs to be calculated only when the relaxation condition is met. In some examples, in S102, the total loss value obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j] includes:

[0040] Perform a relaxation operation on the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]';

[0041] If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', the total loss value is obtained according to the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0042] The relaxation operation usually refers to the appropriate relaxation or adjustment of the constraints of a problem. Specifically, in this example, the relaxation operation is dist[i]=min(edge(i,j)+dist[j],dist[i]), for all i>j>0; this operation simulates the reverse process from time step i to time step 0, by selecting an intermediate point j as a springboard to find a better sampling path. The relaxation condition is that the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]'.

[0043] In some examples, Figure 2 A flow chart of a diffusion model training method is exemplarily shown in FIG. 1 ; performing a relaxation operation according to the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]', specifically comprising:

[0044] S201, establish a unidirectional edge from time step i to time step j, and define an edge weight edge(i, j) based on the unidirectional edge; the edge weight edge(i, j) represents the error between time step i and time step j;

[0045] S202, calculate the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]'.

[0046] Establishing a unidirectional edge usually means adding a directed edge to a graph structure; establishing a unidirectional edge is an operation of graph theory modeling; specifically, in the diffusion model, its reverse process is regarded as a graph, in which each time step i corresponds to a node in the graph, and a unidirectional edge from i to j is established between any two time steps i and j (i>j); establishing unidirectional edges through graph theory modeling in the diffusion model can represent the data of the diffusion model in the form of nodes and edges, so that the propagation path of the data in the diffusion process can be analyzed through nodes and edges; thereby optimizing the path and improving the performance of the model.

[0047] In addition, the edge weights are defined as follows: edge(i,j)=0if j>0, edge(i,0)=reconstruction_error(x0',x0). In graph theory modeling, edge(i,0)=reconstruction_error(x0',x0) means that the edge weight between node i and node 0 is defined as the reconstruction error of data x0. In this way, the graph structure is linked to the reconstruction relationship of the data. x0 represents the original data, and x0' represents the newly generated data. For example, when a time step k is randomly selected in 0-T, the time step k and the noise sample data x corresponding to the time step k are compared. k Input the diffusion model to get the predicted noise ∈′; according to the predicted noise ∈′, use the noise addition formula to solve the

[0048] In some examples, if the first reconstruction error dist[i] is not greater than the third reconstruction error dist[i]', the next iterative training is directly performed. That is, if the relaxation condition is not met, the current iterative training ends and the next iterative training is directly performed.

[0049] In some examples, Figure 3 A flow chart of a diffusion model training method is shown in FIG. 1 ; a total loss value is obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], specifically including:

[0050] S301, calculating the mean square error according to the predicted noise ∈′ and the real noise ∈ to obtain a first loss value;

[0051] S302, obtaining a second loss value by calculating based on a loss function according to the first reconstruction error dist[i] and the second reconstruction error dist[j];

[0052] S303: Calculate the sum of the first loss value and the second loss value to obtain a total loss value.

[0053] The loss function can be designed as loss = ‖dist[i]-(edge(i,j)+dist[j])‖2 based on the relaxation operation; in particular, for the loss function, when j>0, edge(i,j)=0. When i=1, the loss function can be simplified to loss = ‖dist[1]‖2. During the training process, by calculating the gradient of the loss function, the model can update parameters and improve the performance of the model.

[0054] In some examples, the second loss value is calculated based on the loss function according to the first reconstruction error dist[i] and the second reconstruction error dist[j], including:

[0055] According to the loss function, calculate the sum of the edge weight edge(i,j) and the second reconstruction error dist[j];

[0056] Calculate the mean square error between the summed result and the first reconstruction error dist[i] to obtain the second loss value.

[0057] This embodiment provides a method for training a diffusion model, which optimizes the sampling paths by simulating multiple sampling paths in the reverse process of the diffusion model and using the shortest path algorithm in graph theory modeling; for example, assuming that the maximum time step is 1000: before iteration, assuming that i=10, j=2: 10-0 vs 10-2-0, these two paths are generated in 1 step and 2 steps respectively;

[0058] iter 1: Assuming that the relaxation is satisfied, the reconstruction capability of path (10-0) is equivalent to path (10-2-0), and the 1-step generation is equivalent to the original 2-step generation;

[0059] iter 2: Assume i=100, j=10, and the relaxation condition is satisfied, then path(100-0) is equivalent to path(100-10-0)

[0060] According to the result of iter 1, path(100-0) is equivalent to path(100-10-2-0), so the 1-step generation is equivalent to the original 3-step generation;

[0061] iter n:path(T-0) is equivalent to the synthesis of various paths that satisfy the relaxation conditions. After the iterative training, the diffusion model generates images in a process that is converted from 1000-step generation to 1000-0-step generation by continuously optimizing the path and updating the model parameters. This greatly improves the efficiency of the model.

[0062] In practical applications, using a trained diffusion model to generate images can generate high-quality images, such as realistic image details, diverse image styles, and high-resolution images. In some examples, a noisy image is input; the noisy image is input into a trained diffusion model for predictive denoising, and the image currently output by the diffusion model is used as the generated image. The input noisy image may refer to an image randomly extracted from a standard Gaussian noise distribution; the noisy image is input into the model, and the model can directly denoise and generate the noise according to the sampling path trained by the model, thereby achieving faster and more accurate image generation.

[0063] The present application provides a diffusion model training method, first randomly selects a time step i from 0 to T, inputs the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain the predicted noise ∈′; randomly selects a time step j from 0 to i, inputs the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]; update the model parameters by back propagation according to the total loss value; repeat the above steps until the iterative training is completed and a trained diffusion model is obtained. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updates model parameters and optimizes the sampling path of the model, thereby improving the speed and accuracy of the model generating images.

[0064] Embodiment 2

[0065] Figure 4 A structural diagram of a training device for a diffusion model is shown as an example. Figure 4 As shown, the device comprises:

[0066] The training module 11 is used to set a maximum time step T; repeat the iterative training until the preset number of iterations, the iterative training includes: randomly selecting a time step i from 0 to T, inputting the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting a time step j from 0 to i, inputting the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data;

[0067] The processing module 12 is used to obtain a total loss value based on the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], wherein the total loss value represents the loss between the real noise and the predicted noise, as well as the loss caused by the relaxation operation on the sampling path; perform back propagation to update the model parameters according to the total loss value; and return to perform iterative training until the iterative training is completed to obtain a trained diffusion model.

[0068] Among them, back propagation is a technology used to train models. Through the back propagation algorithm, the gradient of the loss function to the model parameters is calculated; the parameters of the model are adjusted according to the gradient of the loss function. By continuously optimizing the parameters of the model, the diffusion model can better learn the distribution of data, thereby improving the ability to recover the original data from noise.

[0069] In the training process of this solution, the total loss value needs to be calculated only when the relaxation condition is met. In some examples, in S102, the processing module 12 is used to obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], including:

[0070] Perform a relaxation operation on the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]';

[0071] If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', the total loss value is obtained according to the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

[0072] The relaxation operation usually refers to the appropriate relaxation or adjustment of the constraints of a problem. Specifically, in this example, the relaxation operation is dist[i]=min(edge(i,j)+dist[j],dist[i]), for all i>j>0; this operation simulates the reverse process from time step i to time step 0, by selecting an intermediate point j as a springboard to find a better sampling path. The relaxation condition is that the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]'.

[0073] In some examples, the processing module 12 is used to perform a relaxation operation according to the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]′, specifically including:

[0074] A one-way edge from time step i to time step j is established between time step i and time step j, and the edge weight edge(i, j) is defined based on the one-way edge; the edge weight edge(i, j) represents the error between time step i and time step j;

[0075] Calculate the sum of the edge weight from time step i to time step j and the second reconstruction error dist[j] to obtain the third reconstruction error dist[i]'.

[0076] Establishing a unidirectional edge usually means adding a directed edge to a graph structure; establishing a unidirectional edge is an operation of graph theory modeling; specifically, in the diffusion model, its reverse process is regarded as a graph, in which each time step i corresponds to a node in the graph, and a unidirectional edge from i to j is established between any two time steps i and j (i>j); establishing unidirectional edges through graph theory modeling in the diffusion model can represent the data of the diffusion model in the form of nodes and edges, so that the propagation path of the data in the diffusion process can be analyzed through nodes and edges; thereby optimizing the path and improving the performance of the model.

[0077] This embodiment provides a training device for a diffusion model. First, a time step i is randomly selected from 0 to T, and the time step i and the noise-added sample data xi corresponding to the time step i are input into the diffusion model to obtain the predicted noise ∈′; a time step j is randomly selected from 0 to i, and the time step j and the noise-reduced sample data x corresponding to the time step j are input into the diffusion model to obtain the predicted noise ∈′; j ′ Input the diffusion model to get the predicted noise ∈ j ′; According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and prediction noise ∈ j ′, calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; obtain the total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j]; update the model parameters by back propagation according to the total loss value; repeat the above steps until the iterative training is completed and a trained diffusion model is obtained. This scheme guides model training by comparing the reconstruction errors of different sampling paths, updates model parameters and optimizes the sampling path of the model, thereby improving the speed and accuracy of the model generating images.

[0078] Embodiment 3

[0079] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device comprising:

[0080] The electronic device includes a processor 591 and a memory 592; it may also include a communication interface 593 and a bus 594. The processor 591, the memory 592, and the communication interface 593 may communicate with each other through the bus 594. The communication interface 593 may be used for information transmission. The processor 391 may call the logic instructions in the memory 592 to execute the above-mentioned example method.

[0081] In addition, the logic instructions in the above-mentioned memory 592 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. The memory 592, as a computer-readable storage medium, can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application. The processor 591 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 592, that is, implementing the method in the above-mentioned method example.

[0082] The memory 592 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 592 may include a high-speed random access memory and may also include a non-volatile memory.

[0083] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method in any embodiment.

[0084] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in any embodiment when the computer program is executed by a processor.

[0085] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0086] It should be further noted that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0087] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0088] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0091] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for training a diffusion model, characterized in that: include: Set the maximum time step T; Repeat the iterative training until the preset number of iterations, the iterative training includes: randomly selecting a time step i from 0 to T, inputting the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting a time step j from 0 to i, inputting the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model j ′ Input the diffusion model to get the predicted noise ∈ j ′ According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and the prediction noise ∈ j ′ , calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data; A total loss value is obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], wherein the total loss value represents the loss between the real noise and the predicted noise, as well as the loss generated under the influence of the relaxation operation on the sampling path; back propagation is performed to update the model parameters according to the total loss value; and the iterative training is returned to execute until the iterative training is completed to obtain a trained diffusion model.

2. The method according to claim 1, characterized in that: The total loss value obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j] includes: Perform a relaxation operation on the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]′; If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained according to the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

3. The method according to claim 2, characterized in that The performing a relaxation operation according to the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]' specifically includes: A unidirectional edge from time step i to time step j is established between the time step i and the time step j, and an edge weight edge(i, j) is defined according to the unidirectional edge; the edge weight edge(i, j) represents the error between the time step i and the time step j; The sum of the edge weight from the time step i to the time step j and the second reconstruction error dist[j] is calculated to obtain the third reconstruction error dist[i]'.

4. The method according to claim 2, characterized in that: The method further comprises: If the first reconstruction error dist[i] is not greater than the third reconstruction error dist[i]′, the next iterative training is directly performed.

5. The method according to claim 1, characterized in that Then the total loss value is obtained according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], which specifically includes: Calculate the mean square error according to the predicted noise ∈′ and the real noise ∈ to obtain a first loss value; Obtaining a second loss value by calculating based on a loss function according to the first reconstruction error dist[i] and the second reconstruction error dist[j]; The total loss value is obtained by calculating the sum of the first loss value and the second loss value.

6. The method according to claim 5, characterized in that The step of calculating a second loss value based on a loss function according to the first reconstruction error dist[i] and the second reconstruction error dist[j] includes: According to the loss function, a sum of the edge weight edge(i,j) and the second reconstruction error dist[j] is calculated; Calculate the mean square error between the summed result and the first reconstruction error dist[i] to obtain the second loss value.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Input noisy image; The noise image is input into the trained diffusion model for predictive denoising operation, and the image currently output by the diffusion model is used as the generated image.

8. A diffusion model training device, characterized in that: include: Training module, used to set the maximum time step T; Repeat the iterative training until the preset number of iterations, the iterative training includes: randomly selecting a time step i from 0 to T, inputting the time step i and the noise-added sample data xi corresponding to the time step i into the diffusion model to obtain the predicted noise ∈′; randomly selecting a time step j from 0 to i, inputting the time step j and the noise-reduced sample data x corresponding to the time step j into the diffusion model j ′ Input the diffusion model to get the predicted noise ∈ j ′ According to the noise-added sample data xi and the predicted noise ∈′, the noise-reduced sample data x j ′ and the predicted noise ∈ j ′ , calculate the first reconstruction error dist[i] from time step i to time step 0 and the second reconstruction error dist[j] from time step j to time step 0; the reconstruction error represents the difference between the reconstructed data generated by the sampling path from the randomly selected time step to 0 and the original data; A processing module is used to obtain a total loss value according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j], wherein the total loss value represents the loss between the real noise and the predicted noise, and the loss generated under the influence of the relaxation operation on the sampling path; perform back propagation to update the model parameters according to the total loss value; return to execute the iterative training until the iterative training is completed to obtain a trained diffusion model.

9. The device according to claim 8, characterized in that The total loss value obtained by the processing module according to the predicted noise ∈′, the real noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j] is used to: Perform a relaxation operation on the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]′; If the first reconstruction error dist[i] is greater than the third reconstruction error dist[i]', then the total loss value is obtained according to the predicted noise ∈′, the true noise ∈, the first reconstruction error dist[i], and the second reconstruction error dist[j].

10. The device according to claim 9, characterized in that The processing module performs a relaxation operation according to the second reconstruction error dist[j] to obtain a third reconstruction error dist[i]', which is specifically used for: A unidirectional edge from time step i to time step j is established between the time step i and the time step j, and an edge weight edge(i, j) is defined according to the unidirectional edge; the edge weight edge(i, j) represents the error between the time step i and the time step j; The sum of the edge weight from the time step i to the time step j and the second reconstruction error dist[j] is calculated to obtain the third reconstruction error dist[i]'.

11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images