Diffusion model-based inference acceleration method and system for text-to-image generation task

By searching for the best approximation process in the linear subspace of the hidden variables of the diffusion model, a lightweight OLSS scheduler is designed, which solves the problem of inefficient calculation of the diffusion model, and realizes the inference acceleration of efficient text and picture generation tasks, which is highly adaptable and The production quality is better than other methods.

WO2024239827A1PCT designated stage expired Publication Date: 2024-11-28ALIBABA CLOUD COMPUTING CO LTD

Patent Information

Application Number
PCT/CN2024/087704
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-04-15
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In the diffusion model, due to iterative forward reasoning, the calculation efficiency of the diffusion model is ineffective due to the iterative forward reasoning, and the existing scheduler method cannot effectively reduce the number of steps and affect the generation quality.

Method used

By analyzing the generation process of the diffusion model, the scheduler design problem is transformed into parameter determination, and the best approximation process is searched in the linear subspace of hidden variables. A lightweight OLSS scheduler is used to pass the least squares method and path optimization algorithm Reduce the inference step count to achieve high-quality image generation.

Benefits of technology

The number of inference steps is significantly reduced, from hundreds or thousands to dozens or even a few steps, improving the computing efficiency and ensuring the generation effect, strong adaptability and easy to train, and adapting to the diffusion model takes only a few minutes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024087704_28112024_PF_FP_ABST
    Figure CN2024087704_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a diffusion model-based inference acceleration method and system for a text-to-image generation task. The method comprises: running a full generation process to acquire intermediate variables; selecting n steps from the full process, and in the iterative computation of the n steps: using a linear combination of a previous latent variable and output intermediate variables as an estimated value of a latent variable obtained in the current step; obtaining a coefficient in the linear combination by minimizing a mean squared error between the estimated value and a corresponding latent intermediate variable; and on the basis of the obtained coefficient, generating an image as the inference result of a text-to-image generation task. According to the present invention, the full generation process is analyzed, the design problem of a scheduler is transformed into the determination of a plurality of parameters, and an accelerated generation process is further transformed into a span process of a linear subspace, so that by reconstructing a short-step approximation process, the number of iterations is reduced from hundreds or thousands of steps to dozens of steps or even a few steps, thereby ensuring the generation effect while greatly improving the efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Method and system for accelerating text-image generation task reasoning based on diffusion model

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 24, 2023, with application number 202310598298.X and invention name “Method and system for accelerating reasoning of text and image generation tasks based on diffusion model”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of deep learning, and in particular to a method and system for accelerating reasoning for text-image generation tasks based on a diffusion model. Background Art

[0003] The task of text-to-image generation involves using a computer program to generate an image that matches the textual description given a piece of text. In recent years, with the development of diffusion models, significant progress has been made in this area. Leading models in this field, through large-scale pre-training, are now capable of generating exquisite images comparable to those produced by human artists. However, while diffusion models significantly improve the quality of generated images, they also present significant computational efficiency issues due to the need to iteratively call the model to perform multiple forward inferences to generate sufficiently clear images.

[0004] To this end, a method is needed to accelerate the reasoning of the diffusion model and thus accelerate the completion of the text-image generation task.

[0005] Summary of the Invention

[0006] The technical problem to be solved by this disclosure is to provide a method and system for accelerating reasoning for text and image generation tasks based on a diffusion model. This method analyzes the complete generation process, transforming the design problem of the scheduler into the determination of several parameters, and further transforming the accelerated generation process into the spanning process of a linear subspace. By reconstructing the approximate process with a short number of steps, hundreds or thousands of iterations are reduced to dozens or even a few steps, significantly improving efficiency while ensuring the generation effect. This method is extremely lightweight and adaptable, and can be adapted to a diffusion model through just a few minutes of training, resulting in a significant improvement in efficiency.

[0007] According to a first aspect of the present disclosure, a method for accelerating reasoning of a text-image generation task based on a diffusion model is provided, comprising: running T steps of a complete generation process to collect intermediate variables obtained from the extended model, wherein the intermediate variables include a set of latent intermediate variables and a set of output intermediate variables; selecting n steps from the T steps, wherein n<T (optionally, n<<T); in the iterative calculation of the n steps: taking a linear combination of the previous latent variable and the output intermediate variable as an estimated value of the latent variable obtained in the current step; obtaining coefficients in the linear combination by minimizing the mean error between the estimated value and the corresponding latent intermediate variable; and generating an image as an inference result of the text-image generation task based on the obtained coefficients.

[0008] Optionally, taking a linear combination of the previous latent variable and the output intermediate variable as the estimated value of the latent variable obtained in the current step includes: taking a linear combination of the final latent variable and the output intermediate variable before the current step as the estimated value of the latent variable obtained in the current step.

[0009] Optionally, the collected intermediate variables include a group of implicit intermediate variables and a group of output intermediate variables corresponding to the n steps.

[0010] Optionally, obtaining the parameters in the linear combination by minimizing the average error between the estimated value and the corresponding implicit intermediate variable includes: using the orthogonal triangular decomposition method to solve the least squares problem to minimize the average error between the estimated value and the corresponding implicit intermediate variable.

[0011] Optionally, the n steps are selected from the T steps at uniform intervals.

[0012] Optionally, the n steps are selected from the T steps through an optimized path search. The step of selecting the n steps from the T steps through an optimized path search may include: performing the optimized path search with the optimization goal that the error of each of the n steps does not exceed an error threshold and the error of each step gradually decreases. The step of performing the optimized path search with the optimization goal that the error of each of the n steps does not exceed an error threshold and the error of each step gradually decreases may include: obtaining the optimized path based on a first binary search algorithm. The step of selecting the n steps from the T steps through an optimized path search may also include: calculating the minimum value of the error threshold when a path exists based on a second binary search algorithm.

[0013] According to a second aspect of the present disclosure, a diffusion model-based text-image generation task reasoning acceleration system is provided, comprising: a storage node for storing a trained diffusion model; and an image generation node for loading the trained diffusion model; obtaining text input from a user; generating an image as an inference result of the text-image generation task using the method described in the first aspect; and returning the image as an inference result to the corresponding user.

[0014] According to a third aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method described in the first aspect above.

[0015] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising an executable code. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described in the first aspect above.

[0016] According to a fifth aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor executes the method described in the first aspect above.

[0017] Therefore, in this disclosure, we first propose a detailed theoretical and empirical analysis of the generation process of a diffusion model based on a scheduler. We transform the design problem of the scheduler into the determination of several parameters, and further transform the accelerated generation process into the spanning of a linear subspace. Based on these analyses, we propose an inference acceleration scheme for text-image generation tasks based on a diffusion model. This scheme accelerates the generation process by searching for the best approximation of the complete generation process in the linear subspace spanned by latent variables. This acceleration scheme can generate high-quality images in a very small number of steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components in the exemplary embodiments of the present disclosure.

[0019] Figure 1 shows a schematic diagram of the diffusion process in physics.

[0020] Figure 2 shows a schematic diagram of the forward chain adding noise and the backward chain generating images.

[0021] Figure 3 shows a heatmap of latent variable correlation coefficients that records the complete generation process.

[0022] FIG4 shows three methods of generating current latent variables using previous latent variables.

[0023] FIG5 shows a schematic flowchart of a method for accelerating reasoning of text-image generation tasks based on a diffusion model.

[0024] FIG6 shows a specific operation example of n-step iterative calculation according to an embodiment of the present invention.

[0025] FIG7 shows an example of using the inference acceleration method of the present invention and an existing model to generate an image based on an input text prompt.

[0026] FIG8 shows a schematic diagram of the composition of a diffusion model-based text-image generation task reasoning acceleration system according to an embodiment of the present invention.

[0027] FIG9 shows a schematic structural diagram of a computing device that can be used to implement the above-mentioned diffusion model-based text-image generation task reasoning acceleration method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] In recent years, diffusion models have made great progress in the field of image generation. Based on a large amount of image and text data obtained from the internet, through large-scale training, diffusion models have been able to generate clear images related to given text content, even reaching a level comparable to that of human painters. The emergence of diffusion models has ushered in a new era of image generation.

[0030] However, diffusion models currently face severe computational efficiency challenges. Unlike methods based on generative adversarial networks (GANs), diffusion models require multiple iterations of the model for forward inference when generating images. This iterative generation process ensures image quality, but it also contributes to its computational inefficiency. To address this computational efficiency issue, research has been proposed, including reducing the number of inference steps through distillation and controlling the generation process through mathematically based schedulers. While distillation can guarantee effective generation, it requires additional training, significantly compromising usability. Training requires computational resources in most application scenarios, making it unsuitable for large-scale deployment. While using schedulers to control the generation process to reduce inference steps does not require additional training, it lacks adaptability to specific models, resulting in suboptimal generation results. Furthermore, while schedulers can reduce the number of steps without requiring training, using smaller step sizes introduces new noise, thus still facing a trade-off between computational efficiency and generation quality.

[0031] Based on related research work, the present disclosure focuses on designing an adaptive scheduler that requires only a very small amount of time for learning. However, for ease of understanding, the generation process of the diffusion model will first be theoretically analyzed. Using a given number of steps, most existing schedulers generate the next sample in the linear subspace spanned by the previous sample and the model output, so the generation process is essentially a process of spanning the linear subspace. In addition, the embodiment of the present invention also analyzes the correlation of intermediate variables during image generation, and observes that the model output contains redundant repetitive information at each step. Therefore, there is a basis for avoiding redundant calculations by reducing the number of steps. Based on the above analysis, the present disclosure replaces the coefficients in the iterative formula with trainable parameters to control the spanning of the linear subspace, and uses a simple least squares method to solve these parameters. The present disclosure can also utilize a path optimization algorithm to further improve performance by adjusting the actual steps used. Path optimization and parameter solution can be completed in a few minutes, so it is easy to train and deploy.

[0032] Therefore, the present invention proposes a new lightweight adaptive scheduler, namely OLSS (Optimal Linear Subspace Search). This method analyzes the complete generation process, searches in the linear subspace spanned by the intermediate variables of the latent space at each step, and reconstructs an approximate process of the complete generation process, significantly reducing the number of steps and thereby improving computational efficiency.

[0033] To facilitate understanding of the principles of this disclosure, we first introduce the relevant knowledge of diffusion models and summarize the key issues in scheduler design. We then conduct an empirical analysis of the generation process of a general diffusion model. Finally, we propose the scheduler designed in this disclosure based on existing work.

[0034] 1. Introduction to the relevant knowledge of diffusion model

[0035] The diffusion model's theory originates from the diffusion process in physics. Figure 1 illustrates this process. Diffusion is essentially the process of particles moving from areas of high concentration to areas of low concentration. As shown in the figure, pigment molecules are highly dispersed when dropped into water, then begin to move toward areas of low concentration, ultimately achieving uniform diffusion and equilibrium.

[0036] The diffusion model is a generative image model that includes a forward diffusion process and a reverse diffusion process. The diffusion process, also known as the "forward process," refers to the process of gradually adding Gaussian noise to the image. The model is trained to remove noise, so its generative process (also known as the "reverse process") is the inverse of the diffusion process, that is, the process of gradual denoising.

[0037] More specifically, the diffusion model is a latent variable model that is mapped to a latent space using a Markov chain. The latent space is the linear space to which the intermediate variables of the iterative generative process of the diffusion model belong. The denoising diffusion probability model (DDPM) consists of two parameterized Markov chains and uses variational inference to generate samples that match the original data after a finite time. The forward chain (i.e., the "diffusion process") perturbs the data distribution by gradually adding Gaussian noise using a pre-designed schedule until the data distribution converges to a given prior, namely the standard Gaussian distribution. The backward chain (i.e., the "generation process") starts from the given prior and uses a parameterized Gaussian transformation kernel to learn how to gradually restore the undisturbed data structure. Formally, given a data distribution x0~q0, a forward noise process q is defined that gradually adds noise to the data x at each time step t through the Markov chain. i To obtain the posterior probability q(x 1:T |x0), where x1, x2, ..., x T It represents the input data and is also the latent space. In other words, the latent space of the diffusion model has the same dimension as the input data x0.

[0038] Figure 2 shows a schematic diagram of the forward chain adding noise and the reverse chain generating images. As shown in the upper part of the figure, from x0 to the last x TCorresponding to a Markov chain. The transition probability of each step of the Markov chain is essentially adding noise. This is the origin of "diffusion" in the diffusion model: noise gradually enters the diffusion system during the evolution of the Markov chain. As time goes by, the added noise (similar to the pigment molecules added to the water as solutes in Figure 1) becomes less and less, and the noise in the system (all the solutes before this moment) gradually diffuses in the system until it is uniform. In the end, the real image input from x0 is asymptotically transformed into a picture of pure Gaussian noise x T The model training mainly focuses on the reverse diffusion process as shown in the lower part of Figure 2. The goal of training the diffusion model is to learn the reverse process of the forward direction: that is, to train the probability distribution p θ (x t-1 |x t ). By traversing backward along the Markov chain, new data x0 can be regenerated.

[0039] Unlike GAN-based generative models, diffusion-based models require multi-step reasoning. The iterative generation process of diffusion models significantly increases computational time. During the training phase, the number of steps can be very high. For example, a leading model typically uses 1,000 steps during training.

[0040] Joint distribution p θ (x 0:T ) is called the generation process (or reverse process). In the complete generation process, from the random Gaussian noise x T To begin, we need to calculate x step by step T-1 ,...,x0. At each step t, the diffusion model ∈ θ Use x t As input and output e t =∈ θ (x t , t), here, e t is the noise value predicted by the diffusion model. Therefore, x can be obtained by the following formula t-1 :

[0041] Among them, α t and σ t is a hyperparameter used for training. Note that it is an additional random noise σ used to increase the diversity of the generated results. t ∈ t , where ∈ t is a multidimensional random Gaussian noise without weights. In the existing method A for the scheduler, the hyperparameter σ t is set to 0, so that the process is given by x T decision.

[0042] In order to reduce the number of steps, in most existing schedulers, a small number of steps t(1), ..., t(n) are selected as a subsequence of {T, T-1, ..., 0}, and the scheduler only calls the model to calculate e in these n steps. t(i) For example, the existing method A directly transforms formula (1) into an n-step generation process:

[0043] Note that the hyperparameter σ in Equation (1) is t is 0, and the increment direction of i∈{1, 2, ..., n} is opposite to the increment direction of {T, T-1, ..., 0}, so the formula (2) is composed of x t(i) Find x t(i+1) .

[0044] The final latent variable x0 obtained by method A is an approximation of the complete generative process. Another method, B, focuses on modeling the generative process as an ordinary differential equation (ODE). Therefore, the numerical solution of x0 can be calculated using forward Euler (a numerical ODE algorithm):

[0045] Method C is an ODE-based scheduler that uses a linear multi-step method and constructs a pseudo-numerical method. Therefore, the iterative equation can be simplified to:

[0046] in

[0047] In formula (2) of method A and formula (3) of method B, x t(i+1) is {x t(i) , e t(i) In the formula (4) using method C, x t(i+1) is {x t(i) , e t(i) , e t(i-1) , e t(i-2) , e t(i-3) In summary, all of these schedulers satisfy: t(i+1) ∈span{x t(i) , e t(1) ,...,e t(i)}. (7)

[0048] So, recursively, it is easy to prove that: t(i+1) ∈span{x t(1) , e t(1) ,...,e t(i)}. (8)

[0049] That is, xt(i+1) In x t(i) , e t(1) ,...,e t(i) In the linear subspace of span.

[0050] As can be seen from the above, the generation process is the process of spanning a linear subspace. This linear subspace is spanned by the initial Gaussian noise and the previous model output. In each step, the model output is obtained and added to the vector set. The core problem of scheduler design is to determine the coefficients of the iterative equation. The number of non-zero coefficients does not exceed

[0051] 2. Empirical Analysis of the Generation Process

[0052] In order to empirically analyze what happens during the entire generation process, the applicant uses the existing latent diffusion model to generate several images and store latent variables, including x T ,...,x0,e T , ..., e1. Figure 3 shows a heat map of latent variable correlation coefficients that records the complete generation process. According to Figure 3, the following findings are made:

[0053] 1)x t is similar to x at adjacent steps but different from x at non-adjacent steps. As the denoising process progresses, x t Being continuously updated.

[0054] 2)x t With the prediction noise e t The correlation between them is strong at the beginning and becomes weak at the end. This is because x t There is very little noise in the last few steps.

[0055] 3)e T ,...,e1 are quite correlated, indicating that the model output contains redundant and repeated information.

[0056] 3. Build a faster scheduler

[0057] Based on existing schedulers and the above analysis, this paper proposes a new diffusion scheduler, OLSS. In this method, instead of using mathematical theory to design iterative equations, a complete generation process is first run to collect intermediate variables and then an approximate process is constructed using these variables.

[0058] 3.1 Evaluation of Skipped Steps

[0059] Assume that n steps t(1), ..., t(n) have been selected as a subsequence of {T, T-1, ..., 0}, where t(1) = T. The diffusion model is only allowed to be called for inference at these n steps. At step i among the n steps t(1), ..., t(n), the intermediate variable x has been obtained. t(1) ,...,x t(i) and e t(1) ,...,e t(i) .

[0060] Figure 4 shows three methods for generating current latent variables using previous latent variables. The third method, method (c), is the method adopted by the inference acceleration solution proposed in the present invention.

[0061] Specifically, Figure 4 presents the proposed method for generating x t(i+1) The complete generation method is shown in method (a). It requires calling the model t(i+1)-t(i)-1 times. In order to reduce time consumption, the applicant has come up with a simple method. As shown in the analysis based on Figure 3, e T , ..., e1 are quite correlated, so the applicant thought of using a simple linear model trained by the intermediate variables to estimate the model output e t(i)-1 . Formally,

[0062] Among them, the feasible region is ε=span{x t(1) , e t(1) ,...,e t(i)}. (10)

[0063] In other words, you can use e t(i)-1 Orthogonal projection in ε as e t(i)-1 Similarly, we can obtain all the estimated intermediate variables in the skipped steps before t(i+1). The flowchart of this naive method is shown in method (b) in Figure 4.

[0064] Therefore, the present disclosure uses trainable parameters to replace the coefficients in the iterative formula to control the span of the linear subspace, through the coefficient w i,0 , w i,1 ,...,w i,i The solution can realize the reasoning acceleration of the text-image generation task based on the diffusion model.

[0065] Figure 5 shows a schematic flowchart of a diffusion model-based method for accelerating inference for the text-image generation task. In step S510, T steps of the complete generation process are run to collect intermediate variables obtained from the extended model. These intermediate variables include a set of latent intermediate variables and a set of output intermediate variables. In step S520, n steps are selected from the T steps, where n < T. In step S530, an image is generated as the inference result of the text-image generation task through iterative calculation of these n steps.

[0066] In the method shown in FIG5 , the selection of the n steps can be based on a predetermined rule, such as uniform selection, or can be determined by solving an optimization path as described in the following optional embodiments of the present disclosure. Furthermore, the selection of the n steps can be performed before or simultaneously with step S510 of collecting intermediate variables, and the present invention does not limit this.

[0067] In the inference acceleration method implemented by the present invention, the focus is on replacing the coefficients in the iterative formula with trainable parameters as shown in the above formula (11) to control the span of the linear subspace. i,0 , w i,1 ,...,w i,i The solution realizes the reasoning acceleration of the text-image generation task based on the diffusion model. To this end, FIG6 shows a specific operation example of n-step iterative calculation according to an embodiment of the present invention. Specifically, in the iterative calculation of the n steps: in step S610, the linear combination of the previous latent variable and the output intermediate variable is used as the estimated value of the latent variable obtained in the current step; in step S620, the coefficients in the linear combination are obtained by minimizing the mean error between the estimated value and the corresponding latent intermediate variable; and in step S630, based on the obtained coefficients, an image is generated as the reasoning result of the text-image generation task. As described above, steps S610 to S630 can be regarded as a refinement of the operation in step S530. Thus, by converting the design problem of the scheduler into a problem of determining several parameters, a short-step approximate generation process of the diffusion model is constructed, which can greatly reduce the number of reasoning steps of the diffusion model, improve computational efficiency while ensuring generation quality.

[0068] In the step S610 of obtaining the estimated value of the latent variable in the current step, the final latent variable (i.e., x T , at this time t(1)=T) and the output intermediate variable before the current step (ie e t(j) The linear combination of j={1, ..., i}) is used as the estimated value of the latent variable obtained in the current step. At this time, the collected intermediate variables include a set of latent intermediate variables and a set of output intermediate variables corresponding to the n steps. Specifically, the collected intermediate variables can be x t(1) ,...,x t(i)and e t(1) ,...,e t(i) .

[0069] In conventional scheduler-based inference acceleration, although n < T, n still needs to be quite large to ensure the quality of the generated image. In contrast, in the present invention, n can be much smaller than T, that is, n << T. Specifically, n can be smaller than T by at least one order of magnitude, or even two orders of magnitude. For example, in the example shown in Figure 5 below, T = 1000 and n = 5.

[0070] In order to find the parameters of the linear combination, the least square method can be used to easily minimize the mean square error Note that in formula (11) only x t(1) (i.e., x T ) instead of {x t(1) ,...,x t(i)}, because {x T , e t(1) ,...,e t(i)} is a linearly independent set, and {x t(2) ,...,x t(i)} can be linearly represented by these vectors. In addition, linear independence enables the use of the QR (orthogonal triangular) decomposition algorithm to solve the least squares problem. This is faster and more numerically stable than directly calculating the pseudo-inverse matrix. To this end, in an optional embodiment, step S620 may include: using the orthogonal triangular decomposition method to solve the least squares problem to minimize the mean error between the estimated value and the corresponding implicit intermediate variable. Thus, the least squares problem is solved using the QR decomposition algorithm, and w is obtained. i,0 , w i,1 ,...,w i,i value.

[0071] 3.2 Path Optimization

[0072] Another problem that needs to be solved is how to select these n steps t(1), ..., t(n) in step S520. As shown in the analysis based on Figure 3 above, we observe that x t With e t The correlation between is different at each step, indicating that low error x is generated tThe difficulty level also varies. In most existing schedulers, the n steps are selected from {T, T-1, ..., 0} at equal intervals. For example, in method C, equation (5) utilizes a linear multi-step method that requires forced uniform selection of steps. However, in the present invention, there is no restriction on the selection of these n steps. In one embodiment, the n steps can be selected from the T steps at uniform intervals. In another embodiment, an optimization path (e.g., an optimal path) comprising n steps can be searched to generate a high-quality image. In an optional embodiment, selecting the n steps from the T steps by optimizing the path search may include: performing the optimization path search with the error of each of the n steps not exceeding an error threshold (e.g., the following error upper limit D) and the error of each step gradually decreasing as the optimization goal. The search for the optimization path can be obtained based on a binary search algorithm. Similarly, the minimum value of the error threshold when the path exists can also be calculated based on the binary search algorithm. In other words, the value of the error upper limit D can also be determined by the binary search algorithm. To distinguish them, the binary search algorithm that finds the optimal path can be called the "first" binary search algorithm, and the binary search algorithm that minimizes the error threshold can be called the "second" binary search algorithm. Here, "first" and "second" are used only to distinguish different instances of the same object, and do not imply importance or order.

[0073] For ease of understanding, we can use d(t(1),...,t(i+1)) to represent the number of cells from x t(i+1) To its {x t(1) , e t(1) ,...,e t(i)}The distance of the orthogonal projection in the spanned subspace (i.e., Compared to x t(i+1) Here, we can add an extra step t(n+1)=0. Let the upper limit of the error be D, and we hope that the error of each step does not exceed D. On the other hand, we need to reduce the error to generate high-quality images; we also need to ensure that the path ends at 0. Therefore, the step search problem is modeled as the following optimization problem:

[0074] Based on the analysis conclusion of Figure 3, the more steps skipped, the greater the error. Therefore, a simple binary search algorithm can be used to solve the optimization problem represented by Equation (12). Based on this step search algorithm, the present invention designs a path search algorithm. Algorithm 1 shown below gives the pseudo code for finding an error-limited path. If D is large enough, such a path always exists, so another binary search algorithm can be used to calculate the minimum error limit when the path exists. Finally, we obtain the optimized path for the construction generation process.

[0075] 4. Efficiency Analysis

[0076] Here we analyze the time complexity of building an OLSS scheduler. The most time-consuming part is solving the least squares problem. If the path is fixed, the least squares problem to be solved is Second, execution path optimization is times, where ∈ is the absolute error of the optimal D. According to experience, when OLSS is used to improve the efficiency of the diffusion model (for example, a specific diffusion model that is leading in the field can be used in the experiment), after collecting all the required intermediate variables from the complete generation process, it is usually sufficient to calculate the optimal path and solve all parameters {ω i,j}.

[0077] In summary, this paper designs a new scheduler that first uses an existing scheduler to perform a complete generation process, captures and stores the intermediate variables, and then analyzes and reconstructs an approximate process with a shorter number of steps. Specifically, the complete generation process consists of T steps. We use Algorithm I to select n steps from this, and then use Algorithm II to construct the generation process for these n steps.

[0078] First, consider Algorithm II (corresponding to steps S610 to S630 shown in Figure 6). Assuming that n steps have been selected, the model is called only on these n steps for forward reasoning to construct an approximate generation process. A naive approach is to use a linear model to supplement the missing intermediate steps and continue to use the original scheduler to complete them. However, since the applicant discovered that this method can lead to error accumulation, in order to avoid this defect, the present invention further proposes a simpler and more direct method, namely, directly using a linear model to predict the intermediate variables of the next step. The results obtained by these two methods are vectors in the linear subspace spanned by the intermediate variables, so our method can ensure extremely low error.

[0079] Next, consider Algorithm I, which involves selecting n steps from T. We want to minimize the maximum error generated by these n steps, so we devise a heuristic algorithm. Given an upper bound D on the error, the more steps skipped, the greater the error. Based on this monotonicity, a binary search algorithm can be used to determine the maximum skip distance for the next step. Another binary search algorithm can be used to find the n steps with the minimum error by searching for the upper bound D.

[0080] Figure 7 shows examples of image generation based on input text prompts using the inference acceleration method of the present invention and existing models. Here, Models 1 and 2, as well as the OLSS model of the present invention, are diffusion models that generate corresponding images through multiple iterations when receiving a text prompt as input. Figure 7 shows two examples. As shown in the upper portion (a), for example, the user input prompt is "Fantasy magic castle on floating island. A very beautiful art painting." Model 1 can generate an image with theoretically optimal results using a complete generation process without considering cost, as shown in the 1000-step iteration on the far right. While the commonly used Models 1 and 2 perform poorly when n is small, such as when n = 5 as shown in the figure, the OLSS model of the present invention can generate images that are far better than those generated by conventional models at 5 steps, even comparable to the theoretically optimal image on the far right. Similarly, in the second example, as shown in (b) at the bottom of the figure, the user input prompt is, for example, "The leader of the wolves bared its ferocious fangs. High-resolution digital painting." The image generated by Model 1 using the full generation process without cost consideration is also shown here. Similarly, when n = 5, the OLSS model of the present invention can generate an image that is far better than the conventional model at 5 steps, even comparable to the theoretical optimal image on the far right.

[0081] In one embodiment, the present invention can also be implemented as a text-image generation task reasoning acceleration system based on a diffusion model. Figure 8 shows a schematic diagram of the composition of a text-image generation task reasoning acceleration system based on a diffusion model according to an embodiment of the present invention. As shown in the figure, the system 800 includes a storage node 810 and an image generation node 820. The storage node 810 can be used to store the trained diffusion model. The image generation node 820 can be used to perform text-image generation task reasoning. Specifically, the image generation node 820 is used to: load the trained diffusion model; obtain text input from the user (for example, the text prompt shown in (a) or (b) of Figure 7); use the text-image generation task reasoning acceleration method based on the diffusion model as described above to generate an image as the reasoning result of the text-image generation task (for example, the image generated by OLSS through 5 steps of iteration shown in (a) or (b) of Figure 7), and return the image as the reasoning result to the corresponding user.

[0082] FIG9 shows a schematic structural diagram of a computing device that can be used to implement the above-mentioned diffusion model-based text-image generation task reasoning acceleration method according to an embodiment of the present invention.

[0083] 9 , a computing device 900 includes a memory 910 and a processor 920 .

[0084] The processor 920 may be a multi-core processor or may include multiple processors. In some embodiments, the processor 920 may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU) or a digital signal processor (DSP). In some embodiments, the processor 920 may be implemented using a customized circuit, such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA), or a GPU with powerful parallel computing capabilities.

[0085] The memory 910 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 920 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 910 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 910 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0086] The memory 910 stores executable codes. When the executable codes are processed by the processor 920 , the processor 920 can execute the above-mentioned method for accelerating the text-image generation task reasoning based on the diffusion model.

[0087] The method and system for accelerating text-image generation task reasoning based on a diffusion model according to the present invention have been described above in detail with reference to the accompanying drawings.

[0088] Specifically, compared with methods that require training, the OLSS of the present invention requires much less computation (for example, it only takes a few minutes), and therefore has the characteristic of being lightweight; and compared with other schedulers that do not require training, OLSS can directly approximate the complete generation process, with excellent generation effects.

[0089] This paper constructs a short-step approximate generation process for the diffusion model, significantly reducing the number of inference steps, improving computational efficiency while maintaining generation quality. Extensive experimental results demonstrate that our method significantly outperforms other acceleration methods at the same number of steps.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0091] In addition, the method according to the present invention may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing the above steps defined in the above method of the present invention.

[0092] Alternatively, the present invention can also be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) on which executable code (or computer program, or computer instruction code) is stored. When the executable code (or computer program, or computer instruction code) is executed by a processor of an electronic device (or computing device, server, etc.), the processor executes the various steps of the above-mentioned method according to the present invention.

[0093] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0094] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems and methods according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the part of the module, program segment or code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for accelerating reasoning of text-image generation tasks based on a diffusion model, comprising: Running T steps of the complete generation process to collect intermediate variables obtained from the extended model, wherein the intermediate variables include a set of latent intermediate variables and a set of output intermediate variables; Select n steps from the T steps, where n<T; In the iterative calculation of the n steps: The linear combination of the previous latent variable and the output intermediate variable is used as the estimated value of the latent variable obtained in the current step; Obtaining coefficients in the linear combination by minimizing the average error between the estimated value and the corresponding latent intermediate variable; and Based on the obtained coefficients, an image is generated as the inference result of the text-image generation task.

2. The method according to claim 1, wherein: The linear combination of the previous latent variable and the output intermediate variable as the estimated value of the latent variable obtained in the current step includes: A linear combination of the final latent variable and the output intermediate variable before the current step is used as an estimated value of the latent variable obtained in the current step.

3. The method according to claim 1, wherein: The collected intermediate variables include a group of implicit intermediate variables and a group of output intermediate variables corresponding to the n steps.

4. The method according to claim 1, wherein: n<<T.

5. The method according to claim 1, wherein: By minimizing the average error between the estimated value and the corresponding implicit intermediate variable, obtaining the parameters in the linear combination includes: The least squares problem is solved using an orthogonal triangular decomposition method to minimize the average error between the estimated value and the corresponding latent intermediate variable.

6. The method according to claim 1, wherein: The n steps are selected from the T steps at uniform intervals.

7. The method according to claim 1, wherein: The n steps are selected from the T steps by optimizing path search.

8. The method according to claim 7, wherein: Selecting the n steps from the T steps by optimizing the path search includes: The optimization path search is performed with the optimization goal that the error of each step in the n steps does not exceed the error threshold and the error of each step is gradually reduced.

9. The method according to claim 8, wherein: Taking the error of each step in the n steps not exceeding the error threshold and the error of each step gradually decreasing as the optimization goal, the optimization path search includes: The optimized path is obtained based on a first binary search algorithm.

10. The method according to claim 8, wherein: Selecting the n steps from the T steps by optimizing the path search further includes: Based on the second binary search algorithm, the minimum value of the error threshold when a path exists is calculated.

11. A text-image generation task reasoning acceleration system based on a diffusion model, comprising: A storage node, used to store the trained diffusion model; as well as Image generation node for loading the trained diffusion model; Get text input from the user; Using the method according to any one of claims 1 to 10, generating an image as the inference result of the text-image generation task; as well as The image is returned to the corresponding user as an inference result.

12. A computing device comprising: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 10.

13. A computer program product, comprising executable codes, which, when executed by a processor of an electronic device, causes the processor to perform the method according to any one of claims 1 to 10.

14. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • License plate image generation method and system based on adaptive diffusion prior variational auto-encoder

    CN115223158A

  • Diffusion model-based text and graph generation task reasoning acceleration method and system

    CN116863012A

  • Computer-implemented method for generating training data and computer-implemented method for training a computer-implemented control algorithm using such training data

    DE102022004282A1

Cited By

  • Multi-GPU parallel accelerated diffusion model reasoning method and system

    CN121349713A

  • A multi-gpu parallel acceleration diffusion model inference method and system

    CN121349713B