A diffusion model inference acceleration method and system

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a diffusion model reasoning acceleration method and system, which comprises the following steps: firstly, a sampling trajectory alignment strategy based on bias correction is adopted, key routing nodes are identified by analyzing the closeness of continuous routing values and binary threshold values, and a correction term is adaptively applied to effectively eliminate trajectory bias caused by inconsistent routing representation between the offline training stage and the online reasoning stage; secondly, a terminal error compensation mechanism based on a light-weight error compensation network is adopted, the network learns a direct mapping relationship from a sampling result with final error to an error-free sampling result through offline training, and efficient compensation can be completed through only one forward propagation in the reasoning stage, thereby avoiding frequent calculation and performance limitation problems caused by traditional step-by-step local compensation. The application significantly improves the stability and fidelity of the generated results while maintaining a high acceleration ratio.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for accelerating diffusion model inference. Background Technology

[0002] Existing diffusion models (such as U-Net, DiT, and U-ViT) typically require T time steps of iterative sampling (from T to 0) to gradually denoise the initial Gaussian noise and generate a clear image. This process is computationally expensive and time-consuming, making it difficult to meet the efficiency requirements of practical applications. Research has found that intermediate features of the same module in diffusion models often exhibit high redundancy between adjacent time steps. This can be mitigated by caching historical features and reusing them directly in subsequent time steps to reduce redundant computation and accelerate sampling. The key challenge lies in determining the optimal reuse path—that is, for the same module, at which time steps should cached features be reused and at which time steps should recalculation be performed—to maximize acceleration while ensuring that the generated quality is as close as possible to the result of the complete computation path. Current mainstream methods mostly adopt the technical paradigm of "offline training to construct the optimal reuse path and directly guide online inference." Recent research has attempted to introduce error compensation during reuse to reduce single-step reuse error (the difference between the reused value and the recalculated result), but this is still in the early exploratory stage. Research trends are shifting from focusing on acceleration to considering both generation quality and acceleration efficiency.

[0003] Although existing research has made some progress in accelerating diffusion models based on feature caching and reuse, current methods still have the following key defects and shortcomings:

[0004] 1. Sampling trajectory deviation affects sampling quality: Existing methods suffer from routing value issues between the offline training and online inference phases. The inconsistency in format leads to a deviation in the sampling trajectories between the two, resulting in a decrease in the sampling quality of online inference.

[0005] 2. Frequent compensation consumes a lot of time: Existing methods use single-step multiplexing error compensation, which requires an error compensation operation every time multiplexing is completed, accumulating a large amount of time consumption.

[0006] 3. Indirect error compensation effect is not obvious: Existing methods indirectly compensate for the final overall error by compensating for local errors within the time step. However, the sampling process is complex and variable, and intermediate operations can affect the superposition of local error compensation benefits, resulting in an insignificant final error compensation effect.

[0007] 4. Error contamination limits compensation capability: Existing compensation parameter training methods based on the distillation paradigm require the output of the student model in the previous time step as input to maintain the relative consistency between the teacher model and the student model and the stability of the training process. However, the teacher model is easily contaminated by the error of the student model, resulting in limited error compensation capability. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for accelerating diffusion model inference, in order to solve the technical problems of sampling trajectory deviation caused by the inconsistency of routing representation between the offline training stage and the online inference stage, as well as the low efficiency and limited effectiveness of traditional single-step error compensation methods in accelerating diffusion model inference, thereby significantly improving the inference speed of diffusion models while ensuring the quality of generation.

[0009] The present invention achieves the above objectives through the following technical solutions:

[0010] Firstly, the present invention proposes a method for accelerating inference in a diffusion model, the method comprising:

[0011] Obtain the offline training results of the diffusion model, including continuous route values ​​and the optimal reuse path obtained by discretizing the continuous route values ​​through binarization thresholding;

[0012] During the online inference phase of the model, key routing nodes are identified based on the proximity of continuous routing values ​​to the binarization threshold.

[0013] Based on the difference between the continuous routing values ​​and the cached historical features, a deviation correction term is applied to the key routing nodes to align the sampling trajectory and output the intermediate sampling results;

[0014] The intermediate sampling results are input into a pre-trained lightweight error compensation network, which performs a single forward propagation and outputs the final sampling results after error compensation.

[0015] The lightweight error compensation network learns the mapping relationship through offline training. The training data for the mapping relationship includes: based on the same initial noise, accelerated sampling based on the optimal reuse path and the bias correction method and complete sampling without feature reuse are used to obtain paired results with error and results without error.

[0016] Furthermore, obtaining the offline training results of the diffusion model specifically includes:

[0017] During the offline training phase, the routing values ​​of each reusable computing module in the diffusion model are used. The routing values ​​are used for training to optimize the target. Indicates the reuse ratio of the corresponding module at the current time step t;

[0018] The output features of module k at time step t are constructed using an interpolation continuity method. , ;in, , The output of the previous module k+1 at the current time step t. The result is obtained by inference calculation using the current time step t as input and the diffusion model. Let this be the cached output feature of module k at the previous time step t+1. Symbolic representation of computational operations for diffusion model inference;

[0019] The loss function used for training includes an error loss term and a time loss term. The error loss term is used to constrain the difference between the output under route guidance and the output of the complete computation path. The time loss term is positively correlated with the sum of all route values.

[0020] After training is complete, the learned continuous routing values ​​will be... The optimal reuse path is obtained by binarizing the data based on a preset threshold. The binarized route value indicates the feature reuse performed by each module at each time step during the online inference phase. Or recalculate .

[0021] Furthermore, the step of identifying key routing nodes based on the proximity of consecutive routing values ​​to a binarization threshold specifically includes:

[0022] Calculate continuous route values With binarization threshold The absolute difference;

[0023] If the absolute difference is less than the preset deviation threshold If so, the corresponding routing node is determined to be a critical routing node.

[0024] Furthermore, based on the difference between the continuous routing values ​​and the cached historical characteristics, a deviation correction term is applied to the critical routing nodes, specifically including:

[0025] when The deviation correction term is applied using the following formula:

[0026] ;

[0027] when The deviation correction term is applied using the following formula:

[0028] ;

[0029] in, The decay coefficient is related to the number of time backtracking steps. ; This refers to the feature difference value obtained from the historical difference cache queue during the most recent recalculation of the module containing the critical routing node. b is the time step backtracking step number, used to indicate the time step interval from the most recent recalculation to the current node.

[0030] Furthermore, the method also includes:

[0031] Maintain feature cache queue and historical difference cache queue;

[0032] The feature cache queue is used to cache the output of each module at the previous time step. And update at each time step;

[0033] The historical difference cache queue is used for... The module corresponding to the key routing node dynamically stores and updates the feature difference D obtained during its most recent recalculation.

[0034] After these modules perform recalculation, the historical difference cache queue will use the feature difference generated in this calculation to update the cache record corresponding to that module.

[0035] Furthermore, the lightweight error compensation network is trained offline through the following steps:

[0036] Based on the same initial noise, the following two sampling processes are performed in parallel: sampling is performed based on the optimal multiplexing path and in conjunction with the deviation correction term to obtain sampling results with errors. Sampling is performed using the complete computation path without any feature reuse, resulting in error-free sampling results. ,Will As a pair of training samples, and generate a dataset containing multiple pairs of said training samples. P is the number of datasets;

[0037] With dataset In As input, with the corresponding As a supervision objective, this is achieved by minimizing the loss function. To optimize the weight parameters of the lightweight error compensation network; wherein, This represents the forward computation of the lightweight error compensation network.

[0038] Furthermore, the lightweight error compensation network adopts any one of the following: multilayer perceptron, simplified U-Net architecture, or lightweight self-attention Transformer architecture.

[0039] Furthermore, the training data specifically includes:

[0040] The optimal reuse path is used and the sampling trajectory deviation correction method is enabled for sampling to obtain a sampling result with final error as the input sample.

[0041] Sampling is performed using a complete computation path without any feature reuse, resulting in error-free sampling as the target sample;

[0042] The training data of the lightweight error compensation network consists of multiple pairs of input samples and target samples as training samples.

[0043] Secondly, the present invention proposes a diffusion model inference acceleration system for implementing the steps of the diffusion model inference acceleration method described above. The system includes:

[0044] The path acquisition module is used to acquire the offline training results of the diffusion model, including continuous route values ​​and the optimal reuse path obtained by discretizing the continuous route values ​​through binarization thresholding.

[0045] The key node identification module is used to identify key routing nodes during the online inference phase of the model based on the proximity of continuous routing values ​​to a binarization threshold.

[0046] The trajectory alignment module is used to apply a deviation correction term to the key routing nodes based on the continuous routing values ​​and the difference of cached historical features, so as to align the sampling trajectory and output the intermediate sampling results.

[0047] The error compensation module includes a pre-trained lightweight error compensation network, which is used to input the intermediate sampling results into the pre-trained lightweight error compensation network, and the network performs a single forward propagation to output the final sampling results after error compensation.

[0048] The lightweight error compensation network learns the mapping relationship through offline training. The training data for the mapping relationship includes: based on the same initial noise, accelerated sampling based on the optimal reuse path and the bias correction method and complete sampling without feature reuse are used to obtain paired results with error and results without error.

[0049] Furthermore, it also includes a cache management module for maintaining feature cache queues and historical difference cache queues, and supports the calculation of deviation correction terms for the trajectory alignment module.

[0050] The beneficial effects of this invention are as follows:

[0051] 1. Regarding trajectory alignment, this invention proposes a selective alignment strategy for key nodes based on deviation correction. This is achieved by analyzing continuous routing values. By assessing the proximity to the binarization threshold θ, the key routing nodes causing sampling trajectory deviations are accurately identified, and based on... The bias correction term is adaptively applied. This design not only effectively bridges the path difference between offline training and online inference and significantly improves the stability of sampling quality, but also achieves maximum performance gains with minimal computational overhead through a selective correction mechanism.

[0052] 2. Regarding error control, this invention abandons the traditional stepwise local error compensation paradigm and instead introduces a lightweight error compensation network for one-time final error compensation. This network learns the mapping relationship from sampling results with accumulated errors to error-free sampling results through offline training, and compensation can be completed in a single forward propagation during the inference phase. This design avoids the problem of the teacher model being contaminated by student model errors in traditional distillation training, and significantly reduces the time consumption caused by frequent compensation, making the error compensation effect more direct and efficient. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating a diffusion model inference acceleration method in this invention;

[0054] Figure 2 This is a comparison diagram of the output methods of the single time step module in offline training and online inference in this invention;

[0055] Figure 3 The routing value in this invention Distribution diagram;

[0056] Figure 4 This is a schematic diagram of the offline training dataset construction method for the lightweight error compensation network in this invention;

[0057] Figure 5 This is a framework diagram of online reasoning in this invention;

[0058] Figure 6 This is another flowchart illustrating the diffusion model inference acceleration method of the present invention. Detailed Implementation

[0059] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0060] To accurately describe the technical solution of this disclosure, the following core feature terms will be used in the description of specific embodiments:

[0061] Optimal reuse path: This refers to a set of discrete decision sequences obtained through optimization learning during offline training. These sequences guide the various computational modules in the model (such as the attention module (ATTN) and the feedforward network module (MLP)) during online inference. The decision is whether to "reuse" the cached output from the previous time step or "recalculate" the current output at different time steps. This path is obtained by binarizing continuous routing values.

[0062] Continuous route value During offline training, to perform differentiable optimization, discrete reuse / recomputation decisions are transformed into continuous weight coefficients. For any module's output at a specific time step... , Values ​​between 0 and 1 are used in the weighted fusion formula. In the middle, adjust "Recalculate Results". With "reused cache feature" The proportion of contribution.

[0063] Critical routing nodes: refer to those consecutive routing values With binarization threshold Very close routing decision points. These nodes are key locations that cause significant deviations between the online inference path and the offline training path, and are the core objective of the trajectory alignment operation in this disclosure.

[0064] Bias Correction Term: A compensation amount calculated and applied to compensate for the bias introduced by path switching (from weighted fusion to binary decision-making) at the aforementioned key nodes. This correction term is based on... The size of the value (greater than or less than) Different formulas are used for calculation, with the aim of making the module output during online inference as close as possible to the weighted fusion output during offline training, thereby achieving sampling trajectory alignment.

[0065] Lightweight Error Compensation Network: This disclosure introduces a small neural network module for one-time, direct compensation of the final error accumulated throughout the accelerated sampling process in the final stage of online inference. Its training data consists of paired results obtained from the same noise source through both a bias-corrected accelerated path and a complete standard path, thereby learning the mapping relationship from erroneous images to high-quality images.

[0066] Example 1

[0067] Please see Figure 1 , Figure 2 ,and Figure 6 This disclosure proposes a method for accelerating diffusion model inference in a specific embodiment, the method comprising the following steps:

[0068] Before initiating the entire methodology, necessary data and model preparation is required. Specifically, a random noise dataset for offline training needs to be prepared, such as 1000 samples of pure Gaussian noise from a standard normal distribution as training samples, with each training batch randomly assigned a class label. Simultaneously, the pre-trained weight file and corresponding architecture code of the target diffusion model need to be downloaded and loaded; in this embodiment, the DiT-XL-2 model is preferably used as the base. These preparations provide the necessary inputs and basic models for subsequently building the reuse path and training the error compensation network.

[0069] S1. Obtain the offline training results of the diffusion model, including continuous route values ​​and the optimal reuse path obtained by discretizing the continuous route values ​​through binarization thresholding.

[0070] As a preferred approach, step S1 involves obtaining the offline training results of the diffusion model, specifically including:

[0071] S1.1 During the offline training phase, the routing values ​​of each reusable computation module in the diffusion model are used. To optimize the training process, the routing values ​​are... Indicates the reuse ratio of the corresponding module at the current time step t;

[0072] S1.2 Constructing the output features of module k at time step t using the interpolation continuity method. , ;in, , The output of the previous module k+1 at the current time step t. The result is obtained by inference calculation using the current time step t as input and the diffusion model. Let be the cached output features of module k at the previous time step t+1; where Symbolic representations of computational operations for diffusion model inference, for example, when module k is an attn module. This indicates the operation of calculating the attention score.

[0073] S1.3 The loss function used for training includes an error loss term. and time loss items Error loss term Used to constrain routing values The difference between the output of a guided model and the output of a model using the complete computation path (i.e., without any reuse) is typically calculated using mean squared error (MSE), with the core objective of ensuring sampling quality. Time loss term. With all route values The sum is positively correlated.

[0074] Specifically, the error loss term The calculation method is as follows:

[0075] ;

[0076] in, This represents the output of the model at time step t, guided by route r. This represents the output of the model at time step t, guided by the complete computational path. Responsible for controlling sampling quality during training. Time loss. The calculation method is as follows:

[0077] ;

[0078] in, During training, the person in charge is responsible for managing the time consumption. The total loss consists of the two losses mentioned above:

[0079] ;

[0080] Where the coefficient γ>0, the continuous route r needs to be updated in reverse according to the loss.

[0081] S1.4 After training is complete, the learned continuous routing values ​​will be... Based on preset threshold Binarization is performed to obtain the optimal reuse path W. Specifically, for each... Value, if If it is, then set it to 1, indicating that the corresponding module should "recalculate" at the corresponding time step; if If the condition is not met, it is set to 0, indicating that the corresponding module should "directly reuse" the cached feature at the corresponding time step. The continuous route r and the optimal reuse path W will be stored to guide the subsequent online inference process.

[0082] In one specific implementation, the process of constructing the optimal reuse path W through offline training is as follows:

[0083] 1000 samples of clean Gaussian noise were randomly sampled as the dataset for offline training of the optimal reuse path W. For each batch, a single class label was randomly sampled. The DiT model code and corresponding weight file (DiT-XL-2) were downloaded. The optimized target route r contains T*56 route values. k = 1, 2, 3...56, t = 1, 2, 3...T, where T is the total number of time steps and 56 represents the total number of reused targets. Continuous routes r, i.e., route values, are initialized by random sampling from a standard normal distribution. , Using the interpolation continuity method, the output of module k at time step t is defined as... :

[0084] ;

[0085] in, This represents the output of the previous module k+1 at the current time step t. The result is obtained by inference calculation using the current time step t as input and the diffusion model. This represents the cached output features of module k at the previous time step t+1. After training is complete, the training results are first continuously routed r and stored in disk space for later use, then a fixed threshold is applied. Binarize the routing value β in the continuous route r into a discrete value.

[0086] ;

[0087] The final discrete route r is the optimal multiplexing path W, which is then stored in disk space for later use.

[0088] S2, please refer to Figure 5 During the online inference phase of the model, key routing nodes are identified based on the proximity of continuous routing values ​​to the binarization threshold.

[0089] As a preferred approach, key routing nodes are identified based on the proximity of consecutive routing values ​​to a binarization threshold. Specifically, this includes calculating consecutive routing values. With binarization threshold The absolute difference; if the absolute difference is less than the preset deviation threshold. If so, the corresponding routing node is determined to be a critical routing node.

[0090] Understandably, this is because when Value far away When the difference between the output before and after binarization is small, while when... When the value is close to θ, the weighted fusion output differs significantly from the output of the binarization decision, which is the main source of trajectory deviation. The purpose of identifying key nodes is to achieve selective correction, maximizing trajectory alignment benefits with minimal computational overhead.

[0091] S3. Based on the difference between continuous routing values ​​and cached historical features, apply deviation correction terms to key routing nodes to align the sampling trajectory and output intermediate sampling results.

[0092] Combination Figure 3 During the offline training phase, the module output is generated by interpolating the continuous routing values ​​β using the formula... Construction. During the online runtime phase, based on the binarized discrete routing values, the output construction method is divided into two types: during recalculation... When reused Analysis revealed that when β is close to 0 or 1, the deviation between the outputs of the two stages is small; however, when β approaches the threshold of 0.5, the deviation increases significantly. Therefore, this disclosure will... Nodes with a value of δ=0.15 are defined as critical nodes. During the online inference phase, bias correction is only performed on critical nodes to achieve maximum performance improvement with minimal computational overhead. Figure 3 In this process, approximately 15% of the nodes are critical nodes that require deviation correction.

[0093] As a preferred approach, a deviation correction term is applied to key routing nodes based on the difference between continuous routing values ​​and cached historical characteristics. Specifically, this includes:

[0094] when Binarization to 1 means that the online reasoning decision is "recalculated," and a bias correction term is applied using the following formula:

[0095] ;

[0096] After applying this correction, the output of the online inference is adjusted to approximate the weighted fusion result of the offline training.

[0097] when Binarization to 0 indicates that online reasoning and decision-making are "direct reuse," and a bias correction term is applied using the following formula:

[0098] ;

[0099] in, The decay coefficient is related to the number of time backtracking steps. ; This refers to the feature difference value obtained from the historical difference cache queue during the most recent recalculation of the module corresponding to this critical routing node. 'b' represents the time step backtracking step number, used to indicate the time step interval since the most recent recalculation from the current node. This number cannot be directly obtained under reuse decisions. This disclosure uses a historical approximation D supplemented by an attenuation coefficient. To perform approximate compensation. Coefficients This ensures that the longer the time interval between historical feature differences, the lower their contribution weight to the current correction.

[0100] In this disclosure, the historical difference is used from the most recent recalculation performed by the module containing the current node. Approximate the difference that cannot be directly obtained from the current node. Where b represents the time step backtracking step, used to indicate the time step interval between the most recent feature change (recalculation) and the current node. During online inference, a historical data cache queue S is maintained to cache and reuse the relevant difference data of the module containing the key node (i.e., the key node β≤θ) after the most recent recalculation. By traversing consecutive routes r, it can be determined which modules need to cache the relevant difference data. Using (1-0.1b)( To approximate () To construct the bias correction term, (1-0.1b) is used. Here, (1-0.1b) is the attenuation coefficient, and the compensation strength needs to decrease as the number of backtracking steps increases. The smaller b is, the smaller the time step spacing for the approximate substitution, the higher the similarity of the difference used for the approximate substitution, and the better the effect of the approximate substitution. Therefore, the best case is b=0, i.e., using ( To approximate () This is used to construct the deviation correction term. The final output of this node... .

[0101] By using the above-mentioned sampling trajectory alignment method based on deviation correction, selective deviation correction is performed on key nodes with large deviations in the online inference stage. This effectively bridges the sampling trajectory deviation caused by inconsistent routing representations, thereby more accurately reproducing the preset optimal sampling effect in the offline training stage and improving sampling quality and stability.

[0102] In this disclosure, the method also includes maintaining two cache queues:

[0103] Feature cache queue: Used to cache the output of all modules at the previous time step. It is updated once at each time step, providing data for reuse and correction of calculations.

[0104] Historical difference cache queue: Dedicated to dynamically storing and updating the feature difference D generated during the most recent recalculation of the module containing the critical routing node that performs reuse. Whenever these modules perform a recalculation operation, the system calculates the current difference. - This data is then used to update the corresponding record for that module in queue S. This mechanism ensures that the data (D) used for approximate compensation is always the latest feature change information for that module.

[0105] In one specific implementation, the process of sampling trajectory alignment based on deviation correction is as follows:

[0106] like Figure 5As shown, the offline training constructs the optimal reuse path W, which is partially stored on disk as a continuous route r=[0.45,0.12,0.89...0.97,0.34,0.74] and an optimal reuse path W=[0,0,1...1,0,1]. A random sample of pure Gaussian noise is then used. A random category label is sampled, and diffusion model inference sampling is performed for T time steps guided by the optimal reuse path W. The optimal reuse path W contains routing values ​​of T*56 nodes, used to indicate whether each node should be recalculated or reused directly during the sampling process. Let the output of the current node be... t=1,2,3...T, where T is the total number of sampling time steps. A buffer queue L is maintained to buffer the output of all modules from the previous time step. The queue L is updated once at each time step. The loaded consecutive routes r are traversed to determine the location index and module of the reuse critical node (critical node β≤θ). A cache queue S is maintained to cache the relevant difference data from the most recent recalculation of the module containing the reuse critical node. Each time these modules are recalculated, the queue S is updated. During online inference, when processing node i (corresponding to module k at time step t), the continuous route r is traversed, and the sampling trajectory deviation is corrected based on the continuous route r. Then, sampling trajectory deviation correction needs to be performed on node i. If If the value is greater than 0.5, then a recalculation will be performed first, and then... As the output of the model at node i. If Take module k from queue S ,Will As the output of the model at node i. If Then the repetitive calculations during the online inference phase will proceed normally. Or reuse No sampling trajectory deviation correction is performed. This process is repeated until the sampling task is completed for T time steps, yielding the sampling result X. 0 .

[0107] S4. Input the intermediate sampling results into the pre-trained lightweight error compensation network, which performs a single forward propagation and outputs the final sampling results after error compensation.

[0108] The lightweight error compensation network learns mapping relationships through offline training. The training data for these mapping relationships includes pairs of results with and without errors, obtained by accelerating sampling based on the optimal reuse path and bias correction method, and by performing full sampling without feature reuse, all based on the same initial noise. These two sets of results are strictly aligned and constitute training pairs.

[0109] As a preferred option, the training of the lightweight error compensation network in step S4 includes:

[0110] Combination Figure 4 Based on the same initial noise, the following two sampling processes are performed in parallel: sampling is performed based on the optimal multiplexing path and in conjunction with the deviation correction term to obtain sampling results with errors. Sampling is performed using the complete computation path without any feature reuse, resulting in error-free sampling results. ,Will As a pair of training samples, and generate a dataset containing multiple pairs of said training samples. The number of datasets, P, can be set as needed, for example, several thousand pairs.

[0111] With dataset In As input, with the corresponding As a supervision objective, this is achieved by minimizing the loss function. To optimize the weight parameters of the lightweight error compensation network; wherein, This represents the forward computation of the lightweight error compensation network.

[0112] As a preferred option, the lightweight error compensation network adopts any one of the following architectures: multilayer perceptron, simplified U-Net architecture, or lightweight self-attention Transformer architecture. Their common characteristics are small parameter count and fast forward propagation speed, thus ensuring that they do not become a bottleneck for online inference as a post-processing module.

[0113] As a preferred approach, obtaining the offline training results of the diffusion model also includes the dataset for constructing a lightweight error compensation network during the offline training phase. This step is performed after S1.4, and based on the same randomly sampled initial noise, pairs of training samples are generated through the following two paths:

[0114] Sampling is performed using the optimal reuse path and with the sampling trajectory deviation correction method enabled, and the sampling result with final error is used as the input sample; sampling is performed using the complete computation path without any feature reuse, and the sampling result without error is used as the target sample; the training dataset of the lightweight error compensation network is composed of multiple pairs of input samples and target samples as training samples.

[0115] In one specific implementation, the process of training the lightweight error compensation network is as follows:

[0116] Randomly sample a pure Gaussian noise The sampled path is obtained by M using the optimal multiplexing path W and enabling the sampling trajectory alignment method. Sampled using the complete computation path without any acceleration (i.e., no reuse, no trajectory deviation correction). , and A data pair that constitutes strict alignment Repeat the generation process P times to generate a lightweight error-compensated network training dataset containing P data pairs. During training, the lightweight error compensation network uses... As input, after T time steps of inference computation, the result is obtained. The result y after error compensation is determined according to the loss function. Adjust and optimize the objective (weight parameters of the lightweight error compensation network), randomly sample class labels once per batch, and repeat the training for E epochs.

[0117] Lightweight error compensation networks learn from large amounts of pairwise data. It implicitly models the mapping function from an "error-laden image" to an "error-free image." Its lightweight design ensures that the computational cost of this mapping process is extremely low, preventing it from becoming a bottleneck for online inference. The lightweight error compensation network can be, but is not limited to, any of the following: an MLP, a simplified U-Net architecture, or a lightweight self-attention-based Transformer architecture. Their common characteristics are small parameter count, fast processing speed, and direct output of the compensation result y via forward propagation.

[0118] After generating the final result through online inference, the performance of the method needs to be quantitatively evaluated. The evaluation mainly includes two aspects: generation quality and acceleration effect. Quality evaluation typically uses the Fréchet Inception Distance (FID) metric, which compares the distribution difference between the final sampled result y generated by the method of this disclosure and the baseline result y* generated using the standard, unaccelerated full computation path under the same initial noise and conditions. Acceleration effect is measured by the computational speedup ratio, i.e., the ratio of the sampling time t* of the full computation path to the overall online inference time t of the method of this disclosure. 0 The speedup effect of the proposed method is evaluated by the ratio of (including trajectory alignment sampling and error compensation).

[0119] According to the above embodiments, the diffusion model inference acceleration method provided in this embodiment solves the problem of inconsistent offline / online paths by using a sampling trajectory alignment strategy based on deviation correction, and solves the problem of low efficiency and poor effect of local compensation by using a one-time final error compensation mechanism based on lightweight networks. Thus, while significantly improving the inference speed, it effectively ensures the quality of generated samples.

[0120] The diffusion model inference acceleration method disclosed herein has a wide range of applications, including real-time image generation and interactive creation systems, large-scale content generation platforms, video generation and temporal media processing, cloud-based diffusion model services, performance optimization of research and development frameworks, and interactive media and immersive experience applications. It can effectively meet the practical application requirements for balancing generation efficiency and quality.

[0121] Example 2

[0122] Another specific embodiment of this disclosure proposes a diffusion model inference acceleration system for implementing the steps of the diffusion model inference acceleration method in Embodiment 1. The system includes a path acquisition module, a key node identification module, a trajectory alignment module, and an error compensation module. The path acquisition module is used to acquire the offline training results of the diffusion model, including continuous routing values ​​and the optimal reuse path obtained by discretizing the continuous routing values ​​through a binarization threshold. The key node identification module is used to identify key routing nodes during the online inference phase of the model based on the proximity of the continuous routing values ​​to the binarization threshold. The trajectory alignment module is used to identify key routing nodes based on the continuous routing values ​​and cached historical data. The difference value is used to apply a deviation correction term to key routing nodes to align the sampling trajectory and output intermediate sampling results. The error compensation module includes a pre-trained lightweight error compensation network, which is used to input the intermediate sampling results into the pre-trained lightweight error compensation network. The network performs a single forward propagation and outputs the final sampling result after error compensation. The lightweight error compensation network learns the mapping relationship through offline training. The training data of the mapping relationship includes: based on the same initial noise, pairs of error-bearing results and error-free results are obtained by accelerating sampling based on the optimal multiplexing path and full sampling without feature reuse.

[0123] The system also includes a cache management module, which maintains the feature cache queue and the historical difference cache queue, and supports the calculation of the deviation correction term of the trajectory alignment module.

[0124] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0126] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for accelerating inference in a diffusion model, characterized in that, The method includes: Obtain the offline training results of the diffusion model, including continuous route values ​​and the optimal reuse path obtained by discretizing the continuous route values ​​through binarization thresholding; During the offline training phase, the routing value β of each reusable computation module in the diffusion model is used as the optimization target for training. The routing value β indicates the reuse ratio of the corresponding module at the current time step t. After training, the learned continuous routing value β is binarized based on a preset threshold θ to obtain the optimal reuse path W. The binarized routing value indicates that each module performs feature reuse or recalculation at each time step. During the online inference phase of the model, key routing nodes are identified based on the proximity of continuous routing values ​​to the binarization threshold. Based on the difference between the continuous routing values ​​and the cached historical features, a deviation correction term is applied to the key routing nodes to align the sampling trajectory and output the intermediate sampling results; The intermediate sampling results are input into a pre-trained lightweight error compensation network, which performs a single forward propagation and outputs the final sampling results after error compensation. The lightweight error compensation network learns the mapping relationship through offline training. The training data for the mapping relationship includes: based on the same initial noise, accelerated sampling based on the optimal reuse path and the bias correction method and complete sampling without feature reuse are used to obtain paired results with error and results without error.

2. The diffusion model inference acceleration method according to claim 1, characterized in that, The process of obtaining the offline training results of the diffusion model also includes: The output features of module k at time step t are constructed using an interpolation continuity method. , ;in, , The output of the previous module k+1 at the current time step t. The result is obtained by inference calculation using the current time step t as input and the diffusion model. The cached output characteristics of module k at the previous time step t+1; Symbolic representation of computational operations for diffusion model inference; The loss function used for training includes an error loss term and a time loss term. The error loss term is used to constrain the difference between the output under route guidance and the output of the complete computation path, and the time loss term is positively correlated with the sum of all route values.

3. The diffusion model inference acceleration method according to claim 2, characterized in that, The identification of key routing nodes based on the proximity of consecutive routing values ​​to a binarization threshold specifically includes: Calculate continuous route values With binarization threshold The absolute difference; If the absolute difference is less than the preset deviation threshold If so, the corresponding routing node is determined to be a critical routing node.

4. The diffusion model inference acceleration method according to claim 3, characterized in that, Based on the difference between the continuous routing values ​​and the cached historical characteristics, a deviation correction term is applied to the critical routing nodes, specifically including: when The deviation correction term is applied using the following formula: ； when The deviation correction term is applied using the following formula: ； in, The decay coefficient is related to the number of time backtracking steps. ; For retrieved from the historical difference cache queue The feature difference during the most recent recalculation of the module containing the critical routing node. , b This represents the number of time steps back, used to indicate the time step interval since the most recent recalculation from the current node.

5. The diffusion model inference acceleration method according to claim 4, characterized in that, The method further includes: Maintain feature cache queue and historical difference cache queue; The feature cache queue is used to cache the output of each module at the previous time step. And update at each time step; The historical difference cache queue is used for... The module corresponding to the key routing node dynamically stores and updates the feature difference D obtained during its most recent recalculation. After these modules perform recalculation, the historical difference cache queue will use the feature difference generated in this calculation to update the cache record corresponding to that module.

6. The diffusion model inference acceleration method according to claim 1, characterized in that, The lightweight error compensation network is trained offline through the following steps: Based on the same initial noise, the following two sampling processes are performed in parallel: sampling is performed based on the optimal multiplexing path and in conjunction with the deviation correction term to obtain sampling results with errors. Sampling is performed using the complete computation path without any feature reuse, resulting in error-free sampling results. ,Will As a pair of training samples, and generate a dataset containing multiple pairs of said training samples. P is the number of datasets; With dataset In As input, with the corresponding As a supervision objective, this is achieved by minimizing the loss function. To optimize the weight parameters of the lightweight error compensation network; wherein, This represents the forward computation of the lightweight error compensation network.

7. The diffusion model inference acceleration method according to claim 6, characterized in that, The lightweight error compensation network adopts any one of the following: multilayer perceptron, simplified U-Net architecture, or lightweight self-attention Transformer architecture.

8. The diffusion model inference acceleration method according to claim 1, characterized in that, The training data specifically includes: The optimal reuse path is used and the sampling trajectory deviation correction method is enabled for sampling to obtain a sampling result with final error as the input sample. Sampling is performed using a complete computation path without any feature reuse, resulting in error-free sampling as the target sample; The training data of the lightweight error compensation network consists of multiple pairs of input samples and target samples as training samples.

9. A diffusion model inference acceleration system, used to implement the steps of the diffusion model inference acceleration method as described in any one of claims 1-8, characterized in that, The system includes: The path acquisition module is used to acquire the offline training results of the diffusion model, including continuous route values ​​and the optimal reuse path obtained by discretizing the continuous route values ​​through binarization thresholding. The key node identification module is used to identify key routing nodes during the online inference phase of the model based on the proximity of continuous routing values ​​to a binarization threshold. The trajectory alignment module is used to apply a deviation correction term to the key routing nodes based on the continuous routing values ​​and the difference of cached historical features, so as to align the sampling trajectory and output the intermediate sampling results. The error compensation module includes a pre-trained lightweight error compensation network, which is used to input the intermediate sampling results into the pre-trained lightweight error compensation network, and the network performs a single forward propagation to output the final sampling results after error compensation. The lightweight error compensation network learns the mapping relationship through offline training. The training data for the mapping relationship is generated in the following way: based on the same initial noise, pairs of error-bearing results and error-free results are obtained by accelerating sampling based on the optimal reuse path and the bias correction method and by full sampling without feature reuse.

10. The diffusion model inference acceleration system according to claim 9, characterized in that, It also includes a cache management module for maintaining feature cache queues and historical difference cache queues, and supports the calculation of deviation correction terms for the trajectory alignment module.

Images