Diffusion transformer model inference acceleration method and system based on multi-granularity pruning

Abstract

The application provides a diffusion Transformer model inference acceleration method and system based on multi-granularity pruning, which comprises the following steps: based on a first correlation feature between a time step pruning error and an input prompt, a time step set for cache multiplexing is determined through offline analysis, and time steps not belonging to the time step set are pruned in the process of online inference; for the time steps not pruned, based on a second correlation feature between a layer pruning error and the input prompt, an online adaptive method is used to evaluate the importance of each Transformer layer in the current inference context, and the Transformer layers pruned in the current time step are dynamically determined. The method designs a multi-granularity pruning architecture of time step granularity and layer granularity based on the correlation features found through experiments, does not need to consume additional computing power for training or fine-tuning, and can realize efficient diffusion Transformer model inference.

Description

Technical Field

[0001] This invention relates to the field of diffusion Transformer model inference technology, and in particular to a method and system for accelerating diffusion Transformer model inference based on multi-granularity pruning. Background Technology

[0002] In recent years, multimodal artificial intelligence technology has continued to develop, with related research making continuous progress in areas such as computer vision, natural language processing, and cross-modal generation. Among these, generation methods based on diffusion models have been widely applied to tasks such as image generation, video generation and editing, and audio generation due to their good modeling stability and generation consistency. In text-guided image and video generation scenarios (Text to Image, Text to Video), diffusion models have become one of the mainstream technical approaches, demonstrating significant advantages in terms of generation quality and diversity. Building on this foundation, the Diffusion Transformer (DiT) model combines the diffusion model framework with the Transformer architecture, employing a sequence modeling approach based on a self-attention mechanism to replace the U-Net-style encoder-decoder structure commonly used in traditional diffusion models. This type of model has certain advantages in terms of model expressiveness, scalability, and multimodal conditional modeling, providing a new structural option for high-resolution image generation and long-sequence video generation tasks.

[0003] However, the global self-attention mechanism introduced in the Transformer architecture introduces significant computational overhead during the inference phase. Specifically, the time and space complexity of self-attention computation typically increases quadratically with the length of the input sequence. In applications such as high-resolution image generation or long-sequence video generation, input features often need to be flattened into longer sequences, leading to a significant increase in attention computation costs and limiting model inference efficiency. Therefore, in practical applications, attention computation has gradually become one of the main performance bottlenecks in the inference process of diffusion Transformer models. With the gradual application of generative artificial intelligence technology in fields such as industrial content creation, virtual reality, film and television production, intelligent manufacturing, digital human systems, and large-scale model engineering deployment, how to effectively reduce the inference cost and improve the inference efficiency of diffusion models while ensuring generation quality has become a critical technical problem that needs to be solved in the engineering implementation and large-scale application of related technologies.

[0004] Because videos are unfolded into one-dimensional token sequences in both spatial and temporal dimensions, the sequence length increases dramatically with resolution and duration. Meanwhile, the Diffusion Transformer model widely employs 3D full-attention, calculating self-attention in the temporal, height, and width dimensions. Its computational complexity is quadratic with the token sequence length *n*, i.e., O(n^2), making attention computation a major bottleneck in image and video inference. When generating a 720×1280 resolution, 5-second video in HunyuanVideo, the model processed over 120,000 contexts at once, with the latency of self-attention computation accounting for the majority of the inference time. With further increases in resolution or video frame rate, the sequence length and attention computation will increase dramatically.

[0005] Therefore, how to improve the inference efficiency of the diffusion Transformer model is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method and system for accelerating inference of a diffusion Transformer model based on multi-granularity pruning, in order to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a method for accelerating inference using a diffusion Transformer model based on multi-granularity pruning. The method includes the following steps: determining a set of time steps for cache reuse through offline analysis based on a first correlation feature between time step pruning error and input prompt words; pruning time steps not belonging to the set of time steps during online inference using the diffusion Transformer model; for unpruned time steps, evaluating the importance of each Transformer layer in the current inference context using an online adaptive method based on a second correlation feature between layer pruning error and input prompt words, and dynamically determining the Transformer layers to be pruned in the current time step based on the evaluation results; wherein the first correlation feature and the second correlation feature are pre-obtained based on the diffusion Transformer model under different prompt word conditions, time step pruning error, and layer pruning error, and the correlation level indicated by the second correlation feature is greater than the correlation level indicated by the first correlation feature.

[0008] In some embodiments of the present invention, the method further includes a step of pre-determining a set of time steps for cache reuse through offline analysis, and using a cumulative error strategy to determine the time steps in the set that will not be pruned. Specifically, this includes: setting an initial value for the cumulative time step pruning error; traversing all time steps, calculating the increment of the time step pruning error caused by skipping each time step, and accumulating the increment of the time step pruning error to a preset cumulative time step pruning error; determining whether the cumulative time step pruning error exceeds a first preset threshold when traversing to the current time step; if it exceeds the threshold, selecting the current time step into the set of time steps for cache reuse, resetting the cumulative time step pruning error to the initial value, and proceeding to the next time step; if it does not exceed the threshold, directly proceeding to the next time step.

[0009] In some embodiments of the present invention, the time step pruning error increment caused by the time step skipping is measured by calculating one or a combination of the relative L1 distance, relative L2 distance, cosine similarity, KL divergence and JS divergence between features of adjacent time steps.

[0010] In some embodiments of the present invention, the step of pruning time steps that do not belong to the time step set during online inference using the diffusion Transformer model includes: performing a full forward computation on time steps that belong to the time step set and caching the output result obtained from performing the full forward computation; reusing the cached output result obtained from the most recent full forward computation on time steps that do not belong to the time step set, thereby achieving pruning of time steps that do not belong to the time step set.

[0011] In some embodiments of the present invention, the step of evaluating the importance of each Transformer layer in the current inference context using an online adaptive method and dynamically determining the Transformer layer to be pruned in the current time step based on the evaluation results includes: calculating a first relative difference between the input of the current Transformer layer in the current time step and the input of the previous time step during online inference; multiplying the first relative difference by a pre-determined importance score of the current Transformer layer in the current time step to obtain the layer pruning error increment; and determining the Transformer layer to be pruned in each time step using an accumulated error strategy based on the layer pruning error increment. The method further includes a step of pre-determining the importance score of each Transformer layer in each time step offline. For each time step, this specifically includes: traversing all Transformer layers; adding a fixed intensity of perturbation noise to the intermediate calculation results of each Transformer layer; calculating a second relative difference between the output result of the current time step with added perturbation noise and the output result of the current time step without added perturbation noise; and using the second relative difference as the importance score of the current Transformer layer in the current time step.

[0012] In some embodiments of the present invention, the step of determining the pruned Transformer layer in each time step using a cumulative error strategy based on the layer pruning error increment includes: setting an initial value for the cumulative layer pruning error; traversing each time step that performs a complete forward computation and accumulating the layer pruning error increment to a preset cumulative layer pruning error; determining whether the cumulative layer pruning error exceeds a second preset threshold when traversing to the current Transformer layer of the current time step; if it exceeds, performing a complete computation in the current Transformer layer of the current time step, caching the intermediate computation results of the complete computation, resetting the cumulative layer pruning error to the initial value, and proceeding to the processing of the next Transformer layer; if it does not exceed, reusing the cached intermediate computation results in the current Transformer layer of the current time step, and then proceeding to the processing of the next Transformer layer.

[0013] In some embodiments of the present invention, the importance score of the current Transformer layer at the current time step is measured individually or jointly by sensitivity analysis based on the parameters or intermediate features of each layer, the magnitude of the change of the output features of each layer with time step, the stability of the change of the output features of each layer with time step, and the importance based on the internal structure of the attention mechanism.

[0014] In some embodiments of the present invention, the method further includes: in the offline stage, constructing a set of linear mapping relationships based on feature pairs of adjacent time steps in advance, and using the least squares estimation method to fit the linear transformation parameters of each channel in the Transformer layer, thereby constructing a linear regression accuracy compensation model; during the online inference process, for the pruned time step and / or Transformer layer, using the linear regression accuracy compensation model to transform the reuse cache, and using the transformed content as the output result of the pruned time step and / or Transformer layer.

[0015] Corresponding to the above methods, the present invention also provides a diffusion Transformer model inference acceleration system based on multi-granularity pruning, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described in the above embodiments.

[0016] In accordance with the above methods, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments.

[0017] The proposed method for accelerating diffusion Transformer model inference based on multi-granularity pruning constructs a time-step granularity pruning strategy independent of the input prompt based on pre-analysis. By evaluating the layer pruning error increment of each Transformer layer in each time step, the Transformer layers with less impact on the final prediction are dynamically cached and reused to construct the layer granularity pruning strategy. This method does not require additional computing power for training or fine-tuning and can achieve high-efficiency diffusion Transformer model inference in application scenarios with limited computing power or high deployment efficiency requirements.

[0018] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0019] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart of an embodiment of the present invention for accelerating inference of a diffusion Transformer model based on multi-granularity pruning.

[0021] Figure 2 This is a flowchart of a method for determining the set of time steps for cache reuse through offline analysis in one embodiment of the present invention.

[0022] Figure 3 This is a flowchart of a method for determining the pruned Transformer layer at each time step in an embodiment of the present invention.

[0023] Figure 4 This is a diagram of an architecture for an accelerated inference system for a diffusion Transformer model based on multi-granularity pruning, according to an embodiment of the present invention.

[0024] Figure 5 This is a diagram illustrating the change pattern of intermediate variables in one embodiment of the present invention.

[0025] Figure 6 This is a flowchart of offline time-step pruning in one embodiment of the present invention.

[0026] Figure 7 This is a flowchart illustrating the calculation of layer importance scores in one embodiment of the present invention.

[0027] Figure 8 This is a schematic diagram of linear regression accuracy compensation in one embodiment of the present invention.

[0028] Figure 9 This is a comparison chart of the speedup ratio and image quality of an embodiment of the present invention with other inference acceleration methods. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0030] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0031] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0032] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0033] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0034] To address the issue of low inference efficiency in the diffusion Transformer model, existing research has focused on optimization in several areas.

[0035] Some methods achieve acceleration by introducing sparse attention mechanisms, such as Sparse VideoGen and Minference, which utilize local or dynamic sparse patterns derived from attention heatmaps to reduce computational complexity. However, these methods typically rely on specific hardware platforms or customized CUDA kernels to achieve actual acceleration, thus limiting their versatility and portability, making them difficult to deploy directly across different computing environments or inference frameworks. Another type of method employs distillation techniques or other post-training paradigms to compress the denoising process, which originally required multiple iterative steps, into fewer inference steps. Although this type of method theoretically has strong acceleration potential, it usually requires additional large-scale post-training processes, resulting in high training costs, and its effectiveness largely depends on the scale and quality of the training data, leading to high deployment complexity. Furthermore, quantization methods reduce storage and computational overhead by compressing model parameters and activation values ​​to low bit widths, and typically require quantization-aware training to reduce accuracy loss after quantization. However, in the diffusion Transformer inference process, the activation tensor is large, and the associated data transfer and processing overhead may offset the acceleration gains from parameter compression, thus limiting its practical effect on accelerating image generation inference.

[0036] In recent years, a class of methods that can directly accelerate inference without retraining has attracted attention: diffusion caching and computational pruning. These methods are often referred to as diffusion caching strategies or intermediate state caching in different literature. The core idea is that during the stepwise denoising process of a diffusion model, the feature representations of adjacent or nearby time steps have high similarity, resulting in significant computational redundancy. By caching intermediate features from the first few steps and reusing these cached results in subsequent steps, redundant computations can be skipped, achieving stepwise inference acceleration. Specifically, in a diffusion model, the generation process typically transforms noisy samples into target data stepwise over multiple time steps, with each time step corresponding to a complete forward computation of the model.

[0037] For example, DeepCache[1] analyzed the similarity of feature representations between different denoising time steps for the diffusion model of U-Net structure, and proposed to reuse some intermediate features between adjacent or slowly changing time steps to reduce the repeated computation of convolution modules. This method is mainly aimed at U-Net architecture and uses the computational redundancy in its neural network structure to achieve acceleration. FORA[2] adopts a fixed time step interval strategy, that is, a complete forward propagation computation is performed every N time steps, and the intermediate output results of the latest cached sub-modules are directly reused in the remaining time steps to skip the execution of high computational modules. This method is simple in structure and easy to implement, but due to the use of fixed interval rules, it fails to adaptively adjust according to the differences in importance of different time steps. PAB[3] takes into account the differences in the stability of different attention computations in the time dimension (including temporal attention and spatial attention), models and analyzes the temporal stability of attention modules, and designs a more flexible caching strategy accordingly. Compared with the fixed interval method, PAB improves the flexibility of caching decision to a certain extent, but its caching granularity is still mainly concentrated at the time step level. TeaCache[4] uses the time step modulation features of the first layer of the diffusion Transformer model (i.e., using time step embedding as a basis to scale and offset the intermediate features of the input) as a criterion to estimate whether the cached result can be reused in the current time step before the start of each time step. If it is determined to be reusable, the final layer output of the latest cached layer is directly used to replace the complete calculation of the current time step, thereby achieving full-step pruning. This method has a certain degree of adaptability in time step selection, but its error estimation is mainly based on simple fitting of input features, and its ability to characterize complex nonlinear changes is limited. TaylorSeer[5] further extends the simple cache reuse, no longer relying solely on the cached result of a single historical time step, but using the cached features of multiple past time steps to predict the calculation result of the next time step through fitting. This method attempts to improve the prediction accuracy through multi-step historical information, but still relies on a fixed number of historical caches, and lacks fine modeling of the importance of different time steps in the time step selection strategy, still relying on a cache strategy with a fixed time step interval. Overall, the above methods have verified to varying degrees that there is significant time dimension computational redundancy in the diffusion model inference process, and have achieved acceleration through the cache reuse mechanism. These methods are essentially structured computational pruning for the diffusion process, which can improve inference efficiency while maintaining the original model structure.

[0038] The existing technology has at least the following drawbacks: (1) Insufficient modeling of the importance of time steps. Existing diffusion Transformer model pruning and caching strategies generally take time step as the smallest decision unit, and do not finely model and evaluate the differences in importance of different denoising time steps in the generation process, making it difficult to implement differentiated computation strategies for key time steps and non-key time steps. In existing representative technical solutions, such as the FORA and PAB methods mentioned above, the outputs of the attention module and the multilayer perceptron (MLP) module are usually cached, and the complete forward inference is performed according to the preset fixed time step interval. In the time step where the cache takes effect, the intermediate results previously cached are directly reused, thereby skipping the computation of attention or MLP. In the design of this type of method, all denoising time steps are regarded as equivalent objects, and the influence of different time steps on the final generation result is not distinguished, nor is an adaptive evaluation mechanism for the importance of time steps introduced. AdaCache[6] establishes a mapping table between different input differences and specific time step intervals, which is simple and easy to implement, but still lacks flexibility in the selection of time step intervals. The Teacache method uses the time-step modulated input of the first layer of the diffuse Transformer model as a reference signal, and determines whether to reuse the cached result before each time step. If the current time step is determined to require complete computation, a complete forward inference is performed, and the output of the last layer of the Transformer model is cached; if the current time step is determined to be reusable, the cached result is directly used to replace the output of the current time step, thereby pruning all Transformer layer computations corresponding to that time step. However, this method mainly relies on simple function fitting of input changes for error estimation, which is insufficient for characterizing the complex nonlinear relationships between time steps, and has limited applicability in practical deployment scenarios. Therefore, existing solutions mostly rely on fixed rules or coarse-grained estimation for pruning decisions in the time dimension, and have not yet established an accurate evaluation mechanism for the importance of different denoising time steps, making it difficult to identify and control the image fidelity after pruning.

[0039] (2) Existing pruning solutions have coarse granularity and do not fully utilize the redundancy within the Transformer's internal structure. Most current diffusion Transformer inference acceleration solutions mainly reuse computations in a single dimension. For example, PAB, FORA, and Teacache mentioned above all use whole-step computation as the basic unit, caching and reusing only the overall output of the model at a certain denoising time step. These methods usually treat the diffusion Transformer model as a whole black box, making pruning decisions only at the time step level, without further analyzing and utilizing the computational redundancy between layers and sub-modules within the model. In such solutions, once a time step is determined to be reusable, the computation of all Transformer layers in that time step is usually skipped; when computation is needed, the forward propagation of the entire network is still performed. This coarse-grained pruning method limits the upper limit of the amount of computation that can be pruned, making it impossible to effectively eliminate the potential fine-grained redundancy within the model. Some existing works have revealed layer-dimensional redundancy. For example, Duca proposed a dual pruning strategy, which uses both aggressive (full-step pruning) and conservative (pruning only some layers) pruning methods simultaneously. Studies have shown that conservative pruning can significantly reduce the quality loss caused by aggressive pruning by recalculating some key layers. Therefore, by alternating between the two strategies at a certain frequency, both acceleration and generation quality can be taken into account. Block Caching[7] is based on the U-Net diffusion model and analyzes and designs a caching and recalculation schedule for each neural network module. Learning-to-Cache[8] adds a Router module to the diffusion Transformer model and trains it together with the whole model. It is used to dynamically predict which layers need to be pruned in each time step during inference. Although these works have revealed the initial signs of layer dimensional redundancy, these works are only based on fine-grained caching of sub-modules within the neural network. Experimental results show that when the entire inference process is excessively and irregularly pruned by fine-grained computation, it will cause a larger error than using only time step caching. Therefore, the overall speedup is no more than 2 times, which is very limited. Compared to the traditional U-Net structure, which contains various heterogeneous modules (such as convolution, downsampling, upsampling, skip connections, etc.), the diffusion Transformer model is usually composed of multiple Transformer layers with consistent structure stacked sequentially. Its internal structure is more regular, and it has higher separability and module independence. This structural characteristic makes it possible to implement finer-grained computational pruning at the layer, sub-module, and even operator levels. However, existing technical solutions generally do not fully utilize this structural advantage, limiting the potential space for inference acceleration. Together with point (1), existing diffusion caching-based methods lack systematic modeling of error propagation and amplification mechanisms, which can easily lead to the continuous amplification of errors in subsequent time steps, resulting in blurred or even completely distorted generated results.Therefore, current methods of this type still have significant shortcomings in terms of controllability and stability.

[0040] (3) Existing technical solutions are highly dependent on specific hardware or underlying implementations, resulting in poor versatility and portability. Current partial diffusion Transformer inference acceleration methods, such as Sparse VideoGen, Sparse-vDiT, and MInference, which combine sparse attention, block sparse computation, or specific operator fusion, often rely on specific hardware platforms or customized CUDA kernels to achieve actual acceleration effects. These methods typically require deep optimization for specific GPU architectures or memory access patterns during design, and their performance gains are highly dependent on the underlying implementation details. Once the deployment environment changes, such as replacing different GPU models, using different inference frameworks, or running on heterogeneous computing platforms, the acceleration effect is often difficult to reproduce, and performance degradation may even occur. Therefore, these solutions face high adaptation and maintenance costs in practical engineering applications, limiting their versatility and portability.

[0041] (4) Some existing solutions rely on post-training or additional model training, resulting in high deployment costs. Some diffusion model acceleration technologies, such as Mean Flow and Learning-to-Cache, achieve inference optimization by introducing distillation, retraining, or additional module learning, such as training student models and fine-tuning after training. These methods usually require a large amount of computing resources for additional training, and the training process is highly sensitive to the data scale, data distribution, and objective function design. For scenarios such as high-resolution image generation or long video generation, the retraining cost is particularly high. In addition, when the model structure or application scenario changes, retraining or fine-tuning is often required, making it difficult to achieve plug-and-play inference acceleration and limiting its practical availability in multi-model, multi-task scenarios.

[0042] To overcome the problems existing in diffusion caching technology of the diffusion Transformer model, this invention proposes a method and system for accelerating diffusion Transformer model inference based on multi-granularity pruning. By utilizing the temporal similarity features exhibited by the diffusion Transformer model in the stepwise denoising process, the computation in the image or video generation inference process is structurally pruned, and a multi-granularity computation pruning mechanism that integrates time-step pruning and layer-by-layer structural pruning within the model is constructed.

[0043] The diffusion Transformer model includes a training phase and an inference phase. The training phase establishes the mapping relationship between noise and features, while the inference phase is a progressive denoising generation process. The inference phase includes multiple time steps, and the complete processing at each time step involves multiple Transformer layers. This invention introduces a multi-granularity pruning strategy during the inference phase, including time-step pruning and layer-level pruning. For time-step pruning, the core is to quantify the importance of each time step. There are various methods for calculating the time-step pruning error increment, and also various strategies for determining the importance of time steps. Layer-level pruning focuses on evaluating the differences in contribution of different Transformer layers to the final noise prediction result during inference, dynamically determining online whether to skip the computation of the corresponding Transformer layer, retaining key computational layers, and skipping redundant ones. Layer importance (contribution difference) can be evaluated in various ways.

[0044] Figure 1 This is a flowchart of a method for accelerating inference of a diffusion Transformer model based on multi-granularity pruning, according to an embodiment of the present invention. The method includes the following steps: Step S110: Based on the first correlation feature between the time step pruning error and the input prompt word, determine the set of time steps for cache reuse through offline analysis, and prune the time steps that do not belong to the set of time steps during the online inference process of applying the diffusion Transformer model.

[0045] In this application, a first and second correlation feature were observed through a prompt word replacement experiment. The first correlation feature includes: the correlation between time-step pruning error and the input prompt word is lower than a preset level, meaning the time-step pruning error is not significantly affected by changes in different input prompt words. The second correlation feature includes: the correlation between layer pruning error and the input prompt word is higher than a preset level, meaning the layer pruning error is significantly affected by changes in different input prompt words. Specifically, the intermediate computational features of each Transformer layer within the model exhibit significantly different error distribution characteristics under different prompt word conditions, and the error introduced by layer pruning is more sensitive to changes in prompt words.

[0046] The time-step granularity pruning based on the first correlation feature can be predetermined offline, while the layer-level granularity pruning based on the second correlation feature cannot be determined offline and requires online dynamic pruning. The first and second correlation features observed in the above experiments are the theoretical basis and motivation for combining offline time-step granularity pruning with online layer-level granularity pruning in this invention.

[0047] Optionally, based on a small batch of randomly selected calibration datasets, it can be pre-determined offline which time steps in the diffusion Transformer model will be cached and reused, forming a set of time steps for cache reuse. The theoretical basis is that the time step pruning strategy is very stable across different prompt words, so only the set of time steps pre-determined through offline analysis needs to be cached and reused. Specifically, the diffusion Transformer model is divided into multiple time steps. Time steps belonging to the set of time steps undergo complete forward computation to obtain and cache the output results, while time steps not belonging to the set are skipped by reusing the cached output results. The reused cached output results refer to the most recent output result obtained from the complete forward computation in the cache, which is the most recently cached output result in the set of time steps.

[0048] Optionally, in the step of offline determination of the set of time steps for cache reuse, a cumulative error strategy can be used to determine which time steps can be skipped and which time steps require full forward computation to obtain and cache the output results. First, a complete diffusion denoising inference process is performed on a small batch of randomly selected calibration datasets (prompt), meaning that a full forward computation is performed at each time step to obtain the output result. The output result of each time step is cached, and the model output results of each time step are statistically averaged to generate a corresponding time step calibration feature table. This time step calibration feature table records the difference between the model output result of each time step and the previous time step, allowing for rapid calculation using the cumulative error strategy in feature table (matrix) form. Subsequently, using the aforementioned offline error analysis strategy, the set of time steps suitable for cache reuse is determined based on this time step calibration feature table.

[0049] Optionally, the cumulative error strategy used in the offline determination of the time step set for cache reuse refers to: 1) initializing the cumulative time step pruning error, generally set to 0; 2) traversing all time steps, calculating the time step pruning error increment caused by skipping each time step, and accumulating the time step pruning error increment to a preset cumulative time step pruning error; 3) determining whether the cumulative time step pruning error exceeds a preset threshold α when traversing to the current time step; 4) if it exceeds, selecting the current time step into the time step set for cache reuse, resetting the cumulative time step pruning error to its initial value, and proceeding to the next time step; if it does not exceed, pruning the current time step, that is, the current time step is not selected into the time step set for cache reuse, and then proceeding to the next time step.

[0050] Based on the above traversal process, a set of time steps for cache reuse can be determined by a small batch of randomly selected calibration datasets (prompts). Generally, although the prompts are different, the set of time steps for cache reuse is highly overlapping. Optionally, for non-overlapping parts, it can be determined whether to finally select them into the set of time steps for cache reuse based on the probability / number of times they are selected.

[0051] Step S120: For time steps that are not pruned, based on the second correlation feature between layer pruning error and input prompt words, an online adaptive method is used to evaluate the importance of each Transformer layer in the current inference context, and the Transformer layer to be pruned in the current time step is dynamically determined according to the evaluation results; wherein, the first correlation feature and the second correlation feature are obtained in advance based on the diffusion Transformer model under different prompt word conditions, time step pruning error and layer pruning error, and the correlation degree indicated by the second correlation feature is greater than the correlation degree indicated by the first correlation feature.

[0052] For each time step in the complete forward computation, the input / input activation values ​​of the current time step and the previous time step in each Transformer layer are obtained. The relative L1 distance between the input of the current time step and the previous time step in each Transformer layer is calculated. The relative L1 distance between the input of the current time step and the previous time step in the current Transformer layer is multiplied by the importance score of the current time step in the current Transformer layer determined offline in advance to obtain the layer pruning error increment. The layer pruning error increment can reflect the contribution of the current Transformer layer to the final noise prediction result. During the online inference of the model, the cumulative error strategy is used to dynamically determine the Transformer layer to be pruned online. The pruned Transformer layer reuses the cached results and skips the computation of the attention module or MLP module of the current layer.

[0053] The theoretical basis for the above scheme steps is that the importance scores of each Transformer layer in each time step are almost unaffected by the prompt. Therefore, the importance scores of each Transformer layer in each time step can be determined offline in advance. However, the input / input activation values ​​of each Transformer layer are correlated with the prompt. Therefore, it is necessary to dynamically determine whether to prune a certain Transformer layer in a certain time step during the inference phase of the diffusion Transformer.

[0054] The importance score of each Transformer layer at each time step is determined offline in advance by injecting a fixed-intensity perturbation noise. The sensitivity of the layer to the final output is quantified by calculating the relative L1 distance between the final output of the model before and after the perturbation. This relative L1 distance is defined as the importance score of the Transformer layer at the current time step. , indicating the first The Transformer layer at the _ The importance score for each time step.

[0055] The steps described above for dynamically determining the Transformer layers to be pruned online using a cumulative error strategy during online model inference include: 1) initializing the cumulative layer pruning error, typically set to 0; 2) traversing each time step of the complete forward computation, calculating the relative L1 distance between the input / input activation values ​​of each Transformer layer at the current time step and the previous time step, obtaining the importance score of the current time step in the current Transformer layer as determined offline in advance, multiplying the relative L1 distance between the input of the current time step and the previous time step in the current Transformer layer by the importance score of the current time step in the current Transformer layer as determined offline in advance to obtain the layer pruning error increment, and accumulating the layer pruning error increment. 3) Determine whether the accumulated layer pruning error exceeds the preset threshold β when traversing to the current Transformer layer at the current time step; 4) If it exceeds the threshold β, the current Transformer layer at the current time step participates in the calculation, that is, the complete attention module or MLP module calculation is performed, the accumulated layer pruning error is reset to the initial value, and the processing of the next Transformer layer begins; if it does not exceed the threshold β, the current Transformer layer at the current time step is pruned, that is, the current Transformer layer at the current time step only reuses the cached result, skips the attention module or MLP module calculation in the current Transformer layer at the current time step, and then enters the processing of the next Transformer layer.

[0056] Among them, layer-level pruning maintains a real-time updated intermediate computation result cache for each Transformer layer, and dynamically determines whether the current Transformer layer needs to participate in the computation during online inference. The cached result reused in the current Transformer layer at the current time step is the intermediate computation result cache maintained by the current Transformer layer.

[0057] The proposed method for accelerating diffusion Transformer model inference based on multi-granularity pruning constructs a time-step granularity pruning strategy independent of the input prompt based on pre-analysis. By evaluating the layer pruning error increment of each Transformer layer in each time step, the Transformer layers with less impact on the final prediction are dynamically cached and reused to construct the layer granularity pruning strategy. This method does not require additional computing power for training or fine-tuning and can achieve high-efficiency diffusion Transformer model inference in application scenarios with limited computing power or high deployment efficiency requirements.

[0058] Figure 2 This is a flowchart of a method for determining a set of time steps for cache reuse using offline analysis, according to an embodiment of the present invention. The method further includes a step of pre-determining the set of time steps for cache reuse through offline analysis. A cumulative error strategy can be used to determine the time steps in the set that are not pruned, such as... Figure 2 As shown, it specifically includes: Step S210: Set the initial value for the cumulative time step pruning error.

[0059] Step S220: Traverse all time steps, calculate the time step pruning error increment caused by skipping each time step, and accumulate the time step pruning error increment to the preset cumulative time step pruning error.

[0060] Step S230: Determine whether the cumulative time step pruning error exceeds the first preset threshold when traversing to the current time step.

[0061] Step S240: If the time step exceeds the limit, select the current time step into the set of time steps for cache reuse, reset the accumulated time step pruning error to the initial value, and proceed to the next time step; if the time step does not exceed the limit, proceed directly to the next time step.

[0062] By employing this embodiment of the invention, a pattern was discovered through small-scale experiments: the time-step pruning strategy is highly stable across different prompt words. Therefore, it is only necessary to cache and reuse the time-step set predetermined through offline analysis. This enables a time-step granular pruning strategy that is entirely based on offline statistical analysis results. This method eliminates the need for additional computational overhead in the actual online inference stage, achieving stable and reliable time-step-level inference acceleration with lower complexity.

[0063] In some embodiments of the present invention, the time step pruning error increment caused by the time step skipping is measured by calculating one or a combination of the relative L1 distance, relative L2 distance, cosine similarity, KL divergence and JS divergence between features of adjacent time steps.

[0064] Using this embodiment of the invention, the incremental time step pruning error caused by skipping time steps can be measured in multiple ways, providing a great degree of freedom in the scheme, which can be flexibly adjusted according to different application scenarios and object-oriented approaches.

[0065] Optionally, the time-step pruning error can be measured by the relative L1 distance between features of adjacent time steps, and skippable time steps can be selected based on a cumulative error threshold. The measurement method for time-step pruning error can also be replaced with, but is not limited to, other forms: for example, other distance or similarity measures based on feature differences (L2 distance, cosine similarity, KL divergence, JS divergence, or combinations thereof), and error indices based on changes in statistical features (mean, variance, energy distribution changes, etc.). Simultaneously, the strategy for determining the importance of time steps can also be implemented in different ways, such as: determining whether to skip the current time step based on whether the single-step error exceeds a preset upper limit; comprehensively judging the error changes of multiple consecutive time steps based on a sliding statistical method using a time window; or selecting a set of time steps with the least impact on generation quality for pruning based on a greedy selection or optimization strategy across the entire time step range. It should be noted that although the above alternatives differ in their specific implementation, they all follow the technical idea of ​​achieving granular pruning of time steps by quantifying their importance, and should be considered equivalent technical solutions of this invention.

[0066] In some embodiments of the present invention, the step of pruning time steps that do not belong to the time step set during online inference using the diffusion Transformer model includes: (1) performing a full forward computation on time steps that belong to the time step set and caching the output result obtained from the full forward computation; (2) reusing the cached output result obtained from the most recent full forward computation on time steps that do not belong to the time step set, thereby achieving pruning of time steps that do not belong to the time step set.

[0067] By employing this embodiment of the invention, a time-step granularity pruning strategy based entirely on offline statistical analysis results can be implemented simply by caching and reusing the time-step set predetermined through offline analysis. This method eliminates the need for additional computational overhead during the actual online inference stage, enabling stable and reliable acceleration of time-step-level inference with lower complexity.

[0068] In some embodiments of the present invention, the step of evaluating the importance of each Transformer layer in the current inference context using an online adaptive method and dynamically determining the Transformer layers to be pruned in the current time step based on the evaluation results may include: calculating a first relative difference between the input of the current Transformer layer in the current time step and the input of the previous time step during online inference; multiplying the first relative difference by a pre-determined importance score of the current Transformer layer in the current time step to obtain the layer pruning error increment; and determining the Transformer layers to be pruned in each time step based on the layer pruning error increment using a cumulative error strategy. The present invention is not limited thereto; the method of evaluating the importance of Transformer layers using "first relative difference * importance score" is merely an example and can be flexibly adjusted according to the specific application scenario.

[0069] Furthermore, the method also includes a step of pre-determining the importance score of each Transformer layer at each time step offline. Specifically, for each time step, the method includes: traversing all Transformer layers, adding a fixed intensity of perturbation noise to the intermediate computation results of each Transformer layer, calculating the second relative difference between the current time step output result with added perturbation noise and the current time step output result without added perturbation noise, and using the second relative difference as the importance score of the current Transformer layer at the current time step.

[0070] The first relative difference occurs during the online inference layer granular pruning process, while the second relative difference occurs during the offline importance score calculation process. Both the first and second relative differences can be represented by calculating the relative L1 distance, but the subjects involved in the calculation are different.

[0071] In the specific implementation process, the complete diffusion denoising inference process can be performed on a calibration dataset (prompt) randomly selected in a small batch, and the importance score of each Transformer layer in each time step can be calculated.

[0072] By employing this embodiment of the invention, a pattern was discovered through small-scale experiments: the importance scores of each Transformer layer at each time step are almost unaffected by the prompt word. Therefore, the importance scores of each Transformer layer at each time step can be determined offline in advance. Using this method, the computational overhead of the online inference stage can be further reduced, which helps to achieve stable and reliable layer-level pruning inference acceleration with lower complexity.

[0073] In some embodiments of the present invention, the step of determining the pruned Transformer layer at each time step using a cumulative error strategy based on the layer pruning error increment is described. Figure 3 This is a flowchart of a method for determining the pruned Transformer layer at each time step in an embodiment of the present invention, as shown below. Figure 3 As shown, it specifically includes: Step S310: Set the initial value for the cumulative layer pruning error.

[0074] Step S320: Iterate through each time step of the complete forward computation and accumulate the layer pruning error increment to the preset cumulative layer pruning error.

[0075] Step S330: Determine whether the cumulative layer pruning error exceeds the second preset threshold when traversing to the current Transformer layer at the current time step.

[0076] Step S340: If the time limit is exceeded, perform a complete computation in the current Transformer layer at the current time step, cache the intermediate computation results of the complete computation, reset the accumulated layer pruning error to the initial value, and proceed to the processing of the next Transformer layer; if the time limit is not exceeded, reuse the cached intermediate computation results in the current Transformer layer at the current time step, and then proceed to the processing of the next Transformer layer.

[0077] Using this invention, a small-scale experiment revealed a pattern: the input / input activation values ​​of each Transformer layer are correlated with the prompt word. Therefore, it is necessary to dynamically determine whether to prune a Transformer layer at a certain time step during the inference phase of the Diffusion Transformer. Based on this pattern, a dynamic layer-level pruning strategy for the online inference phase was designed. This strategy can achieve a reduction in computational cost (such as a reduction in FLOPs) similar to that of a simple layer-level caching method, while significantly improving the fidelity and stability of the generated images or videos. As a result, it is more suitable for efficient inference scenarios of the Diffusion Transformer model.

[0078] In some embodiments of the present invention, the importance score of the current Transformer layer at the current time step is measured individually or jointly by sensitivity analysis based on the parameters or intermediate features of each layer, the magnitude of the change of the output features of each layer with time step, the stability of the change of the output features of each layer with time step, and the importance based on the internal structure of the attention mechanism.

[0079] Using this embodiment of the invention, the impact of Transformer layer pruning can be measured in multiple ways, providing a great degree of freedom in the scheme, which can be flexibly adjusted according to different application scenarios and object-oriented approaches.

[0080] Optionally, regarding hierarchical pruning, the decision to skip a corresponding layer's computation can be dynamically made by evaluating the differences in contribution of different Transformer layers to the final noise prediction result during inference. This method of evaluating hierarchical importance is not limited to the specific method used in the embodiments. Hierarchical importance can also be evaluated in other ways, such as sensitivity analysis based on layer parameters or intermediate features (judging the importance of a layer by statistically analyzing the gradient magnitude or trend of the corresponding layer's weight parameters), judgment based on the magnitude or stability of each layer's output features changing over time, and based on the internal structure of the attention mechanism (statistically analyzing the importance of different attention heads or attention channels and comprehensively evaluating the contribution of the corresponding layer accordingly). It should be noted that the above methods all belong to means of characterizing the degree of influence of each computational module within the model on the generated result from different perspectives, and their essence lies in distinguishing between critical computational layers and redundant computational layers.

[0081] Optionally, the cached content includes the final output of the model from the previous denoising time step, as well as the attention output or MLP output from each Transformer layer, and the time step information modulation output of each layer is selected as the reference variable for adaptive hierarchical pruning. Without changing the core mechanism of cache reuse and pruning determination, possible alternatives include: the cached content can be the residual connection output of each layer, the residual output of the entire Transformer stack, or the intermediate computation results within a submodule; the cached features can be complete feature representations or features that have been compressed, reduced in dimensionality, or processed with low precision; the reference variables for adaptive hierarchical pruning can be replaced with the input features, residual features, attention weight statistics, or other intermediate variables that can reflect the computational state of the current layer. While these alternatives differ in the specific selection of cached objects or reference variables, they all revolve around the technical idea of ​​dynamically skipping hierarchical computations by utilizing the feature similarity between adjacent time steps or adjacent inference states.

[0082] In some embodiments of the present invention, the method further includes: (1) in the offline stage, based on the feature pairs of adjacent time steps, a set of linear mapping relationships are constructed in advance, and the least squares estimation method is used to fit the linear transformation parameters of each channel in the Transformer layer, thereby constructing a linear regression accuracy compensation model; (2) during the online inference process, for the pruned time step and / or Transformer layer, the linear regression accuracy compensation model is used to transform the reuse cache, and the transformed content is used as the output result of the pruned time step and / or Transformer layer.

[0083] By adopting this embodiment of the invention, a channel-level linear regression accuracy compensation mechanism is further introduced, which can effectively compensate for the statistical bias introduced by cache reuse without increasing the additional computational burden, and significantly restore the generation accuracy of the diffusion model without retraining or fine-tuning the model.

[0084] In a specific embodiment of the present invention, the offline time-step pruning process is as follows: (1) At each denoising time step t, input the noise image. First, it is mapped to the latent space via a variational autoencoder (VAE), and then further expanded into a one-dimensional token sequence. The sequence passes through n Transformer layers and a linear output layer in sequence to generate the noise prediction result of the corresponding time step, that is, the final output of the model at time step t. (2) According to the time step granularity caching mechanism, the final output result of the diffusion Transformer model at each time step is used as the cache object. The time step caching module always saves the output result of the diffusion Transformer model obtained from the most recent complete forward computation and reuses the cached result as needed in subsequent inference. Forward computation refers to the data passing through the input layer, hidden layer and output layer to obtain the output result of the current time step. (3) Starting from the starting time step of the inference stage of the diffusion Transformer model, the relative error between adjacent time steps is accumulated in sequence. The time step with the accumulated error exceeding the preset threshold α is marked as the time step that needs to perform complete computation. For the time step with the accumulated error not exceeding the preset threshold α, the cached result is reused and skipped (equivalent to pruning).

[0085] In a specific embodiment of the present invention, the adaptive online Transformer layer pruning process is as follows: (1) For different Transformer layers in the diffusion Transformer model, priority is given to identifying and retaining layer structures that are more critical to the generated results and more sensitive to noise. A real-time updated intermediate calculation result cache is maintained for each Transformer layer, and the current Transformer layer is dynamically judged during the inference process to determine whether it needs to participate in the calculation. (2) By injecting fixed-intensity perturbation noise into the intermediate calculation results of each specific Transformer layer in sequence, and calculating the relative L1 distance between the final output of the model before and after the perturbation, the sensitivity of the layer to the final output is quantified. The relative L1 distance is defined as the importance score of the layer at the current time step. , where i represents the layer index and t represents the time step index. (3) Scoring based on importance Calculate the expected pruning error contribution of the layer, and then accumulate the expected pruning error contributions between adjacent layers. For layers whose accumulated expected pruning error contribution exceeds a preset threshold β, the entire attention module or MLP module needs to be calculated. For layers whose accumulated expected pruning error contribution does not exceed the preset threshold β, the cached results are reused and the attention module or MLP module calculation of the current layer is skipped.

[0086] To address the problems existing in current diffusion caching techniques, this invention proposes a method for accelerating inference of diffusion Transformer models based on multi-granularity pruning. This method fully utilizes the temporal similarity features exhibited by the diffusion Transformer model during stepwise denoising to perform structured pruning of the computations in the image or video generation inference process. This achieves efficient, stable, and universal inference acceleration without requiring model retraining. With the overall goals of efficient inference, high-fidelity generation, and good generalization and portability, this invention constructs a multi-granularity computational pruning mechanism that integrates time-step pruning and layer-by-layer structured pruning within the model. While maintaining a high degree of consistency between the generated results and the original model output, it achieves efficient, training-free inference acceleration for both image and video diffusion models.

[0087] To achieve the above objectives, this invention primarily focuses on solving the following key problems: (1) The problem of accurately measuring the importance of time steps in the diffusion Transformer inference process. In the inference process of the diffusion Transformer model, the influence of different denoising time steps on the final generation result varies significantly. However, existing inference acceleration methods usually adopt fixed step size jump or simple error fitting strategies, which lack systematic modeling of the evolution law of characteristics at different stages of the diffusion process, and make it difficult to accurately characterize the nonlinear contribution of each time step to the generation quality.

[0088] Therefore, the first key technical problem this invention needs to solve is: how to construct an effective time step importance measurement and discrimination mechanism that can accurately quantify the relative importance of the diffusion Transformer model at different denoising time steps, thereby identifying non-critical time steps that have little impact on the final generation quality. By pruning redundant denoising calculations at the time step granularity, coarse-grained computational reduction can be achieved in the early stages of inference, providing a reliable foundation for subsequent finer-grained structured pruning and overall inference acceleration.

[0089] (2) Fine-grained structured pruning within the diffusion Transformer model. Most current diffusion model pruning strategies are performed at the time-step level, but significant computational redundancy still exists in the deep network during the stepwise denoising inference process. The speedup achievable by existing diffusion model hierarchical pruning methods is usually limited, making it difficult to fully exploit the redundant computational potential within the model.

[0090] Therefore, the second key technical problem this invention needs to solve is how to reveal the evolution of the structure and attention mechanism of each layer of the diffusion Transformer model at different time steps, and further realize fine-grained structured pruning of the model's internal structure based on time-step pruning. To this end, this invention constructs a model-level sensitivity evaluation and identification mechanism, dynamically adjusts the participation level of Transformer layers according to the differences in the importance of computation at each layer at different time steps, and designs an adaptive hierarchical structured pruning strategy to further reduce computational redundancy in the repeated denoising process of the diffusion model, and further enhance the acceleration potential of the diffusion Transformer model pruning mechanism.

[0091] To address the shortcomings of existing diffusion caching strategies in terms of pruning granularity, time step selection, and redundancy mining within the model, this invention proposes a multi-granularity pruning-based inference acceleration method for diffusion Transformer models. This method focuses on the differences in the importance of different computational modules during the stepwise denoising inference process of the diffusion Transformer model. It prunes the inference computation process at both the time step granularity and the granularity of the model's internal layers, achieving efficient and stable inference acceleration without changing model parameters or introducing additional training.

[0092] The core idea of ​​this invention lies in the fact that during the inference process of the diffusion Transformer model, significant temporal redundancy exists between different denoising time steps and between different Transformer layers within the model. Furthermore, the impact of computational pruning on the final generated image or video quality varies depending on the time step or layer, exhibiting distinct differences. Based on this understanding, this invention no longer treats the inference computation of the diffusion model as a uniform and equivalent process. Instead, it constructs a multi-granularity (time step granularity and layer granularity) fusion pruning mechanism, starting from two granularities: time step and Transformer layer within the model. This mechanism identifies the importance of various computational modules during the diffusion Transformer model's inference process and selectively skips redundant computational modules with less impact on the generated results. This achieves a systematic reduction of inference computational redundancy while maintaining the stability of the generated results. Specifically, this invention employs a two-stage pruning framework.

[0093] Figure 4 This is a system architecture diagram of an accelerated inference system for the diffusion Transformer model based on multi-granularity pruning, according to one embodiment of the present invention. Figure 4 As shown, in the first stage, the diffusion inference process is analyzed and pruned at the time-step granularity. Based on the characteristics of the diffusion Transformer model, offline analysis is used to statistically evaluate the intermediate feature propagation patterns between different denoising time steps during the inference process. This analysis identifies time steps suitable for full-step cache reuse and with minimal impact on the generated results. During actual inference, for the selected time step, the model inference output of the preceding time step cache is directly reused, skipping the forward computation of the entire Transformer network in the corresponding time step, thus achieving coarse-grained computational pruning in the time dimension. In the second stage, within the remaining time steps that have not been pruned, fine-grained mining of computational redundancy is further performed at the model's internal structure level. The input of the attention module or feedforward network module of each layer is selected as the judgment criterion to dynamically evaluate whether the attention module or feedforward network module in the current Transformer layer needs to participate in the computation. When it is determined that the calculation result of a certain layer can be approximately replaced by the cached result, the calculation result in the layer cache is used to skip the calculation of the corresponding module in that layer, and only the necessary calculation process is retained, thereby performing structured pruning of the remaining calculations at the layer granularity.

[0094] By employing the aforementioned two-stage pruning strategy, this invention can collaboratively mine computational redundancy in the inference process of the diffusion Transformer model at both the time-step granularity and the model's internal layer granularity. This pruning strategy possesses both global consistency and the ability to dynamically adjust according to the actual computational needs of different generated samples, thereby significantly reducing the overall inference computational overhead while maintaining the stability of the generated results. Based on the above fundamental principles, this invention implements a multi-dimensional diffusion caching inference acceleration mechanism that requires no training and is applicable to both image and video diffusion models. It enables efficient generation of high-resolution images and long videos, improving the usability of diffusion models in practical application scenarios.

[0095] Before designing the above two-stage pruning strategy, a cue word relevance analysis must first be performed to provide theoretical support for the feasibility of this invention.

[0096] First, a small set of calibration data is selected to analyze the evolution patterns of various intermediate variables during the inference process of the diffusion Transformer model, in order to evaluate the magnitude of the error introduced in both time step caching and hierarchical caching scenarios, and further analyze whether the error changes significantly with the input prompt.

[0097] In the specific implementation, the relative L1 distance can be used as a metric to quantify the error introduced by cache reuse. For any denoising time step t, the relative error between the intermediate feature of the current time step and the corresponding feature of the previous time step is calculated. Let... In the t-th time step, the first... The output features of a Transformer layer, or the final output features of the model at that time step. Cache the final output of the model to achieve pruning and skipping of the entire time step computation; cache the intermediate outputs of the attention module or MLP module to achieve finer-grained, layer-based pruning.

[0098] The formula for calculating the relative L1 distance is as follows: Figure 5 This is a diagram illustrating the change pattern of intermediate variables in one embodiment of the present invention. For example... Figure 5As shown, the solid line represents the average error under different prompt word conditions, and the shaded area represents the corresponding standard deviation. It can be observed that the standard deviation of the model's final output error under time-step pruning is small and does not change significantly across different prompt words. This indicates that the error introduced by time-step pruning has a low correlation with specific prompt words and exhibits strong consistency. In contrast, the intermediate computational features of each layer within the model show significantly different error distribution characteristics under different prompt word conditions, indicating that the error introduced by hierarchical pruning is more sensitive to changes in prompt words. Based on the above analysis results, it can be determined that the time-step pruning strategy is very stable across different prompt words and can be predetermined through offline analysis, while the hierarchical pruning strategy is more suitable for an online adaptive dynamic decision-making approach, adaptively deciding whether to reuse cached results based on the error assessed in real time during inference. The intermediate computational features refer to the internal feature representations generated by each computational submodule during the model's forward inference process before obtaining the final output, including but not limited to the attention module output, feedforward network output, residual connection output, or other intermediate variables.

[0099] Based on the above analysis, it can be determined that the error introduced by time-step pruning has a low correlation with specific prompt words and exhibits strong consistency characteristics. This provides a theoretical basis for pre-determining the time-step pruning strategy using offline analysis. However, the error introduced by hierarchical pruning is more sensitive to changes in prompt words, which determines that a static or uniform hierarchical pruning strategy cannot be used; an online, adaptive, dynamic decision-making mechanism must be introduced. Therefore, this invention proposes a two-stage diffusion Transformer model pruning technique. In the first stage, an offline-analysis-based time-step pruning strategy is adopted, pre-determining the set of time steps for cache reuse offline. During online inference using the diffusion Transformer model, time steps not belonging to this set are pruned. In the second stage, for the unpruned time steps, an online adaptive hierarchical pruning strategy is adopted to evaluate the importance of each Transformer layer in the current inference context and dynamically determine the Transformer layers to be pruned in the current time step based on the evaluation results.

[0100] Based on the findings and conclusions of the aforementioned cue word relevance analysis, this invention proposes a method for accelerating inference in a diffusion Transformer model based on multi-granularity pruning. In the first stage, an offline analysis-based time-step pruning strategy is employed to select denoised time steps suitable for full-step pruning. In the second stage, for time steps not subject to full-step pruning, an online adaptive hierarchical pruning strategy is used to dynamically cache and prune the attention modules or feedforward network modules in the Transformer layer. The specific schemes for these two stages will be further explained in subsequent sections.

[0101] In the first stage, a time-step caching strategy based on offline analysis is adopted to determine a set of general time-step pruning before inference, thereby achieving a stable acceleration of the diffusion inference process.

[0102] For the current mainstream Diffusion Transformer (DiT) model, at each denoising time step t, the input noisy image... First, it is mapped to the latent space via a variational autoencoder (VAE), and then further expanded into a one-dimensional token sequence. The sequence passes through n Transformer layers and a linear output layer, generating the noise prediction results for the corresponding time steps, i.e., the final output of the model at time step t. In the time step granularity caching mechanism of Stage 1, the final output result of the model at each time step is used as a cache object. The time step caching module always saves the model output result obtained from the most recent complete forward computation and reuses the cached result as needed in subsequent inference processes.

[0103] To quantify the error introduced by pruning computation through cache reuse at a specific time step, this invention uses the relative L1 distance as an error metric, calculated as shown in formula (1). A cumulative error strategy is also used; if at time step... and Time-based cache reuse (corresponding time steps) If the final output of the complete forward computation is taken and pruned, then the accumulated error should be expressed as: ,in, Indicates the model at time step The final output features.

[0104] Figure 6 This is a flowchart of offline time-step pruning in one embodiment of the present invention. Figure 6 As shown, starting from the initial time step of diffusion inference, the relative errors between adjacent time steps are accumulated sequentially. When the accumulated error exceeds a preset threshold α (i.e., the first threshold), the current time step is marked as the time step that requires a complete computation, while time steps that were previously within the threshold constraint range can be skipped by reusing cached results. The parameter α is used to adjust the cache reuse ratio and acceleration level in the time step dimension, and can be set according to optimization experience.

[0105] Formalistically, if at time step If the complete model forward computation is performed, then when the condition of formula (2) is satisfied, the time step... Perform the full forward computation, without being included in the time-step granularity pruning set.

[0106] The formula for determining whether the cumulative error exceeds the preset threshold α is as follows: In its implementation, this invention first performs a complete diffusion denoising inference process offline on a small batch of randomly selected calibration datasets, and then statistically averages the final output of the model at each time step to generate a corresponding time-step calibration feature table. The time-step calibration table records the difference between each step and the previous step's model output. Based on this table, it determines which time steps should not be skipped. Skipped time steps reuse the complete forward computation output of the model from the most recent non-skipped time step during actual inference. Based on the aforementioned offline error analysis strategy, a set of time steps suitable for caching and reuse is determined from this feature table.

[0107] The first-stage time-step granularity pruning strategy generates a cached and reused time-step set entirely based on offline statistical analysis results. No additional computational overhead is required in the actual online inference stage, enabling stable and reliable time-step-level inference acceleration with low complexity.

[0108] In the second stage, an online, adaptive hierarchical pruning mechanism was constructed.

[0109] Although time-step pruning can significantly reduce the overall computational cost of diffusion inference, the self-attention module and feedforward network (MLP) within the Transformer model still account for a large amount of computational overhead in the remaining unpruned time steps. Combined with the foregoing analysis, it can be found that the pruning error at the hierarchical level is strongly correlated with the input prompts. Therefore, a static or uniform hierarchical pruning strategy cannot be used; instead, an online, adaptive hierarchical pruning mechanism is needed.

[0110] In the second stage, this invention constructs an online adaptive hierarchical caching framework. This framework maintains a real-time updated cache of intermediate computation results for each layer of the Transformer and dynamically determines whether the current layer needs to participate in the computation during inference. Existing simple strategies directly compare the differences in input features between the current time step and the corresponding layer in the previous time step; when the difference is below a certain threshold, the cached results are reused and the computation of that layer is skipped. However, this simple strategy does not consider the differences in the contribution of different layers to the final noise prediction result, which can easily introduce unstable errors in actual inference, leading to a significant decrease in generation quality. Therefore, it is not suitable for direct application to hierarchical pruning of the diffusion Transformer model.

[0111] Based on the above understanding, this invention proposes an adaptive hierarchical caching and pruning strategy based on layer importance assessment in the second stage, which is used to achieve more stable and controllable computational reduction at the layer granularity. The basic idea is that in the diffusion Transformer model, different Transformer layers contribute differently to the final model prediction results, and their robustness and sensitivity to noise introduced when reusing layer caches also vary. Therefore, when performing hierarchical pruning, it is necessary to prioritize identifying and retaining layer structures that are more critical to the generated results and more sensitive to noise.

[0112] Figure 7 This is a flowchart illustrating the calculation of layer importance scores in one embodiment of the present invention. Figure 7 As shown, in the specific implementation, a fixed-intensity perturbation noise is sequentially injected into the intermediate computation results of each specific Transformer layer, and the relative L1 distance between the final output of the model before and after the perturbation is calculated to quantify the sensitivity of that layer to the final output. This distance value is defined as the importance score of that layer at the current time step. ,in Representation layer index, Indicates the time step index. Importance score. Used to characterize time steps For the first The extent to which layer reuse caches may affect the final output of the model during hierarchical computation pruning.

[0113] This refers to the input activation value of the i-th layer in the t-th time step that is not completely skipped. It skips the error of the i-th layer at the t-th time step. It involves calculating the relative distance between two input activation values ​​and multiplying the relative distance by the importance score. The expected pruning error (contribution) is obtained. Among these, the importance score... According to Figure 7 The importance score is obtained by pre-calculating offline from the flowchart. The calculation requires the injection of perturbation noise, while when using formula (3) to dynamically evaluate the expected pruning error (contribution), no perturbation noise needs to be injected.

[0114] The expected pruning error contribution of this layer is calculated according to formula (4), where This represents the input of the i-th layer at the t-th time step. Similar to the time step pruning strategy in Stage 1, it also employs a cumulative error control mechanism, using the accumulated error during multi-layer pruning as a constraint to prevent excessive amplification of errors across the hierarchical dimensions, thereby ensuring the stability of the inference process and the quality of generation. In practical applications, the importance score [ The result was calculated offline on a small calibration dataset. Analysis of the experimental results revealed that... The values ​​of [] remain highly consistent under different cue word inputs, indicating that the importance score indicates the inherent robustness of each layer, independent of the cue word input, while the layer input [] indicates the robustness of the layer. Its stability is cue word dependent, thus exhibiting dynamism and adaptability during reasoning.

[0115] In practical implementation, to reduce additional computational overhead, the temporal-step modulated inputs from each layer of the Transformer (i.e., scaling and shifting intermediate input features using temporal-step embeddings) can be selected as the criterion to determine whether to reuse cached results and skip the attention module or MLP module computation of the current layer. This strategy can maintain the accuracy of the judgment while avoiding the introduction of significant online inference overhead.

[0116] The second-stage adaptive pruning strategy based on layer importance evaluation achieves a similar reduction in computational cost (such as a reduction in FLOPs) as the simple hierarchical caching method, while significantly improving the fidelity and stability of the generated images or videos, making it more suitable for efficient inference scenarios of the diffusion Transformer model.

[0117] Furthermore, in some embodiments of the present invention, a lightweight linear regression accuracy compensation mechanism may also be introduced.

[0118] In the original caching strategy, directly reusing intermediate features or model outputs from the previous time step can significantly reduce computation, but it introduces significant feature drift errors during diffusion inference, thus affecting the stability and fidelity of the generated results. This linear regression accuracy compensation mechanism helps to further suppress the performance degradation caused by direct caching and reuse.

[0119] Specifically, for the final output features of the diffusion model at a certain time step, this invention uses feature pairs from adjacent time steps in the calibration dataset. A set of linear mapping relationships is constructed. Using the least squares estimation (LSE) method, the linear transformation parameters are fitted for each channel c, thus establishing the following linear regression accuracy compensation model, expressed by the formula below: in, and They represent the first During the step, the first The scaling factor and bias term of the c-th channel in the layer. This represents the feature value of the current time step obtained through linear regression prediction.

[0120] Figure 8 This is a schematic diagram illustrating linear regression accuracy compensation in one embodiment of the present invention. Figure 8 As shown, during the actual inference phase, when the time-step caching strategy determines that a certain feature result can be reused to prune the current time step, it does not directly copy the cached features from the previous time step. Instead, it first transforms the cached features using the aforementioned linear regression model to generate corrected predicted features, which are then used as the output of the current time step. Through this linear regression accuracy compensation mechanism, the statistical bias introduced by cache reuse can be effectively compensated with almost no additional computational burden, significantly restoring the generation accuracy of the diffusion model without requiring retraining or fine-tuning of the model.

[0121] Starting from the temporal redundancy in the inference process of the diffusion Transformer model, this scheme proposes a multi-granularity computational pruning mechanism based on time step granularity and layer granularity, supplemented by a lightweight precision compensation strategy. Without changing the model structure or relying on the underlying hardware characteristics, it achieves efficient, stable and universal inference acceleration, with the following significant advantages.

[0122] 1) It does not depend on a specific hardware platform or custom operators, and has strong versatility and portability.

[0123] In existing technologies, many diffusion model acceleration solutions rely on specific hardware platforms (such as high-end GPUs and dedicated accelerator cards) or customized CUDA kernels, fusion operators, and underlying operator rearrangements. Although these methods can achieve certain performance improvements in specific environments, they are complex to implement, have poor portability, and are difficult to reuse across different hardware platforms or inference frameworks.

[0124] In contrast, the technical solution proposed in this invention is entirely based on a decision-making mechanism at the model inference process level. By analyzing the computational redundancy characteristics of the diffusion Transformer in the time step and hierarchical dimensions, it reduces computational load by pruning redundant and repetitive computation modules using cache reuse. This solution only involves the scheduling and control of the existing model inference process and does not rely on any specific hardware instruction set, dedicated operators, or underlying parallel mechanisms. Therefore, this solution can be directly deployed on different platforms (such as GPUs or other general-purpose acceleration devices) and mainstream deep learning frameworks, possessing good cross-platform adaptability and engineering portability, significantly reducing actual deployment and maintenance costs.

[0125] 2) No post-training or model fine-tuning is required, and the deployment process is simple and efficient.

[0126] In existing technologies, some methods accelerate diffusion models through distillation steps such as denoising, model distillation, or model weight compression. However, these methods typically require additional post-training or fine-tuning before model inference to recover the accuracy loss caused by model compression. This not only increases computational overhead but also places higher demands on training data, computing resources, and engineering processes, limiting the application of such methods in real-world scenarios.

[0127] This solution requires no computational resources for training or fine-tuning. In the first stage, based on offline statistical analysis, a general time-step caching strategy independent of the input prompt is constructed, allowing the set of skippable time steps to be determined before inference. Secondly, at the hierarchical level, an importance evaluation mechanism dynamically caches and reuses Transformer layers with minimal impact on the final prediction, thus avoiding disruption to critical computational paths. Finally, a lightweight precision compensation mechanism based on linear regression is introduced to finely correct feature drift introduced by cache reuse. These error controls and precision compensations are all completed through statistical modeling and analytical computation during the inference stage, without involving re-optimization of model parameters. Therefore, this invention can achieve stable inference while maintaining the original model weights without any form of post-training or parameter updates. This feature significantly simplifies the model deployment process, making this invention particularly suitable for practical applications with limited computational resources or high deployment efficiency requirements.

[0128] 3) Significantly accelerates the diffusion Transformer inference process while maintaining high-fidelity generation quality.

[0129] The inference process of the diffusion Transformer model is characterized by a large number of time steps and a large amount of computation per step. Existing diffusion caching and pruning methods do not have a mechanism to accurately identify the importance of each computation module in the inference process, which often introduces obvious error accumulation, resulting in problems such as blurred details, structural damage or inconsistency in timing when generating images or videos.

[0130] In this scheme, large-scale quantitative experiments revealed significant differences in computational redundancy between different time steps and between different Transformer layers during the inference process of the diffusion Transformer, and these redundancies are not uniformly distributed. Based on this understanding, this scheme adopts a phased, multi-granular pruning and cache reuse strategy. By accurately identifying the importance of computational modules and ensuring robustness against noise, it reduces the number of times repetitive computation modules are executed, thereby achieving a substantial reduction in computational load. Furthermore, to prevent error accumulation and amplification caused by multiple cache reuses, an error threshold constraint mechanism and a linear regression accuracy compensation strategy are used, ensuring that the error caused by skipped computations is always limited to a controllable range. Under the same or similar image and video generation fidelity metrics, this scheme can significantly reduce the number of floating-point calculations and inference latency of the model, achieving an acceleration effect exceeding that of existing technologies. In terms of generation quality, the inference acceleration scheme proposed in this invention outperforms existing diffusion model pruning and caching methods on mainstream objective evaluation metrics such as peak signal-to-noise ratio (PSNR), and can maintain a high degree of visual quality consistent with the original model generation results while significantly accelerating the inference process. Among them, PSNR, as an important indicator for measuring pixel-level reconstruction error, indicates that the error introduced by this technical solution at the pixel level is extremely small. Figure 9 This is a comparison chart of the speedup ratio and image quality of an embodiment of the present invention with other inference acceleration methods, such as... Figure 9 As shown, the specific image comparison results further demonstrate that, under the same or higher speedup conditions, the proposed solution outperforms existing diffusion model pruning methods in terms of image detail fidelity, structural consistency, and perceptual quality. This verifies that the multi-granularity pruning and caching mechanism proposed in this invention can achieve more efficient inference acceleration while maintaining high-fidelity generation quality.

[0131] In summary, the proposed multi-granularity pruning-based inference acceleration method and system for diffusion Transformer models significantly improves inference efficiency by systematically reducing redundant computations at both the time-step and hierarchical granularities of the diffusion Transformer model, supplemented by lightweight accuracy compensation techniques. This is achieved without introducing additional training costs or relying on dedicated hardware. This method reduces the computational overhead of diffusion model inference, enabling efficient inference for images or videos, and enhancing the practical usability of high-resolution image and long video generation models. It possesses significant practical value and promising prospects for wider application.

[0132] In summary, the key technical points of the diffusion Transformer model inference acceleration method and system based on multi-granularity pruning proposed in this invention include, but are not limited to: 1) Based on the differences in the relevance of error prompt words, differentiated pruning and caching strategies are adopted for different computational dimensions in the diffusion Transformer inference process. A key technical foundation of this invention lies in the systematic analysis of the sources of inference errors in the diffusion Transformer model. Through statistical research on the changes in intermediate features of the model under different prompt word conditions, the differences in the relevance of prompt words to pruning errors at the time step granularity and layer granularity were mainly discovered. At the time step dimension, the error introduced by pruning the entire denoising time step through caching and reuse shows a highly consistent variation pattern across different prompt words, exhibiting significant prompt word independence. However, at the internal layer dimension, the error introduced by pruning specific Transformer layers (such as self-attention modules or feedforward network modules) is highly correlated with the prompt words, showing obvious prompt word correlation. Based on this key observation, this difference is further used as the core basis for designing pruning strategies, distinguishing between prompt word-independent time step pruning and prompt word-dependent layer pruning, and employing different decision-making methods for each. One of the important innovations of this invention is the discovery of the above understanding and its systematic application to the design of pruning and caching strategies.

[0133] 2) A multi-granularity pruning and caching fusion mechanism that integrates time-step pruning and hierarchical pruning can uncover the computational redundancy of the diffusion Transformer model across multiple computational dimensions. In the first stage, based on the irrelevance of prompt words in time-step pruning errors, the importance of each time step in the complete inference process of the diffusion model is evaluated offline on a small-batch calibration dataset. This selects time steps with less impact on the generated result as a candidate set that can be skipped as a whole, forming a general, static time-step caching and reuse strategy. In the second stage, for the remaining unskipped time steps, an adaptive online hierarchical pruning mechanism based on the importance of modules within the model is introduced. By evaluating the contribution of different Transformer layers in the current inference context, computational modules with less impact on the current generated result are selectively pruned, further reducing computational load in a finer-grained manner. This multi-granularity, phased, and collaborative pruning design concept is one of the core technical solutions that this invention focuses on protecting.

[0134] 3) A lightweight linear regression accuracy compensation mechanism for suppressing cache reuse errors. In the process of computational pruning based on cache reuse, directly reusing intermediate features from the previous time step or layer inevitably introduces certain feature drift errors. To further improve the stability and controllability of the pruning strategy, this invention introduces a lightweight linear regression accuracy compensation mechanism. This mechanism utilizes the highly linear correlation between intermediate features between adjacent time steps or layers to construct a simple linear mapping relationship to correct the features obtained from cache reuse, without introducing additional model parameter training or complex nonlinear transformations. This compensation mechanism effectively reduces the error shift caused by cache reuse while maintaining extremely low additional computational and storage overhead, suppressing the gradual amplification of errors, thereby further ensuring the quality of the generated results. This linear regression-based accuracy compensation idea and its unified application in time step caching and hierarchical caching are also important innovations and protected content of this invention.

[0135] Corresponding to the above method, the present invention also provides a diffusion Transformer model inference acceleration system based on multi-granularity pruning. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0136] Corresponding to the methods described above, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the method as described in any of the above embodiments. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0137] Corresponding to the above methods, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the above embodiments.

[0138] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0139] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0140] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0142] References [1] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. "Deepcache: Accelerating diffusion models for free." Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2024. [2] Selvaraju, Pratheba, et al."Fora: Fast-forward caching indiffusion transformer acceleration." arXiv preprint arXiv:2407.01425 (2024). [3] Zhao, Xuanlei, et al."Real-Time Video Generation with PyramidAttention Broadcast." The Thirteenth International Conference on LearningRepresentations. [4] Liu, Feng, et al."Timestep Embedding Tells: It's Time to Cachefor Video Diffusion Model."Proceedings of the Computer Vision and PatternRecognition Conference. 2025. [5] Liu, Jiacheng, et al."From reusing to forecasting: Acceleratingdiffusion models with taylorseers."Proceedings of the IEEE / CVF InternationalConference on Computer Vision. 2025. [6] Kahatapitiya, Kumara, et al."Adaptive caching for faster videogeneration with diffusion transformers."Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2025. [7] Wimbauer, Felix, et al."Cache me if you can: Acceleratingdiffusion models through block caching."Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2024. [8] Ma, Xinyin, et al."Learning-to-cache: Accelerating diffusiontransformer via layer caching."Advances in Neural Information ProcessingSystems 37 (2024): 133282-133304.

Claims

1. A method for accelerating inference in a diffusion Transformer model based on multi-granularity pruning, characterized in that, The method includes: Based on the first correlation feature between time step pruning error and input prompt words, the set of time steps for cache reuse is determined through offline analysis. During online inference using the diffusion Transformer model, time steps that do not belong to the set of time steps are pruned. For time steps that are not pruned, based on the second correlation feature between layer pruning error and input prompt words, an online adaptive approach is used to evaluate the importance of each Transformer layer in the current inference context, and the Transformer layer to be pruned in the current time step is dynamically determined according to the evaluation results; wherein, the first correlation feature and the second correlation feature are obtained in advance based on the diffusion Transformer model under different prompt word conditions, time step pruning error and layer pruning error, and the correlation degree indicated by the second correlation feature is greater than the correlation degree indicated by the first correlation feature.

2. The method according to claim 1, characterized in that, The method further includes a step of pre-determining a set of time steps for cache reuse through offline analysis, and using an accumulated error strategy to determine the time steps in the set that will not be pruned, specifically including: Set the initial value for the cumulative time step pruning error; Traverse all time steps, calculate the time step pruning error increment caused by skipping each time step, and accumulate the time step pruning error increment to the preset cumulative time step pruning error. Determine whether the accumulated time step pruning error exceeds a first preset threshold when traversing to the current time step; If the time limit is exceeded, the current time step is selected into the set of time steps for cache reuse, the accumulated time step pruning error is reset to the initial value, and the process proceeds to the next time step; if the time limit is not exceeded, the process proceeds directly to the next time step.

3. The method according to claim 2, characterized in that, The time step pruning error increment caused by skipping a time step is measured by calculating one or a combination of the following: relative L1 distance, relative L2 distance, cosine similarity, KL divergence, and JS divergence between features of adjacent time steps.

4. The method according to claim 1, characterized in that, The step of pruning time steps that do not belong to the set of time steps during online inference using the diffusion Transformer model includes: Perform a full forward computation at a time step belonging to the set of time steps, and cache the output results obtained from the full forward computation. The output result of the most recent complete forward computation is obtained from the time step multiplexing cache that does not belong to the time step set, thereby pruning the time steps that do not belong to the time step set.

5. The method according to claim 1, characterized in that, The steps of evaluating the importance of each Transformer layer in the current inference context using an online adaptive method and dynamically determining the Transformer layers to be pruned in the current time step based on the evaluation results include: calculating the first relative difference between the input of the current Transformer layer in the current time step and the input of the previous time step during online inference; multiplying the first relative difference by the importance score of the current Transformer layer in the current time step determined offline in advance to obtain the layer pruning error increment; and using a cumulative error strategy based on the layer pruning error increment to determine the Transformer layers to be pruned in each time step. The method also includes a step of pre-determining the importance score of each Transformer layer at each time step offline, specifically including the following for each time step: Iterate through all Transformer layers, add a fixed intensity of perturbation noise to the intermediate computation results of each Transformer layer, calculate the second relative difference between the current time step output with added perturbation noise and the current time step output without added perturbation noise, and use the second relative difference as the importance score of the current Transformer layer at the current time step.

6. The method according to claim 5, characterized in that, The step of determining the pruned Transformer layer at each time step using a cumulative error strategy based on the layer pruning error increment includes: Set the initial value for the cumulative layer pruning error; Iterate through each time step of the complete forward computation and accumulate the layer pruning error increment to a preset cumulative layer pruning error; Determine whether the cumulative layer pruning error exceeds the second preset threshold when traversing to the current Transformer layer at the current time step; If the error exceeds the limit, perform a full computation in the current Transformer layer at the current time step, cache the intermediate computation results of the full computation, reset the accumulated layer pruning error to the initial value, and proceed to the next Transformer layer for processing; if the error does not exceed the limit, reuse the cached intermediate computation results in the current Transformer layer at the current time step, and then proceed to the next Transformer layer for processing.

7. The method according to claim 1 or 6, characterized in that, The importance score of the current Transformer layer at the current time step is measured individually or jointly by sensitivity analysis based on the parameters or intermediate features of each layer, the magnitude of the change of the output features of each layer with time step, the stability of the change of the output features of each layer with time step, and the importance of the internal structure of the attention mechanism.

8. The method according to claim 1, characterized in that, The method further includes: In the offline stage, a set of linear mapping relationships are constructed in advance based on feature pairs of adjacent time steps. The least squares estimation method is used to fit the linear transformation parameters of each channel in the Transformer layer, thereby constructing a linear regression accuracy compensation model. During online inference, for the pruned time steps and / or Transformer layers, the linear regression accuracy compensation model is used to transform the reuse cache, and the transformed content is used as the output of the pruned time steps and / or Transformer layers.

9. A system for accelerating inference of a diffusion Transformer model based on multi-granularity pruning, comprising a processor, a memory, and a computer program / instructions stored in the memory, characterized in that, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 8.

Images