Multi-modal diffusion model reasoning optimization method based on approximate multiplexing mechanism
By introducing an approximate reuse mechanism into the multimodal diffusion model and utilizing reuse matrices and caching mechanisms, the inference process of the Transformer architecture is optimized, solving the problems of high computational overhead and slow inference speed, and achieving efficient generation acceleration in a multi-GPU parallel environment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-04-07
AI Technical Summary
Existing multimodal diffusion models based on the Transformer architecture suffer from high computational overhead and slow inference speed in high-resolution and multimodal generation tasks. Furthermore, existing optimization methods are difficult to adapt to the emerging Stable Diffusion 3 model, and are particularly inefficient in multi-GPU parallel environments.
A multimodal diffusion model inference optimization method based on approximate reuse mechanism is adopted. By caching and reusing intermediate calculation results, the reuse strategy is dynamically determined by the reuse matrix. The inference process is optimized by combining fine-grained and coarse-grained approaches. Parallel reuse is performed in a multi-GPU environment to reduce redundant calculation and communication overhead.
It significantly improves inference efficiency, reduces computational load and communication overhead, ensures generation quality, adapts to multi-GPU parallel environments, solves computational bottleneck problems, and improves generation speed and hardware utilization.
Smart Images
Figure CN121809648A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to a multimodal diffusion model inference optimization method based on an approximate reuse mechanism, which is particularly suitable for large-scale multimodal generative models based on the Transformer architecture. Background Technology
[0002] With the rapid development of AI-generated content (AIGC), large-scale generative models have demonstrated enormous potential in multiple fields, from text generation to image and video generation. Among them, diffusion models have become the mainstream method for image and video generation due to their ability to generate high-quality and diverse content. In recent years, DiffusionTransformer (DiT), based on the Transformer architecture, has gradually replaced the traditional U-Net architecture, significantly improving the model's scalability and generation performance. However, with the increase in model size and the existence of multi-round denoising features, the computational overhead of the inference process increases dramatically. Especially in high-resolution and multimodal generation tasks, the complexity of attention computation increases quadratically with sequence length, becoming a major bottleneck.
[0003] To alleviate the above problems, researchers have proposed several methods to accelerate inference, mainly including: 1. Reduce the number of sampling steps, for example, by refining the sampling trajectory or using probabilistic flow ODE, to shorten the generation process; 2. Reduce the complexity of each prediction step, for example, by compressing model parameters or using low-precision calculations to reduce the computational burden; 3. An approximate reuse mechanism is adopted to reduce redundant calculations by caching and reusing intermediate features of adjacent steps, thereby accelerating the inference process.
[0004] In recent years, approximate reuse techniques have become a research hotspot. On the one hand, heuristic methods utilize the feature similarity between denoising steps to cache and reuse features in attention layers or MLP layers; on the other hand, learning-based methods determine reuse strategies through training, further improving reuse performance. Furthermore, some studies have explored parallel reuse in multi-GPU environments to alleviate the problem of insufficient computing power on a single device.
[0005] However, most existing methods focus on the U-Net architecture or local optimization strategies, and their applicability and optimization effects remain limited for Transformer-based multimodal diffusion models. Although some studies have proposed optimizations for DiT models, most are based on earlier versions (such as SD-XL) and have not been fully adapted to the emerging StableDiffusion 3 model with stronger inference capabilities. Since SD3 differs from its predecessors in both architecture design and inference mechanism, past optimization methods are difficult to transfer directly. Furthermore, SD3 still faces challenges such as high computational overhead, insufficient reuse granularity, and poor generalization in high-resolution generation and complex cross-modal inference scenarios.
[0006] Therefore, there is an urgent need to propose a more efficient approximate reuse mechanism to further improve the inference performance of Transformer-based multimodal diffusion models. Summary of the Invention
[0007] To address the issues of high computational overhead and slow inference speed faced by the new generation of multimodal diffusion models based on the Transformer architecture, represented by Stable Diffusion 3 (SD3), this invention proposes an optimization method for multimodal diffusion model inference based on an approximate reuse mechanism. By leveraging the high feature similarity between adjacent denoising steps of the diffusion model, and by caching and reusing intermediate computation results, redundant computation is avoided. This significantly improves inference efficiency without significantly sacrificing generation quality and enables it to effectively adapt to multi-GPU parallel environments.
[0008] The technical solution of the present invention is as follows: A multimodal diffusion model inference optimization method based on an approximate reuse mechanism is proposed to improve inference efficiency and ensure generation quality. Its features include the following steps: S1. Strategy Configuration: Based on the specified approximate reuse strategy, generate a corresponding reuse matrix to control the inference process for the current inference task. The reuse matrix has a dimension of M×N, where M is the total number of denoising steps in the multimodal diffusion model, N is the total number of Transformer layers in the multimodal diffusion model, and the element R(i, j) in the matrix is used to specify the reuse operation type used in the i-th denoising step and the j-th layer. The reuse operation type includes a first operation type, a second operation type, and a third operation type. S2. Inference Execution: During the iterative denoising inference process of the multimodal diffusion model, for each denoising step i and each model layer j, the following sub-steps are executed: S21. Reuse decision: Query the element R(i,j) in the reuse matrix; S22, Conditional Execution: - If R(i,j) indicates the first operation type, it means that it is not reused. The complete forward computation of the model layer is performed, and the calculated output features are written into the cache management module at the position corresponding to the denoising step and the model layer. - If R(i,j) indicates the second operation type, it represents coarse-grained reuse, reading the corresponding features previously cached and skipping all or part of the forward computation of this layer of the model; - If R(i,j) indicates the third operation type, it represents fine-grained reuse, which is a hybrid operation of computation and reuse at the token level. Only some tokens are recalculated, and the rest reuse cache features. S3. Result Generation: After completing all M denoising steps of inference, decode the final latent features and generate the output image.
[0009] Furthermore, the reuse strategy upon which the reuse matrix is generated in step S1 is based is a dynamic reuse strategy, including: In the preprocessing stage or the initial denoising step of inference, for the candidate reuse position, the output difference between performing feature reuse and performing full computation is calculated; The output difference is compared with a preset quality loss threshold. Based on the comparison results, the second or third operation type is dynamically allocated in the reuse matrix.
[0010] Furthermore, when calculating the output difference, for model layers involving the interaction of text tokens and image tokens, a cross-modal consistency constraint is introduced. This is achieved by increasing the sensitivity to changes in text-related features or setting an independent threshold to ensure that reuse decisions do not lead to significant semantic bias.
[0011] Furthermore, the reuse strategy on which the reuse matrix is generated in step S1 is a hybrid reuse strategy. This strategy systematically and alternately configures the second operation type and the third operation type in the reuse matrix, so that after performing fine-grained reuse of the third operation type in multiple consecutive denoising steps, a denoising step that performs coarse-grained reuse of the second operation type is inserted to achieve an optimized balance between inference efficiency and generation quality.
[0012] Furthermore, the fine-grained reuse operation in the conditional execution step specifically includes: Based on the semantic context of the current denoising step, calculate the feature change sensitivity of each token; Based on the sensitivity, selective recalculation is performed on the subset of tokens with high sensitivity, while feature reuse is performed on the subset of tokens with low sensitivity.
[0013] Furthermore, when the multimodal diffusion model is deployed in a pipelined parallel manner on multiple processors, step S22 includes skipping the repeated calculation of Key and Value tensors in the attention mechanism and passing the cached Key and Value tensors between processors, thereby reducing the amount of computation and compressing pipeline cavitation.
[0014] Furthermore, when the multimodal diffusion model is deployed in a sequential parallel manner on multiple processors, step S22 includes reading historical key and value caches stored on other processors, thereby reducing the number of cross-processor communications and the amount of data required to perform the current computation.
[0015] Second, the present invention also provides an electronic device, characterized in that it includes: One or more processors; Memory, which stores one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.
[0016] Third, a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the above-described method.
[0017] The technical effects of this invention are as follows: 1) For novel Transformer multimodal diffusion models such as Stable Diffusion 3 that employ text-image stitching and Self-Attention, a systematic approximate reuse optimization scheme was designed to solve the problem of transfer incompatibility of older methods. A unified "reuse matrix" abstraction was proposed, which can flexibly integrate various reuse strategies such as fixed-step, greedy dynamic, and coarse-fine granularity hybrid, providing a manageable and scalable unified interface for the reuse mechanism.
[0018] 2) The reuse mechanism incorporates dynamic judgment based on feature similarity thresholds and token-level fine-grained control, with particular consideration given to cross-modal semantic consistency, effectively balancing inference acceleration and generation quality. Extending the approximate reuse mechanism from a single-card to a multi-card parallel environment and deeply integrating it with parallel paradigms such as PP and SP not only reduces computational load but also significantly lowers communication overhead, solving the parallel efficiency bottleneck problem in large-scale deployments. Attached Figure Description
[0019] Figure 1 It is an approximate reuse of the inference process in the SD3 model.
[0020] Figure 2 It is a reuse scheme for the SD3 model in sequence parallelism. Detailed Implementation
[0021] To better understand the embodiments of the present invention, further descriptions are provided below with reference to specific implementation examples. It is worth noting that the following examples are for illustrative purposes only and do not limit the implementation of the present invention.
[0022] This document describes the implementation of an optimization scheme to improve the inference performance of the stable diffusion 3 multimodal model based on an approximate reuse mechanism. In a preferred embodiment, the following steps can be taken to implement this optimization: 1. Determining the reuse matrix Several approximate reuse mechanisms have been implemented for the SD3 model, including: fixed-step approximate reuse, greedy algorithm-based approximate reuse, a combination of fine-grained and coarse-grained dimensions, and approximate reuse under multi-GPU parallelism. The core of these methods is to define a reuse matrix, the size of which is (the number of denoising steps in the image generated by the SD3 model × the number of transformer layers in SD3). The matrix represents the measures taken at the current denoising step and current layer. For example, "0" can represent no reuse and normal calculation, "1" represents coarse-grained reuse of the previous content, and "2" represents fine-grained reuse of the previous content. The SD3 model has a default denoising step count of 28 steps and 24 layers, equivalent to a reuse matrix size of 28 × 24.
[0023] Different reuse matrices are determined based on different approximate reuse mechanisms. For example, based on an approximate reuse mechanism with a fixed number of steps of 2, the first column of every two columns of the reuse matrix can be set to all 0s, indicating that all layers of the model in the current denoising step are calculated normally and not reused. The second column can be set to all 1s, indicating that all layers of the model in the previous denoising step reuse the results of the corresponding model layers in the previous step.
[0024] For the approximate reuse mechanism of the greedy algorithm, a reuse matrix needs to be calculated in advance, and a threshold needs to be set. During image generation and inference, the loss value between the content before reuse and the content before non-reuse in the current layer at the current denoising step is calculated. If it is greater than the threshold, the corresponding position in the reuse matrix is not reused; if it is less than the threshold, it is reused. The final reuse matrix result is determined according to the greedy algorithm.
[0025] In the approximate reuse mechanism that combines fine-grained and coarse-grained dimensions, fine-grained reuse refers to selecting a portion of tokens for computation at the token level, while reusing the previous content for the remaining tokens. Coarse-grained reuse, on the other hand, involves full computation or full reuse of the entire sequence output. As described above, fine-grained reuse reduces the impact of approximate reuse on image quality, while coarse-grained reuse significantly accelerates the image generation process. Therefore, combining the two allows for maximizing the speed of the generation process while effectively controlling image quality loss. The corresponding reuse matrix can be achieved by combining multiple rounds of fine-grained reuse with one round of coarse-grained reuse.
[0026] 2. Establishment of a feature caching mechanism In addition to determining the reuse matrix, approximate reuse also requires establishing a complete caching mechanism. This means writing a cache manager to create the corresponding reuse matrix during model inference based on the user-specified reuse mechanism, cache and update the relevant features according to the reuse matrix, and read the features at the corresponding positions for reuse when needed, thereby skipping computation and completing approximate reuse throughout the entire denoising loop.
[0027] 3. Adaptation for multi-card parallel operation Approximate reuse, besides its application in single-card acceleration, can also be used to optimize communication and improve parallel acceleration in multi-card parallel processing. Specifically, in PP (Programming Point) or SP (Single Point) parallel processing, previously stale key-value (KV) values are passed to skip some communication operations and reduce communication overhead. Furthermore, when performing attention calculations across multiple cards, the aforementioned approximate reuse mechanism can be combined to improve model inference efficiency.
[0028] 4. Model Reasoning Model inference preparation: Run the script to initialize the SD3 model, including user input of text prompts, the length and width of the generated image, the selected approximate reuse mechanism, parallel mode, and other parameters.
[0029] The model completes inference: based on the user's input parameters, it performs inference, fulfills the user's request, and returns the generated image to the user.
[0030] 5. Implementation details and innovative points: In its specific implementation, this invention mainly embodies the following implementation details and innovations: Firstly, in terms of reuse strategy design, this invention proposes a unified method for constructing a reuse matrix. This reuse matrix can flexibly adapt to various reuse mechanisms, including fixed-step reuse, dynamic reuse based on greedy algorithms, and hybrid reuse methods combining fine-grained and coarse-grained approaches. By uniformly mapping different mechanisms into matrix form, it not only facilitates management and expansion but also provides a unified interface for subsequent cache scheduling and computation reuse, thereby improving the flexibility and scalability of the implementation.
[0031] Secondly, regarding the feature caching mechanism, this invention establishes a cache management module to uniformly store and retrieve features at different levels and time steps of the model. This mechanism can dynamically update the cached content based on the reuse matrix, realizing a flexible "store-retrieve-reuse" process, avoiding redundant calculations, improving inference efficiency, and ensuring the feasibility of the reuse strategy under complex model architectures.
[0032] Furthermore, regarding multi-card parallel adaptation, this invention not only considers the pipeline cavitation problem in pipeline parallel (PP) scenarios, but also effectively reduces the amount of data in cross-card communication by combining an approximate multiplexing mechanism. In sequence parallel (SP) scenarios, this invention further utilizes multiplexed KV caches to replace some repetitive communication operations, thereby significantly reducing communication bandwidth pressure while ensuring accuracy and improving the overall parallel efficiency of multi-card deployment.
[0033] Finally, in practical inference applications, this invention combines the aforementioned reuse matrix with a caching mechanism, allowing users to flexibly choose reuse schemes based on parameters such as input prompts, image resolution, and parallel processing methods, thus achieving an efficient model inference process. Compared to existing technologies, this invention effectively reduces inference time and computational resource consumption while ensuring multimodal consistency, improving the response speed and scalability of large-scale multimodal diffusion models in practical applications.
[0034] This embodiment not only demonstrates how to improve the inference efficiency of the Stable Diffusion 3 model by using an approximate reuse mechanism, but also highlights an innovative method to solve the problems of computational overhead and generality in multimodal generation tasks through adaptive optimization under different reuse strategies and parallel modes.
[0035] This invention achieves significant inference acceleration and reduced response time in high-resolution image generation and multimodal tasks by skipping a large amount of redundant computation. It reduces the computational burden and memory usage of the GPU in core modules such as attention and MLP, improving hardware utilization and system throughput. Dynamic reuse and consistency constraint mechanisms ensure semantic alignment between the generated image and the input text, avoiding detail blurring, semantic deviation, or quality degradation caused by reuse. On multi-GPU clusters, it effectively improves parallel efficiency and reduces inference latency by reducing computational and communication overhead, providing a feasible technical foundation for large-scale commercial applications. It is not only applicable to SD3 but can also be easily extended to other future multimodal diffusion models based on the Transformer architecture, possessing long-term technical viability.
[0036] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. Where there is no conflict, the above embodiments and features described therein can be combined with each other.
Claims
1. A multimodal diffusion model inference optimization method based on an approximate reuse mechanism, used to improve inference efficiency and ensure generation quality, characterized in that, Includes the following steps: S1. Strategy Configuration: Based on the specified approximate reuse strategy, generate a corresponding reuse matrix to control the inference process for the current inference task. The reuse matrix has a dimension of M×N, where M is the total number of denoising steps in the multimodal diffusion model, N is the total number of Transformer layers in the multimodal diffusion model, and the element R(i, j) in the matrix is used to specify the reuse operation type used in the i-th denoising step and the j-th layer. The reuse operation type includes a first operation type, a second operation type, and a third operation type. S2. Inference Execution: During the iterative denoising inference process of the multimodal diffusion model, for each denoising step i and each model layer j, the following sub-steps are executed: S21. Reuse decision: Query the element R(i,j) in the reuse matrix; S22, Conditional Execution: - If R(i,j) indicates the first operation type, it means that it is not reused. The complete forward computation of the model layer is performed, and the calculated output features are written into the cache management module at the position corresponding to the denoising step and the model layer. - If R(i,j) indicates the second operation type, it represents coarse-grained reuse, reading the corresponding features previously cached and skipping all or part of the forward computation of this layer of the model; - If R(i,j) indicates the third operation type, it represents fine-grained reuse, which is a hybrid operation of computation and reuse at the token level. Only some tokens are recalculated, and the rest reuse cache features. S3. Result Generation: After completing all M denoising steps of inference, decode the final latent features and generate the output image.
2. The multimodal diffusion model inference optimization method based on an approximate reuse mechanism according to claim 1, characterized in that, The reuse strategy upon which the reuse matrix is generated in step S1 is based is a dynamic reuse strategy, including: In the preprocessing stage or the initial denoising step of inference, for the candidate reuse position, the output difference between performing feature reuse and performing full computation is calculated; The output difference is compared with a preset quality loss threshold. Based on the comparison results, the second or third operation type is dynamically allocated in the reuse matrix.
3. The multimodal diffusion model inference optimization method based on an approximate reuse mechanism according to claim 2, characterized in that, When calculating the output differences, for model layers involving the interaction of text tokens and image tokens, a cross-modal consistency constraint is introduced. This is achieved by increasing the sensitivity to changes in text-related features or setting an independent threshold to ensure that reuse decisions do not lead to significant semantic bias.
4. The multimodal diffusion model inference optimization method based on an approximate reuse mechanism according to claim 1, characterized in that, The reuse strategy on which the reuse matrix is generated in step S1 is a hybrid reuse strategy. This strategy systematically alternates the second operation type and the third operation type in the reuse matrix, so that after performing fine-grained reuse of the third operation type in multiple consecutive denoising steps, a denoising step that performs coarse-grained reuse of the second operation type is inserted to achieve an optimized balance between inference efficiency and generation quality.
5. The multimodal diffusion model inference optimization method based on an approximate reuse mechanism according to claim 1, characterized in that, The fine-grained reuse operation in the conditional execution step specifically includes: Based on the semantic context of the current denoising step, calculate the feature change sensitivity of each token; Based on the sensitivity, selective recalculation is performed on the subset of tokens with high sensitivity, while feature reuse is performed on the subset of tokens with low sensitivity.
6. The multimodal diffusion model inference optimization method based on an approximate reuse mechanism according to claim 1, characterized in that, When the multimodal diffusion model is deployed in a pipelined parallel manner on multiple processors, step S22 includes skipping the repeated computation of Key and Value tensors in the attention mechanism and passing the cached Key and Value tensors between processors, thereby reducing computation and compressing pipeline cavitation.
7. The multimodal diffusion model inference optimization method based on an approximate reuse mechanism according to claim 1, characterized in that, When the multimodal diffusion model is deployed in a sequential parallel manner on multiple processors, step S22 includes reading historical key and value caches stored on other processors, thereby reducing the number of cross-processor communications and the amount of data required to perform the current computation.
8. An electronic device, characterized in that, include: One or more processors; Memory, which stores one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.