A diffusion large model acceleration decoding method based on spatiotemporal stability guidance

Abstract

This invention relates to an accelerated decoding method for a large diffusion model based on spatiotemporal stability guidance, comprising: acquiring text or image input and an initial mask sequence, initializing a mask token position set; inputting the currently determined token set and the text or image into the large diffusion model to obtain the predicted distribution of each mask position, and determining the predicted token and its confidence level; constructing an initial threshold map based on the mask token position set; performing local smoothing processing to obtain a spatial threshold; calculating the consistency length and generating a temporal relaxation factor; generating a spatiotemporal threshold and constructing a submission set; and updating the mask token position set.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and generative artificial intelligence, and in particular to a decoding acceleration method for Diffusion Multi-Modal Large Model (dMLLM), specifically a decoding method that generates an adaptive threshold for tokens based on spatial stability and temporal stability information. Background Technology

[0002] In recent years, the concept of diffusion models has been introduced into the fields of language generation and multimodal understanding, gradually forming a diffusion-based language / multimodal modeling paradigm oriented towards discrete tokens. These methods typically represent the output as a string of length [length missing]. The discrete token sequence is gradually "eroded" into a sequence containing masked tokens through a multi-step Markov forward process. In the inference stage, the model uses a bidirectional Transformer to predict the vocabulary distribution of the mask position in each denoising step and gradually recovers the target sequence to complete the generation or understanding task. Furthermore, this framework can be naturally extended to multimodal conditional generation: visual inputs such as images and videos are encoded into conditional information and interacted with text tokens in the same denoising process, thereby realizing capabilities such as visual question answering, image-text reasoning and instruction following. Especially in the autonomous driving scenario, the diffuse multimodal large model can be used for scene understanding and decision expression in complex road environments - for example, reasoning about the relationship between traffic participants in the forward-looking camera / surround-view video, generating text / structured instructions related to driving intention explanation, risk analysis, behavior suggestions or planning, while ensuring multimodal consistency and improving the robustness of long-link reasoning. Relevant representative works include LLaDA[1], Dream[2], and LaViDa

[13] .

[0003] Regarding the training and scaling of large diffusion models, existing studies have proposed training paradigms and modeling strategies from different perspectives. For example, some studies have expanded the scale and training stability of masked diffusion models in text scenarios [3]; other studies have expanded large diffusion models by adapting them to autoregressive models to improve the trainability and performance of diffusion modeling [4]; in addition, some studies have provided more systematic analysis and method design for the mechanism of “token denoising order” and “planned denoising” in discrete diffusion [5][6]. Meanwhile, in order to more flexibly adjust the diffusion process during the inference stage, some studies have proposed mechanisms such as remasking and scaling during inference to adjust the generation process under different computational budgets [7].

[0004] In the inference stage of the diffusion model, multiple iterative denoising steps are usually required, which results in a large inference time and computational overhead [1][8]. In each denoising step, the decoding strategy needs to determine which mask positions should be "decoded" into specific tokens and kept unchanged in subsequent steps. The most common approach in existing schemes is to use a global unified criterion to screen mask positions, such as using a unified threshold or a unified rule to compare the confidence of all mask positions, thereby selecting and fixing the set of tokens that meet the conditions [1][2]. This type of strategy is simple to implement, but due to the differences in the degree of contextual dependence and prediction uncertainty of different positions, it is often difficult to differentiate between different positions using a single global criterion, so there is still room for improvement in further improving decoding efficiency while ensuring generation quality [5][6].

[0005] To reduce the inference cost of large diffusion models, one type of research proposes a cache reuse mechanism from the perspective of engineering implementation. For example, for the attention calculation process of diffusion Transformer, a reuse strategy such as key-value cache (KV Cache) is proposed to reduce the overhead of repeated calculation between different denoising steps, thereby improving throughput[9]. Furthermore, some works have introduced more adaptive cache strategies or dynamic eviction mechanisms to balance memory usage and throughput improvement at different stages

[10] .

[0006] In addition to cache reuse, another type of research proposes acceleration methods from the perspective of parallel decoding process and decoding framework redesign. Furthermore, in the development path of diffusion models, large-scale diffusion models and conditional diffusion models for general tasks have also emerged, providing a broader model foundation for diffusion generation and reasoning

[11]

[12] .

[0007] Based on the above existing technologies, it can be seen that: on the one hand, the diffusion language model has formed a relatively systematic technical route in terms of training paradigm, scaling and multi-task expansion[3][4]

[11] ; on the other hand, the inference acceleration methods are mainly concentrated on cache reuse[9]

[10] or unified rule adjustment of the overall process[7]. However, for the observable location neighborhood correlation (which can be regarded as "spatial stability") and cross-denoising step prediction consistency (which can be regarded as "temporal stability") in the decoding process of the diffusion large model, the existing decoding strategies usually still use a globally unified criterion for processing[1][5][6]. There is a lack of a decoding method that integrates spatial neighborhood information and historical consistency information in the inference stage, forms differentiated submission criteria (such as token-level adaptive threshold) for different token positions to improve decoding efficiency and maintain generation quality.

[0008] References: [1] Nie S, Zhu F, You Z, et al. Large language diffusion models[C] / / Advances in Neural Information Processing Systems (NeurIPS). 2025. [2] Ye J, Xie Z, Zheng L, et al. Dream 7B: Diffusion Large LanguageModels. arXiv preprint arXiv:2508.15487 , 2025. [3] Nie S, Zhu F, Du C, et al. Scaling up Masked Diffusion Models onText[C] / / International Conference on Learning Representations (ICLR). 2025. [4] Gong S, Agarwal S, Zhang Y, et al. Scaling Diffusion Language Modelsvia Adaptation from Autoregressive Models[C] / / International Conference onLearning Representations (ICLR). 2025. [5] Kim J, Shah K, Kontonis V, et al. Train for the Worst, Plan for theBest: Understanding Token Ordering in Masked Diffusions[C] / / InternationalConference on Machine Learning (ICML). 2025. [6] Liu S, Nam J, Campbell A, et al. Think while You Generate: DiscreteDiffusion with Planned Denoising[C] / / International Conference on LearningRepresentations (ICLR). 2025. [7] Wang G, Schiff Y, Sahoo S, Kuleshov V. Remasking DiscreteDiffusion Models with Inference-Time Scaling[C] / / Advances in NeuralInformation Processing Systems (NeurIPS). 2025. [8] Arriola M, Gokaslan A, Chiu J T, et al. Block Diffusion:Interpolating Between Autoregressive and Diffusion Language Models[C] / / International Conference on Learning Representations (ICLR). 2025. [9] Ma X, Yu R, Fang G, Wang X. dKV-Cache: The Cache for DiffusionLanguage Models[C] / / Advances in Neural Information Processing Systems(NeurIPS). 2025.

[10] Song Y, Liu X, Li R, et al. Sparse-dLLM: Accelerating DiffusionLLMs with Dynamic Cache Eviction[C] / / Proceedings of the AAAI Conference onArtificial Intelligence (AAAI). 2026.

[11] Tae J, Ivison H, Kumar S, Cohan A. TESS 2: A Large-ScaleGeneralist Diffusion Language Model[C] / / Proceedings of the Association forComputational Linguistics (ACL). 2025.

[12] Koh H, Jhang M, Kim D, et al. Conditional [MASK]Discrete Diffusion Language Model[C] / / Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). 2025.

[13] Li S, Kallidromitis K, Bansal H, et al. LaViDa: A Large Diffusion Model for Vision-Language Understanding[C] / / Advances in Neural Information Processing Systems (NeurIPS). 2025.

[14] Yang L, Tian Y, Li B, et al. MMaDA: Multimodal Large Diffusion Language Models[C] / / Advances in Neural Information Processing Systems (NeurIPS). 2025.

[15] Chang S, Zhang X, Liu Q, Niu L. D 3 ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs[C] / / Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2026. Summary of the Invention

[0009] The main technical problem addressed by this invention is that, in the iterative denoising decoding process of large diffusion models, existing decoding strategies based on a single global threshold fail to simultaneously utilize the local neighborhood information of the token at its sequence position and the historical consistency information across decoding steps. This results in a limited number of tokens that can be reliably decoded in each step, thus affecting inference efficiency. To address this problem, this invention provides a spatiotemporal stability-guided decoding method for large diffusion models. This invention constructs a "token-level adaptive threshold" to replace the global threshold used in each decoding step, thereby increasing the number of tokens that can be decoded in each step while ensuring output quality, and thus improving overall decoding throughput.

[0010] A spatiotemporal stability-guided accelerated decoding method for diffusion large models is proposed for the inference decoding process of diffusion large models, wherein the diffusion large model accelerates decoding at each decoding step. For the set of mask token positions Output prediction confidence With a predicted token identifier; the method includes: 1) Obtain text or image input With the initial mask sequence Let the length of the mask sequence be... Initialize the mask token location set And set the maximum number of iterations. ; 2) For the iteration step The currently determined set of tokens With the text or image input Input the large diffusion model to obtain the position of each mask. Predicted distribution The prediction token is determined by the prediction distribution. and its confidence level ; 3) Based on the aforementioned mask token location set Constructing an initial threshold map Among them, for the mask position Set a high threshold For non-mask positions Set a low threshold ; 4) For the initial threshold map Perform local smoothing to obtain the spatial threshold. ; 5) Based on mask position Calculate the consistency length based on the predicted token consistency in the historical iteration step. And generate a time relaxation factor based on the consistency length. ; 6) For each mask position Spatiotemporal thresholds are generated based on the spatial threshold and the temporal relaxation factor. And construct the submission set The submission set The tokens at each mask position are fixed to the corresponding predicted tokens. And update the mask token location set. ; 7) When Empty or iteration step Reaching the maximum number of iterations Output the decoding result when the time is right, otherwise let Then return to step 2) and continue iterating.

[0011] Furthermore, in step 3), .

[0012] Furthermore, in step 4), the local smoothing process is Gaussian smoothing and the smoothing scale is determined by parameters. control, ; Furthermore, step 5) includes: for each mask location Record its predicted tokens in historical iteration steps The consistency length is calculated by counting the number of consecutive iterations in which the predicted token ID remains consistent. ;when Time relaxation factor generated ,when Time relaxation factor generated ,in And only if the mask position The time relaxation factor is activated only when the confidence level in the previous iteration meets the preset gating condition. Otherwise ; Furthermore, the preset gating condition is: only when the confidence level of mask position i in the previous iteration step is higher than the low threshold. ; Furthermore, step 6) includes: for each mask location The spatiotemporal threshold is obtained by fusing the spatial threshold with the temporal relaxation factor. ,in

[0013] The submission set is determined based on the confidence level and the spatiotemporal threshold.

[0014] Fix the position in the submission set to the corresponding predicted token and update the mask token position set. .

[0015] The present invention has the following beneficial effects: (1) By utilizing the submitted information in the neighborhood through the spatial threshold map, the threshold of the mask position surrounded by more submitted tokens is lower, making it easier to submit and increasing the number of single-step submissions; (2) By utilizing the consistency of step prediction through the time relaxation factor, tokens with stable historical predictions are submitted earlier, reducing repeated predictions and repeated corrections; (3) Since only the threshold generation and submission strategy in the decoding stage is changed, without changing the model parameters and structure, it is easy to integrate into the existing diffusion large model inference process. Attached Figure Description

[0016] Figure 1 A schematic diagram of the overall process of a spatiotemporal stability-guided decoding method Figure 2 Specific implementation of the proposed method Figure 3 Diagram of LaViDA inference results for a large diffusion model in a traffic scenario. Figure 4 Diagram of LaViDA inference results for a large diffusion model in a general scenario Detailed Implementation

[0017] The present invention will now be described in conjunction with the accompanying drawings and embodiments.

[0018] A method for accelerating decoding of large diffusion models based on spatiotemporal stability guidance like Figure 1 As shown, the spatiotemporal stability-guided decoding method proposed in this invention is used for the iterative decoding process of large diffusion models. In each decoding step... The diffusion of large language models for the current set of token positions, which is still a mask. Output the predicted distribution and obtain: (1) Location Predicted token ID: (2) Location Prediction confidence level: .in, For sequence position index, For sequence length, It is a discrete word list (codebook).

[0019] Based on this, the present invention no longer uses a "single global threshold per step", but instead applies a threshold to each location. Generate the corresponding token-level threshold and with " Has it been achieved? "This serves as the criterion for whether to submit decoding, thus forming a submission set." This completes the write operation and mask set update. The generation of relevant thresholds is jointly performed by the spatial adaptation module and the temporal consistency module.

[0020] The basic technical solution of this invention is as follows: 1) Obtain the diffusion model in the first... The prediction results of each denoising step for the current mask token position. Let the sequence length be... Let the first The set of mask token positions for the step is For any mask position The confidence level of the predicted token output by the model is denoted as .

[0021] 2) Construct the initial threshold map. Based on the... Constructing an initial threshold map from the mask states at each position in the step sequence An initial high threshold is assigned to the mask position. Assign an initial low threshold to the decoded positions. And satisfy .

[0022] 3) Perform spatial stability guidance to obtain a spatial threshold map. For the initial threshold map Smoothing is performed (e.g., Gaussian smoothing), and the smoothing scale is determined by parameters. control spatial threshold map This makes it easier to submit a mask position in the current step when there are many submitted positions in the neighborhood around that mask position.

[0023] 4) Perform time stability guidance to obtain the time relaxation factor. For any mask position The number of prediction token consistency steps in the historical continuous denoising steps is counted. ( The meaning is: the number of consecutive steps in which the predicted token ID remains unchanged at this position, including the current step and several immediately preceding steps. When this happens, the relaxation factor at that location is set to... ;when At this point, the relaxation factor is set to 1. To further suppress premature submission of uncertain tokens, a constraint can be added: "Only when the confidence level of the position in the previous denoising step is higher than a threshold..." Only then should the relaxation factor be applied. ".

[0024] 5) Combine the spatial threshold and the temporal relaxation factor to obtain the token-level adaptive threshold. For example, they can be combined in a multiplicative manner: .

[0025] 6) Determine the submission set and update the mask set based on the token-level adaptive threshold. Assign confidence scores to each mask position. Its corresponding threshold Compare and choose the one that satisfies the criteria. The position constitutes the submission set The predicted tokens at these positions are then fixed as the decoded results, while the mask set is updated. .

[0026] 7) Repeat steps 1) to 6) until the termination condition is met and the final sequence is output.

[0027] The embodiments of the present invention are described in detail below.

[0028] First, we introduce the spatial adaptive threshold generation module. This module aims to construct an adaptive spatial threshold map by utilizing the presence and number of decoded tokens within the spatial neighborhood.

[0029] (1) Spatial adaptive threshold generation module In one embodiment, the spatial adaptive threshold generation module includes the following process: 1) Construct an initial threshold map based on the current mask state. Assign a low threshold to decoded (non-masked) positions. A high threshold is assigned to positions that are still masks. .in, and This is a preset threshold parameter.

[0030] 2) Perform Gaussian smoothing on the initial threshold map to obtain a spatially adaptive threshold map. ,in This is the Gaussian smoothing scale parameter, used to control the smoothing intensity.

[0031] With the above design, when a certain mask position is adjacent to more decoded positions in the sequence, its threshold will be reduced accordingly, making it easier for that position to meet the submission condition in the current or earlier decoding step, thereby improving decoding efficiency.

[0032] Then, we introduce the time consistency relaxation module to further relax the time-stable token decoding threshold.

[0033] (2) Time Consistency Relaxation Module In one embodiment, the time consistency relaxation module is used to reduce the threshold for certain locations based on historical prediction consistency, and it includes the following process: 1) For each position that is still a mask The number of consistency steps is obtained by counting whether the predicted token ID remains consistent in the current decoding step and several consecutive decoding steps before it. .in, This means "tracing back from the current step to predict the maximum number of consecutive steps with consistent IDs".

[0034] 2) Based on the number of consistency steps Generate relaxation factor : Take 1 when consistency is insufficient (no relaxation); take 1 when consistency meets the preset conditions (e.g., continuous consistency for at least two steps). (Relaxation), in which It is a relaxation constant less than 1.

[0035] 3) To avoid introducing relaxation at uncertain positions, the implementation is further restricted: relaxation only occurs when the confidence level of that position in the previous decoding step has reached a preset threshold (e.g., reaching a certain threshold). The relaxation factor is only allowed to be applied when ( ).

[0036] Finally, the spatiotemporal adaptive threshold fusion and decoding rules are introduced. In one embodiment, the spatial adaptive threshold map is... With time relaxation factor Fusion yields a spatiotemporally adaptive threshold:

[0037] in, For position In the The token-level threshold for the decoding step.

[0038] Based on this, the submission set is determined. For each If and only if the confidence level at that position is Not less than its threshold At that time, the position is included in the decoding set; and the position is written into the prediction token. Once fixed, it will no longer participate in subsequent corrections; update the mask set. .

[0039] The proposed spatiotemporal stability-guided accelerated decoding method for large diffusion models is applied to inference decoding of large diffusion models, primarily including a testing phase. In the testing phase, the proposed spatiotemporal stability-guided decoding framework is used to infer from the input text or image and answer the input question. The method proposed in this invention is applicable to general large diffusion model inference processes. Furthermore, the proposed spatiotemporal stability-guided decoding framework can support inference for different large diffusion models (such as LLaDA-8B-Instruct, Dream-7b-Instruct, LaViDa, etc.). Figure 2A brief description of the specific implementation of the method proposed in this invention is given, and the specific steps are as follows: Step 1: Input Initialization and Parameter Setting. Obtain the text or image input to be generated or inferred. (Text tasks involve questions / instructions, multimodal tasks involve images and questions, etc.) Maximum number of denoising steps and sequence length Initialize the mask sequence (Set as a full mask or a mask sequence containing partially known tokens according to the diffusion model), and set the hyperparameters of the method of the present invention, including a high threshold for constructing the initial threshold map. With low threshold Spatial smoothing scale parameters and time relaxation constant Initialize the mask token location set And establish a cache to record the historical prediction token IDs for each location, so as to ensure the stability of subsequent calculation time.

[0040] Step Two: Model Prediction and Confidence Calculation. In the... One denoising step, to determine the sequence state of the currently determined tokens. With conditional input Input a large diffusion model to obtain all mask locations. vocabulary distribution ; Calculate the predicted token for each mask position and its confidence level This serves as the basis for the judgment in this step.

[0041] Step 3: Construct the initial threshold map and generate spatial thresholds. Construct the initial threshold map based on the current mask state. : for mask position Assigned value For the already determined location Assigned value This creates a threshold difference between the determined region and the region to be determined; subsequently, the threshold difference is... Perform local smoothing (preferably Gaussian smoothing, with the smoothing scale determined by...). (Control), to obtain spatial threshold map This allows mask positions in the "neighborhood of already determined tokens" to obtain a lower threshold, thus having a higher probability of early submission.

[0042] Step 4: Calculate temporal stability and generate temporal relaxation factors. For each mask position... Update the prediction token ID record for that position in the historical denoising steps, and calculate the length of consecutive consistent predictions. That is, the number of steps in which the predicted token ID at this position remains consistent with the prediction results of the previous steps; when At that time, the time relaxation factor at that location is set to (in Otherwise set to Preferably, to prevent uncertain tokens from being submitted prematurely due to accidental consistency, a gating constraint can be added: gating only occurs when the confidence level of the position in the previous denoising step reaches a preset lower limit (e.g., higher than a certain threshold). Relaxation factors are only allowed when... Otherwise keep .

[0043] Step 5: Fuse spatiotemporal information to generate an adaptive threshold and determine the decoding set. For each mask position... The token adaptive threshold is obtained by fusing the spatial threshold with the temporal relaxation factor. Multiplicative fusion is preferred: ; Calculate the confidence level at each location With the corresponding threshold Compare and construct the submission set and will The token in the middle position is fixed as .

[0044] Step Six: Update the mask set and iterate until the output terminates. Remove the committed positions from the mask set to obtain the next mask set. The updated sequence state is then input into the next denoising step, and steps two through six are repeated. The final result is output when the termination condition is met. The termination condition includes: the mask set is empty. Or reach the maximum number of noise reduction steps .

[0045] The specific application scenarios and examples of the method of the present invention are described below.

[0046] (1) Traffic Scene Question Answering: Applying the method of this invention to the diffusion multimodal large model LaViDa and using it for visual language question answering tasks in traffic scenes can improve the model's inference efficiency. See also Figure 3 Under the same test conditions, compared with the benchmark model, the inference time of the model using the method of this invention is reduced from 156.49 seconds to 22.23 seconds, thereby accelerating the inference process.

[0047] (2) General Scene Question Answering: Applying the method of this invention to the diffusion multimodal large model LaViDa and using it for visual language question answering tasks in general scene description can improve the model's inference efficiency. See also Figure 4 Under the same test conditions, compared with the benchmark model, the inference time of the model using the method of this invention is reduced from 155.44 seconds to 23.27 seconds, thereby accelerating the inference process.

[0048] Furthermore, the method of this invention can also be extended to general visual question answering tasks, and can achieve stable reasoning acceleration under different input scenarios.

Claims

1. A method for accelerating decoding of a large diffusion model based on spatiotemporal stability guidance, characterized in that, The inference decoding process for the diffusion large model, wherein the diffusion large model at each decoding step For the set of mask token positions Output prediction confidence With a predicted token identifier; the method includes: 1) Obtain text or image input With the initial mask sequence Let the length of the mask sequence be... Initialize the mask token location set And set the maximum number of iterations. ; 2) For the iteration step The currently determined set of tokens With the text or image input Input the large diffusion model to obtain the position of each mask. Predicted distribution The prediction token is determined by the prediction distribution. and its confidence level ; 3) Based on the aforementioned mask token location set Constructing an initial threshold map Among them, for the mask position Set a high threshold For non-mask positions Set a low threshold ; 4) For the initial threshold map Perform local smoothing to obtain the spatial threshold. ; 5) Based on mask position Calculate the consistency length based on the predicted token consistency in the historical iteration step. And generate a time relaxation factor based on the consistency length. ; 6) For each mask position Spatiotemporal thresholds are generated based on the spatial threshold and the temporal relaxation factor. And construct the submission set The submission set The tokens at each mask position are fixed to the corresponding predicted tokens. And update the mask token location set. ; 7) When Empty or iteration step Reaching the maximum number of iterations Output the decoding result when the time is right, otherwise let Then return to step 2) and continue iterating.

2. The accelerated decoding method for diffusion large models based on spatiotemporal stability guidance according to claim 1, characterized in that, In step 3), .

3. The accelerated decoding method for diffusion large models based on spatiotemporal stability guidance according to claim 1, characterized in that, In step 4), the local smoothing process is Gaussian smoothing, and the smoothing scale is determined by parameters. control, .

4. The accelerated decoding method for diffusion large models based on spatiotemporal stability guidance according to claim 1, characterized in that, Step 5) includes: for each mask position Record its predicted tokens in historical iteration steps The consistency length is calculated by counting the number of consecutive iterations in which the predicted token ID remains consistent. ;when Time relaxation factor generated ,when Time relaxation factor generated ,in And only if the mask position The time relaxation factor is activated only when the confidence level in the previous iteration meets the preset gating condition. Otherwise .

5. The method according to claim 4, characterized in that, The preset gating condition is: only when the mask position is... If the confidence level in the previous iteration is higher than the low threshold .

6. The method according to claim 1, characterized in that, Step 6) includes: for each mask position The spatiotemporal threshold is obtained by fusing the spatial threshold with the temporal relaxation factor. ,in The submission set is determined based on the confidence level and the spatiotemporal threshold. Fix the position in the submission set to the corresponding predicted token and update the mask token position set. .

Images