Discrete diffusion language model adaptive decoding method and system based on tail awareness

Abstract

The application provides a tail-aware-based discrete diffusion language model adaptive decoding method and system. The method comprises the following steps: setting decoding parameters and initializing; determining the boundary of a current decoding block based on the set decoding parameters and the residual tail position of a previous decoding block, performing in-block decoding on all decoding blocks one by one, and obtaining an in-block mask fluctuation rate; adaptively adjusting the length of a next decoding block according to the residual tail position of the current decoding block and the mask fluctuation rate; repeating the process of in-block decoding and adaptive adjustment of the length of the decoding block until a preset iteration termination condition is reached, and finally outputting a complete generated sequence. The application realizes the joint optimization of the block boundary problem and the fixed block length problem by adjusting the starting position of the decoding block and adaptively scaling the length of the decoding block.

Description

Technical Field

[0001] This invention relates to the field of training-free inference acceleration technology for block-based discrete diffusion language models. Specifically, it relates to an adaptive decoding method and system for discrete diffusion language models based on tail perception, and also to a corresponding computer terminal and computer-readable storage medium. Background Technology

[0002] Discrete Diffusion Language Model (dLLM) is one of the important research directions in the field of large language models in recent years. Its core idea is to generate complete text output by progressively denoising and predicting the mask positions in the sequence. Unlike the word-by-word generation method of autoregressive models, dLLM can achieve higher inference throughput under a parallel decoding framework and has broad application prospects.

[0003] However, in practical deployments, dLLM typically employs a block-wise decoding strategy, dividing the generated sequence into several fixed-length blocks and decoding them sequentially to achieve a balance between generation quality and inference speed. Existing block-wise decoding schemes have the following two shortcomings: Firstly, the block boundary cutting method is relatively fixed. When the number of unmaskings in the first step of a certain block is low (i.e., the "hard start" phenomenon), there are still a large number of mask positions at the end of the block that need to be forcibly unmasked within a limited number of steps, which causes low-confidence "hard tokens" to be forced to be written into the output, seriously affecting the local generation quality. Secondly, all blocks are decoded using a uniform fixed length, making it impossible to dynamically adjust the allocation of decoding resources based on the actual complexity of each block's content. For semantically complex blocks with unstable predictions, the fixed number of steps is often insufficient; while for blocks with simple content, it results in a waste of computational resources.

[0004] Existing methods such as FastdLLM, dLLM-Cache, and D2Cache do not systematically address the aforementioned block boundary and block length adaptation issues, thus limiting the generation quality and efficiency of block-based dLLM in practical applications. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, this invention provides an adaptive decoding method and system for a discrete diffusion language model based on tail perception, and also provides a corresponding computer terminal and computer-readable storage medium.

[0006] According to a first aspect of the present invention, an adaptive decoding method for discrete diffusion language models based on tail awareness is provided, characterized in that it includes: Set decoding parameters, including: initial block length [ Hard start determination threshold Residual tail trigger threshold Multiplicative contraction factor Additive expansion quantity MVR volatility benchmark and the starting position; initialize the block index. And set the starting position of the first decoded block. ; Based on the set decoding parameters and the residual tail position of the previous decoding block, the boundary of the current decoding block is determined, and all decoding blocks are decoded block by block to obtain the intra-block mask volatility. The length of the next decoding block is adaptively adjusted based on the residual tail position and mask volatility of the current decoding block. The process of repeatedly performing intra-block decoding and adaptively adjusting the length of the decoding block continues until the preset iteration termination condition is met, and finally the complete generated sequence is output.

[0007] Preferably, the step of determining the boundary of the current decoding block based on the set decoding parameters and the residual tail position of the previous decoding block, performing intra-block decoding on all decoding blocks one by one, and obtaining the intra-block mask volatility includes: For the Decoding blocks Perform intra-block decoding; where: Based on the current block Starting position and block length Determine the current block The decoding range; if there is a residual tail position migrated from the previous block, then that position is incorporated into the initial mask set of this block. This allows the current block to continue processing from the location of the token that was difficult to decode in the previous block; if there is no residual tail position migrated from the previous block, then the mask position within the current block range is used as the initial mask set. Steps within the current decoding block The previous loop performs denoising prediction; where, for each step Invoke the diffusion language model for the current residual mask set Predict all positions and obtain the top-1 prediction result and corresponding confidence score for each position; for positions with confidence scores higher than a set threshold... Normal unmasking is performed at the current location; if no location reaches the threshold in the current step... Then, the position with the highest unmasking confidence is forced to be unmasked; after unmasking is completed, the residual mask set is updated. ; During intra-block decoding, the set of positions that maintain the mask state in two consecutive steps. Statistics were compiled, among which Statistical analysis of the set The total number of block-level changes is obtained by counting the number of times the top-1 prediction result changes. Compare the total number And calculate the mask volatility. ;when When =0, let = ; During or at the end of intra-block decoding, if the current decoding block meets the preset residual tail determination condition, the residual tail position of that block is moved to the next decoding block as the residual tail. The initial mask set is then used for joint decoding by the next decoding block.

[0008] Preferably, the confidence level threshold is set as follows: .

[0009] Preferably, the preset residual tail determination conditions include: Number of first-step mask releases ; The number of remaining masks does not exceed If so, it is determined that the block has a residual tail.

[0010] Preferably, the step of adaptively adjusting the length of the next decoding block based on the residual tail position and mask volatility of the current decoding block includes: Mask volatility calculated based on the current decoded block The length of the next decoded block is determined according to the following rules. : like ,but ; like ,but ; Update the start position of the next decoding block based on the current decoded block length and residual tail migration status. That is, the position of the residual tail that is not fully decoded in the current decoding block is used as the starting position of the next decoding block, and the block is continuously covered from this starting position. Decoding begins at each sequence position.

[0011] Preferably, the iteration termination condition includes: Generate the sequence end marker EOS token, or, when the preset generation length is reached. .

[0012] According to a second aspect of the present invention, an adaptive decoding system for a discrete diffusion language model based on tail awareness is provided, comprising: The initialization decoding parameter module is used to set decoding parameters, including: initial block length, hard start decision threshold, residual tail trigger threshold, multiplication shrinkage factor, additive expansion amount, MVR fluctuation benchmark, and starting position; it also initializes the block index and sets the starting position of the first decoded block. The intra-block decoding module determines the boundary of the current decoding block based on the set decoding parameters and the residual tail position of the previous decoding block, performs intra-block decoding on all decoding blocks one by one, and obtains the intra-block mask volatility. An adaptive length adjustment module is used to adaptively adjust the length of the next decoding block based on the residual tail position and mask volatility of the current decoding block. The sequence output module generates a complete sequence by repeatedly executing the block-based decoding module and the adaptive length adjustment module until the preset iteration termination condition is met.

[0013] Based on the same inventive concept, other embodiments of the present invention provide a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform any of the methods in the above embodiments of the present invention, or to run any of the systems in the above embodiments of the present invention.

[0014] Based on the same inventive concept, other embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0015] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: This invention introduces a tail-aware fusion mechanism to migrate the tail position of the hard-start block residual to the next block for continued decoding, avoiding premature forced unmasking of low-confidence tokens near the block boundary, and effectively improving the generation coherence and text quality of the block boundary region.

[0016] This invention enables the model to dynamically allocate decoding resources based on the prediction complexity of different decoding blocks through an MVR-based block length adaptive strategy: shrinking the block length of high-fluctuation blocks to concentrate computing power, and expanding the block length of low-fluctuation blocks to improve throughput efficiency, thereby achieving a better balance between generation quality and inference efficiency.

[0017] This invention is a completely training-free inference scheduling scheme that does not require parameter fine-tuning or structural modification of the original discrete diffusion language model. It can be used as a plug-and-play module compatible with existing block-based dLLM inference frameworks, and has good versatility and deployment convenience.

[0018] This invention employs a tail-sensing fusion mechanism and a mask volatility control mechanism, which can be enabled independently or work in tandem. It is applicable to a variety of dLLM base models and generation tasks and has strong practical value.

[0019] This invention defines volatility as the uncertainty of a token, that is, the situation where the top-1 prediction result of the same token is reversed between two adjacent denoising steps, and quantifies it as mask volatility (MVR) (the proportion of top-1 prediction changes in positions where the mask state is maintained for two consecutive steps). This volatility is a time-dimensional, quantifiable block-level complexity metric, which, unlike existing technologies that only use cross-sectional statistics of the built-in confidence of the same denoising step, can directly characterize the unstable phenomenon of repeated prediction jumps.

[0020] The present invention addresses the residual tail position of the previous decoding block (i.e., the number of demasks in the first step is less than a threshold). The hard start block, and the remaining mask at the end does not exceed the threshold. The position of the residual mask determines the starting position of the next decoding block. The block boundary determination does not depend on any explicit semantic delimiters. It belongs to the boundary determination based on the decoding state and is unrelated to whether the generated text contains punctuation or delimiters. Therefore, it can naturally cover generation scenarios without obvious delimiters, such as code generation.

[0021] This invention addresses the problem that existing technologies only use volatility to reveal semantic structure. Based on the relationship between mask volatility (MVR) and a preset volatility benchmark, volatility is directly incorporated into the block length decision, and the length of the next decoding block is asymmetrically adaptively scaled: high volatility blocks are multiplicatively shrunk to concentrate computing power, and low volatility blocks are additively expanded to improve throughput, so that volatility truly participates in the dynamic allocation of decoding resources.

[0022] This invention addresses the problem of low-confidence tokens being prematurely submitted at block boundaries in existing technologies. It abandons the passive rollback to the default block length and actively migrates the residual tail at the end of the hard-start block to the next block for joint decoding through a tail-aware fusion mechanism, thereby preventing premature forced unmasking of such tokens from the mechanism. Attached Figure Description

[0023] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating the workflow of an adaptive decoding method for a discrete diffusion language model based on tail perception in a preferred embodiment of the present invention.

[0024] Figure 2This is a schematic diagram of the constituent modules of the tail-aware discrete diffusion language model adaptive decoding system in a preferred embodiment of the present invention. Detailed Implementation

[0025] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0026] Existing block-based decoding methods for Discrete Diffusion Language Models (dLLM) typically suffer from two main problems: First, they usually divide the generation sequence according to fixed boundaries. If a decoding block can only unmask a small number of tokens in the first step, it indicates that the block is in a state of prediction instability or insufficient context. If the residual tail position is forcibly unmasked in the current block, it is easy to generate low-confidence hard tokens, which in turn affects the local generation quality. Second, they usually use the same block length for all decoding blocks, making it impossible to dynamically adjust computational resources according to the prediction difficulty of different blocks. For blocks with unstable predictions, a fixed block length may lead to insufficient decoding resources; for blocks with stable predictions, a fixed block length may result in wasted computation.

[0027] To address the hard-start and low-confidence forced unmasking issues caused by fixed block boundaries, as well as the uneven distribution of decoding resources due to fixed block lengths, one embodiment of this invention provides an adaptive decoding method for discrete diffusion language models based on tail-aware fusion. This method, based on tail-aware fusion and mask volatility control, consists of two parts: adjusting the starting position of the decoding block and adaptively scaling the decoding block length. The starting position of the decoding block is determined by the location of the difficult-to-decode tokens in the previous decoding block, thus mitigating the negative impact of block boundary truncation on generation quality. The size of the decoding block is determined by the inversion rate of the mask tokens in the previous decoding block to assess the stability of the current sequence generation. This stability is then used to determine the size of the decoding block, thereby improving overall throughput efficiency. This method achieves joint optimization of both block boundary and fixed block length issues by adjusting the starting position of the decoding block and adaptively scaling the decoding block length.

[0028] Specifically, such as Figure 1 As shown, the tail-aware adaptive decoding method for discrete diffusion language models provided in this embodiment may include: S1, Initialize decoding parameters. Set decoding parameters, including: initial block length. [ The hard start determination threshold is... The residual tail trigger threshold is The multiplicative contraction factor is The addition expansion amount is MVR fluctuation benchmark is and the starting position; initialize the block index. And set the starting position of the first decoded block to be ; S2, Decoding is performed block by block. Based on the set decoding parameters and the residual tail position of the previous decoding block, the boundary of the current decoding block is determined. Intra-block decoding is performed on all decoding blocks one by one, and the intra-block mask volatility is obtained. S3, Adaptively adjust the length of the next block. Based on the residual tail position and mask volatility of the current decoded block, the length of the next decoded block is adaptively adjusted. S4, iterate until the end. Repeat steps S2 and S3 until the preset iteration termination condition is met, and finally output the complete generated sequence.

[0029] In some preferred embodiments, the above-mentioned S1, initializing the decoding parameters, may further include: Setting: Initial block length is Initial block length The range is [ The hard start threshold is... The residual tail trigger threshold is The multiplicative contraction factor is The additive expansion amount is MVR volatility benchmark is ; Initialize block index And set the starting position of the first decoded block to be .

[0030] In some preferred embodiments, S2, based on the set decoding parameters and the residual tail position of the previous decoding block, determines the boundary of the current decoding block, performs intra-block decoding on all decoding blocks one by one, and obtains the intra-block mask volatility, and may further include: For the Decoding blocks Perform intra-block decoding; where: S21, Determine block boundaries: based on the current block start position. and block length Determine the decoding range of the current block; if there is a residual tail position migrated from the previous block, then incorporate that position into the initial mask set of this block. This allows the current block to continue processing from the location of the token that was difficult to decode in the previous block; if there is no residual tail position migrated from the previous block, the mask position within the current block range is used as the initial mask set; where the location of the decoded token refers to the token whose confidence level can never reach the threshold set for decoding, corresponding to the residual tail position migrated from the previous block. S22, Perform intra-block denoising iteration: In the current decoding block, step... The previous loop performs denoising prediction; where, for each step Invoke the diffusion language model for the current residual mask set Predict all positions and obtain the top-1 prediction result and corresponding confidence score for each position; for positions with confidence scores higher than a set threshold... (Preferred, Normal unmasking is performed at the position where the threshold is reached; if no position reaches the threshold in the current step, the position with the highest confidence is forcibly unmasked to avoid the decoding process stalling; after unmasking is completed, the residual mask set is updated. Among them, the residual mask set This refers to the set of sequence positions within the nth decoding block that are still in a masked state (not yet demasked and assigned values) after the denoising prediction step t has been performed. Initial value This represents all mask positions at the start of decoding this block (including the residual tail positions migrated from the previous block, if any); after each demasking step, the assigned positions are removed from the set and updated to... The size of the set decreases monotonically as the steps progress. When the decoding within the block ends, if the set is not empty and the residual tail determination condition is met, the remaining positions are moved to the next block as the residual tail. S23, Statistical Mask Fluctuation Rate: During intra-block decoding, the set of positions where the mask state is maintained for two consecutive steps. Statistics were compiled, among which Statistical analysis of the set The total number of block-level changes is obtained by counting the number of times the top-1 prediction result changes. Compare the total number And calculate the mask volatility. ;when When =0, let Among them, the total number of comparisons It refers to all the positions in the nth decoding block where the mask state is maintained for two consecutive steps (i.e., the set). The cumulative number of elements in each step, i.e. Each matching condition (position, step pair) contributes one comparison opportunity; The number of times the top-1 prediction changes is counted; the ratio of these two is the mask volatility. .when When this occurs, it indicates that there are no comparable locations within the block, and the preset fluctuation benchmark is used. As the default value; S24, Perform tail-aware fusion: During or at the end of intra-block decoding, if the current decoding block meets the following two conditions, the residual tail position of that block is migrated to the next decoding block as the residual tail. In the initial mask set, the next decoding block continues joint decoding, instead of forcibly demasking that position within the current decoding block: First, the number of masks unmasked in the first step. ; Second, the number of remaining masks does not exceed ; If both of the above conditions are met, the block is determined to have a residual tail.

[0031] In some preferred embodiments, S3, which adaptively adjusts the length of the next decoding block based on the residual tail position and mask volatility of the current decoding block, may further include: S31, Mask fluctuation rate calculated based on the current decoded block The length of the next decoded block is determined according to the following rules. : like > ,but ; like ≤ ,but ; S32, update the start position of the next decoding block based on the current decoded block length and residual tail migration status. That is, the position of the residual tail that is not fully decoded in the current decoding block is taken as the starting position of the next decoding block, and the block is continuously covered from this starting position. Decoding begins at each sequence position.

[0032] In some preferred embodiments, the iteration termination condition S4 above may further include: Generate the sequence end marker EOS token, or, when the preset generation length is reached. .

[0033] Based on the same inventive concept, another embodiment of the present invention provides an adaptive decoding system for a discrete diffusion language model based on tail perception.

[0034] Specifically, such as Figure 2 As shown, the tail-aware discrete diffusion language model adaptive decoding system provided in this embodiment may include: The initialization decoding parameter module is used to set decoding parameters, including: initial block length, hard start decision threshold, residual tail trigger threshold, multiplication shrinkage factor, additive expansion amount, MVR fluctuation benchmark, and starting position; it also initializes the block index and sets the starting position of the first decoded block. The intra-block decoding module determines the boundary of the current decoding block based on the set decoding parameters and the residual tail position of the previous decoding block, performs intra-block decoding on all decoding blocks one by one, and obtains the intra-block mask volatility. An adaptive length adjustment module is used to adaptively adjust the length of the next decoding block based on the residual tail position and mask volatility of the current decoding block. The sequence output module generates a complete sequence by repeatedly executing the block-based decoding module and the adaptive length adjustment module until the preset iteration termination condition is met.

[0035] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.

[0036] The tail-aware adaptive decoding method and system for discrete diffusion language models provided in the above embodiments of the present invention address the hard-start and low-confidence forced unmasking problems caused by fixed block boundaries by setting a tail-aware fusion mechanism (TAM). This mechanism detects the number of unmasks in the first step of each block and identifies blocks with a first-step unmasking count below a threshold. Hard-start blocks. For this type of block, if the size of its residual mask set at the end does not exceed a threshold. If the residual tail is not forcibly decoded within the current block, it is instead moved to the beginning of the next block for joint decoding, thus mitigating the negative impact of block boundary truncation on generation quality. To address the uneven distribution of decoding resources caused by fixed block lengths, a mask volatility control mechanism is implemented, introducing Mask Volatility (MVR) as a measure of block-level complexity. MVR quantifies the instability of predictions within a block by statistically analyzing the frequency of changes in the top-1 predictions at each position during decoding. With preset fluctuation threshold The relationship allows for asymmetric adaptive adjustment of the next block length: when > When using the multiplicative contraction factor Shorten the next block to concentrate decoding resources; when ≤ At that time, the additive expansion was used. Increase the length of the next block to improve overall throughput efficiency.

[0037] The following detailed explanation, using a specific application example, further illustrates the application effect of the technical solution provided by the above embodiments of the present invention in a specific task.

[0038] In this specific application example, taking code generation as an example, the LLaDA discrete diffusion language model is used to generate a corresponding code sequence based on a given natural language description (such as "implement a Python function to merge and sort an input list and return the sorted result"). The preset generation length is G=256 tokens. Based on the method provided in the above embodiments of the present invention, the following steps are performed: S1, Initialize decoding parameters. Set the initial block length. =32, =8, =64, hard start threshold =4, residual tail trigger threshold =3, multiplicative contraction factor =0.75, additive expansion amount =8, MVR volatility benchmark =0.3, starting position =0; initialize block index =0, setting the starting position of the first decoded block to 0. .

[0039] S2, Decoding is performed block by block. For the 0th decoded block... (Covering generated sequence positions 0 to 31) Perform intra-block decoding: Use all 32 mask positions as the initial mask set. The denoising prediction is performed iteratively, with each step calling the LLaDA model to predict each position in the current residual mask set, and for positions with a confidence level higher than 100%, the prediction is performed. At positions with a confidence level of 0.9, normal demasking is performed. If no position reaches the threshold in the current step, the position with the highest confidence level is forcibly demasked. After demasking is completed, the residual mask set is updated. During intra-block decoding, the set of positions that maintain the mask state for two consecutive steps is counted. The total number of block-level changes is obtained by counting the number of times the top-1 prediction changes. Compare the total number ,calculate If the number of masks released in the first step And the number of residual masks at the end of the block does not exceed If the value is 3, then the tail position of the residual is moved to the starting mask set of the next decoding block, and the next block continues to decode.

[0040] S3, adaptively adjusts the length of the next block. If (This indicates that there are unstable situations such as syntactic ambiguity or repeated jumps in function name prediction within the code block.) Centralized computing power is used to process this complex segment; if (This indicates that the current code snippet prediction is stable, such as numeric literals or simple assignment statements.) The block length is increased to improve throughput efficiency. If there is residual tail migration in the previous block, the next block starts from the position of that residual tail. Continuous coverage backwards Decode each sequence position; otherwise, use... This is the starting position.

[0041] S4, iterate until completion. Repeat steps S2 and S3 until an EOS token appears in the generation sequence or the cumulative generation length reaches the preset generation length. =256, and finally output the complete code sequence.

[0042] As can be seen from this specific task, the technical solution provided by the above embodiments of the present invention has the following application effects: In code generation scenarios, code sequences typically include segments with strong syntactic constraints such as function definitions, control flow, and indentation structures (high volatility, requiring concentrated computing power) and segments with fixed structures such as variable assignments, numerical constants, and simple expressions (low volatility, allowing for expansion of block length); the present invention adaptively adjusts the block length through MVR, enabling the LLaDA model to reduce the block length of high-volatile code segments to concentrate decoding resources and reduce the forced unmasking rate, and to expand the block length of low-volatile code segments to reduce block switching overhead and improve inference throughput; at the same time, through the tail-aware fusion mechanism, residual tokens that are difficult to decode (such as keywords in cross-line syntax structures, closing brackets, etc.) are migrated to the next block for joint decoding, avoiding premature forced submission of low-confidence tokens that lead to syntax errors, effectively improving the syntax accuracy and overall generation quality of code generation.

[0043] Based on the same inventive concept, other embodiments of the present invention provide a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to perform any of the methods in the above embodiments of the present invention, or to run any of the systems in the above embodiments of the present invention.

[0044] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0045] A processor is used to execute computer programs stored in memory to implement the various steps in the methods or modules in the systems described in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0046] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0047] Based on the same inventive concept, other embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0048] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0049] The tail-aware adaptive decoding method and system for discrete diffusion language models provided in the above embodiments of the present invention, by introducing a tail-aware fusion mechanism, migrates the tail position of the hard-start block residual to the next block for continued decoding, avoiding premature forced unmasking of low-confidence tokens near the block boundary, and effectively improving the generation coherence and text quality of the block boundary region. Through an MVR-based block length adaptive strategy, the model can dynamically allocate decoding resources according to the prediction complexity of different decoding blocks: shrinking the block length for high-volatility blocks to concentrate computing power, and expanding the block length for low-volatility blocks to improve throughput efficiency, thereby achieving a better balance between generation quality and inference efficiency. Employing a completely training-free inference scheduling scheme, it requires no parameter fine-tuning or structural modification of the original discrete diffusion language model, and can be used as a plug-and-play module compatible with existing block-based dLLM inference frameworks, exhibiting good versatility and deployment convenience. The tail-aware fusion mechanism and the mask volatility control mechanism can be enabled independently or work together, applicable to various dLLM base models and generation tasks, and have strong practical value.

[0050] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0051] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. An adaptive decoding method for discrete diffusion language models based on tail-awareness, characterized in that, include: Set decoding parameters, including: initial block length [ Hard start determination threshold Residual tail trigger threshold Multiplicative contraction factor Additive expansion quantity MVR volatility benchmark and the starting position; initialize the block index. And set the starting position of the first decoded block. ; Based on the set decoding parameters and the residual tail position of the previous decoding block, the boundary of the current decoding block is determined, and all decoding blocks are decoded block by block to obtain the intra-block mask volatility. The length of the next decoding block is adaptively adjusted based on the residual tail position and mask volatility of the current decoding block. The process of repeatedly performing intra-block decoding and adaptively adjusting the length of the decoding block continues until the preset iteration termination condition is met, and finally the complete generated sequence is output.

2. The adaptive decoding method for discrete diffusion language models based on tail perception according to claim 1, characterized in that, The process involves determining the boundary of the current decoding block based on the set decoding parameters and the residual tail position of the previous decoding block, performing intra-block decoding on each of all decoding blocks, and obtaining the intra-block mask volatility, including: For the Decoding blocks Perform intra-block decoding; where: Based on the current block Starting position and block length Determine the current block The decoding range; if there is a residual tail position migrated from the previous block, then that position is incorporated into the initial mask set of this block. This allows the current block to continue processing from the location of the token that was difficult to decode in the previous block; if there is no residual tail position migrated from the previous block, then the mask position within the current block range is used as the initial mask set. Steps within the current decoding block The previous loop performs denoising prediction; where, for each step Invoke the diffusion language model for the current residual mask set Predict all positions and obtain the top-1 prediction result and corresponding confidence score for each position; for positions with confidence scores higher than a set threshold... Normal unmasking is performed at the current location; if no location reaches the threshold in the current step... Then, the position with the highest unmasking confidence is forced to be unmasked; after unmasking is completed, the residual mask set is updated. ; During intra-block decoding, the set of positions that maintain the mask state in two consecutive steps. Statistics were compiled, among which Statistical analysis of the set The total number of block-level changes is obtained by counting the number of times the top-1 prediction result changes. Compare the total number And calculate the mask volatility. ;when When =0, let = ; During or at the end of intra-block decoding, if the current decoding block meets the preset residual tail determination condition, the residual tail position of that block is moved to the next decoding block as the residual tail. The initial mask set is then used for joint decoding by the next decoding block.

3. The adaptive decoding method for discrete diffusion language models based on tail perception according to claim 2, characterized in that, The set threshold Set as =0.

9.

4. The adaptive decoding method for discrete diffusion language models based on tail perception according to claim 2, characterized in that, Preset residual tail determination conditions, including: Number of first-step mask releases And the number of remaining masks does not exceed If so, it is determined that the block has a residual tail.

5. The adaptive decoding method for discrete diffusion language models based on tail awareness according to claim 1, characterized in that, The step of adaptively adjusting the length of the next decoding block based on the residual tail position and mask volatility of the current decoding block includes: Mask volatility calculated based on the current decoded block The length of the next decoded block is determined according to the following rules. : like ,but ; like ,but ; Based on the current decoded block length and residual tail migration, determine the starting position of the next decoded block. Update to the residual tail position of the current decoded block, and start from that position. backward continuous coverage Decoding begins at each sequence position.

6. The adaptive decoding method for discrete diffusion language models based on tail awareness according to claim 1, characterized in that, The iteration termination conditions include: Generate the sequence end marker EOS token, or, when the preset generation length is reached. .

7. An adaptive decoding system for a discrete diffusion language model based on tail perception, characterized in that, include: The initialization decoding parameter module is used to set decoding parameters, including: initial block length, hard start decision threshold, residual tail trigger threshold, multiplication shrinkage factor, additive expansion amount, MVR fluctuation benchmark, and starting position; it also initializes the block index and sets the starting position of the first decoded block. The intra-block decoding module determines the boundary of the current decoding block based on the set decoding parameters and the residual tail position of the previous decoding block, performs intra-block decoding on all decoding blocks one by one, and obtains the intra-block mask volatility. An adaptive length adjustment module is used to adaptively adjust the length of the next decoding block based on the residual tail position and mask volatility of the current decoding block. The sequence output module generates a complete sequence by repeatedly executing the block-based decoding module and the adaptive length adjustment module until the preset iteration termination condition is met.

8. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program can be used to perform the method of any one of claims 1-6.

Images