Text acceleration generation method and system based on dynamic masking and parallel decoding

Abstract

This invention provides a text acceleration generation method and system based on dynamic masking and parallel decoding. The technical solution of this invention introduces a dynamic masking mechanism into a large model with parallel decoding capabilities. Based on the attention weight matrix calculated by the large model in each round, the information entropy of each element in the filling sequence is dynamically calculated. The input and output masks are dynamically determined based on the information entropy to dynamically shield noise and accelerate the decoding process. Furthermore, this invention performs parallel iterative decoding in stages. In the early stages of iteration, a momentum acceleration mode is used in each round to accelerate decoding efficiency through momentum terms. After each round, the convergence speed of the information entropy is determined. When the convergence speed of the information entropy is less than or equal to the convergence threshold, a normal iterative mode without momentum acceleration is used in each round to reduce later oscillations and facilitate stability at the optimal convergence point, enabling faster and better parallel decoding.

Description

Technical Field

[0001] This invention relates to the field of deep learning acceleration, specifically to the field of natural language processing, and more specifically to a method and system for accelerating text generation based on dynamic masking and parallel decoding. Background Technology

[0002] In recent years, generative artificial intelligence (AIGC) technology, driven by large language models (LLMs), has made groundbreaking progress and triggered a new wave of technological innovation globally. AIGC has profoundly changed the way content is created, automatically generating various forms of content such as text, images, code, audio, and video. It has been widely applied in many industries, including media, finance, healthcare, and education, demonstrating enormous commercial value and social impact.

[0003] The core of this technological revolution lies in the sequence generation task, where a model generates a completely new and coherent sequence of symbols based on input information. Currently, Autoregressive Decoding (AR) is the most mainstream paradigm for sequence generation. When generating each token, the AR model relies on all previously generated tokens; this "word-by-word" generation method ensures the internal consistency and high quality of the generated sequence. However, this comes at the cost of significant computational latency, as the generation process cannot be parallelized, and the total latency increases linearly with the sequence length. In applications that prioritize real-time interaction and efficient service, such as online translation, intelligent dialogue, and code assistants, this high latency becomes a key bottleneck affecting technology implementation and user experience.

[0004] To overcome the performance bottleneck of autoregressive decoding, academia and industry have turned their attention to parallel decoding techniques, also known as non-autoregressive generation. Unlike autoregressive decoding, parallel decoding allows the generation of the entire sequence or most of the sequence at once, continuously refining it through iterative optimization. This highly parallelizes the computational process and significantly reduces generation latency. For example, decoding methods based on the Jacobi Iteration approach offer a novel solution for achieving efficient sequence generation by iteratively updating the candidate sequences of the entire sequence in parallel.

[0005] However, despite the enormous potential of parallel decoding technology in terms of acceleration, existing implementation methods still face three core challenges, making it difficult for them to fully match mature autoregressive models in terms of convergence speed and generation quality:

[0006] 1. Convergence efficiency bottleneck: Traditional Jacobi decoding uses a fixed number of iteration rounds (usually equal to the maximum sequence length), which cannot actively suppress invalid oscillations of high-entropy noise tokens. Experiments show that in machine translation tasks, approximately 40% of the iteration rounds are consumed in redundant computation of low-information tokens;

[0007] 2. Lack of information selection mechanism: Existing masking strategies mostly use static templates (such as diagonal masks), which affects the modeling ability and fails to dynamically perceive the distribution of key information in the decoding process. For example, although some hierarchical masks improve local convergence, their fixed parameters lead to a decline in long-range dependency modeling ability;

[0008] 3. Rigid Termination Conditions: Preset thresholds or fixed rounds are difficult to adapt to the dynamic entropy changes in the generation process. While reinforcement learning is effective in optimizing the termination point, its online decision-making module adds extra computational overhead, making it difficult to meet real-time requirements.

[0009] Considering the current state of technology, the core shortcomings of existing parallel decoding methods that this invention aims to address can be summarized as follows:

[0010] 1. Low convergence efficiency: The conventional iterative mode results in slow convergence speed, leading to low convergence efficiency;

[0011] 2. Mask rigidity: Static templates cannot reflect the spatiotemporal evolution of attention weights, resulting in the attenuation of key semantic information.

[0012] The following is information on the references for some existing technologies:

[0013] [1] Santilli, Andrea et al. “Accelerating Transformer Inference for Translation via Parallel Decoding.” Annual Meeting of the Association for Computational Linguistics (2023).

[0014] [2] Kou, Siqi et al. “CLLMs: Consistency Large Language Models.” ArXiv abs / 2403.00835 (2024): n. pag.

[0015] [3] Che Wanxiang, Dou Zhicheng, Feng Yansong, et al. Natural Language Processing in the Era of Large Models: Challenges, Opportunities and Development [J]. Science in China: Information Science, 2023, 53(9): 1645-1687.

[0016] [4] Huang Fei. Research on Natural Language Generation Based on Non-Autoregressive Model [D]. Tsinghua University, 2023. DOI:10.27266 / d.cnki.gqhau.2023.000107.

[0017] It should be noted that the background information presented here is only for illustrating relevant information about the present invention to aid in understanding the technical solution of the present invention, and does not imply that the relevant information is necessarily prior art. The relevant information was submitted and disclosed together with the present invention, and should not be considered prior art unless there is evidence that the relevant information was disclosed before the filing date of the present invention. Summary of the Invention

[0018] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a text acceleration generation method and system based on dynamic masking and parallel decoding.

[0019] The objective of this invention is achieved through the following technical solution:

[0020] According to a first aspect of the present invention, a method for text generation is provided, comprising: S1, acquiring a large model with parallel decoding capability, wherein the input tensor of the large model is composed of an input text sequence and a padding sequence, the large model is configured for multi-round iterative decoding, wherein in each round a forward computation is performed based on the input tensor to obtain the hidden state output at each position, and the input tensor of the next round is composed of the input text sequence and the updated padding sequence extracted from the hidden state output at each position; S2, introducing a dynamic masking mechanism for the large model to dynamically calculate the information entropy at each position based on the attention weight matrix calculated by the large model in each round, and determining the input mask and the output mask based on the information entropy, wherein the input mask is used to filter the input tensor padding sequence by the input text sequence. Elements with entropy values ​​less than the low entropy threshold are retained, while elements with entropy values ​​greater than or equal to the low entropy threshold are set to zero. The output mask is used to retain elements in the updated output filling sequence whose corresponding entropy values ​​are greater than the high entropy threshold, while setting elements with entropy values ​​less than or equal to the high entropy threshold to zero. S3: The large model performs parallel iterative decoding in stages by introducing a dynamic mask mechanism. In the first stage, each round adopts a momentum acceleration mode, and the convergence speed of the information entropy is determined after each round. When the convergence speed is less than or equal to the convergence threshold, the process switches to the second stage to adopt a normal iterative mode without momentum acceleration in each round. S4: When the iterative decoding meets the convergence condition, the final token sequence is extracted from the filling sequence obtained in the last round of iteration and converted into output text.

[0021] Optional, the normal iteration mode is:

[0022]

[0023] in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. This indicates the output mask.

[0024] Optional, momentum acceleration mode is:

[0025]

[0026] in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. Indicates the output mask. This represents the parameter of the first momentum term. This represents the parameter of the second momentum term.

[0027] Optionally, the input and output masks can be calculated in the following ways:

[0028]

[0029]

[0030] in, Indicates the first The input mask of the wheel regarding position mask value, Indicates the first Wheel position Information entropy Indicates the low entropy threshold. Indicates the first The output mask of the wheel regarding position mask value, This represents the high entropy threshold.

[0031] Optionally, when the variance of information entropy is detected to indicate that the entropy value oscillation exceeds a preset range, the low entropy threshold is forcibly increased and the high entropy threshold is decreased.

[0032] Optionally, the parallel decoding logic of the large model is configured as follows: obtain the maximum generation length, initialize the padding sequence with the padding length determined by the maximum generation length and the length of the input text sequence; concatenate the input text sequence and the padding sequence into the input tensor of the large model; the large model iteratively performs decoding calculations based on the input tensor, updating the content of the padding sequence in each round; after each round of decoding, determine whether the convergence condition is met. If so, the decoding terminates; otherwise, proceed to the next round of iteration based on the new input tensor obtained from the input text sequence and the updated padding sequence. When the decoding terminates, construct the output text based on the final padding sequence.

[0033] Optionally, convergence is considered satisfied when any of the following conditions are met: Condition 1: When the oscillation amplitude of information entropy tends to stabilize, and when the variance of information entropy for consecutive preset rounds is less than a preset variance threshold; Condition 2: When the cumulative number of iterations of iterative decoding reaches the maximum number of iterations.

[0034] According to a second aspect of the present invention, a method for text generation is provided, comprising: acquiring a large model with parallel decoding capability, wherein the input tensor of the large model is composed of an input text sequence and a padding sequence, the large model is configured for multi-round iterative decoding, wherein in each round a forward computation is performed based on the input tensor to obtain the hidden state output at each position, and the input tensor of the next round is composed of the input text sequence and the updated padding sequence extracted from the hidden state output at each position; introducing a dynamic masking mechanism for the large model to dynamically calculate the information entropy at each position based on the attention weight matrix calculated by the large model in each round, determining an input mask and an output mask based on the information entropy, wherein the input mask is used to retain elements in the padding sequence of the input tensor whose entropy value is less than a low entropy threshold and to set elements whose entropy value is greater than or equal to the low entropy threshold to zero, and the output mask is used to retain elements in the updated padding sequence of the output tensor whose entropy value is greater than a high entropy threshold and to set elements whose entropy value is less than or equal to the high entropy threshold to zero; and performing each iterative decoding according to a preset iterative mode using the input mask and the output mask, wherein the preset iterative mode is a normal iterative mode. Or momentum acceleration mode: , Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. Indicates the output mask. This represents the parameter of the first momentum term. This represents the second momentum term parameter; when the iterative decoding satisfies the convergence condition, the final token sequence is extracted from the filling sequence obtained in the last iteration, and the final token sequence is converted into the output text.

[0035] According to a third aspect of the present invention, a system with parallel decoding capability is provided for implementing the method described in the first aspect. The system comprises: a large model scheduling module for acquiring a large model with parallel decoding capability, wherein the input tensor of the large model is formed by concatenating an input text sequence and a padding sequence, the large model is configured for multi-round iterative decoding, wherein in each round a forward computation is performed based on the input tensor to obtain the hidden state output at each position, and the input tensor of the next round is formed by concatenating the input text sequence and the updated padding sequence extracted from the hidden state output at each position; an input preprocessing module for generating the input tensor of the large model in each iteration; and a dynamic mask calculation module for dynamically calculating the information entropy at each position based on the attention weight matrix calculated by the large model in each round, and determining the input mask and output mask based on the information entropy, wherein the input mask is used to... In the input tensor's filling sequence, elements with entropy values ​​less than the low entropy threshold are retained, while elements with entropy values ​​greater than or equal to the low entropy threshold are set to zero. The output mask is used to retain elements with entropy values ​​greater than the high entropy threshold in the updated output filling sequence, while setting elements with entropy values ​​less than or equal to the high entropy threshold to zero. A scheduler is used to perform parallel iterative decoding in stages using a large model with a dynamic masking mechanism. In the first stage, each round uses momentum acceleration mode and determines the convergence speed of information entropy after each round. When the convergence speed is less than or equal to the convergence threshold, it switches to the second stage to use normal iterative mode without momentum acceleration in each round. A termination judgment module is used to monitor whether the iterative decoding meets the convergence condition. If so, it triggers the output module to be enabled. The output module is used to extract the final token sequence from the filling sequence obtained in the last round of iteration and convert it into output text.

[0036] According to a fourth aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to perform the steps of the method as described in the first or second aspect by executing the executable instructions. Attached Figure Description

[0037] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0038] Figure 1 This is a flowchart illustrating a text generation method according to an embodiment of the present invention;

[0039] Figure 2This is a schematic diagram illustrating the impact of each module on the speed and quality of parallel decoding generation according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram illustrating the impact of momentum-free acceleration, pure momentum acceleration, and mixed momentum acceleration on the convergence of parallel decoding iterations according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0042] As mentioned in the background section, existing technologies suffer from iterative redundancy and mask rigidity. The technical solution of this invention introduces a dynamic masking mechanism into a large model with parallel decoding capabilities. Based on the attention weight matrix calculated by the large model in each round, the information entropy of each element in the filling sequence is dynamically calculated. The input mask and output mask are dynamically determined based on the information entropy to dynamically shield noise and accelerate the decoding process. In addition, this invention also performs parallel iterative decoding in stages. In the early stage of iteration, a momentum acceleration mode is used in each round to accelerate the decoding efficiency through the momentum term. After each round, the convergence speed of the information entropy is determined. When the convergence speed of the information entropy is detected to be less than or equal to the convergence threshold, a normal iterative mode without momentum acceleration is used in each round to reduce later oscillations and make it easier to stabilize at the optimal convergence point, thus achieving parallel decoding faster and better.

[0043] The purpose of this application is to provide a model acceleration method based on information entropy engineering and momentum equation optimization to fill the gap in parallel decoding acceleration and low-entropy system regulation, improve the inference speed of models, especially generative large language models based on transformers, and provide an explanation for the operation process of the model system.

[0044] According to one embodiment of the present invention, see Figure 1 This invention provides a method for text generation, including steps S1, S2, S3, and S4. To better understand this invention, each step will be described in detail below with reference to specific embodiments.

[0045] Step S1: Obtain a large model with parallel decoding capability. The input tensor of the large model is composed of the input text sequence and the padding sequence. The large model is configured for multi-round iterative decoding. In each round, forward computation is performed based on the input tensor to obtain the hidden state output at each position. The input tensor of the next round is composed of the input text sequence and the updated padding sequence extracted from the hidden state output at each position.

[0046] According to one embodiment of the present invention, in order to achieve parallel decoding, a parallel decoding framework needs to be pre-built. The generation mechanism of the existing token-by-token (also called term) autoregressive model is modified to enable parallel decoding. Optionally, a large model with parallel decoding capabilities can be built based on the Transformer architecture. The parallel decoding logic of the large model is configured as follows:

[0047] Set maximum generation length The initial padding sequence is set to a length of [length missing]. Placeholder;

[0048] The input text sequence and the padding sequence are concatenated to form the model input. ,in, This represents the input text sequence. Indicates the initial filling sequence. That is ;

[0049] The decoding calculation is performed iteratively, updating the contents of the fill sequence in each round until the difference between adjacent iteration results is lower than a preset threshold, or the number of iterations reaches a certain threshold. ;

[0050] After decoding terminates, extract the first sequence terminator from the last obtained padding sequence. The preceding content is converted into output text.

[0051] According to one embodiment of the present invention, the generation mechanism of a large model with parallel decoding capability is configured as follows: a padding sequence of a preset maximum generation length is concatenated into the input text sequence as an initial input tensor, and the padding content is dynamically updated through multiple rounds of iterative decoding, as expressed in: ),in, It is a parallel decoding function, representing a large model with parallel decoding capabilities; This represents the sequence of input text (i.e., the question text) for generating the task. This represents the output obtained in the previous iteration. In other words, the function receives a concatenated structure of the input text sequence for the generation task and the output obtained in the previous iteration. Indicates the large model in Output of each round.

[0052] According to an embodiment of the present invention, the parallel decoding logic of a large model includes: determining the maximum generation length, constructing the initial padding sequence, input concatenation processing, constructing the parallel decoding iterative execution process, and setting the termination condition. The following describes each sub-process:

[0053] Maximum generation length determination: The maximum generation length is set based on the average sentence length of the target language and the maximum input length supported by the large model. In the initial stage, the maximum generation length parameter is set. Its value needs to be determined comprehensively based on task characteristics (such as the typical sentence length distribution of the target language) and the model's context window capacity (such as the maximum positional encoding length of the Transformer). For example, in machine translation tasks, if the average sentence length of the target language is approximately 100 words and the model supports a maximum length of 512, then... It can take values ​​between 100 and 512, and can be set to 128 or 256. Then, based on the original input sequence... actual length Dynamically calculate the length of the padding sequence required: ,in, The length of the input text sequence is 2, minus 2 is to preserve the special markers at the beginning and end.

[0054] Construction of the padding sequence: Using predefined placeholders (such as the special symbol [PAD] or a zero vector) as initial content, a sequence is generated containing... A sequence of filler characters .

[0055] Input concatenation processing: Concatenate the input text sequence with the padding sequence to form the model input. ,in, This represents the input text sequence. This represents the padding sequence; the initial padding sequence is... If compatibility with the model's positional encoding mechanism is required, positional encoding injection must be performed on the concatenated complete sequence. For absolute positional encoding, the encoding range covers from 1 to... All positions; if relative position encoding is used, the encoding strategy needs to be adjusted according to the specifications of the model implementation.

[0056] Construction of the parallel decoding iterative execution process: In the first round of decoding, the concatenated sequence (input tensor) or the position-encoded sequence (obtained by position encoding the input tensor) is input into the pre-trained model to perform a complete forward computation and obtain the hidden state output at each position. The corresponding part of the filling region (i.e., the position) is extracted from the hidden state sequence. to As the initial generation result Starting from the second iteration, the input structure is dynamically updated: the original input is preserved. Partially unchanged, the filler portion from the previous output is used. Replace the original padding characters and reorganize them into a new input order. The model is input again for forward computation to generate updated fill results. This process is repeated cyclically, with each iteration progressively correcting / updating the content of the padding sequence based on model predictions. During the first decoding round, the concatenated input tensor... Input the model, perform multi-head attention and fully connected layer computation, and output the hidden state sequence. Extract the hidden states corresponding to the regions of the filled sequence. The initial prediction results are generated through the Softmax layer. (V represents the vocabulary size).

[0057] Termination condition setting: After each iteration of decoding calculation, it is determined whether the termination condition has been met. The termination condition is: the content of the fill sequence is updated in each round until the difference between adjacent iteration results is lower than a preset threshold, or the number of iterations reaches a certain threshold. .

[0058] According to one example of the present invention, the construction and position encoding of the input tensor are processed as follows:

[0059] Construction of input tensors: First, define the maximum generation length parameter. Its value is dynamically calculated based on task requirements and the model context window length. For example, when the input text length is N and the model's maximum supported sequence length is 512, the value is set as follows: Initialize the padding sequence to contain A placeholder vector of [PAD] tags, with embedding dimensions consistent with the model input layer (e.g., ...). The original input text sequence is concatenated with the padding sequence to form a complete input tensor. .

[0060] Position encoding: Inject absolute position encoding into the input tensor. The position encoding formula is:

[0061]

[0062]

[0063] in, For location index; The dimension index represents the dimensional index in the input tensor. One dimension; For embedded dimensions, It is a sine function. It is a cosine function. If the model uses relative position encoding, then the position bias matrix is ​​generated according to the sliding window rule. ,satisfy A linear mapping.

[0064] Step S2: Introduce a dynamic masking mechanism for the large model to dynamically calculate the information entropy at each position based on the attention weight matrix calculated by the large model in each round. Determine the input mask and output mask based on the information entropy. The input mask is used to retain elements in the filling sequence of the input tensor whose entropy value is less than the low entropy threshold and to set elements whose entropy value is greater than or equal to the low entropy threshold to zero. The output mask is used to retain elements in the updated filling sequence of the output whose entropy value is greater than the high entropy threshold and to set elements whose entropy value is less than or equal to the high entropy threshold to zero.

[0065] According to an embodiment of the present invention, the execution logic of the dynamic masking mechanism in step S2 includes: weight matrix extraction, information entropy quantization calculation, and dynamic mask generation. Wherein:

[0066] Weight matrix extraction: In the k-th iteration, extract the attention (attention weights or normal FC weights) weight matrix from the multi-head attention module of the n-th layer of the large model. Where h is the number of attention heads, This represents the total sequence length. In principle, it can be extracted from any multi-head attention module of layer n selected by the implementer. However, it is preferable to extract the attention weight matrix from the multi-head attention module of the last layer of the large model.

[0067] Information entropy quantization calculation: This step calculates the information entropy at each position based on the extracted weight matrix. For each position, this is done using the entropy-attention mapping function. The information entropy at each location is calculated based on the attention weights. The entropy value reflects the information certainty of the location under the attention mechanism: a low entropy value indicates that the attention weights are highly concentrated in a few locations (high information effectiveness), while a high entropy value indicates that the attention distribution is more dispersed (possibly noise or a fuzzy signal).

[0068] Indicative, for the first The first iteration The information entropy of each location is calculated as follows:

[0069]

[0070] in, Represents the weight matrix The Middle The position and the first The attention value between positions. Of course, a correction factor greater than 0 and less than 1 can be added before the above summation symbol to obtain other alternative calculation methods for information entropy.

[0071] Dynamic mask generation: This step determines the mask matrix for each location based on the information entropy of that location, including: input mask. and output mask The process is represented as: = .

[0072] According to one embodiment of the present invention, The function can be a neural network. Or, The function can also be a predefined calculation rule. Based on predefined low-entropy and high-entropy thresholds, the entropy value at each position is binarized and filtered to generate input and output mask matrices. Optionally, the low-entropy and high-entropy thresholds can be predefined constants. For example, the rule for generating a dynamic mask matrix based on entropy values ​​can be expressed as follows:

[0073] Input mask ( Only input signals from the low-entropy region are allowed to participate in subsequent calculations, suppressing the propagation of high-entropy noise. The calculation formula is as follows:

[0074]

[0075] Output mask ( ): Only the output results in the high-entropy region are retained for the next iteration, while overfitting signals in the low-entropy region are filtered out. This can be formalized as follows:

[0076]

[0077] in, Low entropy threshold This is the high entropy threshold.

[0078] According to one embodiment of the present invention, the low-entropy threshold and the high-entropy threshold can also be dynamically calculated values ​​to dynamically generate a more suitable mask. For example, let the dynamic low-entropy threshold be... The dynamic high entropy threshold is Then input the mask. and output mask It can be calculated as follows:

[0079] Input mask ( ):when hour, Otherwise ,in, , This represents the maximum number of iterations.

[0080] Output mask ( ):when hour, Otherwise ,in, .

[0081] According to one embodiment of the present invention, the input mask and output mask can be used for input adjustment and output filtering. For example, after the dynamic mask tensor is generated, the current input and output are adjusted by element-wise multiplication, wherein:

[0082] Input adjustment: Before the model receives the intermediate sequence generated in the previous round, it is multiplied with the input mask matrix to mask out high-entropy noise regions. ;

[0083] Output adjustment: Apply an output mask to the preliminary results generated by the current iteration, forcing low-entropy regions to be set to zero to prevent overfitting. ;

[0084] Therefore, a dynamic masking mechanism is introduced during the engineering optimization phase to perform masking operations and generate adjusted iterative inputs and outputs that satisfy the equation: .

[0085] In addition, to enhance the dynamic adaptability of the system, the threshold parameter and This can be adjusted as the decoding stage progresses. For example, a lower setting can be used in earlier iterations. To allow more candidate tokens to survive, the threshold is gradually increased as the convergence rate improves to filter out redundant signals. Through this mechanism, step S2 reduces computational redundancy while ensuring the effective transmission of key information, thereby accelerating the convergence speed of the decoding process.

[0086] Step S3: By introducing a dynamic masking mechanism, the large model performs parallel iterative decoding in stages. In the first stage, each round adopts momentum acceleration mode and the convergence speed of information entropy is determined after each round. When the convergence speed is less than or equal to the convergence threshold, the process is switched to the second stage to adopt normal iterative mode without momentum acceleration in each round.

[0087] According to one embodiment of the present invention, in this step, when the convergence speed is greater than a preset convergence threshold, the next round remains in the first stage; when the convergence speed is less than or equal to the preset convergence threshold, subsequent rounds proceed to the second stage. Each round always adopts a normal iteration mode until the iteration is completed. The convergence speed can be defined in different ways and can be set according to the implementer's needs. Illustratively, the convergence speed of the information entropy in each round can be calculated as follows: ,in, Indicates the length of the padding sequence. Indicates the first Wheel position Information entropy Indicates the first Wheel position Information entropy. When Higher than the preset convergence threshold At that time, momentum acceleration mode is used; Less than or equal to the convergence threshold At that time, the regular update mode, i.e., the normal iteration mode, is adopted. ).

[0088] According to one embodiment of the present invention, in order to avoid an incorrect timing for transitioning to the second stage, which would lead to premature entry into the second stage and affect the overall decoding efficiency, other methods can be adopted to determine the timing for transitioning to the second stage. For example, the second stage can be entered when the convergence speed is less than a preset convergence threshold for m consecutive rounds (m rounds are greater than or equal to 2, such as m being 2, 3, or 4).

[0089] According to one embodiment of the present invention, the normal iteration mode is represented as follows:

[0090]

[0091] in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. This indicates the output mask.

[0092] According to one embodiment of the present invention, the momentum acceleration mode is as follows:

[0093]

[0094] in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. Indicates the output mask. This represents the parameter of the first momentum term. This represents the parameter of the second momentum term.

[0095] According to an embodiment of the present invention, the momentum term parameter is dynamically determined by the entropy value and satisfies... = ,in, This represents a mapping function (which can be a neural network) that generates the parameters of the first and second momentum terms based on information entropy. This represents information entropy. The function analyzes the distribution characteristics of entropy values ​​(such as mean, variance, gradient change, etc.) and outputs the optimal combination of parameters that fits the current generation stage.

[0096] According to one embodiment of the present invention, the convergence threshold may be a specified threshold. Threshold In practice, this can be specifically set and customized. Alternatively, in this embodiment, in order to better improve the performance threshold... It can also be dynamically generated based on the f2 function, such as updating f2 to... = .

[0097] According to one embodiment of the present invention, the momentum term parameter can also be dynamically calculated according to a predetermined calculation rule:

[0098]

[0099]

[0100] in, For the current round ( (Round) average entropy value, This represents the theoretical maximum entropy. This design enhances the weight of historical information in the low-entropy stage (when the generated content is stable) and increases the weight of the current prediction in the high-entropy stage (when the content has not converged).

[0101] It should be noted that although the momentum acceleration mode has more computational terms than the normal iteration mode, it can accelerate the iterative decoding as a whole. The acceleration principle is: adding... and After that, it can make the subsequent This brings the result closer to the final outcome, reducing the number of iterations required. For example, instead of needing to start from... Calculated to Only then can convergence be achieved; after adopting the momentum acceleration mode, it may only be necessary to start from... Calculated to If convergence is achieved, then the number of calls to the large model (function F) is reduced by two. The overall computational cost of a single call to function F is significantly higher than the computational cost of the additional terms added by the momentum acceleration mode (only one millionth or even less of the computational cost of function F), so the additional computational cost can be ignored.

[0102] Step S4: When the iterative decoding meets the convergence condition, extract the final token sequence from the padding sequence obtained in the last round of iteration and convert it into output text.

[0103] According to one embodiment of the present invention, the convergence condition can be the convergence condition of the prior art.

[0104] However, existing technologies still suffer from convergence uncertainty: the termination condition is decoupled from the semantic coherence measurement, which can easily lead to syntax errors caused by premature stopping. Therefore, improvements can be made. According to one embodiment of the present invention, the convergence condition is set as follows: convergence is considered complete when the oscillation amplitude of the information entropy tends to stabilize and when the variance of the information entropy for a consecutive preset number of rounds (N, where N can be 3, 4, or 5, etc.) is less than a preset variance threshold. For example, if the oscillation of the information entropy tends to stabilize, when N is set to 5, the variance of the entropy for 5 rounds is calculated, and when the variance of the entropy value is... If the condition is met, the iteration is considered to have converged. Additionally, to avoid iteration timeouts, a maximum number of iterations can be set. Convergence is considered achieved when the maximum number of iterations is reached. That is, if no trigger is achieved... However, the number of iterations reached the preset maximum. (usually set) If the condition is met, the process will be forcibly terminated. This mechanism ensures that the decoding process will not loop indefinitely in extreme cases (such as the presence of high-frequency oscillations or abnormal model generation).

[0105] According to one embodiment of the present invention, the calculation method for the oscillation amplitude of information entropy can be set as follows:

[0106] ,or

[0107] ,

[0108] in, Indicates the preset rounds. Indicates the length of the padding sequence. Indicates the current iteration round. Indicates the first Wheel position Information entropy Indicates the first Wheel position The information entropy is the moving average of the entropy value. = For position The entropy value is the moving average. When If the semantic distribution of the generated sequence is considered to have stabilized, the variance of the information entropy can be checked to see if it is less than a preset variance threshold. If so, decoding is terminated early to save computational resources. For example, the variance of the entropy value within the sliding window can be calculated. ,when And continuous Convergence is determined when the conditions are met. It is a positive value (e.g., 0.01) specified according to actual needs, and can be defined in different forms in different tasks.

[0109] According to one embodiment of the present invention, when the entropy value oscillation amplitude is detected to exceed a preset range based on the variance of the information entropy, the low entropy threshold is forcibly increased and the high entropy threshold is decreased. This reduces the proportion of 0 values ​​in the input mask and increases the proportion of 0 values ​​in the output mask to suppress invalid iterations. For example, if the entropy value oscillation amplitude... Then the mask threshold will be forced. improve , reduce .

[0110] According to one embodiment of the present invention, iterative decoding terminates when the convergence condition is met. After decoding terminates, the concatenated complete sequence is processed. ( Post-processing is performed for the final iteration count: the position of the first occurrence of a sequence terminator (e.g., [EOS]) is detected, and all tokens preceding it are truncated and converted into valid output. If no terminator is detected, the entire sequence is taken. This is converted into the final output text. In addition, to optimize the output quality, a noise reduction and pruning strategy can be introduced, and redundant or contradictory generated segments can be further removed through semantic coherence detection algorithms (such as local segment scoring based on contrast loss).

[0111] According to an embodiment of the present invention, a system with parallel decoding capability is provided, characterized in that the system includes: a large model scheduling module, used to acquire a large model with parallel decoding capability, wherein the input tensor of the large model is composed of an input text sequence and a padding sequence, the large model is configured for multi-round iterative decoding, in each round performing forward computation based on the input tensor to obtain the hidden state output at each position, and the input tensor of the next round is composed of the input text sequence and the updated padding sequence extracted from the hidden state output at each position; an input preprocessing module, used to generate the input tensor of the large model in each round of iteration; and a dynamic mask calculation module, used to dynamically calculate the information entropy at each position based on the attention weight matrix calculated by the large model in each round, and determine the input mask and output mask based on the information entropy, wherein the input mask is used to filter the input tensor. In the filling sequence, elements with entropy values ​​less than the low entropy threshold are retained, while elements with entropy values ​​greater than or equal to the low entropy threshold are set to zero. The output mask is used to retain elements with entropy values ​​greater than the high entropy threshold in the updated filling sequence and set elements with entropy values ​​less than or equal to the high entropy threshold to zero. The scheduler is used to perform parallel iterative decoding in stages using a large model with a dynamic masking mechanism. In the first stage, each round uses momentum acceleration mode and determines the convergence speed of information entropy after each round. When the convergence speed is less than or equal to the convergence threshold, it switches to the second stage to use normal iterative mode without momentum acceleration in each round. The termination judgment module is used to monitor whether the iterative decoding meets the convergence condition. If so, it triggers the output module to be enabled. The output module is used to extract the final token sequence from the filling sequence obtained in the last round of iteration and convert it into output text.

[0112] In addition, although the above method is the optimal implementation method of the present invention, in some cases, the implementer may also adopt other implementation methods. For example, during the iteration process, only one of the normal iteration mode or the momentum acceleration mode is selected to complete the entire iterative decoding process.

[0113] To verify the performance improvement effect of the dynamic masking mechanism and iterative mode proposed in this invention, this embodiment conducted an ablation experiment to analyze the parallel decoding performance and generation quality. The results of the impact of each module on the speed and quality of parallel decoding generation are as follows: Figure 2 As shown.

[0114] Improved Decoding Efficiency: Experimental data shows that the baseline model (i.e., the parallel decoding model without the mechanism of this invention) has a decoding speed of only 45.8 tokens / second. After introducing an input mask, the speed increases to 82.3 tokens / second; after further introducing an output mask, the speed reaches 95.1 tokens / second; and when the system fully runs the dynamic mask and momentum acceleration logic (Full Model) described in this invention, the decoding speed is significantly improved to 153.5 tokens / second, an improvement of approximately 235% compared to the baseline model. This demonstrates that this invention greatly optimizes the inference efficiency of parallel decoding by dynamically pruning redundant computations of low-information tokens.

[0115] Improved Quality / Perplexity: While increasing speed, the perplexity (PPL) of the model shows a decreasing trend. The baseline model has a PPL of 16.2, while the PPL of the full model of this invention is reduced to 12.3. This result demonstrates that this invention effectively suppresses the interference of high-entropy noise tokens on the bitstream through a dynamic masking mechanism. It not only avoids sacrificing quality for speed but also improves the semantic coherence and accuracy of text generation by optimizing the iterative logic.

[0116] In addition, the inventors also conducted a comparative analysis of the convergence process of different iterative acceleration strategies. Figure 3 The relative entropy change curves of the proposed "hybrid momentum acceleration" strategy and the traditional strategy during the iterative convergence process are shown below, with the convergence efficiency comparison results as follows:

[0117] Momentum-free acceleration (dashed line): exhibits a slow convergence characteristic that is close to linear, requiring a large number of iterations to reach a stable state, which is difficult to meet the needs of real-time generation.

[0118] Pure momentum acceleration (dotted line): It exhibits an extremely fast rate of decrease in the early stages of iteration, but due to the inertia of the momentum term in the later stages of convergence, it shows a significant "late-stage oscillation" phenomenon near the target value, making it difficult to stabilize at the optimal convergence point.

[0119] The superiority of the hybrid strategy of this invention (solid line): This invention, through the phased iterative mode in step S3, activates momentum acceleration mode in the early stage of iteration (rapid convergence stage), utilizing nonlinear acceleration characteristics to rapidly reduce information entropy; when the convergence speed is detected to be less than or equal to a preset convergence threshold, it dynamically switches to the normal iterative mode without momentum acceleration. Experimental curves clearly show that the scheme of this invention, while maintaining a high convergence rate in the early stage, perfectly avoids oscillations in the later stage, achieving "smooth convergence" at a faster speed, and ultimately achieving a lower relative entropy level.

[0120] In summary, the method of this invention achieves efficient inference for natural language generation models (especially large language generation models) in sequence generation tasks such as machine translation and text generation by integrating dynamic perception of information entropy, attention-driven mask optimization, and momentum acceleration mechanisms. This method is applicable to latency-sensitive vertical scenarios such as cloud-based large model inference services and edge device text generation, supporting models with hundreds of billions of parameters to achieve generation quality comparable to autoregressive decoding within 1 / 3 of the iteration rounds. Some embodiments of this invention possess at least one or more of the following technical effects:

[0121] Entropy-Attention Joint Modeling: Establish a quantitative mapping relationship between attention weights and information entropy, and construct a dynamic mask generation function to dynamically shield noise and accelerate the decoding process;

[0122] Momentum Iteration Acceleration: Introducing a momentum term based on gradient descent optimization improves the linear convergence rate of traditional Jacobi decoding to superlinear.

[0123] Adaptive Termination Strategy: Dynamic pruning is achieved in each iteration through entropy change feature monitoring and oscillation suppression mechanism.

[0124] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0125] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0126] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0127] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for text generation, characterized in that, include: S1. Obtain a large model with parallel decoding capability. The input tensor of the large model is formed by concatenating the input text sequence and the padding sequence. The large model is configured for multi-round iterative decoding. In each round, forward computation is performed based on the input tensor to obtain the hidden state output at each position. The input tensor of the next round is formed by concatenating the input text sequence and the updated padding sequence extracted from the hidden state output at each position. S2. Introduce a dynamic masking mechanism for the large model to dynamically calculate the information entropy at each position based on the attention weight matrix calculated by the large model in each round. Determine the input mask and output mask based on the information entropy. The input mask is used to retain elements in the filling sequence of the input tensor whose entropy value is less than the low entropy threshold and set elements whose entropy value is greater than or equal to the low entropy threshold to zero. The output mask is used to retain elements in the filling sequence of the output update whose entropy value is greater than the high entropy threshold and set elements whose entropy value is less than or equal to the high entropy threshold to zero. S3. By introducing a dynamic masking mechanism, the large model performs parallel iterative decoding in stages. In the first stage, each round adopts a momentum acceleration mode, and the convergence speed of the information entropy is determined after each round. When the convergence speed is less than or equal to the convergence threshold, the process switches to the second stage, in which a normal iterative mode without momentum acceleration is adopted in each round. The momentum acceleration mode is as follows: in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. Indicates the output mask. This represents the parameter of the first momentum term. Indicates the parameter of the second momentum term; S4. When the iterative decoding meets the convergence condition, extract the final token sequence from the padding sequence obtained in the last iteration and convert it into the output text.

2. The method according to claim 1, characterized in that, The normal iteration mode is: in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. This indicates the output mask.

3. The method according to claim 1 or 2, characterized in that, The calculation methods for input and output masks include: in, Indicates the first The input mask of the wheel regarding position mask value, Indicates the first Wheel position Information entropy Indicates the low entropy threshold. Indicates the first The output mask of the wheel regarding position mask value, This represents the high entropy threshold.

4. The method according to claim 3, characterized in that, When the variance of information entropy is detected to indicate that the entropy value oscillation exceeds the preset range, the low entropy threshold is forcibly increased and the high entropy threshold is decreased.

5. The method according to claim 1 or 2, characterized in that, The parallel decoding logic for the large model is configured as follows: Get the maximum generated length, and initialize the padding sequence based on the padding length determined by the maximum generated length and the length of the input text sequence; Concatenate the input text sequence with the padding sequence to form a large model input tensor; The large model iteratively performs decoding calculations based on the input tensor, updating the contents of the padding sequence in each round; After each round of decoding, it is determined whether the convergence condition is met. If so, the decoding terminates. Otherwise, the next round of iteration is performed based on the new input tensor obtained from the input text sequence and the updated padding sequence. When the decoding terminates, the output text is constructed based on the final padding sequence.

6. The method according to claim 1 or 2, characterized in that, The convergence condition is considered satisfied if any of the following conditions are met: Condition 1: When the fluctuation amplitude of information entropy tends to stabilize, and when the variance of information entropy in consecutive preset rounds is less than a preset variance threshold; Condition 2: When the cumulative number of iterations in the iterative decoding reaches the maximum number of iterations.

7. A method for text generation, characterized in that, include: Obtain a large model with parallel decoding capability. The input tensor of the large model is composed of the input text sequence and the padding sequence. The large model is configured for multi-round iterative decoding. In each round, forward computation is performed based on the input tensor to obtain the hidden state output at each position. The input tensor of the next round is composed of the input text sequence and the updated padding sequence extracted from the hidden state output at each position. A dynamic masking mechanism is introduced for the large model to dynamically calculate the information entropy at each position based on the attention weight matrix calculated by the large model in each round. The input mask and output mask are determined based on the information entropy. The input mask is used to retain the elements in the filling sequence of the input tensor whose entropy value is less than the low entropy threshold and set the elements whose entropy value is greater than or equal to the low entropy threshold to zero. The output mask is used to retain the elements in the filling sequence of the output update whose entropy value is greater than the high entropy threshold and set the elements whose entropy value is less than or equal to the high entropy threshold to zero. Each iteration of decoding is performed using the input and output masks according to a preset iteration mode. The preset iteration mode is either normal iteration mode or momentum acceleration mode. The normal iteration mode is: ， The momentum acceleration mode is: ， in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. Indicates the output mask. This represents the parameter of the first momentum term. Indicates the parameter of the second momentum term; When the iterative decoding meets the convergence condition, the final token sequence is extracted from the padding sequence obtained in the last iteration, and the final token sequence is converted into the output text.

8. A system for implementing the method as described in any one of claims 1-6, characterized in that, The system includes: The large model scheduling module is used to acquire large models with parallel decoding capabilities. The input tensor of the large model is composed of the input text sequence and the padding sequence. The large model is configured to perform multi-round iterative decoding. In each round, forward computation is performed based on the input tensor to obtain the hidden state output at each position. The input tensor of the next round is composed of the input text sequence and the updated padding sequence extracted from the hidden state output at each position. The input preprocessing module is used to generate the input tensors of the large input model in each iteration. The dynamic mask calculation module is used to dynamically calculate the information entropy of each position based on the attention weight matrix calculated by the large model in each round. The input mask and output mask are determined based on the information entropy. The input mask is used to retain the elements in the filling sequence of the input tensor whose entropy value is less than the low entropy threshold and set the elements whose entropy value is greater than or equal to the low entropy threshold to zero. The output mask is used to retain the elements in the filling sequence of the output update whose entropy value is greater than the high entropy threshold and set the elements whose entropy value is less than or equal to the high entropy threshold to zero. The scheduler is used to perform parallel iterative decoding of a large model in stages by introducing a dynamic masking mechanism. In the first stage, each round uses a momentum acceleration mode, and the convergence rate of the information entropy is determined after each round. When the convergence rate is less than or equal to the convergence threshold, the process transitions to the second stage, where each round uses a normal iterative mode without momentum acceleration. The momentum acceleration mode is as follows: in, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Indicates the first The filling sequence of the wheel, Representing a large model, This represents the input text sequence. Indicates the input mask. Indicates the output mask. This represents the parameter of the first momentum term. Indicates the parameter of the second momentum term; The termination judgment module is used to monitor whether the iterative decoding meets the convergence condition. If so, it triggers the start of the output module. The output module is used to extract the final token sequence from the padding sequence obtained in the last iteration and convert it into output text.

9. An electronic device, characterized in that, include: One or more processors; as well as Memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1-7 by executing the executable instructions.

Images