Large language model parallel speculation decoding method and device, equipment and storage medium

Abstract

The invention provides a large language model parallel speculation decoding method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence and natural language processing. The method comprises the steps that on the basis of a current output sequence, follow-up lexical elements of the current output sequence are predicted in a parallel iteration mode through a large language model, and a prediction track of each prediction position is obtained; continuous lexical element fragments are extracted from the prediction track and serve as candidate n-gram fragments; the candidate n-gram fragments are stored in a cache pool; and selecting a candidate n-gram fragment of which the beginning is matched with the last lexical element of the current output sequence from the cache pool, inputting the candidate n-gram fragment into the large language model for parallel verification, and adding a lexical element sequence contained in the successfully verified candidate n-gram fragment into the current output sequence. The verification acceptance rate of the candidate sequence in speculative decoding is effectively improved, and the reasoning speed of the large language model is remarkably improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to a method, apparatus, device and storage medium for parallel inference decoding of large language models. Background Technology

[0002] Currently, large language models such as GPT and LLaMA generally employ autoregressive decoding for text generation. This method generates only one lexical unit per step, and subsequent lexical unit generation strictly depends on all previously generated lexical units. This results in an essentially serial inference process, high computational latency, and limited throughput. To accelerate inference, speculative decoding techniques have been proposed. The core idea is to use a fast module (such as a small draft model or an N-gram matching-based search mechanism) to pre-generate multiple candidate lexical unit sequences in parallel, which are then verified and accepted by the main language model, thereby reducing the number of serial decoding steps.

[0003] However, existing speculative decoding methods share a common problem: the candidate sequence deviates significantly from the expected sequence of the main model, resulting in a low acceptance rate during the validation phase. Summary of the Invention

[0004] This application provides a parallel speculative decoding method, apparatus, device, and storage medium for large language models, which addresses the deficiency of low acceptance rate in the verification stage of existing speculative decoding methods. It improves the candidate sequence acceptance rate of parallel speculative decoding of large language models, thereby achieving more efficient inference acceleration while ensuring lossless generation quality.

[0005] This application provides a parallel inference decoding method for large language models, including the following steps: Based on the current output sequence, the large language model is used to predict the subsequent words in the current output sequence in a parallel iterative manner to obtain the prediction trajectory at each prediction position. Continuous word segments are extracted from the predicted trajectory as candidate n-gram segments; The candidate n-gram fragments are stored in a cache pool with a fixed capacity and maintained using a hybrid eviction policy that combines access timeliness with generation confidence. Candidate n-gram segments whose beginning matches the end lexicon of the current output sequence are selected from the cache pool, input into the large language model for parallel verification, and the lexicon sequence contained in the successfully verified candidate n-gram segments is appended to the current output sequence.

[0006] In some embodiments, the current output sequence is initialized with a cue word at the start of decoding, and during the decoding process, it is dynamically expanded by appending the lexical sequences contained in the successfully verified candidate n-gram segments to the current output sequence.

[0007] In some embodiments, the step of predicting subsequent lexical units of the current output sequence using a large language model in a parallel iterative manner based on the current output sequence to obtain the prediction trajectory at each prediction position includes: Obtain the sequence to be predicted, which includes W words, where W is an integer greater than 1; Perform N iterations, where N is an integer greater than 1. In each iteration, the current output sequence and the current sequence to be predicted are concatenated and input into the large language model, and the sequence to be predicted is updated according to the word probability distribution output by the large language model. Based on the value changes of the sequence to be predicted in the N iterations, a prediction trajectory is formed for each prediction position.

[0008] In some embodiments, each candidate n-gram fragment entry in the cache pool records the following information: the lexical sequence of the candidate n-gram fragment, the generation position, the preferred probability of each lexical in the candidate n-gram fragment, and the iteration step number when the candidate n-gram fragment was last successfully adopted.

[0009] In some embodiments, storing the candidate n-gram fragment in a cache pool includes: When the cache pool is full, a comprehensive elimination score is calculated for each candidate n-gram fragment in the cache pool. The comprehensive elimination score is a weighted sum of the average confidence score and the least recently used score of the candidate n-gram fragment. The average confidence score is used to characterize the overall generation determinism level of the candidate n-gram fragment, and the least recently used score is used to characterize the length of time that the candidate n-gram fragment has not been reused. Based on the overall elimination score of each candidate n-gram fragment, some candidate n-gram fragments in the cache pool are deleted, where a higher score indicates a higher priority for deletion.

[0010] In some embodiments, the least recently used score is calculated by dividing the number of iterations the candidate n-gram fragment has undergone since its last successful adoption by a preset constant.

[0011] In some embodiments, the average confidence score is calculated by summing the preferred probability values ​​corresponding to each word in the candidate n-gram segment and dividing by the length of the candidate n-gram segment.

[0012] In some embodiments, extracting continuous word segments from the predicted trajectory as candidate n-gram segments includes: From the predicted trajectories at all predicted positions, a continuous subsequence of word terms of length L is extracted as the candidate n-gram segment, where L is an integer greater than 1.

[0013] In some embodiments, selecting candidate n-gram segments from the cache pool whose beginnings match the end tokens of the current output sequence and inputting them into the large language model for parallel validation includes any of the following methods: The system sequentially verifies whether the preferred word units output by the large language model at each corresponding position in the candidate n-gram segment are consistent with the word units at the corresponding positions in the candidate n-gram segment. If they are completely consistent, the verification is considered successful. Alternatively, based on the probability distribution of the large language model at the corresponding position of the candidate n-gram segment, random numbers are generated sequentially to determine whether the lexical units in the candidate n-gram segment can be accepted. If all can be accepted, the verification is considered successful.

[0014] In some embodiments, the method further includes: Based on the evaluation results of the current decoding state, the prediction parameters are dynamically adjusted; the prediction parameters include at least one of the following: the number of lexical units in the sequence to be predicted, the number of iterations, and the length of the candidate n-gram segments; the evaluation results of the current decoding state include at least one of the following: the probability distribution entropy of the large language model at the current prediction position, and the validation acceptance rate of recent candidate n-gram segments.

[0015] In some embodiments, if the evaluation result includes the probability distribution entropy, dynamically adjusting the prediction parameters based on the evaluation result of the current decoding state includes: If the probability distribution entropy is lower than or equal to the first threshold, then at least one of the prediction parameters is increased; If the probability distribution entropy is higher than the second threshold, then at least one of the prediction parameters is reduced.

[0016] In some embodiments, if the evaluation result includes the validation acceptance rate, where the validation acceptance rate is the ratio of the total length of successfully adopted n-gram fragments to the total number of validations in the most recent K validations, and K is a positive integer, then dynamically adjusting the prediction parameters based on the evaluation result of the current decoding state includes: If the verification acceptance rate is higher than the third threshold, then increase at least one of the prediction parameters; If the validation acceptance rate is lower than or equal to the fourth threshold, then at least one of the prediction parameters is reduced.

[0017] This application also provides a parallel inference decoding device for a large language model, comprising the following modules: The trajectory acquisition module is used to predict subsequent words in the current output sequence using a large language model in a parallel iterative manner based on the current output sequence, so as to obtain the predicted trajectory at each prediction position. The fragment extraction module is used to extract continuous word fragments from the predicted trajectory as candidate n-gram fragments; The fragment storage module is used to store the candidate n-gram fragments into a cache pool. The cache pool has a fixed capacity and is maintained using a hybrid eviction policy. The hybrid eviction policy combines access timeliness and generation confidence. The fragment verification module is used to select candidate n-gram fragments from the cache pool whose beginnings match the end lexicons of the current output sequence, input them into the large language model for parallel verification, and append the lexicon sequences contained in the successfully verified candidate n-gram fragments to the current output sequence.

[0018] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the large language model parallel inference decoding method as described above.

[0019] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the parallel speculative decoding method for large language models as described above.

[0020] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the large language model parallel speculative decoding method as described above.

[0021] The parallel inference decoding method, apparatus, device, and storage medium for large language models provided in this application introduce a hybrid elimination strategy based on access timeliness and generation confidence to intelligently manage the candidate n-gram cache, which effectively improves the verification acceptance rate of candidate sequences in inference decoding, thereby significantly reducing computational redundancy and improving the inference speed of large language models while ensuring generation quality. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the parallel inference decoding method for large language models provided in this application embodiment.

[0024] Figure 2 This is a schematic flowchart of a method for obtaining the predicted trajectory at each predicted location, provided in an embodiment of this application.

[0025] Figure 3 This is a flowchart illustrating the method for storing candidate n-gram fragments into a cache pool, as provided in an embodiment of this application.

[0026] Figure 4 This is a schematic diagram of the overall process of the parallel inference decoding method for large language models provided in the embodiments of this application.

[0027] Figure 5 This is a schematic diagram of the structure of the parallel inference decoding device for a large language model provided in an embodiment of this application.

[0028] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] To facilitate a clearer understanding of the technical solutions of the various embodiments of this application, some technical content related to the various embodiments of this application will be introduced first.

[0031] Large language models: refer to natural language processing models with a huge number of parameters and strong generalization ability, such as Tongyi Qianwen and GPT.

[0032] Autoregressive decoding: refers to a serial decoding method in which tokens are generated one by one, and each step depends on all previous tokens as input.

[0033] Speculative decoding: refers to the technique of predicting multiple candidate tokens in parallel and then verifying them by the main model, thereby reducing the number of decoding steps.

[0034] Jacobi iteration: Similar to the parallel iterative method for solving nonlinear equations in numerical analysis, it is used here to generate multiple tokens in parallel.

[0035] Token: The basic semantic unit processed by the language model, which can be a character, word, or subword.

[0036] n-gram: A sequence of n consecutive tokens, such as "fishing enthusiast" which is a 3-gram (fishing-enthusiast-person).

[0037] Trajectory: A sequence of tokens generated at a certain position through multiple Jacobi iterations, used to construct an n-gram.

[0038] Least Recently Used (LRU): A cache eviction policy.

[0039] Key-Value Cache: Caches key / value vectors during Transformer inference to avoid redundant calculations of the attention mechanism.

[0040] Entropy: A measure of the uncertainty of a probability distribution. The higher the entropy, the more uncertain the model.

[0041] Acceptance rate: The proportion of n-gram lengths that are successfully validated and adopted by the main model out of the total validation length.

[0042] W (Guess Window): The number of tokens predicted in a single parallel prediction.

[0043] L (n-gram Length): The length of the n-gram extracted from the trajectory.

[0044] G (Verification Branches): The number of candidate n-grams in a single parallel verification.

[0045] Currently, large language models such as GPT and LLaMA generally employ autoregressive decoding for text generation. This method generates only one lexical unit per step, and subsequent lexical unit generation strictly depends on all previously generated lexical units. This results in an essentially serial inference process, high computational latency, and limited throughput. To accelerate inference, speculative decoding techniques have been proposed. The core idea is to use a fast module (such as a small draft model or an N-gram matching-based search mechanism) to pre-generate multiple candidate lexical unit sequences in parallel, which are then verified and accepted by the main language model, thereby reducing the number of serial decoding steps.

[0046] However, existing speculative decoding methods share a common problem: the candidate sequence deviates significantly from the expected sequence of the main model, resulting in a low acceptance rate during the validation phase.

[0047] In view of this, embodiments of this application provide a parallel inference decoding method for large language models. The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings.

[0048] Figure 1 This is a flowchart illustrating the parallel inference decoding method for large language models provided in this application. The parallel inference decoding method for large language models can be applied to electronic devices, which can be various types of devices with information processing capabilities. For example, the electronic device may include a personal computer, laptop, PDA, or server; the electronic device may also be a mobile terminal, such as a mobile phone, in-vehicle computer, tablet computer, or projector. Figure 1 As shown, the method may include the following steps 101 to 104: Step 101: Based on the current output sequence, use a large language model to predict the subsequent words in the current output sequence in a parallel iterative manner to obtain the prediction trajectory of each prediction position.

[0049] It should be noted that the parallel iteration refers to a method that predicts multiple subsequent words simultaneously. For example, the Jacobi iteration algorithm is used to generate multiple word candidates in parallel in one forward calculation, and to form the word change sequence, i.e., the prediction trajectory, for each prediction position through multiple rounds of iteration.

[0050] Furthermore, the current output sequence is initialized with a prompt word at the start of decoding. During the decoding process, the current output sequence is dynamically expanded by appending the word sequence contained in the successfully verified candidate n-gram segments to it.

[0051] It should be noted that the output sequence grows gradually as the decoding process progresses. Each time a candidate n-gram fragment is successfully verified and adopted, the sequence is automatically expanded to the corresponding length of words.

[0052] Step 102: Extract continuous word segments from the predicted trajectory as candidate n-gram segments.

[0053] It should be noted that the extraction operation can extract a fixed-length continuous word sequence from the predicted trajectory in a sliding window manner, such as extracting 3-grams or 4-grams as candidate segments.

[0054] Step 103: Store the candidate n-gram fragments in a cache pool with a fixed capacity and maintain it using a hybrid eviction policy that combines access timeliness and generation confidence.

[0055] It should be noted that the hybrid eviction strategy is used to determine which n-gram fragments to evict when the cache pool is full. It takes into account the last access time of the fragment and the confidence of the model at the time of generation, so as to retain high-quality, high-reusability fragments.

[0056] Step 104: Select candidate n-gram segments from the cache pool whose beginning matches the end lexicon of the current output sequence, input them into the large language model for parallel verification, and append the lexicon sequence contained in the successfully verified candidate n-gram segments to the current output sequence.

[0057] It should be noted that during verification, candidate segments that can be connected to the end of the current output sequence are selected, and the large language model is used to determine whether they meet the model's expectations. If the verification passes, the entire segment is included in the output sequence, realizing multi-lexical one-step decoding.

[0058] This application generates candidate n-gram fragments by reusing the model's own iterative trajectory and combines an intelligent cache eviction strategy that integrates timeliness and confidence. This significantly improves the verification acceptance rate and cache hit rate of candidate sequences in speculative decoding, thereby achieving more efficient acceleration of large language model inference while ensuring lossless generation quality.

[0059] Figure 2 This is a schematic flowchart illustrating the method for obtaining the predicted trajectory at each predicted location, as provided in an embodiment of this application. Figure 2 As shown, step 101, based on the current output sequence, uses a large language model to predict subsequent words in the current output sequence in a parallel iterative manner to obtain the prediction trajectory for each prediction position, and may include: Step 201: Obtain the sequence to be predicted, which includes W tokens, where W is an integer greater than 1.

[0060] It should be noted that W is the preset prediction window size, representing the number of tokens predicted in a single parallel prediction. At the start of decoding, the W tokens in the sequence to be predicted can be randomly initialized or generated in a preset manner.

[0061] Step 202: Perform N iterations, where N is an integer greater than 1. In each iteration, the current output sequence and the current sequence to be predicted are concatenated and input into the large language model, and the sequence to be predicted is updated according to the word probability distribution output by the large language model.

[0062] It should be noted that N is the preset number of iterations. In each iteration, the model takes the aforementioned concatenated sequence as input and outputs in parallel the probability distribution of the word at each of the W positions; based on this distribution, the word at each position can be updated to the word with the highest probability. Through multiple iterations, the sequence to be predicted gradually approximates the reasonable subsequent generation result of the model in the context.

[0063] Step 203: Based on the value changes of the sequence to be predicted in the N iterations, form the prediction trajectory for each prediction position.

[0064] It should be noted that the updated lexical units at each position after each iteration are recorded. For the i-th prediction position, the lexical units generated sequentially in N iterations are arranged in order, thus forming the prediction trajectory for that position. Each prediction trajectory reflects the intermediate state sequence during the model's inference and generation process at that position.

[0065] This application uses a parallel iterative approach to generate complete trajectories for multiple predicted positions at once, providing a sufficient data foundation for the subsequent construction of high-quality candidate n-gram segments and effectively overcoming the latency bottleneck of traditional serial decoding.

[0066] In some embodiments, step 102, which extracts continuous word segments from the predicted trajectory as candidate n-gram segments, may include: sliding to extract continuous word subsequences of length L from the predicted trajectories at all predicted positions as candidate n-gram segments, where L is an integer greater than 1.

[0067] It should be noted that the sliding extraction refers to sequentially traversing all possible subsequences composed of continuous words in all predicted trajectories with a fixed window length L. For example, for a trajectory formed after N iterations at a certain predicted position, N-L+1 continuous subsequences of length L can be extracted. These subsequences, because they originate from multiple iterations of the same model in similar contexts, usually have high inherent semantic coherence and contextual adaptability, making them suitable as high-quality candidate segments for parallel validation.

[0068] This method, through systematic sliding extraction of predicted trajectories, can fully exploit the structured semantic fragments generated during model iteration, construct a rich and high-quality candidate n-gram library, and provide a key data foundation for subsequent efficient caching and parallel verification.

[0069] In some embodiments, each candidate n-gram fragment entry in the cache pool records the following information: the lexical sequence of the candidate n-gram fragment, the generation position, the preferred probability of each lexical in the candidate n-gram fragment, and the iteration step number when the candidate n-gram fragment was last successfully adopted.

[0070] It should be noted that by recording the complete word sequence, the content of candidate segments can be accurately reproduced; by recording the generation position, the original prediction context of the segment can be associated; by recording the preferred probability of each word (i.e., the highest probability value in the model output), the confidence level at the time of segment generation can be quantified; and by recording the iteration step number of the last successfully adopted segment, the reuse of the segment in recent decoding processes can be tracked. This structured information provides the necessary quantitative basis for the subsequent implementation of a hybrid elimination strategy based on timeliness and confidence.

[0071] By structurally recording the multidimensional information of candidate n-gram fragments, the quality, timeliness, and contextual relevance of each fragment can be accurately evaluated, thereby supporting the implementation of intelligent cache management strategies. Under limited cache capacity, high-quality, high-reusability fragments are prioritized for retention, significantly improving cache hit rate and verification efficiency.

[0072] Figure 3 This is a flowchart illustrating the method for storing candidate n-gram fragments into a cache pool, as provided in an embodiment of this application. Figure 3 As shown, step 103, which stores the candidate n-gram fragment in a cache pool, may include: Step 301: When the capacity of the cache pool is full, calculate the comprehensive elimination score for each candidate n-gram fragment in the cache pool; the comprehensive elimination score is the weighted sum of the average confidence score and the least recently used score of the candidate n-gram fragment, the average confidence score is used to characterize the overall generation determinism level of the candidate n-gram fragment, and the least recently used score is used to characterize the length of time that the candidate n-gram fragment has not been reused.

[0073] It should be noted that the comprehensive eviction score is the core indicator for evaluating the retention value of cached entries. The average confidence score reflects the intrinsic quality of the fragment, while the least recently used score reflects its recent reuse frequency. By combining the two and assigning adjustable weights, this method can flexibly balance the impact of "fragment quality" and "access timeliness" in eviction decisions according to the actual decoding scenario.

[0074] For example, the final score S_final = alpha * S_Conf - beta * S_LRU, where S_final is the final overall eviction score of the cached entry. alpha and beta are positive hyperparameters used to adjust the relative importance of confidence and timeliness in the decision. S_Conf is the average confidence score of the entry. S_LRU is the least recently used score of the entry. The n-gram entry with the highest score is considered the lowest value and is preferentially removed from the cache pool.

[0075] Step 302: Delete some candidate n-gram fragments in the cache pool according to the comprehensive elimination score of each candidate n-gram fragment, where the higher the score, the higher the priority of deletion.

[0076] It should be noted that when it is necessary to free up cache space to store new fragments, the system sorts all existing entries according to the comprehensive elimination score calculated in step 301 and deletes one or more entries with the highest scores. This process ensures that the cache pool always retains the set of candidate n-gram fragments with the highest comprehensive value (combination of quality and timeliness).

[0077] By employing a hybrid eviction strategy that combines generation confidence with access timeliness, this method can intelligently manage a finite-sized n-gram cache pool. It effectively preserves high-quality semantic fragments with high reusability potential, significantly improves cache hit rate and candidate fragment validation acceptance rate, thereby reducing redundant model computations and achieving more efficient inference acceleration.

[0078] Furthermore, the least recently used score is calculated by dividing the number of iterations the candidate n-gram fragment has undergone since its last successful adoption by a preset constant.

[0079] It should be noted that the LRU strategy aims to prioritize retaining recently adopted segments and evict segments that have not been used for a long time, in order to maintain the relevance of cached content to the current decoding context. The above calculation method standardizes the "length of time not used" into a comparable score.

[0080] For example, a specific implementation may include: 1. Maintain the iteration step number S for each n-gram entry in the cache pool when it was last successfully adopted; 2. Whenever an n-gram is validated and successfully adopted by the main model, update its S to the current iteration step number Current_Step; 3. If an n-gram entry is never adopted, its S remains at its initial value (e.g., 0). 4. During the eviction decision, the LRU score (S_LRU) for each entry is calculated as: (Current_Step - S_i) / MaxStepDiff. Where, S_LRU(i): the Least Recently Used (LRU) score of the i-th cached entry. CurrentStep: the current global iteration step number. S_i: the iteration step number when the i-th cached entry was last successfully adopted. MaxStepDiff: a preset normalization constant used to map the time difference to a standardized score range. A higher S_LRU(i) value indicates that the n-gram entry has not been adopted for a longer period, and its reuse value in the current context is likely to be lower, thus making it more likely to be evictioned.

[0081] Furthermore, the average confidence score is calculated by summing the preferred probability values ​​corresponding to each word in the candidate n-gram segment and then dividing by the length of the candidate n-gram segment.

[0082] It should be noted that this score is directly based on the raw probability output by the large language model when generating the segment, measuring the model's "confidence" in the segment sequence. A higher average confidence score generally means that the segment better matches the model's expectations in the current context, and is more likely to be validated and accepted in the future.

[0083] For example, the specific calculation method is as follows: 1. For each n-gram entry, record the model's preferred probability (top-1 probability value) for each constituent word when it is generated.

[0084] 2. Calculate the average confidence score (S_Conf) for this item as: (p1 + p2 + ... + p n ) / n. Where, S_Conf(i): the average confidence score of the i-th cached entry. p j S_Conf(i) represents the preferred probability when generating the j-th lexical unit, and n is the length of the n-gram. The lower the value of S_Conf(i), the lower the average generation confidence of the segment, the higher the risk of failure in the verification stage, and therefore the more likely it is to be discarded to optimize the overall quality of the cache pool.

[0085] The efficient n-gram caching mechanism based on iterative trajectories provided in this application transforms the coherent lexical sequences generated by the Jacobi iteration process into a structured n-gram cache and innovatively introduces a two-factor elimination strategy of "LRU + confidence," significantly improving the reuse rate of high-frequency, high-deterministic semantic segments. This method utilizes the model's own iterative trajectory as a "self-draft," avoiding the training overhead of additional draft models. Simultaneously, it overcomes the inefficiency of traditional simple elimination strategies (such as FIFO) through intelligent cache management, thereby effectively improving the efficiency of parallel acceptance of multiple lexical units, reducing unnecessary redundant computation, and significantly accelerating the inference speed of large language models.

[0086] It is understood that the Least Recently Used (LRU) component in the above cache eviction policy is merely an example. Those skilled in the art will understand that other policies based on access timeliness, such as the Least Frequently Used (LFU) policy, which makes eviction decisions by recording and comparing the historical usage frequency of n-gram fragments, can also achieve the purpose of this application and fall within the protection scope of this application.

[0087] In some embodiments, selecting candidate n-gram segments from the cache pool whose beginnings match the end tokens of the current output sequence and inputting them into the large language model for parallel validation includes any of the following methods: The system sequentially verifies whether the preferred word units output by the large language model at each corresponding position in the candidate n-gram segment are consistent with the word units at the corresponding positions in the candidate n-gram segment. If they are completely consistent, the verification is considered successful. Alternatively, based on the probability distribution of the large language model at the corresponding position of the candidate n-gram segment, random numbers are generated sequentially to determine whether the lexical units in the candidate n-gram segment can be accepted. If all can be accepted, the verification is considered successful.

[0088] It should be noted that the two methods described above correspond to two main decoding strategies in text generation for large language models: greedy decoding and sampling decoding. The first method is suitable for greedy decoding scenarios that require deterministic output results; the second method is suitable for sampling decoding scenarios that expect more diverse output results, and it judges the acceptability of candidate lemmas through random sampling based on probability distribution. Both methods ensure the rigor of the verification process, that is, the adoption or rejection of candidate n-gram sequences completely follows the probability distribution and decoding rules of the main model itself, thereby ensuring the consistency of the final output text with the standard autoregressive decoding in terms of statistical distribution, and achieving lossless acceleration.

[0089] For example, the specific verification process can be described as follows: All candidate n-gram segments whose first word completely matches the last word of the currently generated output sequence are selected from the cache pool. For each candidate segment, it is concatenated to the current output sequence to form an expanded input sequence (format: [prompt word + accepted sequence + candidate n-gram]). This input sequence is then fed into a large language model, and the logits or probability distributions of each corresponding position of the candidate segments are calculated in parallel.

[0090] If a greedy decoding strategy is adopted, it directly checks whether the preferred word (i.e. the word with the highest probability) output by the model at each position is exactly the same as the word at the corresponding position in the candidate segment; if all positions of the candidate segment are successfully matched, the segment is determined to be successfully verified and can be adopted as a whole.

[0091] If a sampling decoding strategy (such as rejection sampling) is adopted, sampling is performed sequentially according to the probability distribution output by the model at each position: for each word in the candidate segment, a random number is generated and compared with the probability of that word in the model distribution. If the probability of that word is not lower than the random sampling result, the word is considered acceptable; if all words in the candidate segment are accepted sequentially, the segment is considered successfully verified. If a word is not accepted at a certain position, the verification process is interrupted, and the model will resample and generate the next word based on the true distribution at that position, thus ensuring that the decoding process can continue.

[0092] The aforementioned verification mechanism provides an adaptation scheme for greedy decoding and sampling decoding scenarios, and can efficiently determine the acceptability of candidate n-gram segments while strictly adhering to the main model's own generation rules. This method not only ensures the quality and consistency of the generated text, but also significantly reduces the overall decoding steps by supporting parallel verification and batch adoption of multi-lexical sequences, thereby effectively improving the inference speed of large language models.

[0093] In some embodiments, the method further includes: dynamically adjusting prediction parameters based on the evaluation result of the current decoding state; the prediction parameters include at least one of the following: the number of lexical units in the sequence to be predicted, the number of iterations, and the length of the candidate n-gram segments; the evaluation result of the current decoding state includes at least one of the following: the probability distribution entropy of the large language model at the current prediction position, and the validation acceptance rate of recent candidate n-gram segments.

[0094] It should be noted that the dynamic adjustment strategy aims to adaptively optimize resource allocation based on real-time feedback during the decoding process. By monitoring the determinism (entropy) of model predictions and the historical validation success rate (acceptance rate), the system can dynamically expand or shrink key parameters such as the prediction window and iteration depth. This allows for a more aggressive parallel strategy to maximize acceleration gains when model generation confidence is high and historical validation success rate is high, while reverting to a conservative strategy to conserve computational resources and ensure decoding robustness when model uncertainty or frequent validation failures occur.

[0095] Furthermore, if the evaluation result includes the probability distribution entropy, the step of dynamically adjusting the prediction parameters based on the evaluation result of the current decoding state includes: If the probability distribution entropy is lower than or equal to the first threshold, then at least one of the prediction parameters is increased; If the probability distribution entropy is higher than the second threshold, then at least one of the prediction parameters is reduced.

[0096] For example, entropy-based adjustments can be implemented as follows: Before each new round of parallel prediction, calculate the entropy of the probability distribution of the word at the current prediction position, using the formula: H(P) = -Σ p(x)·log(p(x)), where H(P) represents the entropy of the probability distribution of the word at the current position, used to quantify the uncertainty of the model's prediction. A higher H(P) value indicates a more uniform probability distribution and greater model uncertainty; a lower value (closer to 0) indicates a more concentrated probability distribution and greater model certainty. P represents the complete probability distribution of the word at the current prediction position. p(x) represents the probability of word x appearing in probability distribution P. Σ (summation symbol): represents the summation of p(x)·log(p(x)) terms for all possible words x in probability distribution P. If the calculated entropy value H(P) is lower than the preset first threshold (e.g., 0.8), it indicates that the model has a high degree of certainty regarding the subsequent generated content. In this case, the prediction window size W, the number of iterations N, or the candidate n-gram length L can be appropriately increased to explore a wider range of parallel acceleration potential. Conversely, if the entropy value H(P) is higher than the preset second threshold (which may be the same as or different from the first threshold), it indicates that the model is in an uncertain state. In this case, the above parameters should be reduced, and a more conservative iterative strategy should be adopted to reduce the risk of resource waste and potential error accumulation caused by blind parallelism.

[0097] Furthermore, if the evaluation result includes the verification acceptance rate, where the verification acceptance rate is the ratio of the total length of successfully adopted n-gram fragments to the total number of verifications in the most recent K verifications, and K is a positive integer, then dynamically adjusting the prediction parameters based on the evaluation result of the current decoding state includes: If the verification acceptance rate is higher than the third threshold, then increase at least one of the prediction parameters; If the validation acceptance rate is lower than or equal to the fourth threshold, then at least one of the prediction parameters is reduced.

[0098] For example, adjustments based on the acceptance rate can be implemented as follows: The system maintains a sliding window that records relevant data from the K most recent validation operations. The validation acceptance rate is defined as the total length of all successfully adopted n-gram segments within this window, divided by the total number of validation operations. If the calculated acceptance rate is higher than a preset third threshold (e.g., 70%), it indicates that the current cached content closely matches the model's expectations, resulting in a high validation success rate. In this case, the prediction parameters can be appropriately increased to further accelerate decoding by leveraging the high acceptance rate trend. Conversely, if the acceptance rate is lower than a preset fourth threshold (e.g., 30%), it indicates that the current prediction or caching strategy deviates significantly from the model's actual generation path. In this case, the prediction parameters should be reduced, reverting to a more refined and conservative iterative mode to recalibrate and improve subsequent validation efficiency.

[0099] Understandably, entropy-based adjustments and acceptance rate-based adjustments can be implemented independently or in tandem. When both are enabled simultaneously, the system can make decisions based on both types of feedback. For example, it might significantly increase parameters when model certainty is high and historical acceptance rates are also high, while tending to reduce parameters when either metric is poor. This dual-feedback mechanism allows the system to more finely adapt to different task types, different text generation stages, and different model context states, achieving a dynamic optimal balance between acceleration benefits and computational overhead. This results in a more stable and efficient inference acceleration effect under resource-constrained conditions.

[0100] These two strategies can work simultaneously, ensuring a dynamic balance between acceleration and computational load, resulting in faster acceleration and less resource consumption.

[0101] By introducing the aforementioned dynamic adjustment strategy, this application achieves for the first time real-time and dynamic control of key parameters (W / L / G) for parallel speculative decoding, where W (Guess Window): the number of tokens predicted in a single parallel run; L (n-gram Length): the length of the n-gram extracted from the trajectory; and G (Verification Branches): the number of candidate n-grams verified in a single parallel run. This dynamic adaptive window adjustment strategy constructs a dual-feedback driven mechanism based on the immediate metric of model generation determinism (entropy) and continuous feedback from historical verification efficiency (acceptance rate). This enables the decoding process to intelligently adapt to different semantic stages and task types: when the model is "confident" and the verification success rate is high, the parallel window is automatically expanded to improve throughput; when "uncertain" or verification efficiency is low, the window is promptly shrunk to ensure generation quality and resource efficiency. This strategy achieves a refined trade-off between computational resources and acceleration benefits at the system level, significantly improving the adaptability, robustness, and overall performance of the speculative decoding method in various scenarios.

[0102] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0103] The workflow of the parallel inference decoding method for large language models includes: 1. Type Prompt; 2. Initialize the KV Cache and randomly initialize W tokens; 3. Parallel Jacobi iteration N times → Generate Token trajectory → Construct n-gram Pool; 4. Efficient cache management: LRU + confidence eviction strategy.

[0104] 5. Parallel verification of G candidate n-grams → Accept high-confidence sequences.

[0105] 6. Dynamically adjust W, N, G: based on entropy value and acceptance rate feedback.

[0106] 7. Update the output sequence & re-iterate if no tokens are accepted.

[0107] 8. Is generation complete? → No → Return to the parallel prediction step; Yes → Output the final sequence.

[0108] Figure 4 This is a schematic diagram of the overall flow of the parallel inference decoding method for large language models provided in the embodiments of this application. Figure 4 As shown, the method includes the following steps 401 to 406: Conditions: Given a large language model M and a user input prompt, the goal is to accelerate the generation of W tokens.

[0109] Step 401: Initialization.

[0110] Input user prompt and build initial KV cache (compute the input key-value pair cache); Randomly initialize a sequence Y = [y1, y2, ..., y] for W future tokens to be predicted. W [ ] serves as the starting point for Jacobi iteration.

[0111] Step 402: Parallel Jacobi iteration and trajectory generation.

[0112] Using the current Prompt + Y as input, perform a complete forward propagation to obtain the position y at each position. i logits; Update Y → Y' based on logits (take the top-1); Repeat the above process N times to form the "trajectory" for each position: for example, the trajectory of position k is [y k (0) , y k (1) , ...,y k ( ᴺ ) ]; Extract all continuous N-grams from the trajectory (e.g., extract [y] when N=2). k (0) , y k (1) ], [y k (1) , y k (2) (etc.), added to the n-gram cache pool.

[0113] Step 403: Efficient n-gram trajectory cache management.

[0114] Cache structure: Maintain a fixed-capacity n-gram cache pool. Each entry contains: n-gram sequence, generation position, confidence (top-1 probability), and last access step S.

[0115] The elimination strategy is generated by combining the LRU strategy and the confidence strategy: a. LRU strategy: Prioritize the elimination of the n-gram that has not been validated and adopted for the longest time, and retain the high-frequency reuse sequence; Retain frequently validated and adopted n-grams to reflect their reuse value in the current context. Avoid the accidental deletion of high-frequency semantic combinations (such as "artificial intelligence" and "fishing enthusiast") due to early insertion caused by a "first-in, first-out" approach. The specific calculation strategy is as follows: 1) Maintain a step S for the last adoption for each n-gram entry in the cache pool; 2) Whenever an n-gram is validated and successfully adopted by the main model, update its S to the iteration step number Current_Step; 3) If a certain n-gram is never adopted, then S remains at its initial value of 0; 4) During the elimination decision, calculate the LRU score (S_LRU) for each n-gram: S_LRU(i) = (CurrentStep - S_i) / MaxStepDiff; MaxStepDiff is a fixed value; the higher S_LRU is, the longer the n-gram has not been adopted.

[0116] b. Confidence Strategy: Prioritize discarding n-gram trajectories with the lowest average confidence to improve cache quality. Retain n-grams generated with high model confidence to improve cache semantic accuracy and reduce validation failures and rollback overhead caused by low-quality guesses. The calculation method is as follows: 1) Record the top-1 probability value of each word in the generation trajectory for each n-gram; 2) Calculate the average confidence score (S_Conf) of this n-gram: S_Conf(i) = (p1+ p2+ ... + p n ) / n; Where, p j is the top-1 probability of the j-th word when it is generated; the score range is [0, 1], the smaller the value, the lower the average confidence, and the more worthy it is to be eliminated.

[0117] The final score is calculated as S_final = alpha * S_Conf - beta * S_LRU, where alpha and beta are hyperparameters. The highest-scoring n-gram will be eliminated.

[0118] Step 404: Parallel verification and sequence acceptance.

[0119] Choose the n-gram whose beginning completely matches the last token of the already accepted sequence. The input becomes [Prompt + accepted sequence + n-gram]. Calculate the logits at each position for verification. If it is greedy decoding, directly verify whether the top-1 prop is the same as each sequence in the n-gram. If it is sampling decoding, use the "reject sampling" algorithm to generate a random number and calculate whether the next token in the n-gram can be accepted. If accepted, continue to the next token; otherwise, use the original model to sample the next token to ensure that the iteration continues.

[0120] Step 405: Dynamically adapt to window adjustments.

[0121] (1) Adjustment based on entropy: Before generating each n-gram, calculate the entropy of the probability distribution at the current position: H(P) = -Σ p(x)·log(p(x)); If H(P) < threshold (e.g., 0.8), it indicates that the model confidence is high, so increase W(+1), N(+1), and G(+1). If H(P) > threshold, it indicates that the model is uncertain. Reduce W(-1), N(-1), and G(-1) and perform conservative iteration.

[0122] (2) Adjustments based on acceptance rate: Calculate the total length of the n-grams that were successfully accepted in the most recent K validations, divided by the total number of validations. If the acceptance rate is >70%, increase W (+1), N (+1), and G (+1); if it is <30%, decrease parameters W (-1), N (-1), and G (-1). These two strategies can work simultaneously, ensuring a dynamic balance between acceleration and computational load, resulting in faster acceleration and less resource consumption.

[0123] Step 406: Iterative convergence and output.

[0124] Repeat steps 402 to 405 until the generated sequence reaches the maximum length or encounters a terminator. Output the final token sequence, maintaining the same generation quality as autoregressive decoding.

[0125] This application aims to address the problems of low acceptance rate, poor caching efficiency, rigid parameters, and large computational redundancy in existing large language model inference decoding methods. It provides a parallel decoding method that does not require additional model training, does not rely on repeated context, can dynamically adapt to generated states, and efficiently utilizes iterative trajectory caching, significantly improving the inference speed of large language models while maintaining lossless output quality.

[0126] Based on the foregoing embodiments, this application provides a parallel speculative decoding device for a large language model. The modules and units included in the device can be implemented by a processor; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0127] The parallel inference decoding apparatus for large language models provided in the embodiments of this application is described below. The parallel inference decoding apparatus for large language models described below can be referred to in correspondence with the parallel inference decoding method for large language models described above.

[0128] Figure 5 This is a schematic diagram of the structure of the parallel inference decoding device for a large language model provided in an embodiment of this application. Figure 5 As shown, the large language model parallel inference decoding device 500 includes: The trajectory acquisition module 510 is used to predict the subsequent words of the current output sequence based on the current output sequence using a large language model in a parallel iterative manner, so as to obtain the predicted trajectory at each prediction position. The fragment extraction module 520 is used to extract continuous word fragments from the predicted trajectory as candidate n-gram fragments; The fragment storage module 530 is used to store the candidate n-gram fragments into a cache pool. The cache pool has a fixed capacity and is maintained using a hybrid eviction policy. The hybrid eviction policy combines access timeliness and generation confidence. The fragment verification module 540 is used to select candidate n-gram fragments from the cache pool whose beginning matches the end lexicon of the current output sequence, input them into the large language model for parallel verification, and append the lexicon sequence contained in the successfully verified candidate n-gram fragments to the current output sequence.

[0129] In some embodiments, the current output sequence is initialized with a cue word at the start of decoding, and during the decoding process, it is dynamically expanded by appending the lexical sequences contained in the successfully verified candidate n-gram segments to the current output sequence.

[0130] In some embodiments, the trajectory acquisition module 510 includes: A sequence acquisition unit is used to acquire a sequence to be predicted, wherein the sequence to be predicted includes W words, where W is an integer greater than 1; The model iteration unit is used to perform N iterations, where N is an integer greater than 1. In each iteration, the current output sequence and the current sequence to be predicted are concatenated and input into the large language model, and the sequence to be predicted is updated according to the word probability distribution output by the large language model. The trajectory acquisition unit is used to form a predicted trajectory for each prediction position based on the value changes of the sequence to be predicted in the N iterations.

[0131] In some embodiments, each candidate n-gram fragment entry in the cache pool records the following information: the lexical sequence of the candidate n-gram fragment, the generation position, the preferred probability of each lexical in the candidate n-gram fragment, and the iteration step number when the candidate n-gram fragment was last successfully adopted.

[0132] In some embodiments, the fragment storage module 530 includes: The score acquisition unit is used to calculate a comprehensive elimination score for each candidate n-gram fragment in the cache pool when the cache pool is full. The comprehensive elimination score is a weighted sum of the average confidence score and the least recently used score of the candidate n-gram fragment. The average confidence score is used to characterize the overall generation determinism level of the candidate n-gram fragment, and the least recently used score is used to characterize the length of time that the candidate n-gram fragment has not been reused. The fragment deletion unit is used to delete some candidate n-gram fragments in the cache pool according to the comprehensive elimination score of each candidate n-gram fragment, wherein the higher the score, the higher the priority of deletion.

[0133] In some embodiments, the least recently used score is calculated by dividing the number of iterations the candidate n-gram fragment has undergone since its last successful adoption by a preset constant.

[0134] In some embodiments, the average confidence score is calculated by summing the preferred probability values ​​corresponding to each word in the candidate n-gram segment and dividing by the length of the candidate n-gram segment.

[0135] In some embodiments, the fragment extraction module 520 is specifically used for: From the predicted trajectories at all predicted positions, a continuous subsequence of word terms of length L is extracted as the candidate n-gram segment, where L is an integer greater than 1.

[0136] In some embodiments, the fragment verification module 540 includes a fragment verification unit, which verifies the fragment using any of the following methods: The system sequentially verifies whether the preferred word units output by the large language model at each corresponding position in the candidate n-gram segment are consistent with the word units at the corresponding positions in the candidate n-gram segment. If they are completely consistent, the verification is considered successful. Alternatively, based on the probability distribution of the large language model at the corresponding position of the candidate n-gram segment, random numbers are generated sequentially to determine whether the lexical units in the candidate n-gram segment can be accepted. If all can be accepted, the verification is considered successful.

[0137] In some embodiments, the large language model parallel inference decoding apparatus further includes: The parameter adjustment module is used to dynamically adjust the prediction parameters based on the evaluation results of the current decoding state. The prediction parameters include at least one of the following: the number of lexical units in the sequence to be predicted, the number of iterations, and the length of the candidate n-gram segments. The evaluation results of the current decoding state include at least one of the following: the probability distribution entropy of the large language model at the current prediction position, and the validation acceptance rate of recent candidate n-gram segments.

[0138] In some embodiments, if the evaluation result includes the probability distribution entropy, the parameter adjustment module is specifically used for: If the probability distribution entropy is lower than or equal to the first threshold, then at least one of the prediction parameters is increased; If the probability distribution entropy is higher than the second threshold, then at least one of the prediction parameters is reduced.

[0139] In some embodiments, if the evaluation result includes the validation acceptance rate, where the validation acceptance rate is the ratio of the total length of successfully adopted n-gram fragments to the total number of validations in the most recent K validations, and K is a positive integer, the parameter adjustment module is specifically used for: If the verification acceptance rate is higher than the third threshold, then increase at least one of the prediction parameters; If the validation acceptance rate is lower than or equal to the fourth threshold, then at least one of the prediction parameters is reduced.

[0140] In this embodiment, a hybrid elimination strategy based on access timeliness and generation confidence is introduced to intelligently manage the candidate n-gram cache, which effectively improves the verification acceptance rate of candidate sequences in speculative decoding, thereby significantly reducing computational redundancy and improving the inference speed of large language models while ensuring generation quality.

[0141] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application. For example... Figure 6As shown, the electronic device 600 may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a large language model parallel inference decoding method. This method includes: based on the current output sequence, using a large language model to predict subsequent words in the current output sequence in a parallel iterative manner, to obtain the prediction trajectory at each prediction position. Continuous word segments are extracted from the predicted trajectory as candidate n-gram segments; The candidate n-gram fragments are stored in a cache pool with a fixed capacity and maintained using a hybrid eviction policy that combines access timeliness with generation confidence. Candidate n-gram segments whose beginning matches the end lexicon of the current output sequence are selected from the cache pool, input into the large language model for parallel verification, and the lexicon sequence contained in the successfully verified candidate n-gram segments is appended to the current output sequence.

[0142] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the large language model parallel inference decoding method provided by the above methods. The method includes: based on the current output sequence, using a large language model to predict the subsequent word units of the current output sequence in a parallel iterative manner to obtain the prediction trajectory of each prediction position. Continuous word segments are extracted from the predicted trajectory as candidate n-gram segments; The candidate n-gram fragments are stored in a cache pool with a fixed capacity and maintained using a hybrid eviction policy that combines access timeliness with generation confidence. Candidate n-gram segments whose beginning matches the end lexicon of the current output sequence are selected from the cache pool, input into the large language model for parallel verification, and the lexicon sequence contained in the successfully verified candidate n-gram segments is appended to the current output sequence.

[0144] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0145] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the large language model parallel inference decoding method provided by the above methods, the method comprising: based on the current output sequence, using a large language model to predict subsequent lexical units of the current output sequence in a parallel iterative manner to obtain a prediction trajectory for each prediction position; Continuous word segments are extracted from the predicted trajectory as candidate n-gram segments; The candidate n-gram fragments are stored in a cache pool with a fixed capacity and maintained using a hybrid eviction policy that combines access timeliness with generation confidence. Candidate n-gram segments whose beginning matches the end lexicon of the current output sequence are selected from the cache pool, input into the large language model for parallel verification, and the lexicon sequence contained in the successfully verified candidate n-gram segments is appended to the current output sequence.

[0146] The aforementioned computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0147] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0148] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, radio frequency (RF), etc., or any suitable combination thereof.

[0149] Computer program code for performing the operations described herein can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0151] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the large language model parallel speculative decoding method described in various embodiments or some parts of embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A parallel inference decoding method for large language models, characterized in that, include: Based on the current output sequence, the large language model is used to predict the subsequent words in the current output sequence in a parallel iterative manner to obtain the prediction trajectory at each prediction position. Continuous word segments are extracted from the predicted trajectory as candidate n-gram segments; The candidate n-gram fragments are stored in a cache pool with a fixed capacity and maintained using a hybrid eviction policy that combines access timeliness with generation confidence. Candidate n-gram segments whose beginning matches the end lexicon of the current output sequence are selected from the cache pool, input into the large language model for parallel verification, and the lexicon sequence contained in the successfully verified candidate n-gram segments is appended to the current output sequence.

2. The parallel inference decoding method for large language models according to claim 1, characterized in that, The current output sequence is initialized as a prompt word at the start of decoding. During the decoding process, the sequence of words contained in the successfully verified candidate n-gram segments is appended to the current output sequence to dynamically expand it.

3. The parallel inference decoding method for large language models according to claim 1 or 2, characterized in that, Based on the current output sequence, a large language model is used to predict subsequent words in the current output sequence in a parallel iterative manner to obtain the prediction trajectory at each prediction position, including: Obtain the sequence to be predicted, which includes W words, where W is an integer greater than 1; Perform N iterations, where N is an integer greater than 1. In each iteration, the current output sequence and the current sequence to be predicted are concatenated and input into the large language model, and the sequence to be predicted is updated according to the word probability distribution output by the large language model. Based on the value changes of the sequence to be predicted in the N iterations, a prediction trajectory is formed for each prediction position.

4. The parallel inference decoding method for large language models according to claim 1, characterized in that, Each candidate n-gram fragment entry in the cache pool records the following information: the lexical sequence of the candidate n-gram fragment, the generation position, the first-choice probability of each lexical in the candidate n-gram fragment, and the iteration step number when the candidate n-gram fragment was last successfully adopted.

5. The parallel inference decoding method for large language models according to claim 4, characterized in that, The step of storing the candidate n-gram fragments into a cache pool includes: When the cache pool is full, a comprehensive elimination score is calculated for each candidate n-gram fragment in the cache pool. The comprehensive elimination score is a weighted sum of the average confidence score and the least recently used score of the candidate n-gram fragment. The average confidence score is used to characterize the overall generation determinism level of the candidate n-gram fragment, and the least recently used score is used to characterize the length of time that the candidate n-gram fragment has not been reused. Based on the overall elimination score of each candidate n-gram fragment, some candidate n-gram fragments in the cache pool are deleted, where a higher score indicates a higher priority for deletion.

6. The parallel inference decoding method for large language models according to claim 5, characterized in that, The least recently used score is calculated by dividing the number of iterations the candidate n-gram fragment has undergone since its last successful adoption by a preset constant.

7. The parallel inference decoding method for large language models according to claim 5, characterized in that, The average confidence score is calculated by summing the preferred probability values ​​corresponding to each word in the candidate n-gram segment and dividing by the length of the candidate n-gram segment.

8. The parallel inference decoding method for large language models according to claim 1 or 2, characterized in that, The step of extracting continuous word segments from the predicted trajectory as candidate n-gram segments includes: From the predicted trajectories at all predicted positions, a continuous subsequence of word terms of length L is extracted as the candidate n-gram segment, where L is an integer greater than 1.

9. The parallel inference decoding method for large language models according to claim 1, characterized in that, The step of selecting candidate n-gram segments from the cache pool whose beginnings match the end tokens of the current output sequence and inputting them into the large language model for parallel validation includes any of the following methods: The system sequentially verifies whether the preferred word units output by the large language model at each corresponding position in the candidate n-gram segment are consistent with the word units at the corresponding positions in the candidate n-gram segment. If they are completely consistent, the verification is considered successful. Alternatively, based on the probability distribution of the large language model at the corresponding position of the candidate n-gram segment, random numbers are generated sequentially to determine whether the lexical units in the candidate n-gram segment can be accepted. If all can be accepted, the verification is considered successful.

10. The parallel inference decoding method for large language models according to claim 1, characterized in that, The method further includes: Based on the evaluation results of the current decoding state, the prediction parameters are dynamically adjusted; the prediction parameters include at least one of the following: the number of lexical units in the sequence to be predicted, the number of iterations, and the length of the candidate n-gram segments; the evaluation results of the current decoding state include at least one of the following: the probability distribution entropy of the large language model at the current prediction position, and the validation acceptance rate of recent candidate n-gram segments.

11. The parallel inference decoding method for large language models according to claim 10, characterized in that, If the evaluation result includes the probability distribution entropy, the step of dynamically adjusting the prediction parameters based on the evaluation result of the current decoding state includes: If the probability distribution entropy is lower than or equal to the first threshold, then at least one of the prediction parameters is increased; If the probability distribution entropy is higher than the second threshold, then at least one of the prediction parameters is reduced.

12. The parallel inference decoding method for large language models according to claim 10, characterized in that, If the evaluation result includes the validation acceptance rate, where the validation acceptance rate is the ratio of the total length of successfully adopted n-gram segments to the total number of validations in the most recent K validations, and K is a positive integer, then dynamically adjusting the prediction parameters based on the evaluation result of the current decoding state includes: If the verification acceptance rate is higher than the third threshold, then increase at least one of the prediction parameters; If the validation acceptance rate is lower than or equal to the fourth threshold, then at least one of the prediction parameters is reduced.

13. A parallel inference decoding device for a large language model, characterized in that, include: The trajectory acquisition module is used to predict subsequent words in the current output sequence using a large language model in a parallel iterative manner based on the current output sequence, so as to obtain the predicted trajectory at each prediction position. The fragment extraction module is used to extract continuous word fragments from the predicted trajectory as candidate n-gram fragments; The fragment storage module is used to store the candidate n-gram fragments into a cache pool. The cache pool has a fixed capacity and is maintained using a hybrid eviction policy. The hybrid eviction policy combines access timeliness and generation confidence. The fragment verification module is used to select candidate n-gram fragments from the cache pool whose beginnings match the end lexicons of the current output sequence, input them into the large language model for parallel verification, and append the lexicon sequences contained in the successfully verified candidate n-gram fragments to the current output sequence.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the large language model parallel speculative decoding method as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the large language model parallel speculative decoding method as described in any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the large language model parallel speculative decoding method as described in any one of claims 1 to 12.

Images