Speculative sampling based large model inference acceleration method and device, equipment and medium

Abstract

The application provides a large model inference acceleration method and device based on speculative sampling, equipment and medium, which constructs a lightweight, adaptive and fused two-stage N-Gram draft model framework, uses high-order N-Gram model to capture long sequence dependence pattern in local context, and uses low-order N-Gram model to provide robust probability prior to alleviate data sparsity problem, and based on this, dynamically determines interpolation weight based on context semantic strength, dynamically weights and fuses the probability distribution output by high-order and low-order N-Gram model respectively by using the interpolation weight, effectively combines the complementary advantages of high-order and low-order N-Gram model to generate mixed probability distribution, uses the mixed probability distribution to generate candidate word sequence, avoids the prediction deviation caused by improper order setting of single model, improves the semantic rationality and verification pass rate of candidate sequence while keeping low computing overhead and storage requirement, and realizes large model inference acceleration.

Description

Technical Field

[0001] This application relates to the field of large model inference technology, and in particular to a method, apparatus, device and medium for accelerating large model inference based on speculative sampling. Background Technology

[0002] The autoregressive decoding process of large models generates tokens one by one, which requires frequent access to massive parameters, causing memory bandwidth to become the inference bottleneck. Speculative sampling introduces a lightweight draft model to pre-generate candidate sequences, which are then verified by the large model. This can improve the inference efficiency of large models without changing the output distribution. Among them, the N-Gram model is often used as a draft model because of its simple structure and efficient inference.

[0003] However, when traditional N-Gram models are used as draft models, high-order N-Grams rely on the co-occurrence frequency of n-grams in the training corpus for probability estimation. Many semantically reasonable but infrequently used word combinations do not appear in the corpus or appear very infrequently, resulting in their probabilities being estimated to be zero or extremely low. This leads to data sparsity issues, affecting the rationality and diversity of candidate sequences. While low-order N-Grams are somewhat robust to data sparsity, their context modeling capabilities are limited, making it difficult to capture complex local dependencies. Therefore, traditional N-Gram models struggle to balance prediction accuracy and robustness, resulting in unstable quality of generated candidate sequences, high failure rates in large-scale model validation, and frequent triggering of backoff mechanisms. This increases computational overhead, weakens the acceleration effect of speculative sampling, and limits their effective application in real-world scenarios. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems, this application provides a method, apparatus, device and medium for accelerating large model inference based on speculative sampling. It can overcome the problem of unstable candidate quality caused by data sparsity, high storage cost and static modeling in traditional N-Gram models by constructing a lightweight and context-adaptive two-order N-Gram hybrid modeling mechanism. It effectively balances the semantic expressive power and prediction robustness of the model, improves the validation pass rate of large models in the speculative sampling process, reduces the backoff frequency caused by candidate errors, reduces the number of forward inferences and memory access overhead of large models, and achieves efficient and stable inference acceleration.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] According to a first aspect of the embodiments of this application, a method for accelerating large model inference based on speculative sampling is provided, the method comprising:

[0007] Obtain a first probability distribution output by a high-order N-Gram model based on a vocabulary and a first context of input. The first probability distribution includes the probability that each word in the vocabulary is the next word in the first context. The first context includes K words, where K+1 is the order of the high-order N-Gram model.

[0008] Obtain a second probability distribution output by the low-order N-Gram model based on the vocabulary and the input second context. The second probability distribution includes the probability that each word in the vocabulary is the next word in the second context. The second context includes M words, where M+1 is the order of the low-order N-Gram model, and M is less than K. The next position of the first context and the next position of the second context are the same position.

[0009] Interpolation weights are obtained based on the text vectors corresponding to the first context, and interpolation operations are performed on the first probability distribution and the second probability distribution based on the interpolation weights to obtain a mixed probability distribution. The target word at the position is determined based on the mixed probability distribution according to the set speculative sampling strategy. The mixed probability distribution includes the mixed probability of each word in the vocabulary as a word at the position.

[0010] If it is determined that lexical verification is required, the target lexical at the specified position and the predicted target lexical at the L positions preceding the specified position are submitted to the large model; otherwise, the first context and the second context are updated based on the target lexical at the specified position, and the process returns to the step of obtaining the first probability distribution and the second probability distribution.

[0011] According to a second aspect of the embodiments of this application, a large-model inference acceleration device based on speculative sampling is provided, the device comprising:

[0012] The high-order prediction result acquisition module is configured to obtain a first probability distribution output by the high-order N-Gram model based on the vocabulary and the first context of the input. The first probability distribution includes the probability that each word in the vocabulary is the next word in the first context. The first context includes K words, where K+1 is the order of the high-order N-Gram model.

[0013] The low-order prediction result acquisition module is configured to obtain a second probability distribution output by the low-order N-Gram model based on the vocabulary and the input second context. The second probability distribution includes the probability that each word in the vocabulary is the next word in the second context. The second context includes M words, where M+1 is the order of the low-order N-Gram model, and M is less than K. The next position of the first context and the next position of the second context are the same position.

[0014] The interpolation module is configured to obtain interpolation weights based on the text vector corresponding to the first context, and to perform interpolation operations on the first probability distribution and the second probability distribution based on the interpolation weights to obtain a mixed probability distribution. Then, according to a set speculative sampling strategy, the target word at the position is determined based on the mixed probability distribution. The mixed probability distribution includes the mixed probability of each word in the vocabulary as a word at the position.

[0015] The speculative sampling and verification module is configured to, if it is determined that lexical verification is required, submit the target lexical at the current position and the predicted target lexical at the previous L positions to the large model; otherwise, update the first context and the second context based on the target lexical at the current position and return to the step of obtaining the first probability distribution and the second probability distribution.

[0016] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a memory and a processor; the memory being used to store a computer program; the processor being used to execute the above-described large model inference acceleration method based on speculative sampling by invoking the computer program.

[0017] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described method for accelerating large model inference based on speculative sampling.

[0018] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0019] In the technical solution provided in this application, a lightweight, adaptive, fusion-based two-order N-Gram draft model framework is constructed. The higher-order N-Gram model captures long-sequence dependency patterns and high-frequency phrase structures in the local context to enhance the modeling ability for specific linguistic expressions. Simultaneously, the lower-order N-Gram model provides robust probabilistic priors to alleviate data sparsity. Based on this, interpolation weights are dynamically determined based on contextual semantic strength awareness. According to the probability distributions output by the higher-order and lower-order N-Gram models, these interpolation weights generate a context-aware hybrid probability distribution, avoiding prediction bias caused by improper order setting in a single model. Furthermore, this hybrid probability distribution maintains high probability estimation accuracy and stability under diverse input conditions, enhancing the semantic rationality, grammatical compliance, and contextual consistency of the final generated candidate word sequences. While maintaining extremely low computational overhead and storage requirements, it improves the semantic rationality and validation pass rate of candidate sequences, thereby reducing the number of forward propagations in large models, effectively overcoming the memory access bottleneck in autoregressive decoding, and accelerating large model inference.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Furthermore, no embodiment in this application needs to achieve all the effects described above. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1A This is a schematic diagram illustrating a prefix lexical sequence and obtaining a first context and a second context, as shown in an exemplary embodiment of this application;

[0023] Figure 1B This is a schematic diagram illustrating an exemplary embodiment of a method for accelerating large model inference based on speculative sampling.

[0024] Figure 1C This is a schematic diagram of a hierarchical probabilistic storage structure shown in an exemplary embodiment of this application;

[0025] Figure 1D This is a schematic diagram illustrating an exemplary embodiment of the present application of a process for interpolating the first probability distribution and the second probability distribution based on the interpolation weight;

[0026] Figure 2A This is a flowchart illustrating a step of long-distance dependency correction of a mixed probability distribution based on historical context dependency information, as shown in an exemplary embodiment of this application.

[0027] Figure 2B This is a schematic diagram illustrating an exemplary embodiment of this application of obtaining the historical hidden state of historical word elements and correcting the mixed probability distribution;

[0028] Figure 2C This is a schematic diagram illustrating an exemplary embodiment of the present application of a process for adjusting a mixed probability distribution based on attention weights and decay factors;

[0029] Figure 3A This is a schematic diagram illustrating the training sample construction process for online incremental training of a gating network, as shown in an exemplary embodiment of this application.

[0030] Figure 3B This is a schematic diagram illustrating the labeling process of sample labels in the training samples of a gating network according to an exemplary embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the structure of a large model inference acceleration device based on speculative sampling, as illustrated in an exemplary embodiment of this application.

[0032] Figure 5 This is a hardware schematic diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another.

[0034] Before explaining and describing the large model inference acceleration method provided in this embodiment, a brief explanation of some of the terms and concepts involved in this application will be given:

[0035] Large models represent deep neural network models with a huge number of parameters (usually in the billions to hundreds of billions), massive training data, and powerful language understanding and generation capabilities, such as Qwen, LLaMA, and GPT series. Their structures are mostly based on the Transformer architecture, which can capture complex semantic relationships and contextual dependencies. In its autoregressive decoding process, tokens are generated one by one, and each step requires calling all model parameters for forward propagation.

[0036] Large-model inference acceleration refers to a general term for techniques that improve the generation speed, reduce latency, and lower energy consumption of large language models during the inference phase through algorithm optimization, architectural improvements, or the introduction of auxiliary mechanisms. Since large-model decoding is an autoregressive serial process, the generation of each new token must wait for the previous one to complete, creating a "word-by-word waiting" bottleneck. As the model size increases, each forward propagation involves reading massive amounts of parameters, and memory access bandwidth becomes a limiting factor for system performance. Common inference acceleration methods include at least caching attention keys, model quantization, model pruning and distillation, and parallel sampling strategies such as speculative sampling / speculated sampling. Among these, speculative sampling is an effective means to maintain the consistency of the original distribution while significantly improving throughput.

[0037] Speculative sampling, as an efficient technique for accelerating large-scale model inference, utilizes a lightweight draft model to pre-generate several candidate tokens in parallel or in multiple steps. The large model then sequentially verifies whether these candidate tokens conform to its true probability distribution. If verification is successful, it achieves "multiple outputs in one execution"; otherwise, it backtracks and corrects erroneous paths. The workflow of speculative sampling can be simply summarized as follows: the draft model generates k candidate tokens based on the current context and submits the current context and k candidate tokens to the large model for verification. For each candidate token, the large model uses rejection sampling to determine whether to accept the token. If a token is rejected at a certain position, it regenerates from that point and outputs all previously accepted tokens; this process is repeated until the entire sequence is generated.

[0038] The draft model is a lightweight language model used to quickly generate candidate token sequences within a speculative sampling framework as an initial guess for large model validation. It does not require completely accurate probabilistic modeling, but should have high generation efficiency and reasonable semantic priors.

[0039] Token verification is the core step in speculative sampling. It involves feeding the candidate token sequence generated by the draft model into the large model, where the large model re-evaluates its true conditional probability in the current context and determines whether to retain the token based on the set acceptance rules (such as rejection sampling based on probability ratio).

[0040] A lexical is the basic unit in text processing, and its granularity is determined by the word segmentation strategy. It can be a complete word, a word fragment, a single character, or a punctuation mark. In this embodiment, the lexical, as the basic language unit constituting the context sequence, affects the N-Gram model's ability to model local language patterns.

[0041] The N-Gram model is a statistical language model widely used in natural language processing. Its core idea is based on the Markov assumption: the probability of a token's occurrence depends on a finite number of preceding context tokens, and is independent of or weakly dependent on earlier historical tokens. The order N of the N-Gram model represents the length of the continuous word sequence the model relies on when predicting the current word. It divides the text sequence into continuous combinations of tokens of length N (called N-grams), and uses the first N-1 tokens to predict the probability distribution of each token in the vocabulary as the Nth token. Specifically: when N=1, the model only considers the current word itself when predicting the current word, without relying on any preceding context. For example, when predicting the probability of "apple," it only counts the frequency of "apple" in the corpus; when N=1, the model predicts the probability of the current word itself, without relying on any preceding context. When N=2, the model considers the combination of the current word and the previous word when predicting the current word. For example, when predicting "apple", it depends on the previous word (such as "eat") and calculates the joint probability of "eat apple". When N=3, the model considers the combination of the current word and the previous two words when predicting the current word. For example, when predicting "apple", it depends on the previous two words (such as "like to eat") and calculates the joint probability of "like to eat apple". And so on. When N=k (k-Gram), the model considers the combination of the current word and the previous k-1 words when predicting the current word.

[0042] Low-order N-gram models typically refer to models with N≤3, such as unigram and bigram models. These models have relatively simple structures, low computational overhead, and strong resistance to data sparsity, exhibiting good practicality and robustness in scenarios with limited resources, high real-time requirements, or insufficient training data. Unigram models, based on the assumption of word independence, ignore contextual information and estimate the probability of a word based on its frequency of occurrence in the corpus. The core idea of ​​its probability prediction is that high-frequency words are more likely to appear in any position. This model is suitable for tasks that are not sensitive to context or have sparse data, such as keyword extraction, word frequency analysis, and initial language model baselines. Bigram models introduce basic contextual dependencies, assuming that the probability of the current word's occurrence depends only on its preceding word. Based on a first-order Markov structure, they can capture local word collocation patterns (e.g., "coffee" is often followed by "beans," "cup," or "cause"), thus reflecting the local coherence of language to some extent.

[0043] Higher-order N-Gram models refer to N-Gram models with larger N values; typically, N≥4 is considered a higher-order model, such as 4-Gram, 5-Gram, etc. As N increases, the model can utilize longer historical context for prediction, thereby capturing longer-distance contextual dependencies and better reflecting local grammatical structures and semantic collocations in natural language. Compared to lower-order N-Gram models, higher-order N-Gram models have stronger linguistic expressive power and generally achieve higher prediction accuracy and perplexity performance in language modeling tasks, more closely resembling the true structure of natural language.

[0044] With the development of artificial intelligence technology, large models are widely used in the field of natural language processing, such as dialogue systems, text generation, code completion, and machine translation. Common large models typically contain billions or even hundreds of billions of trainable parameters and use the Transformer architecture to achieve deep semantic modeling and long-range dependency capture. However, because large models generate tokens sequentially in the autoregressive decoding stage, the generation of each token requires calling the complete model parameters for forward computation, resulting in a serious computational and memory access bottleneck in the inference process. In high-concurrency or real-time response scenarios, the model parameters need to be repeatedly loaded from storage units (such as GPU memory) to computing units (such as CUDA Cores) each time a token is generated, making memory access bandwidth a key factor restricting inference speed.

[0045] Speculative sampling can fundamentally decode and calculate the memory access ratio, improving generation throughput without changing the final sampling distribution. This method introduces a lightweight draft model to quickly generate several candidate token sequences, which are then validated by the original large model to determine whether to accept the generation path. Among the many speculative sampling techniques, the N-Gram model is often chosen as the draft model due to its simple structure, fast inference speed, and lack of neural network computation requirements. For example, 3-Gram or 5-Gram models can be used to generate candidate sequences for subsequent validation by the large model.

[0046] However, traditional N-Gram models suffer from data sparsity. Higher-order N-Gram models (such as 5-Gram and above) rely on the co-occurrence frequency of specific word order combinations in the training corpus. Due to the diversity of language expressions, many reasonable but rare n-gram combinations do not appear in the training set, resulting in probability estimates of zero or extremely low levels, creating a "zero probability" phenomenon that affects the rationality and diversity of candidate sequences. Furthermore, as the model order N increases, the number of n-gram entries required for storage grows exponentially, leading to high storage costs. These limitations in practical applications result in unstable candidate sequence quality when used as a draft model, leading to a higher failure rate in large-scale model validation, frequent triggering of backoff mechanisms, and hindering the overall efficiency and stability of speculative sampling.

[0047] In view of this, this application provides a method for accelerating large-scale model inference based on speculative sampling, which is applicable to autoregressive acceleration scenarios for generative model inference, such as real-time response of dialogue systems and chatbots, text generation and content creation applications, code generation and intelligent programming assistants, machine translation and cross-language understanding, document summarization and information extraction systems, etc. For the prediction of the word at the next position after the last word in the prefix word sequence, this embodiment proposes a lightweight high-low order hybrid modeling mechanism. By using the probabilistic prior of high- and low-order N-Gram language models and speculative sampling strategies, the accuracy and semantic rationality of candidate word generation are improved, thereby increasing the pass rate of large-scale model validation and achieving efficient and stable inference acceleration.

[0048] This embodiment employs both low-order and high-order N-Gram models. Based on the same vocabulary, it independently predicts the word at the next position of the prefix word sequence, outputting their respective probability distributions. Then, based on the contextual semantic strength, it dynamically weights and fuses the probability distributions of the high-order and low-order N-Gram models to generate a mixed probability P representing each word in the vocabulary as the word at that position. current Based on this mixed probability and combined with a preset speculative sampling strategy, the target word at the given position is selected from each word in the given vocabulary as a fast inference result for the output of the large model, replacing some autoregressive decoding steps. This reduces the number of forward inferences of the large model while ensuring the quality of generation, thereby accelerating the inference of the large model.

[0049] Further, the first context on which the high-order N-Gram model predicts the token at the next position of the last token in the prefix token sequence includes K consecutive tokens, where K + 1 represents the order of the high-order N-Gram model (i.e., N = K + 1 in N-Gram). Therefore, the last K tokens can be taken from the prefix token sequence as the first context on which the high-order N-Gram model predicts the token at this position; similarly, the second context on which the low-order N-Gram predicts the token at the next position of the last token in the prefix token sequence includes M tokens, where M + 1 represents the order of the low-order N-Gram model, and M < K is satisfied. Therefore, the last M tokens can be taken from the prefix token sequence as the second context of the low-order N-Gram model predicting the token at this position.

[0050] It should be clear that the next position of the first context and the next position of the second context are both the same position as the next position of the last token in the prefix token sequence, ensuring that the high-order N-Gram and the low-order N-Gram models rely on contexts of different lengths but have the same prediction target, with comparability and a basis for fusion.

[0051] For example, see Figure 1A The schematic diagram of an exemplary prefix token sequence and obtaining the first context and the second context. Assume that the current prefix token sequence is "the quick brown fox jumps":

[0052] Let the high-order N-Gram model be a 5-Gram (i.e., the order is 5), then the first context of this high-order N-Gram model includes the last 4 tokens in the prefix token sequence, that is, "quick brown fox jumps"; let the low-order N-Gram model be a bigram model, then the first context of this low-order N-Gram model includes the last 1 token in the prefix token sequence, that is, "jumps". Both the high-order N-Gram model and the low-order N-Gram model are used to predict the token at the next position of the last token jumps in the current prefix token sequence. By interpolating and fusing the respective probability distributions of the high-order N-Gram model and the low-order N-Gram model, the mixed probability of the token at this position is generated.

[0053] Furthermore, after completing one prediction—that is, selecting the target word at the next position after the last word in the prefix word sequence—if it is determined that word verification is needed (e.g., reaching a preset verification period or the number of predicted target words reaches a set number), the target word at the position to be predicted, along with the predicted target words at the previous L positions, are submitted to the large model for a one-time parallel forward computation. This allows the large model to perform probability verification on each predicted target word, i.e., by comparing the true conditional probability P of the target word output by the large model. LLM The mixture probability P of the target word when selected by speculative sampling current Does the confidence condition meet? The process involves evaluating the reliability of the inference of the target word and deciding whether to adopt the target word as the inference output for the corresponding position based on the verification results; otherwise, the prefix word sequence is directly updated based on the currently selected target word, that is, the target word is added to the end of the prefix word sequence to form an updated prefix word sequence, and the latest K and M words are extracted again based on the updated prefix word sequence to redetermine the first context and the second context, and then returning to the process of continuing to predict the next position of the last word in the prefix word sequence.

[0054] For example, after the diagram determines that the next word after "jumps" is "over", it is determined that no word verification is needed at this point. Therefore, "over" is added to the prefix word sequence to form the sequence "the quick brown fox jumps over". The first context "brown fox jumps over" and the second context "over" are redefined, and the word at the next position of "over" is predicted. If word verification is triggered subsequently, the target word generated when word verification is triggered, as well as the previously generated target word to be verified, including "over" and the target word after it, are submitted to the large model for verification.

[0055] Based on this, see Figure 1B The flowchart shown is an example of a method for accelerating large model inference based on speculative sampling, which may include at least the following steps:

[0056] S101, obtain the first probability distribution output by the high-order N-Gram model based on the vocabulary and the first context of the input. The first probability distribution includes the probability that each word in the vocabulary is the next word in the first context. The first context includes K words, and K+1 is the order of the high-order N-Gram model.

[0057] The vocabulary represents a finite set of all words that an N-Gram model can recognize and output, denoted as . This vocabulary defines the model's probability output space, ensuring that each prediction task is modeled on a fixed-size set of discrete lexical units. This vocabulary can be obtained by extracting all unique lexical units (including words, subwords, or character-level units, depending on the segmentation strategy used) from the training corpus. A minimum frequency threshold can be set to filter out low-frequency lexical units to control model size, and special markers can be introduced, such as... <unk>(Unknown word) <sos>(Start symbol) <eos>(End markers) etc., to improve generalization ability. For example, if the vocabulary contains {"I", "you", "love", "learning", "artificial intelligence", ...}, then when the model predicts the word in the next position of the prefix word sequence, it will predict the conditional probability of these words in the vocabulary as the word in the next position.

[0058] K represents the historical dependency length used by the higher-order N-Gram model to predict the current word, meaning the model estimates the probability of the next word's occurrence by referring to the previous K known words. The parameter K is also called the context window size. The first context input represents the preceding word sequence used by the higher-order N-Gram model for prediction, i.e., the sequence of K consecutive words preceding the current prediction position. In this step, the first context is a word sequence of length K. Let the current prefix word sequence be:

[0059]

[0060] If the next position of the first context is Wt+1, then the first context can be represented as:

[0061]

[0062] Here, each Wi represents a token, K is the size of the context window on which the higher-order N-Gram model relies, and the conditional probability of the token Wt+1 to be predicted depends on the first K words. The first context constitutes the word element at position Wt+1 in the prediction of the higher-order N-Gram model. For example, for a higher-order 5-Gram model, the word element window K = 4 corresponding to the higher-order N-Gram model, and the first context of the input is ("I", "like", "rice", "and"). Then the model will predict the word element at the next position of the word element "and" based on the first context.

[0063] The first probability distribution represents the higher-order N-Gram model given the first context. Then, the conditional probability distribution of the output of each word in the entire vocabulary V as the word in the next position of the first context is a normalized discrete probability vector: Let the vocabulary be V. The first probability distribution is represented as a vector: And satisfy This first probability distribution can be obtained based on the maximum likelihood estimate in the N-Gram model, i.e. Where count() represents the frequency of occurrence of the word sequence in the training corpus, and the numerator represents the sequence ( vi) The frequency of occurrence as (K+1)-Gram in the training corpus, where the denominator represents the context. The total frequency of its own appearance as a K-Gram.

[0064] Based on the above, this step utilizes a trained high-order N-Gram model to calculate the conditional probability of each word in the vocabulary being the next word in the first context, given a first context of length K. This generates a complete probability distribution, where the j-th probability element represents the conditional probability of the j-th word in the vocabulary being the next word in that position. When calculating the conditional probability of each word in the vocabulary being the next word in the first context, the conditional probability can be determined using the maximum likelihood estimation method in the following way:

[0065] Query the frequency of the first context in the global count statistics (Count( Furthermore, traverse the vocabulary, and for each word vi∈V in the vocabulary, construct a sequence combination ( ,vi), query the co-occurrence frequency count of this sequence combination in the training corpus. ,vi), to obtain the original probability If Count( If the probability of an unseen sequence is zero (i, vi), a smoothing algorithm is applied to avoid it. For example, Kneser-Ney smoothing can be used, which effectively estimates the probability of an unseen sequence through a backtracking mechanism and absolute discounting. Since the first probability distribution is a normalized probability vector, the original probability of each word in the vocabulary is normalized to obtain the first probability distribution output by the higher-order N-Gram model. This first probability distribution can be stored as a multidimensional vector, with the length of the vector being the same as the number of lexical units in the vocabulary. Alternatively, it can be organized into a dictionary or array structure for easy access later.

[0066] {

[0067] "Me": 0.001,

[0068] "You": 0.003,

[0069] "Love": 0.02,

[0070] "Learning": 0.0005,

[0071] "Artificial Intelligence": 0.85,

[0072] " <unk>0.1, ...

[0073] }

[0074] To improve inference efficiency and support fast lookups with large-scale vocabularies, this embodiment provides a hierarchical probabilistic storage structure consisting of a high-frequency layer, a mid-frequency layer, and a backoff layer. This replaces the traditional global frequency table lookup and achieves efficient and robust retrieval of the output probabilities of high-order N-Gram models. The high-frequency and mid-frequency layers ensure low-latency responses to common patterns, while the backoff layer uses low-order N-Gram models and statistical priors to address sparsity issues. The overall mechanism balances speed and accuracy, providing high-quality, lightweight inference for subsequent probability fusion and speculative sampling.

[0075] Referring to Figure 1C, which exemplarily illustrates a hierarchical probability storage structure, the high-frequency layer storage structure is used to cache the top x% of high-order N-Gram sequences with the highest frequency, i.e., sequence combinations of length K+1; the mid-frequency layer storage structure is used to cache high-order N-Gram sequences with frequencies between x% and y%; the fallback layer includes a preset low-order language model used to process low-frequency or zero-frequency high-order N-Gram sequences that are not hit in the high-frequency and mid-frequency layers. The low-order language model does not depend on the complete context and is configured to generate the fallback probability P of a word based on a shorter historical window (such as the last 1-2 words) or the global statistical characteristics of the word itself (such as the frequency of the word in the training corpus). backoff (vi) The backoff strategy can be combined with Kneser-Ney smoothing or linear interpolation to ensure reasonable non-zero probabilities are output in the case of sparse data. This structure adopts a design idea that combines multi-level caching with a backoff mechanism, prioritizing high-frequency N-Gram sequences in the high-speed access layer, while low-frequency or unregistered sequences are dynamically probable through a lightweight language model, thereby reducing query latency and storage overhead while ensuring prediction accuracy.

[0076] Under this hierarchical probabilistic storage structure, the first probability distribution output by the high-order N-Gram model based on the vocabulary and the first context of the input can be obtained in the following way: In the hierarchical probabilistic storage structure composed of high-frequency layer, mid-frequency layer and backoff layer, the high-order N-Gram sequence corresponding to each word in the vocabulary is queried layer by layer to obtain the probability of the word being the word at the position; wherein, the high-order N-Gram sequence corresponding to the word includes the first context and the word.

[0077] In other words, for the first context of the current input For each lexical unit vi in ​​the vocabulary, construct a complete (K+1)-Gram sequence (i.e., a higher-order N-Gram), which represents "in the first context" The language pattern followed by the lexical "vi":

[0078]

[0079] In response to this The sequence is queried hierarchically, retrieving its probability information sequentially in the high-frequency layer, mid-frequency layer, and backtracking layer, ultimately obtaining the word as the first context. The conditional probability of the word at the next position is summed, and the probabilities of all words are normalized to form a complete first probability distribution, which is used as the output of the higher-order N-Gram model.

[0080] For example, suppose the current first context is: ("I", "like", "eat", "apple"), i.e., K=4, the higher-order N-Gram model is 5-Gram, and the vocabulary contains candidate nouns: ["banana", "orange", "watermelon", "study", "run", ...]:

[0081] The top 10% of high-frequency caches are most commonly 5-Gram, such as "I like to eat apples and bananas";

[0082] The mid-frequency layer buffers 10% to 30% of the mid-frequency sequences;

[0083] The fallback layer uses a Bigram model with global word frequency weighting;

[0084]

[0085] When predicting the word at the next position in the current first context, the constructed The sequence is as follows:

[0086] Finally, after normalization, the first probability distribution is obtained:

[0087] {

[0088] "Banana": 0.43,

[0089] "Orange": 0.11,

[0090] "Watermelon": 0.04,

[0091] "Learning": 0.01,

[0092] "Running": 0.01,

[0093] ...

[0094] }

[0095] S102, obtain the second probability distribution output by the low-order N-Gram model based on the vocabulary and the input second context. The second probability distribution includes the probability that each word in the vocabulary is the next word in the second context. The second context includes M words, where M+1 is the order of the low-order N-Gram model, and M is less than K. The next position of the first context and the next position of the second context are the same position.

[0096] M represents the historical dependency length used by the low-order N-Gram model to predict the current lexical unit, meaning the model estimates the probability of the next lexical unit by referring to the previous M known lexical units. The second context input represents the pre-word sequence used by the low-order N-Gram model for prediction, i.e., the sequence of M consecutive lexical units preceding the current position to be predicted. In this step, the second context is a continuous sequence of lexical units of length M. If the next position of the second context is Wt+1, the second context can be represented as:

[0097]

[0098] Here, each Wi represents a token, M is the size of the context window on which the low-order N-Gram model relies, and the conditional probability of the token Wt+1 to be predicted depends on the first M words. This constitutes the second context for predicting the word at position Wt+1 in the low-order N-Gram model. Based on the implementation steps, a low-order N-Gram model is used, whose order is lower than that of the high-order N-Gram model. Correspondingly, the length of the second context on which the low-order N-Gram model depends is shorter than the length of the first context on which the high-order N-Gram model depends.

[0099] For example, for a low-order 3-Gram model with a lexical window M = 2, if the prefix lexical sequence is: ["I", "like", "eat", "apple", "also", "love"], then the second context is "also love". The low-order N-Gram model will predict the lexical in the next position of the lexical "love" based on this second context.

[0100] The second probability distribution represents the low-order N-Gram model given the second context C. low Then, the conditional probability distribution of the output of each word in the entire vocabulary V as the word in the next position of the second context is a normalized discrete probability vector: if the vocabulary is Then the second probability distribution is represented as a vector: And satisfy This second probability distribution can be obtained based on the maximum likelihood estimate in the N-Gram model, i.e. Where count() represents the frequency of occurrence of the word sequence in the training corpus, and the numerator represents the sequence combination ( vi) The frequency of occurrence of (M+1)-Gram sequences in the training corpus, where the denominator represents the context. The total frequency of occurrence of the M-Gram itself. The method for determining this second probability distribution using the maximum likelihood estimation method is the same as the principle for obtaining the first probability distribution described above, and will not be repeated in this embodiment. Similar to higher-order N-Gram models, lightweight caching structures (such as hash tables or Bloom filters) can also be built for lower-order N-Gram models to cache the probabilities of common Bigram / Trigram sequences, improving inference speed.

[0101] The fact that the next position in the first context and the next position in the second context are the same indicates that the prediction objectives of the high-order N-Gram model and the low-order N-Gram model used in this embodiment are completely consistent: both predict the unknown word after the last word in the prefix word sequence. For example, the prefix word sequence... ,based on Higher-order 5-Gram models and based on Low-order 2-Gram models are used to predict the word W at position t+1. t+1 .

[0102] Based on this, the prediction target of the second probability distribution is completely consistent with that of the first probability distribution, which is the word at the position following the end of the current prefix word sequence. In other words, whether based on a first context of length K or a second context of length M, the output is a conditional probability estimate of the word at the same position. They rely on different historical lengths, but the predictor variables are the same; therefore, they are semantically of the same dimension. Furthermore, both probability distributions are defined on the same vocabulary V, i.e., covering the same set of words. Each probability distribution is a normalized vector of length |V|, and their output spaces are perfectly aligned. The conditional probability of word vi in ​​the first probability distribution and the conditional probability of word vi in ​​the second probability distribution both represent a quantitative assessment of the probability that word vi will appear as a word in the same position.

[0103] S103, based on the text vector corresponding to the first context, an interpolation weight is obtained, and based on the interpolation weight, an interpolation operation is performed on the first probability distribution and the second probability distribution to obtain a mixed probability distribution. According to the set speculative sampling strategy, the target word at the position is determined based on the mixed probability distribution. The mixed probability distribution includes the mixed probability of each word in the vocabulary as a word at the position.

[0104] The text vector corresponding to the first context is used to represent the semantic and structural information of the entire context. It can be generated by mapping each word element included in the first context to a word vector and concatenating the word vectors corresponding to each word element according to the order of the word elements in the first context. For example, if each word vector is 512-dimensional and the first context has 5 words, then the concatenated vector is 2560-dimensional.

[0105] For example, given the prefix lexical sequence "The quick brown fox jumps", the first context upon which the high-order 5-Gram model predicts the lexical position following "jumps" depends is "quick brown fox jumps". Each lexical in this first context is then mapped to a fixed-dimensional dense vector, i.e., a word embedding. This word vector can be obtained from the vector layer of Qwen2.5-0.5B. Assuming the word vector dimension d=3, then:

[0106] Embed("quick")=[0.4,0.5,0.2],

[0107] Embed("brown")=[0.6,0.3,0.1],

[0108] Embed("fox")=[0.3,0.7,0.4],

[0109] Embed("jumps")=[0.5,0.2,0.6],

[0110] The text vector E = [0.4, 0.5, 0.2, 0.6, 0.3, 0.1, 0.3, 0.7, 0.4, 0.5, 0.2, 0.6].

[0111] The interpolation weight is a scalar value between 0 and 1, used to control the contribution ratio of the probability distributions output by the higher-order N-Gram model and the lower-order N-Gram model to the final mixed probability distribution. When obtaining the interpolation weight based on the text vector corresponding to the first context, the text vector can be input into a pre-trained gating network to obtain the interpolation weight output by the gating network. This gating network can be implemented by a small neural network (such as a fully connected layer with a sigmoid activation function), receiving the text vector as input and internally calculating to output a suitable interpolation weight β based on the features of the current context. For example, the gating network outputs this interpolation weight through a linear transformation and a non-linear activation function. This interpolation weight represents the confidence level of the first probability distribution output by the higher-order N-Gram model based on the first context conditions. A larger β indicates a more reliable first probability distribution, while a smaller β indicates a lower confidence level, meaning a greater reliance on the smoothed probability estimate provided by the lower-order N-Gram model.

[0112] The mixed probability distribution represents the final mixed probability distribution obtained by interpolating the first and second probability distributions. It is a vector containing the probabilities of all lexical terms in the vocabulary. Each lexical probability element represents the mixed probability of the lexical term being the next lexical term after the last lexical term in the prefix lexical sequence. The first and second probability distributions are generated by a high-order N-Gram model and a low-order N-Gram model, respectively. Because the first probability distribution relies on a longer context, it can capture richer local grammatical structures and long-distance semantic dependencies, and has strong expressive power. However, in sparse data scenarios, it is prone to unrecorded sequences, leading to unreliable predictions. The second probability distribution, on the other hand, is based on a shorter context to reflect local word collocations. Because its context combinations are more common, it has stronger generalization ability and resistance to sparsity, and can provide robust backoff estimation when the high-order model fails. Therefore, this embodiment dynamically determines the mixed probability distribution based on the first probability distribution and the second probability distribution by combining the interpolation weights, in order to retain the high expressive power of the high-order model when the context is sufficient, and combine the robustness advantage of the low-order N-Gram model when the context is sparse, thereby improving the prediction accuracy and stability of the language model in different contexts as a whole.

[0113] Speculative sampling strategy refers to a strategy that uses a mixed probability distribution obtained through interpolation and fusion, and selects a word from the vocabulary as the target word at the specified position through a set random or deterministic method. It may include, but is not limited to, any of the following strategies for selecting the target word from the probability distribution: (1) Spinning random sampling: Each word in the mixed probability distribution is regarded as a sector, and the sector size is proportional to its probability. The word that the pointer eventually points to is the selected target word; (2) Top-q sampling: Select the top q words with the highest probability from the mixed probability distribution, renormalize the probability of these q words, and then randomly sample from these q candidate words; (3) Top-p sampling: Sort all words in descending order of probability, and accumulate the probability of each word until the sum first exceeds the preset threshold p (e.g., p). =0.7), the words that enter the cumulative set are used as the candidate set, the probability of words in the candidate set is normalized, and random sampling is performed on them; (4) Elastic search: select the word with the largest probability value in the mixed probability distribution as the target word; (5) Temperature sampling: introduce the temperature parameter T to smooth or sharpen the original probability distribution. When T is greater than 1, the probability distribution is flatter to increase randomness, and when T is less than 1, the probability distribution is sharper to increase the advantage of high probability words.

[0114] In the process of interpolating the first probability distribution and the second probability distribution based on the interpolation weights, in order to achieve dynamic and adaptive fusion of the prediction results of high-order and low-order N-Gram models, and to improve the robustness of the system while ensuring the quality of generation, this embodiment introduces a set threshold to distinguish between the sufficiency of contextual information and the level of model confidence, thereby achieving adaptive fusion of high-order and low-order N-Gram models. See also Figure 1D The illustrative diagram illustrates a process for interpolating the first probability distribution and the second probability distribution based on the interpolation weights. The process dynamically selects or mixes probability distributions output by N-Gram models of different orders based on the interpolation weights to obtain the mixed probability distribution.

[0115] S1031, if the interpolation weight is less than or equal to the first threshold, the second probability distribution is used as the mixed probability distribution;

[0116] Based on the fact that the interpolation weight can reflect the reliability of the higher-order N-Gram model in predicting the next word in the first context under the condition of the first context, the larger the interpolation weight, the more reliable the first probability distribution output by the higher-order N-Gram is. Conversely, the smaller the interpolation weight, the less reliable the first probability distribution output by the higher-order N-Gram is. Therefore, for cases where the interpolation weight is less than or equal to the first threshold, it is considered that the prediction results provided by the higher-order N-Gram model are unreliable or may be overfitted and have no reference value. In this case, it is necessary to regress to the prediction results of the lower-order N-Gram model with better robustness. Therefore, the second probability distribution output by the lower-order N-Gram model can be directly used as the mixed probability distribution.

[0117] S1032, if the interpolation weight is greater than or equal to the second threshold, the first probability distribution is used as the mixed probability distribution; the second threshold is greater than the first threshold;

[0118] Similarly, for cases where the interpolation weights are greater than or equal to the second threshold, it is believed that the higher-order N-Gram model can provide more accurate predictions and is sufficiently reliable. Therefore, the first probability distribution output by the higher-order N-Gram model can be directly used as the mixed probability distribution.

[0119] S1033, when the interpolation weight is greater than the first threshold and less than the second threshold, the first probability distribution and the second probability distribution are linearly weighted and fused to obtain the mixed probability distribution; the weight of the first probability distribution is determined by the interpolation weight; the weight of the second probability distribution is the complement of the interpolation weight.

[0120] When the interpolation weight is greater than the first threshold and less than the second threshold, the prediction results provided by the higher-order N-Gram model are considered to have some reference value but are not entirely reliable. Further adjustments to the higher-order N-Gram model's prediction results are needed, taking into account the generalization advantage of the lower-order N-Gram model. Therefore, this step uses the interpolation weight to perform a weighted fusion of the first probability distribution output by the higher-order N-Gram model and the second probability distribution output by the lower-order N-Gram model to obtain a mixed probability distribution. For example, when the interpolation weight is β, this mixed probability distribution can be expressed as... .

[0121] This embodiment provides a probability acquisition method for prediction results based on context-aware dynamic adaptive fusion of high- and low-order N-Gram models. Compared with the existing method of using a single N-order model for probability distribution calculation, it can not only effectively alleviate the data sparsity problem of high-order N-Gram models, but also make full use of the information of high-order N-Gram models when the context is reliable, thereby improving the accuracy and stability of language modeling as a whole.

[0122] Alternatively, to further accelerate the model inference process and reduce unnecessary computational overhead, before obtaining the first probability distribution, interpolation weights can be obtained based on the text vectors corresponding to the first context on which the higher-order N-Gram model depends. If the interpolation weights are greater than the first threshold, the first probability distribution output by the higher-order N-Gram model based on the vocabulary and the input first context can be obtained. If the interpolation weights are less than or equal to the first threshold, the step of obtaining the first probability distribution can be skipped, thereby avoiding wasting resources in computing a first probability distribution that will eventually be ignored in scenarios with sparse or irrelevant contexts, and reducing resource waste.

[0123] The target lexical at the position is determined based on the mixed probability distribution, which represents a lexical to be verified that is predicted and proposed by the mixed probability distribution at that position and selected from the vocabulary V. For example, after the prefix lexical sequence "The quick brown fox jumps", the target lexical might be "over".

[0124] S104, if it is determined that word verification is required, the target word at the position and the predicted target words at the previous L positions are submitted to the large model; otherwise, the first context and the second context are updated based on the target word at the position and the process returns to the step of obtaining the first probability distribution and the second probability distribution.

[0125] In the current stage where lexical verification has not been triggered, the high-order N-Gram model and the low-order N-Gram model continue to be used to implement conventional autoregressive generation. That is, the first context and the second context are updated using the target lexical at the position obtained in step S103, and the process returns to the step of obtaining the first probability distribution and the second probability distribution to obtain the target lexical at the next position of the updated first context. For example, if the target lexical Wt+1 at position t+1 is output in step S103, then the first context C high Updated to:

[0126] And the second context C low Updated to:

[0127]

[0128] The first probability distribution of the higher-order N-Gram model output is obtained based on the updated first context, and the second probability distribution of the higher-order N-Gram model output is obtained based on the updated second context. The first / second probability distribution includes the conditional probability of each word in the vocabulary as the word Wt+2 at the next position t+2 of the first / second context.

[0129] Determining that lexical verification is currently required can be dynamically determined based on a preset triggering strategy. This triggering strategy includes, but is not limited to, fixed generation cycles, dynamic confidence thresholds, and cumulative generation length reaching a set threshold. Taking the cumulative generation length reaching a set threshold strategy as an example, for instance, if lexical verification is triggered every four consecutive target lexical units generated, then after completing step S103 and obtaining the predicted target lexical unit at the specified position:

[0130] If four target words have been generated consecutively, it is determined that word verification is required. The target word Wt+1 at the specified position, along with the target words Wt-2, Wt-1, and Wt at the three positions preceding it, and the corresponding word sequence before Wt-2, are submitted to the large model for batch verification. Otherwise, the verification process is skipped, and the first context on which the high-order N-Gram model depends and the second context on which the low-order N-Gram model depends are updated using the predicted target words at the specified position. Steps S101-S103 are then executed to continue predicting the target word at the next position of the updated first context.

[0131] The target words predicted at the L positions preceding the current position refer to the L consecutive positions backward from the current prediction position (e.g., Wt+1), generated sequentially through autoregression using steps S101-103 provided in this application. The target words at the current position and the sequence of target words predicted at the L positions preceding the current position constitute a complete speculative generation path, used for submission to a large model for batch validation.

[0132] In the case of triggering lexical verification, the subsequent verification process of the large model is executed, that is, the target lexical at the position output in step S103 and the target lexical predicted at the previous L positions are packaged and submitted to the large model. The preceding context of the candidate sequence can also be input into the large model to perform forward inference once to obtain the true conditional probability of each lexical in the sequence. For example, if the prefix word sequence is "The quick brown fox jumps" and each word in this sequence has been validated by the large model, then if four consecutive target words "over", "the", "lazy", and "dog" are generated to trigger word validation, then when the target word "dog" is obtained in step S103, the candidate sequence "over the lazy dog" formed by the three target words "over", "the", and "lazy" before the target word "dog" and the target word "dog", as well as the prefix word sequence "The quick brown fox jumps" that has been validated before the first target word "over" generated in this round, are submitted to the large model for validation together, and the validation probability P of each target word given by the large model is obtained. LLM .

[0133] For the L+1 target words submitted for large model validation (1≤i≤L+1), compare the validation probabilities P of the target words output by the larger model. LLM The conditional probability P of the target word in the aforementioned mixed probability distribution current For example, setting a tolerance coefficient. ∈(0,1):

[0134] The verification pass conditions are: This indicates that the large model also considers the word to be a reasonable choice, accepts the generated result, and can directly output and advance the decoding pointer;

[0135] The verification failure (rejection) conditions are as follows: This indicates that the large model is skeptical of the lexical and rejects the candidate, requiring a regression and the large model to generate a more reliable lexical.

[0136] After the large model validation is completed, regardless of whether the validation passes or fails, each target word, along with a sample sequence of length K consisting of the K-1 consecutive words preceding it (K+1 represents the order of the aforementioned higher-order N-Gram model), and the validation probability output by the large model for that target word, will be used as training samples to train the gating network used to generate the aforementioned interpolation weights. If the large model accepts the target word after validation, it indicates that the prediction result of the higher-order N-Gram is reliable, and the gating network is incrementally trained online to increase the interpolation weight β, thereby enhancing the contribution of the higher-order model. Conversely, if the large model rejects the target word after validation, it is necessary to appropriately reduce the contribution of the first probability distribution output by the higher-order N-Gram model to the mixed probability distribution. The online incremental update gating network does not directly approximate the distribution of the large model with the high-order N-Gram model. Instead, it dynamically adjusts the mixing degree of the probability distributions of the outputs of the high-order and low-order N-Gram models by learning an interpolation weight. This makes the mixed probability distribution obtained from the interpolation operation as close as possible to the true distribution of the large model. The optimization goal of training is to reduce probability bias. That is, when the large model confirms that a prediction by the high-order model is reliable, the gating network should automatically increase the interpolation weight to make the mixed distribution more dependent on the high-order model, thus preserving its semantic richness. All samples generated during the validation process can be used to fine-tune the gating network online, enabling it to continuously adapt to the text generation needs of new domains and styles, and possessing good generalization ability.

[0137] In this embodiment, a lightweight, adaptive fusion two-order N-Gram draft model framework is constructed. The higher-order N-Gram model captures long-sequence dependency patterns in local contexts, while the lower-order N-Gram model provides robust probabilistic priors to alleviate data sparsity. Based on this, interpolation weights are dynamically determined based on contextual semantic strength awareness. Compared to fixed weights, this adaptive interpolation weight achieves semantic-driven adaptive fusion, allowing the model to automatically switch the dominant model in different contexts. Combining the values ​​of the interpolation weights, a mixed probability distribution is dynamically determined based on the first and second probability distributions. This avoids prediction bias caused by improper model order selection in related technologies that directly select a single model, improving the accuracy and stability of the mixed probability distribution. Overall, this enhances the prediction accuracy and stability of the language model in different contexts, generating high-quality candidate word sequences that balance expressiveness and robustness. These sequences are then applied to a speculative sampling process for batch verification by a large model, improving the inference speed and verification pass rate of the large model.

[0138] This embodiment achieves intelligent enhancement of the candidate lemma generation process without introducing additional neural network components. Compared with the existing fixed use of a single N-Gram model as a draft model, which often leads to candidates deviating from the true distribution due to zero probability or low-quality priors, causing frequent rejection by large models, this embodiment constructs interpolation weights through contextual semantic awareness. It then uses the interpolation weights to adaptively and dynamically fuse high-order and low-order N-Gram models to construct a probability distribution, thereby improving the acceptability of candidate sequences, i.e., increasing the acceptance rate of speculative sampling. While maintaining extremely low computational overhead and storage requirements, it improves the semantic rationality and verification pass rate of candidate sequences, thereby reducing the number of forward propagations in large models, effectively overcoming the memory access bottleneck in autoregressive decoding, and achieving the goal of accelerating inference by synergistically optimizing accuracy and efficiency.

[0139] Based on the above embodiments, since traditional autoregressive models lack explicit memory and feedback control of their own generation path during word-by-word generation, they are prone to semantic breaks, repetitions, or divergences due to local contextual mutations or attention drift. To enhance the local consistency and contextual coherence in the language model generation process, this embodiment further proposes a method to dynamically correct the mixed probability distribution based on the semantic state of historical words. By caching the historical hidden states of each word in the recent generation path in the prefix word sequence and combining the current hidden state corresponding to each word in the vocabulary at the current prediction position, the mixed probability of each word in the vocabulary as a word at the current prediction position in the mixed probability distribution is adaptively corrected, thereby improving semantic fluency.

[0140] See Figure 2A The illustrative diagram illustrates how to obtain the historical hidden state of historical lexical units and correct the mixed probability distribution. After obtaining the mixed probability distribution, it can be corrected through the following steps:

[0141] S201, obtain the current hidden state corresponding to each word in the vocabulary when it is used as a word at the position; the current hidden state represents the semantic state of the new sequence formed by adding the word to the prefix word sequence;

[0142] The current hidden state represents the vector representation output by the neural network after processing an input sequence, encoding the overall semantic information of that input sequence. In this step, for each word in the vocabulary, the word is concatenated with the prefix word sequence of the first context described above, that is, the word is used as the word at the next position of the prefix word sequence to form a new sequence. The new sequence is then vectorized and represented as the current hidden state corresponding to that word.

[0143] Assume the prefix lexical sequence is The current predicted position is Wt+1, and the vocabulary is as follows: ,but:

[0144] For the i-th lexical unit vi in ​​the vocabulary, when vi is used as a lexical unit at position Wt+1, the current hidden state represents the new sequence. The overall semantic state.

[0145] This embodiment can employ another model with the same or similar structure as the large model used for validating lexical units, but with fewer parameters, to extract the current hidden state and the historical hidden state described later. For example, a lightweight decoder-only model can be used as an auxiliary model to extract the current and historical hidden states. Only a small number of hidden states need to be cached during generation and forward inference performed through a forward inference layer, without introducing backpropagation or additional training overhead. In this step, the new sequence formed by adding the prefix lexical unit to the given lexical unit can be used as the model input to obtain the current hidden state output by the model with fewer parameters.

[0146] S202, for h consecutive historical words adjacent to the word at the specified position in the prefix word sequence, obtain the historical hidden state of each historical word; wherein, the historical hidden state is used to reflect the semantic state of the context in the prefix word sequence ending with the historical word; the h is determined according to the total length of the prefix word sequence;

[0147] The h consecutive historical lexical units adjacent to the lexical unit at the stated position in the prefix lexical unit sequence represent the last h words in the generated prefix lexical unit sequence (i.e., the h words closest to the currently predicted position). For example, the prefix lexical unit sequence is... The current predicted position is Wt+1, and the h historical lexical units are the lexical units from Wt-h+1 to Wt.

[0148] The historical hidden state of each historical lexical unit reflects the overall semantic state of the local sequence from the starting lexical unit in the prefix lexical unit sequence to that historical lexical unit. It encodes the contextual semantics from the starting lexical unit in the prefix lexical unit sequence to that historical lexical unit, and can be used to construct long-distance semantic consistency compensation signals. For example, the historical hidden state of historical lexical unit Wt-h+1 represents the contextual sequence. The overall semantic state, the historical latent state of the historical lexical Wt represents the context sequence. The overall semantic state.

[0149] Regarding the acquisition of the historical hidden state, this embodiment uses a lightweight Decoder-only language model (such as Qwen2.5-0.5B) as an auxiliary model. The model structure includes a standard self-attention layer and an output projection layer, namely the Im_head layer, which can be understood as the output head of the language model, used to output a fixed-dimensional vector representation as the historical hidden state.

[0150] like Figure 2B As shown, assuming the prefix lexical sequence is ["the", "quick", "brown", "fox", "jumps"], the mixed probability distribution of each lexical in the vocabulary as a continuation lexical for "jumps" has been obtained. Let h=3, that is, select the three most recent historical lexicals: "brown", "fox", "jumps". The historical hidden state corresponding to this historical lexical can be obtained in the following way: Input the entire prefix lexical sequence into the auxiliary model, and calculate the hidden state of each position in turn. For each historical lexical such as "brown" in the window, the model will obtain the hidden state corresponding to the historical lexical based on the context sequence [the quick brown] from the starting lexical in the existing word order sequence. This hidden state represents the semantic state used for the context sequence [the quick brown]. By mapping the hidden state corresponding to this historical lexical through the output projection layer, a fixed-dimensional vector representation is obtained as the historical hidden state of this historical lexical.

[0151] In this embodiment, to ensure both context coverage and computational efficiency, the number of historical lexical units h is selected using an adaptive strategy, and its value is dynamically determined based on the total length of the prefix lexical sequence. When the total length of the prefix lexical sequence is D, the number of historical lexical units h can be calculated as follows:

[0152]

[0153] Here, ⌊⌋ represents the floor function, which makes the length of the historical lexical unit grow logarithmically with the sequence length. This allows for rapid response to local changes when the sequence is short, and stable capture of generation trends when the sequence is long, avoiding computational redundancy and increased latency caused by an excessively large window. For example, when the length of the prefix lexical unit sequence L=100, the calculated... This method selects the seven most recent consecutive historical lexical units as the analysis objects within a dynamic window. This approach can maintain a balance between sensitivity to local context and perception of global trends during the generation process of different lengths, thereby improving the robustness and adaptability of the correction process.

[0154] Alternatively, to further control the length h of the historical window to avoid the number of historical lexical units being too large or too small, a maximum number of historical lexical units Lmax and a minimum number of historical lexical units Lmin can be introduced. The number h of historical lexical units can be expressed as: This allows for adaptive control of the number of historical lexical units, preventing them from being too large, which would lead to a surge in computational overhead, or too small, which would result in insufficient contextual information.

[0155] S203, based on the historical hidden states of the h historical lexical units and the current hidden state corresponding to each lexical unit in the vocabulary when it is a lexical unit at the position, the mixed probability corresponding to each lexical unit in the mixed probability distribution when it is a lexical unit at the position is corrected to obtain the corrected mixed probability distribution.

[0156] This step utilizes the historical hidden states of the h most recent historical words before the current prediction position, as well as the current hidden state of each word in the vocabulary at the current prediction position. Based on the contextual semantic relevance between the current hidden state of each word and the historical hidden states of the h historical words, the mixture probability of each word in the vocabulary at the current prediction position in the mixture probability distribution is readjusted. The aim is to enhance context awareness, increase the mixture probability of words that are more consistent with the contextual semantics, and thus obtain a more accurate mixture probability distribution with stronger semantic consistency with the context. This effectively alleviates the problem of decreased model attention to early key contextual information due to excessively long prediction sequences (i.e., long-distance dependency decay problem), reduces the occurrence of semantically incoherent, repetitive, or off-topic predicted words due to contextual forgetting or attention distraction, thereby increasing the probability that predicted words are accepted by the large model.

[0157] The contextual semantic relevance between the current hidden state corresponding to each word element and the historical hidden states of h historical word elements can be determined by the similarity between the current hidden state and the historical hidden states of h historical word elements.

[0158] See also Figure 2C An exemplary flowchart illustrates an adjustment of the mixed probability distribution based on attention weights and decay factors. When correcting the mixed probability corresponding to each word in the mixed probability distribution as a word at the specified position, the following steps can be taken:

[0159] S2031, for each word element in the vocabulary, based on the similarity between the current hidden state corresponding to the word element when it is a word element at the position and each of the historical hidden states, determine the attention weight of the current hidden state to each of the historical hidden states.

[0160] Similarity is used to represent the degree of matching between the current hidden state and each historical hidden state in semantics or feature space, and can be obtained through methods such as cosine similarity, scaled dot product, and dot product similarity. For example, this embodiment uses cosine similarity to determine the similarity. Assuming that the current hidden state corresponding to the i-th word vi in ​​the aforementioned vocabulary is the word at position Wt+1, it is represented as follows. h historical word elements to The corresponding historical hidden states are [ , ..., ], then the current hidden state The historical latent state of the j-th historical lexicon (1≤j≤h) among h historical lexicons The similarity between them can be expressed using cosine similarity as:

[0161]

[0162] The attention weight of the current hidden state to each of the historical hidden states reflects the degree of attention the model pays to each historical context when generating the current lexical. Therefore, to ensure that the attention allocation of all historical positions constitutes an effective probability distribution, the similarity needs to be normalized to obtain the attention weight. Thus, for any lexical in the vocabulary, after obtaining the similarity between the current hidden state corresponding to the lexical at the given position and each of the historical hidden states, the attention weight of the current hidden state to each of the historical hidden states can be determined through normalization. Based on this, assume the current hidden state corresponding to the i-th lexical vi in ​​the vocabulary... The historical hidden state of the j-th historical word The attention weights are represented as The attention weight calculation process can then be expressed as:

[0163]

[0164] in, This represents the current hidden state corresponding to the i-th lexical unit vi in ​​the vocabulary. The historical latent state of the j-th historical term Similarity; This is a temperature parameter used to control the sharpness of attention distribution, for example, the... It can take the value 0.1.

[0165] S2032, Based on the distance between the historical word and the word at the position, determine the decay factor corresponding to the historical hidden state of the historical word; the distance is negatively correlated with the decay factor;

[0166] This attenuation factor is used to control the strength of the influence of historical trend information. The farther the historical word is from the current prediction position, the smaller the influence of the attention weight of the historical word on the current prediction position, thus avoiding the suppression of novelty expression due to excessive accumulation of historical information.

[0167] The attenuation factor can be set to... Where λ is an adjustable hyperparameter, acting as a decay coefficient to control the rate at which the influence of historical hidden states on the current prediction decays with distance. Its specific value can be determined by combining the model design objectives and experimental verification. r represents the distance between the historical word and the word at the given position, with a value of [1, h], where h is the number of historical words. When r is large, it indicates that the distance between the historical word and the word at the given position is far. In this case, the dependence on the historical hidden state of the historical word should be reduced to prevent excessive smoothing from leading to a decrease in generation diversity.

[0168] S2033, based on the attention weight of the current hidden state corresponding to the word when it is the word at the position, and combined with the decay factor corresponding to each historical hidden state, the mixed probability corresponding to the word when it is the word at the position in the mixed probability distribution is corrected.

[0169] After obtaining the attention weight of the current hidden state corresponding to the word as a word at the stated position for each of the historical hidden states, the attention weight corresponding to each historical hidden state can be modified using the decay factor corresponding to that historical hidden state to obtain the probability modification magnitude value corresponding to the word as a word at the stated position. For example, the current hidden state corresponding to the i-th word vi in ​​the vocabulary. The historical hidden state of the j-th historical word The attention weights are represented as The decay factor corresponding to the j-th historical word is Then the current hidden state corresponding to the i-th word element vi The historical hidden state of the j-th historical word The corrected attention weights can be expressed as:

[0170]

[0171] Based on this, the attention weights of each historical hidden state can be fused by the current hidden state corresponding to the word when it is the word at the position, for example by direct summation or weighted calculation, to obtain the probability correction magnitude value corresponding to the word when it is the word at the position, and the mixed probability corresponding to the word when it is the word at the position in the mixed probability distribution can be corrected using the probability correction magnitude value, to obtain the corrected mixed probability distribution.

[0172] For example, the mixing probability corresponding to the i-th lexical unit vi in ​​the vocabulary as the lexical unit at the stated position is: Then the corrected mixture probability can be expressed as:

[0173]

[0174] Where μ is a preset hyperparameter, representing the correction intensity coefficient, which can be set according to actual needs, for example, μ=0.1; The sum of the attention weights of the current hidden state corresponding to the word as a word at the position, after correction for each of the historical hidden states, is used as the probability correction magnitude value corresponding to the word as a word at the position.

[0175] The positional encoding decay mechanism of attention weights formed by the above method makes the influence of the historical hidden state of historical words farther away from the current prediction position decay exponentially on the current prediction, thereby effectively suppressing the cumulative effect of long-distance noise while preserving long-range dependency capabilities.

[0176] For example, the current prediction may deviate from the original topic due to sudden changes in local context. The historical latent state of historical lexical units can provide historical inertia signals to prevent generation abrupt changes. For instance, when describing an animal jumping, even if the current context is ambiguous, the historical latent state of historical lexical units tends to retain action-related words such as "over" and "high." In this case, the probability of action-related words in the corrected mixed probability distribution may be moderately increased to generate a more coherent result that fits the current context.

[0177] In this embodiment, by introducing semantic state information based on historical context and using it to weighted correct the current mixed probability distribution, an external feedback adjustment mechanism that does not require modification of the main model structure is constructed. By dynamically fusing historical context information with the current candidate word representation using an attention mechanism, the mixed probability of each word in the vocabulary as a word at the current prediction position is adjusted. An interpretable and controllable external long-term memory cache is established outside the main model to capture and maintain semantic coherence. Historically relevant information is injected into the mixed probability distribution, which effectively suppresses generation problems such as abnormal jumps and semantic deviations, and enhances the semantic coherence, naturalness and readability of the predicted sequence.

[0178] In some embodiments, to continuously improve the gating network's dynamic adjustment capability for the fusion weights of high- and low-order N-Gram models, this embodiment further performs online incremental training on the gating network to achieve fine-tuning. Based on this, see [link to relevant documentation]. Figure 3A The exemplary diagram illustrates a training sample construction process for online incremental training of a gating network. After submitting the target word at the specified position and the predicted target words at the L positions preceding the specified position to the large model, the method further includes:

[0179] S301, for the p-th target word submitted to the large model, obtain the verification probability output by the large model when predicting the p-th target word under historical context conditions; wherein, the historical context includes: the verified prefix word sequence to which the first context on which the high-order N-Gram model depends when generating the first target word among the L+1 target words, and the first p-1 target words; p is an integer satisfying 1≤p≤L+1;

[0180] The historical context consists of two parts: ① The original validated prefix word sequence preceding the L+1 target words within the current validation window. This can also be understood as the original validated context sequence within the prefix word sequence, i.e., the word sequence validated by the large model before this round of word validation. For example, before the validation of the first predicted word among the L+1 target words, the prefix word sequence is "The quick brown fox jumps". For example, if L+1=4, then four consecutive target words "over", "the", "lazy", and "dog" are generated to trigger word validation. When submitting the target word "dog" and the three target words "over", "the", and "lazy" preceding it to the large model for validation, if "over" is the first target word among the L+1 target words, then the prefix word sequence of the first context relied upon by the higher-order N-Gram model is "The quick brown fox jumps". ② The first p-1 target words, that is, the p-1 target words generated before the p-th target word. For example, for the p-th target word, such as "lazy", the first p-1 target words are "over" and "the". Based on this, for the p-th target word, such as "lazy", its corresponding historical context is "The quick brown fox jumps" and the historical context "The quick brown fox jumps over the" composed of the two target words "over" and "the" before "lazy".

[0181] In this embodiment, when lexical verification is triggered, the prefix lexical sequence of the first context relied upon by the high-order N-Gram model when generating the first target lexical among the L+1 target lexicals, along with the L+1 target lexicals to be verified, are submitted to the large model for parallel verification. The large model outputs the verification probability of the L+1 target lexicals through one forward inference. For example, for the p-th target lexical... Its verification probability is defined as:

[0182]

[0183] Here, the first context refers to the prediction made by the higher-order N-Gram model. At that time The previously validated prefix lexical sequence, i.e., the first predicted target lexical in the L+1 target lexical sequences predicted by the higher-order N-Gram, should be... The prefix word order of the first context to which the time depends.

[0184] For example, taking the prefix word sequence "The quick brown fox jumps", and L+1=4, then generating four consecutive target words "over", "the", "lazy", and "dog" to trigger word verification is an example. The prefix word sequence is concatenated with the four consecutively generated target words to obtain the input sequence "The quick brown fox jumps over the lazydog", which is then input into the large model. The decoder of the large model calculates the hidden state for each word position in the input sequence and obtains the probability distribution of each word in the vocabulary at that word position through the output layer. The probability distribution of the target word is then extracted as the verification probability of that target word. For example:

[0185] P LLM ("over"∣"The quick brown fox jumps")=0.41

[0186] P LLM ("the"∣"... jumps over")=0.38

[0187] P LLM ("lazy"∣"... over the")=0.35

[0188] P LLM ("dog"∣"... the lazy")=0.42

[0189] S302, in the complete sequence consisting of the verified prefix word sequence and the L+1 target words predicted thereafter, obtain a local sequence with length K ending at the p-th target word;

[0190] This implementation step is used to construct a context fragment with semantic locality, which is used as the input feature of the gating network training sample. The local sequence corresponding to the p-th target word is a sequence composed of K words obtained sequentially from the p-th target word in the direction of the sequence start word in the complete sequence, where K is the number of words in the first context on which the high-order N-Gram model used to calculate the first probability distribution in this embodiment depends.

[0191] Taking the aforementioned prefix lexical sequence "The quick brown fox jumps" as an example, and generating four target lexicals "over", "the", "lazy", and "dog" to trigger lexical verification, the complete sequence is "The quick brown fox jumps over the lazy dog". Assuming a higher-order 5-Gram model is used, the length of the first context it depends on is K=4.

[0192] For the first target word "over", the local sequence is ["brown","fox", "jumps", "over"];

[0193] For the second target word "the", the local sequence is ["fox", "jumps", "over","the"];

[0194] For the third target word "lazy", the local sequence is ["jumps", "over", "the", "lazy"];

[0195] For the third target word "dog", the local sequence is ["over","the","lazy","dog"].

[0196] S303, depending on whether the large model accepts the p-th target word as the next word in the historical context, the sample interpolation weights corresponding to the local sequence are constructed by combining the verification probability with the probability of the p-th target word in the second probability distribution through a preset normalization method.

[0197] The validation rules are used to compare the validation probability of the target word output by the large model with the conditional probability of that target word in the mixed probability distribution. If the validation condition is met, the large model accepts the p-th target word as the next word in the historical context. In this case, the prediction result of the higher-order N-Gram model is considered reliable, and the weight interpolation should be increased to enhance the contribution of the probability distribution predicted by the higher-order N-Gram model. If the validation condition is not met, the large model rejects the p-th target word as the next word in the historical context. In this case, the prediction result of the higher-order N-Gram model is considered to deviate from the distribution of the large model, and the weight interpolation should be appropriately reduced to decrease the contribution of the probability distribution predicted by the higher-order N-Gram model. The sample interpolation weights corresponding to the obtained local sequence will be used as the sample labels of the local sequence, representing the fusion ratio of the probability distributions output by the ideal high- and low-order N-Gram models in that local context.

[0198] Based on this, see Figure 3B An exemplary diagram illustrating the labeling process for training samples in a gating network is shown below:

[0199] S3031, when the large model accepts the p-th target word as the next word in the historical context, the verification probability is normalized based on the verification probability and the probability of the p-th target word in the second probability distribution to obtain the sample interpolation weight.

[0200] The sample interpolation weights are used to reflect the dominance of the validation probability of the large model relative to the probability distribution of the output of the low-order N-Gram model. Assuming the validation probability of the p-th target word is expressed as... The conditional probability of the target word in the second probability distribution output by the low-order N-Gram model is expressed as: Then the sample interpolation weights can be expressed as:

[0201]

[0202] Among them, if Then let

[0203] S3032, when the large model refuses to use the p-th target word as the next word in the historical context, the normalized output of the verification probability is subjected to a preset penalty factor of less than 1 for attenuation processing to obtain the sample interpolation weight.

[0204] The penalty factor is used to proactively reduce the target weight when validation fails. For example, the default penalty factor is 0.5. Let's assume the penalty factor is expressed as... The sample interpolation weights obtained by attenuation processing can be expressed as:

[0205]

[0206] S304, the local sequence is used as the input feature, and the sample interpolation weights corresponding to the local sequence are used as the supervision signal to form a training sample pair; the training sample pair is used to update the parameters of the gating network online.

[0207] Each training sample pair includes [local sequence, sample interpolation weights]. By inputting this training sample pair into the training module of the gating network, the gating network parameters are updated online using optimization algorithms such as gradient descent, so that the output interpolation weights gradually approach the real sample interpolation weights, thereby improving its ability to dynamically adjust the fusion ratio of high- and low-order N-Gram models in different contexts.

[0208] This embodiment utilizes real-world interaction data generated during each candidate generation and large-scale model validation process to construct training sample pairs with local sequences as input features and sample interpolation weights as supervision signals. These pairs are then used to update the gating network parameters online, enabling continuous optimization of the high- and low-order language model fusion strategy. This allows the gating network to learn the reliability patterns of the high-order N-Gram model under different contextual modes. For example, in fixed collocations (such as "quick brown fox") or high-frequency phrase scenarios, the large-scale model validation probability is significantly higher than the low-order N-Gram model prediction probability, causing the sample interpolation weights to approach 1. This prompts the gating network to automatically increase the contribution ratio of the high-order model. Conversely, in sparse, low-frequency, or complex semantic contexts, the low-order N-Gram model has a larger prediction bias, and the large-scale model output probability dominates. The sample interpolation weights decrease, and the gating network correspondingly reduces the influence of the high-order model, avoiding generation bias caused by the zero probability problem of the high-order model.

[0209] Furthermore, since the training samples are generated in real time during inference and used for parameter updates, the gating network possesses online adaptive capabilities, dynamically adapting to changes in the distribution of input content (such as domain switching, style transfer, etc.), and can achieve continuous optimization without additional labeled data. Compared to traditional static interpolation methods, this embodiment improves the alignment between the mixed probability distribution and the true output of the large model, effectively reducing probability shift phenomena.

[0210] In some embodiments, such as Figure 1C The first probability distribution of the high-order N-Gram model output is obtained under the hierarchical probabilistic storage structure shown. When performing a hierarchical query on the N-Gram sequence corresponding to each word in the vocabulary, the query can be performed according to the priority order of the hierarchical structure, that is:

[0211] The system checks whether the N-Gram sequence corresponding to the given word exists in the high-frequency layer. If it does, the probability value of the N-Gram sequence stored in the high-frequency layer is used as the probability that the word is predicted by the high-order N-Gram model to be the word at the given position. If not, the system checks whether the N-Gram sequence exists in the mid-frequency layer. If it does, the probability value of the N-Gram sequence stored in the mid-frequency layer is used as the probability that the word is predicted by the high-order N-Gram model to be the word at the given position. If not, the system calls a preset low-order language model in the fallback layer and calculates the fallback probability of the word based on the frequency of each word in the vocabulary in the training corpus. This fallback probability is then used as the probability that the word is predicted by the high-order N-Gram model to be the word at the given position.

[0212] Specifically, when querying whether the N-Gram sequence corresponding to the term exists in a specified layer, which is either the high-frequency layer or the mid-frequency layer, and using a Bloom filter that supports counting as the existence discrimination structure, it can be implemented in the following way:

[0213] Based on multiple independent hash functions preset in the specified layer, the N-Gram sequence corresponding to the word is input into each hash function one by one for mapping calculation to obtain a set of hash positions; the set of hash positions jointly identify the existence characteristics of the N-Gram sequence corresponding to the word in the specified layer; the counter value corresponding to each of the set of hash positions is queried, and if the counter value of each hash position is greater than or equal to 1, it is determined that the N-Gram sequence corresponding to the word exists in the specified layer.

[0214] Based on the objective of this implementation step, which is to quickly determine whether a certain high-order N-Gram sequence exists in the high-frequency or mid-frequency layer, in order to save memory and improve query efficiency, this embodiment uses a Bloom filter and a counter array as the existence judgment structure of the specified layer, thereby achieving high-speed query capability at a small space cost.

[0215] A Bloom filter and a counter array are used as the existence determination structure for a specified layer. The Bloom filter maintains a fixed-length array at its core, which can be a binary bit array (for standard Bloom filters) or an integer counter array (for counting Bloom filters). Each element is called a "slot," and its array index serves as the target position for hash mapping. When an N-Gram sequence is inserted into the Bloom filter of a specified layer, the system performs a hash mapping on the N-Gram sequence based on z independent hash functions preset for that layer, generating z hash values. Each hash value corresponds to an index position (i.e., a hash position) in the array. Subsequently, these z hash positions are traversed, and the storage unit at each position is incremented: in the bit array, the corresponding bit is set from 0 to 1; in the counter array, the current value of the corresponding counter is incremented by 1. For example, if the Bloom filter has 1000 storage slots, numbered from 0 to 999, then each hash position is a valid index in the array.

[0216] For this specified layer, multiple independent hash functions are pre-defined, meaning there is no correlation between the hash functions to prevent collision clustering. When a high-order N-Gram sequence, such as a 5-Gram sequence, is expected to be stored in the specified layer, such as a high-frequency layer, z hash functions h1, h2, ..., hz are used to calculate the hash value of the 5-Gram sequence respectively:

[0217] h1 (5-Gram sequence) = 17;

[0218] h2 (5-Gram sequence) = 42;

[0219] ...

[0220] hz (5-Gram sequence) = 89;

[0221] The z hash values ​​(e.g., 17, 42, 89) obtained after being calculated by z hash functions indicate the 17th, 42nd, and 89th slots in the array, respectively. The counter values ​​of the z hash positions are then updated to indicate that the N-Gram sequence has been registered in this layer and its existence characteristics have been jointly marked by the z hash positions.

[0222] Based on this, when querying whether the N-Gram sequence corresponding to the term exists in a specified layer, if the counter value corresponding to the hash position of the N-Gram sequence corresponding to the term under each hash function is greater than or equal to 1, it means that the N-Gram may exist in the specified layer. Otherwise, if any hash position is 0, it means that at least one hash is missed, and the N-Gram sequence corresponding to the term definitely does not exist in the specified layer.

[0223] For example, to query whether "quick brown fox jumps over" exists in the high-frequency layer, the three independent hash functions set in the high-frequency layer are used for calculation:

[0224] h1("quick brown fox jumps over") = 24

[0225] h2("quick brown fox jumps over") = 75

[0226] h3("quick brown fox jumps over") = 99

[0227] A set of hash positions [24,75,99] is obtained. Further checking the counter values ​​of hash positions 24, 75, and 99 in the Bloom filter, counter

[24] = 2, counter

[75] = 1, and counter

[99] = 3, satisfies that the counter values ​​at all hash positions are greater than or equal to 1. Therefore, it is believed that "quick brown fox jumps over" may exist in the high-frequency layer, and the result returned by the Bloom filter indicates that it exists.

[0228] This embodiment achieves efficient and low-overhead querying of N-Gram sequences by deploying an existence determination mechanism based on counting Bloom filters in the high-frequency and mid-frequency layers. Multiple independent hash functions are used to map and calculate the target N-Gram sequence, generating a unique set of hash positions as its existence fingerprint. By simply checking whether the counter values ​​at these positions are all greater than zero, it is quickly possible to determine whether the sequence has been recorded in the specified layer. Compared to traditional hash table lookups, this embodiment reduces memory usage and access latency.

[0229] Corresponding to the aforementioned embodiments of the large model inference acceleration method based on speculative sampling, see [link to relevant documentation]. Figure 4 As shown, this application also provides an embodiment of a large model inference acceleration device based on speculative sampling, the device comprising:

[0230] The high-order prediction result acquisition module 401 is configured to obtain a first probability distribution output by the high-order N-Gram model based on the vocabulary and the first context of the input. The first probability distribution includes the probability that each word in the vocabulary is the next word in the first context. The first context includes K words, where K+1 is the order of the high-order N-Gram model.

[0231] The low-order prediction result acquisition module 402 is configured to obtain a second probability distribution output by the low-order N-Gram model based on the vocabulary and the input second context. The second probability distribution includes the probability that each word in the vocabulary is the next word in the second context. The second context includes M words, where M+1 is the order of the low-order N-Gram model, and M is less than K. The next position of the first context and the next position of the second context are the same position.

[0232] The interpolation module 403 is configured to obtain interpolation weights based on the text vector corresponding to the first context, and to perform interpolation operations on the first probability distribution and the second probability distribution based on the interpolation weights to obtain a mixed probability distribution, and to determine the target word at the position based on the mixed probability distribution according to a set speculative sampling strategy; the mixed probability distribution includes the mixed probability of each word in the vocabulary as a word at the position;

[0233] The speculative sampling and verification module 404 is configured to, if it is determined that lexical verification is required, submit the target lexical at the current position and the predicted target lexical at the previous L positions to the large model; otherwise, update the first context and the second context based on the target lexical at the current position and return to the step of obtaining the first probability distribution and the second probability distribution.

[0234] In some embodiments, when the interpolation module is configured to obtain interpolation weights based on the text vector corresponding to the first context, it includes: mapping each word element included in the first context to a word vector, and sequentially concatenating the word vectors corresponding to each word element to generate the text vector; inputting the text vector into a pre-trained gating network to obtain the interpolation weights output by the gating network.

[0235] In some embodiments, after submitting the target word at the said position and the predicted target words at the preceding L positions to the large model, the apparatus further includes:

[0236] The verification probability acquisition module is configured to acquire the verification probability output by the large model when predicting the p-th target word under historical context conditions for the p-th target word submitted to the large model; wherein, the historical context includes: the verified prefix word sequence to which the first context on which the high-order N-Gram model depends when generating the first target word among the L+1 target words, and the first p-1 target words; p is an integer satisfying 1≤p≤L+1;

[0237] The local sequence acquisition module is configured to acquire a local sequence of length K ending with the p-th target word from the complete sequence consisting of the verified prefix word sequence and the L+1 consecutively predicted target words thereafter;

[0238] The sample labeling module is configured to construct the sample interpolation weights corresponding to the local sequence by combining the verification probability and the probability of the p-th target word in the second probability distribution, based on whether the large model accepts the p-th target word as the next word in the historical context, and through a preset normalization method.

[0239] The training sample construction module is configured to use the local sequence as input features and the sample interpolation weights corresponding to the local sequence as supervision signals to form training sample pairs; the training sample pairs are used to update the parameters of the gating network online.

[0240] In some embodiments, when the sample labeling module is configured to construct the sample interpolation weights corresponding to the local sequence through a preset normalization method, it includes:

[0241] When the large model accepts the p-th target word as the next word in the historical context, the verification probability is normalized based on the verification probability and the probability of the p-th target word in the second probability distribution to obtain the sample interpolation weight; when the large model rejects the p-th target word as the next word in the historical context, a preset penalty factor less than 1 is introduced into the normalized output of the verification probability for attenuation processing to obtain the sample interpolation weight.

[0242] In some embodiments, when the interpolation module is configured to perform interpolation operations on the first probability distribution and the second probability distribution based on the interpolation weights to obtain a mixed probability distribution, the following steps are included:

[0243] When the interpolation weight is greater than a first threshold and less than a second threshold, the first probability distribution and the second probability distribution are linearly weighted and fused to obtain the mixed probability distribution; wherein, the second threshold is greater than the first threshold; the weight of the first probability distribution is determined by the interpolation weight; the weight of the second probability distribution is the complement of the interpolation weight; when the interpolation weight is greater than or equal to the second threshold, the first probability distribution is used as the mixed probability distribution; when the interpolation weight is less than or equal to the first threshold, the second probability distribution is used as the mixed probability distribution.

[0244] In some embodiments, the first context is obtained from a prefix lexical sequence, and the next position of the first context is the same as the next position of the last lexical in the prefix lexical sequence; after obtaining the mixed probability distribution, the apparatus further includes:

[0245] The current hidden state acquisition module is configured to acquire the current hidden state corresponding to each word in the vocabulary when it is used as a word at the specified position; the current hidden state represents the semantic state of the new sequence formed by adding the prefix word sequence to the word.

[0246] The historical hidden state acquisition module is configured to acquire the historical hidden state of each historical word in the prefix word sequence for h consecutive historical words adjacent to the word at the position; wherein, the historical hidden state is used to reflect the semantic state of the context in the prefix word sequence ending with the historical word; the h is determined according to the total length of the prefix word sequence;

[0247] The correction module is configured to correct the mixed probability corresponding to each word in the mixed probability distribution when it is a word at the position, based on the historical hidden states of the h historical words and the current hidden state corresponding to each word in the vocabulary when it is a word at the position, so as to obtain the corrected mixed probability distribution.

[0248] In some embodiments, the correction module is configured to correct the mixed probability corresponding to each word in the mixed probability distribution when it is a word at the position, including:

[0249] For each word in the vocabulary, the attention weight of the current hidden state to each historical hidden state is determined based on the similarity between the current hidden state corresponding to the word when it is the word at the position and each historical hidden state.

[0250] Based on the distance between the historical lexical and the lexical at the position, the decay factor corresponding to the historical hidden state of the historical lexical is determined; the distance is negatively correlated with the decay factor.

[0251] Based on the attention weight of the current hidden state corresponding to the word when it is a word at the position, and combined with the decay factor corresponding to each historical hidden state, the mixed probability corresponding to the word when it is a word at the position in the mixed probability distribution is corrected.

[0252] In some embodiments, the higher-order prediction result acquisition module is specifically used for:

[0253] In a hierarchical probabilistic storage structure consisting of a high-frequency layer, a mid-frequency layer, and a fallback layer, the N-Gram sequence corresponding to each word in the vocabulary is queried layer by layer to obtain the probability that the word is the word at the specified position. The N-Gram sequence corresponding to the word includes the first context and the word itself. The high-frequency layer storage structure is used to cache the top x% of the most frequently occurring N-Gram sequences. The mid-frequency layer storage structure is used to cache N-Gram sequences with frequencies between x% and y%. The fallback layer includes a preset low-order language model configured to generate the fallback probability of each word based on its frequency of occurrence in the training corpus.

[0254] In some embodiments, when the higher-order prediction result acquisition module is configured to perform a hierarchical query on the N-Gram sequence corresponding to each word in the vocabulary, it includes:

[0255] The system checks if the N-Gram sequence corresponding to the given word exists in the high-frequency layer. If it does, the probability value of the N-Gram sequence stored in the high-frequency layer is used as the probability predicted by the higher-order N-Gram model as the word at the given position. If not,

[0256] Then, it checks whether the N-Gram sequence exists in the intermediate frequency layer. If it does, the probability value of the N-Gram sequence stored in the intermediate frequency layer is used as the probability of the word predicted by the higher-order N-Gram model as the word at the specified position; otherwise,

[0257] Then, in the fallback layer, a preset low-order language model is called, and the fallback probability of each word in the vocabulary is calculated based on the frequency of occurrence of each word in the training corpus. This probability is used as the probability that the word in the high-order N-Gram model is the word at the specified position.

[0258] In some embodiments, the higher-order prediction result acquisition module is configured to query whether the N-Gram sequence corresponding to the term exists in a specified layer, wherein the specified layer is either the high-frequency layer or the mid-frequency layer, including:

[0259] Based on multiple independent hash functions preset by the specified layer, the N-Gram sequence corresponding to the word is input into each hash function one by one for mapping calculation, resulting in a set of hash positions; the set of hash positions jointly identifies the existence characteristics of the N-Gram sequence corresponding to the word in the specified layer;

[0260] Query the counter value corresponding to each of the set of hash positions. If the counter value of each hash position is greater than or equal to 1, it is determined that the N-Gram sequence corresponding to the term exists in the specified layer.

[0261] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0262] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 5 As shown, the electronic device 500 includes at least one processor 501, a memory 502, and a bus 503. At least one processor 501 is electrically connected to the memory 502. The memory 502 is configured to store at least one computer-executable instruction, and the processor 501 is configured to execute the at least one computer-executable instruction to perform the steps of any of the large model inference acceleration methods based on speculative sampling provided in any embodiment or optional implementation of this application.

[0263] Furthermore, the processor 501 can be an FPGA (Field-Programmable Gate Array) or other devices with logic processing capabilities, such as an MCU (Microcontroller Unit) or a CPU (Central Processing Unit).

[0264] This application also provides another readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the large model inference acceleration methods based on speculative sampling provided in any of the embodiments or optional implementations of this application.

[0265] The readable storage media provided in this application include, but are not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, readable storage media include any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0266] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0267] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.< / unk> < / eos> < / sos> < / unk>

Claims

1. A method for accelerating large-scale model inference based on speculative sampling, characterized in that, The method includes: Obtain a first probability distribution output by a high-order N-Gram model based on a vocabulary and a first context of input. The first probability distribution includes the probability that each word in the vocabulary is the next word in the first context. The first context includes K words, where K+1 is the order of the high-order N-Gram model. Obtain a second probability distribution output by the low-order N-Gram model based on the vocabulary and the input second context. The second probability distribution includes the probability that each word in the vocabulary is the next word in the second context. The second context includes M words, where M+1 is the order of the low-order N-Gram model, and M is less than K. The next position of the first context and the next position of the second context are the same position. Interpolation weights are obtained based on the text vectors corresponding to the first context, and interpolation operations are performed on the first probability distribution and the second probability distribution based on the interpolation weights to obtain a mixed probability distribution. The target word at the position is determined based on the mixed probability distribution according to the set speculative sampling strategy. If word verification is required, the target word at the specified position and the predicted target words at the L positions preceding the specified position are submitted to the large model; otherwise, the first context and the second context are updated based on the target word at the specified position, and the process returns to the step of obtaining the first probability distribution and the second probability distribution.

2. The method according to claim 1, characterized in that, The interpolation weights obtained based on the text vector corresponding to the first context include: Map each word element included in the first context to a word vector, and concatenate the word vectors corresponding to each word element in order to generate the text vector; The text vector is input into a pre-trained gating network to obtain the interpolation weights output by the gating network.

3. The method according to claim 2, characterized in that, After submitting the target word at the stated position and the predicted target words at the preceding L positions to the large model, the method further includes: For the p-th target word submitted to the large model, obtain the verification probability output by the large model when predicting the p-th target word under the historical context; wherein, the historical context includes: the verified prefix word sequence to which the first context on which the high-order N-Gram model depends when generating the first target word among L+1 target words, and the first p-1 target words; p is an integer satisfying 1≤p≤L+1; In the complete sequence consisting of the verified prefix word sequence and the L+1 consecutively predicted target words, obtain a local sequence of length K ending with the p-th target word; Based on whether the large model accepts the p-th target word as the next word in the historical context, and combining the verification probability with the probability of the p-th target word in the second probability distribution, the sample interpolation weights corresponding to the local sequence are constructed through a preset normalization method. The local sequence is used as the input feature, and the sample interpolation weights corresponding to the local sequence are used as the supervision signal to form a training sample pair; the training sample pair is used to update the parameters of the gating network online.

4. The method according to claim 3, characterized in that, The sample interpolation weights corresponding to the local sequence are constructed using a preset normalization method, including: When the large model accepts the p-th target word as the next word in the historical context, the verification probability is normalized based on the verification probability and the probability of the p-th target word in the second probability distribution to obtain the sample interpolation weight. When the large model refuses to use the p-th target word as the next word in the historical context, a preset penalty factor of less than 1 is introduced into the normalized output of the verification probability to attenuate it, thereby obtaining the sample interpolation weight.

5. The method according to claim 1, characterized in that, Based on the interpolation weight, interpolation is performed between the first probability distribution and the second probability distribution to obtain a mixed probability distribution, including: When the interpolation weight is greater than a first threshold and less than a second threshold, the first probability distribution and the second probability distribution are linearly weighted and fused to obtain the mixed probability distribution; wherein, the second threshold is greater than the first threshold; the weight of the first probability distribution is determined by the interpolation weight; and the weight of the second probability distribution is the complement of the interpolation weight. If the interpolation weight is greater than or equal to the second threshold, the first probability distribution is used as the mixed probability distribution; If the interpolation weight is less than or equal to the first threshold, the second probability distribution is used as the mixed probability distribution.

6. The method according to claim 1, characterized in that, The first context is obtained from the prefix lexical sequence, and the next position of the first context is the same as the next position of the last lexical in the prefix lexical sequence; After obtaining the mixed probability distribution, the method further includes: Obtain the current hidden state corresponding to each word element in the vocabulary when it is used as a word element at the specified position; the current hidden state represents the semantic state of the new sequence formed by adding the word element to the prefix word element sequence; For h consecutive historical words adjacent to the word at the specified position in the prefix word sequence, the historical hidden state of each historical word is obtained; wherein, the historical hidden state is used to reflect the semantic state of the context in the prefix word sequence ending with that historical word; the h is determined according to the total length of the prefix word sequence. Based on the historical hidden states of the h historical lexical units and the current hidden state corresponding to each lexical unit in the vocabulary when it is a lexical unit at the specified position, the mixed probability corresponding to each lexical unit in the mixed probability distribution when it is a lexical unit at the specified position is corrected to obtain the corrected mixed probability distribution.

7. The method according to claim 6, characterized in that, The mixing probability corresponding to each word in the mixing probability distribution when it is a word at the specified position is corrected, including: For each word in the vocabulary, the attention weight of the current hidden state to each historical hidden state is determined based on the similarity between the current hidden state corresponding to the word when it is the word at the position and each historical hidden state. Based on the distance between the historical lexical and the lexical at the position, the decay factor corresponding to the historical hidden state of the historical lexical is determined; the distance is negatively correlated with the decay factor. Based on the attention weight of the current hidden state corresponding to the word when it is the word at the position, and combined with the decay factor corresponding to each historical hidden state, the mixed probability corresponding to the word when it is the word at the position in the mixed probability distribution is corrected.

8. The method according to claim 1, characterized in that, Obtain the first probability distribution output by the high-order N-Gram model based on the vocabulary and the first context of the input, including: In a hierarchical probabilistic storage structure consisting of a high-frequency layer, a mid-frequency layer, and a backoff layer, the N-Gram sequence corresponding to each word in the vocabulary is queried layer by layer to obtain the probability that the word is the word at the specified position; wherein, The N-Gram sequence corresponding to the lexical element includes the first context and the lexical element; The high-frequency layer storage structure is used to cache the top x% of N-Gram sequences that occur most frequently; The intermediate frequency layer storage structure is used to cache N-Gram sequences whose occurrence frequency is between x% and y%. The fallback layer includes a preset low-level language model, which is configured to generate the fallback probability of each word based on the frequency of each word in the vocabulary in the training corpus.

9. The method according to claim 8, characterized in that, Perform a hierarchical query on the N-Gram sequence corresponding to each lexical unit in the vocabulary, including: Check if the N-Gram sequence corresponding to the given word exists in the high-frequency layer. If it does, use the probability value of the N-Gram sequence stored in the high-frequency layer as the probability predicted by the higher-order N-Gram model as the word at the given position; otherwise, Then, it checks whether the N-Gram sequence exists in the intermediate frequency layer. If it does, the probability value of the N-Gram sequence stored in the intermediate frequency layer is used as the probability of the word predicted by the higher-order N-Gram model as the word at the specified position; otherwise, Then, in the fallback layer, a preset low-order language model is called, and the fallback probability of each word in the vocabulary is calculated based on the frequency of occurrence of each word in the training corpus. This probability is used as the probability that the word in the high-order N-Gram model is the word at the specified position.

10. The method according to claim 9, characterized in that, Query whether the N-Gram sequence corresponding to the term exists in a specified layer, where the specified layer is either the high-frequency layer or the mid-frequency layer, including: Based on multiple independent hash functions preset by the specified layer, the N-Gram sequence corresponding to the word is input into each hash function one by one for mapping calculation, resulting in a set of hash positions; the set of hash positions jointly identifies the existence characteristics of the N-Gram sequence corresponding to the word in the specified layer; Query the counter value corresponding to each of the set of hash positions. If the counter value of each hash position is greater than or equal to 1, it is determined that the N-Gram sequence corresponding to the term exists in the specified layer.

11. A large-model inference acceleration device based on speculative sampling, characterized in that, The device includes: The high-order prediction result acquisition module is configured to obtain a first probability distribution output by the high-order N-Gram model based on the vocabulary and the first context of the input. The first probability distribution includes the probability that each word in the vocabulary is the next word in the first context. The first context includes K words, where K+1 is the order of the high-order N-Gram model. The low-order prediction result acquisition module is configured to obtain a second probability distribution output by the low-order N-Gram model based on the vocabulary and the input second context. The second probability distribution includes the probability that each word in the vocabulary is the next word in the second context. The second context includes M words, where M+1 is the order of the low-order N-Gram model, and M is less than K. The next position of the first context and the next position of the second context are the same position. The interpolation module is configured to obtain interpolation weights based on the text vector corresponding to the first context, and to perform interpolation operations on the first probability distribution and the second probability distribution based on the interpolation weights to obtain a mixed probability distribution. Then, according to a set speculative sampling strategy, the target word at the position is determined based on the mixed probability distribution. The mixed probability distribution includes the mixed probability of each word in the vocabulary as a word at the position. The speculative sampling and verification module is configured to, if it is determined that lexical verification is required, submit the target lexical at the current position and the predicted target lexical at the previous L positions to the large model; otherwise, update the first context and the second context based on the target lexical at the current position and return to the step of obtaining the first probability distribution and the second probability distribution.

12. An electronic device, characterized in that, include: Memory, processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to implement the method as described in any one of claims 1-10.

13. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images