A large language model-based inference decoding method, device, equipment and medium

Abstract

The application discloses a large language model-based speculation decoding method and device, equipment and medium, and relates to the technical field of large models, comprising: determining the output probability distribution of a plurality of word positions, determining the confidence score based on the output probability distribution; determining the target control parameter according to the target state signal, aggregating and calculating the confidence vector to obtain the aggregated confidence, determining the target draft length through the aggregated confidence and the target control parameter; determining the verification tolerance threshold based on the target control parameter and the confidence score, generating the draft sequence based on the target depth learning model and the target draft length, and verifying each word in the draft sequence based on the preset large language model and the verification tolerance threshold; adding the target word that has passed the verification to the end of the current generated sequence to obtain the target sequence, determining the target sequence as the current generated sequence, and jumping again to perform the next round of speculation decoding. The application can improve the efficiency of speculation decoding.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a speculative decoding method, apparatus, device and medium based on a large language model. Background Technology

[0002] With the development of deep learning technology, large language models have demonstrated outstanding performance in various fields of natural language processing, such as machine translation, text summarization, dialogue generation, and code synthesis. Most of these models adopt an autoregressive decoding approach, that is, generating the next token in the sequence one by one, with the generation of each token depending on all previously generated tokens.

[0003] However, the inherent sequential nature of autoregressive decoding limits its inference efficiency on parallel computing hardware such as GPUs (Graphics Processing Units). Each generation step must wait for the previous step to complete, which creates a significant latency bottleneck, especially when processing long sequences or deployed in latency-sensitive applications.

[0004] To address this issue, researchers proposed a speculative decoding technique. This technique employs a "draft-verify" paradigm: first, a lightweight "drafting model" generates a draft sequence containing multiple candidate lexical units in parallel; then, a primary, more robust "verify model" verifies this draft sequence. If the draft lexical units match or are sufficiently close to the verify model's predictions, they are accepted, thus advancing multiple generation steps at once and accelerating inference.

[0005] Existing speculative decoding methods suffer from a core flaw: they typically employ fixed draft lengths (e.g., generating a fixed number of k tokens each time) and rigid validation criteria. This static strategy cannot adapt to the dynamic changes during language generation. In some text regions, the model may have high certainty about subsequent content, where a conservative draft length limits the potential for speedup; while in other semantically ambiguous or complex regions, the model has high uncertainty, and an aggressive draft strategy leads to frequent validation failures and rollbacks, increasing computational overhead. Furthermore, fixed validation criteria cannot be adjusted based on the model's immediate confidence state, resulting in a poor balance between efficiency and generation quality. Moreover, these static methods completely ignore the computational environment and real-world application requirements during inference, such as hardware load or Quality of Service (QoS) requirements, failing to perform dynamic optimization at a more macroscopic level and lacking flexibility in practical deployments. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a speculative decoding method, apparatus, device, and medium based on a large language model, which can significantly improve decoding efficiency and robustness, optimize resource utilization, and achieve system-level intelligent inference optimization. While ensuring high-quality output, it enhances flexibility and applicability in complex application scenarios. The specific solution is as follows:

[0007] Firstly, this application discloses a speculative decoding method based on a large language model, including:

[0008] Using a target deep learning model, based on the initial input sequence, the currently generated sequence, and a preset decoding time step, the output probability distribution of several word positions in the currently generated sequence is determined. Based on the output probability distribution, the confidence score of each word position is determined, and a confidence vector is determined according to each confidence score.

[0009] The target control parameters are determined based on the target state signal, and the confidence vector is aggregated to obtain the aggregate confidence. The target draft length is then determined using the aggregate confidence and the target control parameters. The target state signal includes hardware resource signals, quality of service signals, and the remaining latency budget signal allocated to the current speculative decoding request.

[0010] The verification tolerance threshold is determined based on the target control parameters and the confidence score. A draft sequence is generated using the target deep learning model based on the target draft length. Each word in the draft sequence is verified based on the preset large language model and the verification tolerance threshold.

[0011] The successfully verified target word is added to the end of the currently generated sequence to obtain the target sequence. The target sequence is then identified as the currently generated sequence, and the process jumps back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence, and the preset decoding time step to perform the next round of speculative decoding.

[0012] Optionally, determining the confidence score for each lexical position based on the output probability distribution includes:

[0013] The confidence score for each term position is determined based on the output probability distribution using a first confidence score determination formula; the first confidence score determination formula is as follows:

[0014] ;

[0015] Where C is the confidence score; P is the output probability distribution at the current word position; H(P) is the information entropy of the output probability distribution at the current word position; The size of the predefined lexical table; The maximum possible entropy value of the lexical table;

[0016] Alternatively, the confidence score for each term position can be determined based on the output probability distribution using a second confidence score determination formula; the second confidence score determination formula is:

[0017] ;

[0018] Wherein, C is the confidence score; The probability of the word with the highest output probability distribution value in the output probability distribution at the current word position; This represents the probability of the word at the current word position that is the second highest output probability distribution value in the output probability distribution.

[0019] Optionally, the aggregation calculation of the confidence vector to obtain the aggregated confidence includes:

[0020] Calculate the average value of the confidence vector to obtain the aggregate confidence score.

[0021] Optionally, determining the target control parameters based on the target state signal includes:

[0022] The target control parameters corresponding to the target state signal are determined according to a predefined target mapping table; the target control parameters include a draft length scaling factor and a verification tolerance scaling factor; wherein, the hardware resource signals include the graphics processor load, temperature, and available video memory; the target mapping table is a mapping table that records the correspondence between the state signals and the control parameters.

[0023] Optionally, determining the target draft length using the aggregated confidence score and the target control parameters includes:

[0024] The target draft length is determined based on the aggregate confidence level and the target control parameters using a formula; the target draft length determination formula is as follows:

[0025] ;

[0026] in, The target draft length; and These are the preset maximum draft length and the preset minimum draft length, respectively; clip() is the clipping function; The draft length scaling factor; The aggregation confidence level; This is the floor operator.

[0027] Optionally, determining the verification tolerance threshold based on the target control parameter and the confidence score includes:

[0028] Determine the target product between the confidence score and the validation tolerance scaling factor;

[0029] The sum between the target product and the preset minimum tolerance is determined as the verification tolerance threshold.

[0030] Optionally, the step of validating each lexical unit in the draft sequence based on a preset large language model and the validation tolerance threshold includes:

[0031] The first word with the highest output probability distribution corresponding to the current word position is calculated using a pre-defined large language model, and the log probability of the first word is determined.

[0032] Determine the second word corresponding to the current word position in the draft sequence, and determine the logarithmic probability of the second word;

[0033] Calculate the target difference between the log probability of the first word and the log probability of the second word;

[0034] Determine the relationship between the target difference and the verification tolerance threshold;

[0035] If the target difference is less than or equal to the verification tolerance threshold, then the second word corresponding to the current word position in the draft sequence is determined to have been successfully verified.

[0036] Secondly, this application discloses a speculative decoding device based on a large language model, comprising:

[0037] The confidence vector determination module is used to determine the output probability distribution of several word positions in the currently generated sequence based on the initial input sequence, the currently generated sequence and the preset decoding time step using the target deep learning model, determine the confidence score of each word position based on the output probability distribution, and determine the confidence vector according to each confidence score.

[0038] The target draft length determination module is used to determine target control parameters based on target status signals, perform aggregation calculations on the confidence vector to obtain aggregate confidence, and determine the target draft length through the aggregate confidence and the target control parameters; the target status signals include hardware resource signals, quality of service signals, and the remaining delay budget signal allocated to the current speculative decoding request;

[0039] The verification module is used to determine the verification tolerance threshold based on the target control parameters and the confidence score, generate a draft sequence based on the target draft length using the target deep learning model, and verify each word in the draft sequence based on the preset large language model and the verification tolerance threshold.

[0040] The jump module is used to add the successfully verified target word to the end of the currently generated sequence to obtain the target sequence, determine the target sequence as the currently generated sequence, and jump back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence and the preset decoding time step to perform the next round of speculative decoding.

[0041] Thirdly, this application discloses an electronic device, including:

[0042] Memory, used to store computer programs;

[0043] A processor is used to execute computer programs to implement speculative decoding methods based on large language models, as described above.

[0044] Fourthly, this application discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned speculative decoding method based on a large language model.

[0045] This application first utilizes a target deep learning model to determine the output probability distribution of several word positions in the currently generated sequence based on the initial input sequence, the currently generated sequence, and a preset decoding time step. Based on the output probability distribution, a confidence score is determined for each word position, and a confidence vector is determined according to each confidence score. Target control parameters are determined based on target state signals, and the confidence vectors are aggregated to obtain an aggregated confidence score. The target draft length is then determined using the aggregated confidence score and the target control parameters. The target state signals include hardware resource signals, quality of service signals, and the current speculative decoding request being divided into... The remaining delay budget signal is allocated; a verification tolerance threshold is determined based on the target control parameters and the confidence score; a draft sequence is generated using the target deep learning model based on the target draft length; each word in the draft sequence is verified based on a preset large language model and the verification tolerance threshold; finally, the successfully verified target words are added to the end of the currently generated sequence to obtain the target sequence, which is then identified as the currently generated sequence, and the process jumps back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence, and the preset decoding time step for the next round of speculative decoding. It is evident that this application determines the draft length through confidence, significantly reducing invalid draft generation and rollbacks caused by verification failures, thereby maximizing decoding speed while ensuring quality. Simultaneously, control parameters are determined based on state signals, enabling dynamic adjustment of these parameters. This dynamic control allows for intelligent risk control; in high-confidence regions, the criteria are relaxed to accept more reasonable candidate terms, increasing generation diversity; in low-confidence regions, the criteria are tightened to strictly control the quality, effectively preventing the introduction of erroneous terms. This ensures both acceleration and text quality. Furthermore, the introduction of state signals allows this application to adapt not only to micro-level changes in text content but also to macro-level changes in the computing environment and business needs, achieving end-to-end intelligent inference optimization and greater flexibility and robustness in practical deployment. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart of a speculative decoding method based on a large language model disclosed in this application;

[0048] Figure 2This is a flowchart of a specific inference decoding method based on a large language model disclosed in this application;

[0049] Figure 3 This is a flowchart of a specific inference decoding method based on a large language model disclosed in this application;

[0050] Figure 4 This is a schematic diagram of the speculative decoding device based on a large language model disclosed in this application;

[0051] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Existing speculative decoding methods suffer from a core flaw: they typically employ fixed draft lengths (e.g., generating a fixed number of k tokens each time) and rigid validation criteria. This static strategy cannot adapt to the dynamic changes during language generation. In some text regions, the model may have high certainty about subsequent content, where a conservative draft length limits the potential for speedup; while in other semantically ambiguous or complex regions, the model has high uncertainty, and an aggressive draft strategy leads to frequent validation failures and rollbacks, increasing computational overhead. Furthermore, fixed validation criteria cannot be adjusted based on the model's immediate confidence state, resulting in a poor balance between efficiency and generation quality. Moreover, these static methods completely ignore the computational environment and real-world application requirements during inference, such as hardware load or Quality of Service (QoS) requirements, failing to perform dynamic optimization at a more macroscopic level and lacking flexibility in practical deployments. To address the shortcomings of the aforementioned technical problems, this application discloses a speculative decoding method, apparatus, device, and medium based on a large language model, which can significantly improve decoding efficiency and robustness, optimize resource utilization, and achieve system-level intelligent reasoning optimization. While ensuring high-quality output, it enhances flexibility and applicability in complex application scenarios.

[0054] See Figure 1 As shown, this embodiment of the invention discloses a speculative decoding method based on a large language model, including:

[0055] Step S11: Using the target deep learning model, based on the initial input sequence, the currently generated sequence, and the preset decoding time step, determine the output probability distribution of several word positions in the currently generated sequence, determine the confidence score of each word position based on the output probability distribution, and determine the confidence vector according to each confidence score.

[0056] In this embodiment, at each decoding time step, this application does not directly generate a draft, but instead first uses the drafting model to prospectively calculate in parallel a preset maximum window (e.g., The output distribution of all potential word positions within the range is calculated. Based on these distributions, one or more confidence scores are calculated for each potential future position, forming a confidence score vector. This step aims to provide sufficient and parallel information input for subsequent adaptive decision-making. Specifically, this application is applied to a decoding system consisting of a drafting model (a lightweight Transformer model) and a validation model (the original large language model). This system can provide inference services for various NLP (Natural Language Processing) applications on a cloud server. First, the system receives an initial input sequence x, such as a source language sentence in a machine translation task. This sequence x serves as a context reference throughout the decoding process and remains unchanged. Simultaneously, the system initializes a target sequence y, which is used to store the final decoding result (e.g., initialized as an empty sequence, or containing a special word indicating the start of the sequence). <bos>The ultimate goal of all subsequent decoding steps is to generate new tokens and append them to the end of the target sequence y. Therefore, as... Figure 2 As shown, at each decoding time step j, the drafting model (a lightweight Transformer model) predictively computes the positions of multiple subsequent tokens in the target sequence y (e.g., from j+1 to j+) based on the initial input sequence x and the currently generated target sequence y. The output probability distribution of the confidence assessment module is used to calculate the confidence score C based on these distributions. Specifically:

[0057] The confidence score for each term position is determined based on the output probability distribution using a first confidence score determination formula; the first confidence score determination formula is as follows:

[0058] ;

[0059] Where C is the confidence score; P is the output probability distribution at the current word position; H(P) is the information entropy of the output probability distribution at the current word position; The size of the predefined lexical table; The maximum possible entropy value of the lexical table;

[0060] Alternatively, the confidence score for each term position can be determined based on the output probability distribution using a second confidence score determination formula; the second confidence score determination formula is:

[0061] ;

[0062] Wherein, C is the confidence score; The probability of the word with the highest output probability distribution value in the output probability distribution at the current word position; This represents the probability of the word at the current word position that is the second highest output probability distribution value in the output probability distribution.

[0063] In other words, confidence can be calculated based on entropy:

[0064] ;

[0065] Where P is the probability distribution of the model's output at the current position (i.e., the probability assigned to all words in the vocabulary), V is the predefined vocabulary of the model (i.e., the set of all possible output words of the model), and |V| is the size of the vocabulary. The smaller the entropy, the higher the confidence. H(P) is the information entropy of the probability distribution P; log|V| is the maximum possible entropy value. The smaller the entropy value, the more concentrated the probability distribution P (i.e., the more "deterministic" the model), and the lower the uncertainty, resulting in a higher confidence level. The higher.

[0066] Confidence can also be calculated based on marginal values:

[0067] ;

[0068] in, It refers to the probability of the word with the highest probability value in the probability distribution P (i.e., the "best predicted word"); This refers to the probability of the word with the second-highest probability value (i.e., the "second-best predicted word"). The larger the difference between the two, the more obvious the model's "preference" for the best predicted word, and the higher the confidence level. The higher.

[0069] Confidence levels can also be integrated:

[0070] ;

[0071] in, These are learnable or pre-set weights that, through weighted summation or other ensemble methods, are combined into a more robust unified confidence score.

[0072] Finally, this application calculates confidence scores in parallel for multiple potential future locations, forming a confidence vector. This avoids excessive speculative calculations in uncertain regions by using confidence scores, allowing computational resources to be allocated more efficiently to high-success-rate decoding steps and reducing the average energy consumption for generating tokens per unit time.

[0073] Step S12: Determine the target control parameters based on the target state signal, perform aggregation calculation on the confidence vector to obtain the aggregate confidence, and determine the target draft length through the aggregate confidence and the target control parameters; the target state signal includes hardware resource signal, quality of service signal, and the remaining delay budget signal allocated to the current speculative decoding request.

[0074] In this embodiment, target control parameters corresponding to the target state signal are determined according to a predefined target mapping table. The target control parameters include a draft length scaling factor and a verification tolerance scaling factor. The hardware resource signals include the graphics processor's load, temperature, and available video memory. The target mapping table records the correspondence between state signals and control parameters. The adaptive decoding controller receives a confidence score vector and may selectively receive an external system state signal S. This application introduces a multi-factor decision framework for system state feedback to achieve a higher level of adaptability. When performing adjustments, the adaptive decoding controller receives not only the confidence score within the model but also the external system state signal (S), for example:

[0075] Hardware resource signals: such as the GPU's current load, temperature, or available video memory;

[0076] Quality of Service (QoS) signal: The operating mode preset by the user or system administrator, such as "high-speed mode", "balanced mode" or "high-quality mode";

[0077] Task specification signals: such as the remaining delay budget allocated to the current inference request.

[0078] The controller fuses internal confidence levels and external system status signals to dynamically adjust global decoding strategy control parameters (e.g., draft length scaling factor α for calculating draft length and verification tolerance scaling factor γ for calculating verification threshold). For example, in "Extreme Speed ​​Mode" or when the latency budget is about to run out, the controller automatically increases the values ​​of α and γ, making the decoding strategy more aggressive; while in "High Quality Mode" or under high GPU load, it automatically decreases these parameter values, switching to a more conservative strategy to prioritize output quality and system stability.

[0079] In other words, the controller first adjusts the global decoding strategy control parameters, namely the draft length scaling factor, based on the external system status signal S. And verify the tolerance scaling factor γ. A specific adjustment mechanism is a rule-based mapping: the system predefines a state-parameter mapping table, which stores the parameter combinations {α, γ} corresponding to different system states S. For example:

[0080] If S is "Extreme Speed ​​Mode" (QoS signal), then it is mapped to (High-motivation combination);

[0081] If S is in "equalization mode" (QoS signal), then it is mapped to (Moderately positive combination);

[0082] If S is "High Quality Mode" (QoS signal), then it is mapped to (Low-motivation combination);

[0083] If S is "high hardware load" (e.g., GPU utilization > 90%), then it is mapped to This is to reduce system pressure.

[0084] During decoding, the controller receives the current external system status signal S (e.g., "Extreme Speed ​​Mode").

[0085] The controller queries the mapping table to obtain the corresponding parameter combination (e.g., ).

[0086] The controller sets the α used in the current decoding iteration. and γ= .

[0087] This application, based on the confidence score vector obtained in the previous step, implements a dual and decoupled adjustment mechanism to determine the "length" of the draft and the "rigor" of the verification, respectively. In this embodiment, regarding the draft length adjustment: based on the aggregated evaluation of multiple confidence scores within the future window (e.g., calculating their average), an aggregated confidence score reflecting the overall difficulty of the short-term generation task is obtained. Based on this aggregated confidence score, the number of draft tokens to be generated at the current decoding time step (i.e., the draft length) is dynamically determined. If the aggregation confidence is high, a longer draft is used to maximize speedup; conversely, if the confidence is low, the draft is shortened to avoid unnecessary computation. First, this application considers the generated confidence score vector (i.e....) Aggregate calculations can be performed, for example, by calculating the average value, to obtain an aggregate confidence level that reflects the difficulty of generating the entire future draft window. Then, the target draft length is determined based on the aggregate confidence level and the target control parameters using the target draft length determination formula; the target draft length determination formula is:

[0088] ;

[0089] in, The target draft length; and These are the preset maximum draft length and the preset minimum draft length, respectively; clip() is the clipping function; The draft length scaling factor can be adjusted according to the application scenario (e.g., prioritizing speed or quality). High time, near ; When low, near ; The aggregation confidence level; This is the floor operator.

[0090] Step S13: Determine the verification tolerance threshold based on the target control parameters and the confidence score; generate a draft sequence using the target deep learning model based on the target draft length; and verify each word in the draft sequence based on the preset large language model and the verification tolerance threshold.

[0091] In this embodiment, the dual and decoupled adjustment mechanism adjusts the verification threshold based on the independent confidence score of each future position, dynamically determining an independent verification tolerance scaling factor for that position. This adjustment mechanism directly links the validation criteria to the word-by-word meta-confidence of the drafting model: when the confidence is high, the tolerance increases (the criteria are relaxed) to improve the pass rate; when the confidence is low, the tolerance decreases (the criteria are tightened) to ensure the accuracy of the generated content. Therefore, this application determines the target product between the confidence score and the validation tolerance scaling factor; the sum of the target product and the preset minimum tolerance is determined as the validation tolerance threshold. Specifically, for each position t in the draft sequence, its validation tolerance scaling factor... Determine using the following formula:

[0092] ;

[0093] in, It is a basic, minimum tolerance level, ensuring a basic verification threshold even at the highest confidence level. γ, the verification tolerance scaling factor, is an adjustment factor used to control the sensitivity of the verification standard to the confidence level. This is the confidence score of a single term at position t. The logic of this formula is: when drafting the model confidence score... When low (e.g., approaching 0), tolerance scaling factor The size will decrease, and the validation criteria will become more stringent. This requires that the draft lexical units be highly consistent with the predictions of the validation model, thereby effectively filtering out uncertain drafts; conversely, when the confidence level of the draft model is low... When high (e.g., approaching 1), tolerance scaling factor The size will increase, and the validation criteria will be more lenient. This allows the validation model to accept some words that are not its best predictions but are still highly probable, thereby increasing the approval rate of the draft.

[0094] Then, the draft model is based on the length determined by the controller. Generate a draft sequence, that is, the drafting model is based on the adjusted draft length. Parallel generation of a containing The model generates a draft sequence of candidate lexical units. Then, it validates each lexical unit in the draft sequence one by one. Crucially, when validating the t-th draft lexical unit, it strictly uses a validation tolerance threshold calculated independently for that unit. The process involves validating each lexical unit in the draft sequence based on a preset large language model and the aforementioned validation tolerance threshold. The validation process continues until the first lexical unit fails validation, or until all draft lexical units are successfully validated. In this way, by generating longer drafts in high-confidence regions of the model and adopting a conservative strategy in low-confidence regions, this application significantly reduces invalid draft generation and rollbacks caused by validation failures, thereby maximizing decoding speed while ensuring quality.

[0095] It should also be noted that this invention adjusts not only the length of the draft but also its structure. When the aggregation confidence is within a preset intermediate range, it indicates that the generation task has some uncertainty but is not completely unpredictable. At this point, the drafting model will generate a tree-like draft structure with branches. The main path of this tree represents the highest probability word sequence, while the side branches represent the next highest probability alternative paths. Accordingly, the refined adaptive verification step is also extended to perform parallel or pruning verification on this tree structure, thereby increasing the probability of finding the correct sequence.

[0096] Step S14: Add the successfully verified target word to the end of the currently generated sequence to obtain the target sequence. Determine the target sequence as the currently generated sequence and jump back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence and the preset decoding time step to perform the next round of speculative decoding.

[0097] In this embodiment, after validating each word in the draft sequence based on a preset large language model and the validation tolerance threshold, all valid words are accepted and appended sequentially to the end of the target sequence y initialized in step S11, thereby increasing the length of sequence y. Decoding time step j is updated to the position of the last accepted word, and then jumps back to the step of using the target deep learning model to determine the output probability distribution of several word positions in the currently generated sequence based on the initial input sequence, the currently generated sequence, and the preset decoding time step, to begin the next round of adaptive inference decoding. In this way, this application can automatically adapt to the complexity of different input texts and the uncertainty fluctuations of the model during the generation process without modifying or retraining the existing large language model structure, making the decoding process robust in various scenarios and possessing strong versatility and practicality.

[0098] In summary, this application first utilizes a target deep learning model to determine the output probability distribution of several word positions in the currently generated sequence based on the initial input sequence, the currently generated sequence, and a preset decoding time step. Based on the output probability distribution, it determines the confidence score for each word position and then determines a confidence vector based on each confidence score. Next, it determines target control parameters based on target state signals, aggregates the confidence vectors to obtain an aggregate confidence score, and determines the target draft length using the aggregate confidence score and the target control parameters. The target state signals include hardware resource signals, quality of service signals, and the current speculative decoding request. The process involves: calculating the allocated remaining delay budget signal; determining the verification tolerance threshold based on the target control parameters and the confidence score; generating a draft sequence using the target deep learning model based on the target draft length; verifying each word in the draft sequence based on a preset large language model and the verification tolerance threshold; finally, adding the successfully verified target words to the end of the currently generated sequence to obtain the target sequence; defining the target sequence as the currently generated sequence; and then returning to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence, and the preset decoding time step for the next round of speculative decoding. As can be seen, this application significantly reduces invalid draft generation and rollbacks caused by verification failures by determining the draft length through confidence, thereby maximizing decoding speed while ensuring quality. Simultaneously, control parameters are determined based on state signals, enabling dynamic adjustment of these parameters. This dynamic control allows for intelligent risk control; in high-confidence regions, the criteria are relaxed to accept more reasonable candidate terms, increasing generation diversity; in low-confidence regions, the criteria are tightened to strictly control the quality, effectively preventing the introduction of erroneous terms. This ensures both acceleration and text quality. Furthermore, the introduction of state signals allows this application to adapt not only to micro-level changes in text content but also to macro-level changes in the computing environment and business needs, achieving end-to-end intelligent inference optimization and greater flexibility and robustness in practical deployment.

[0099] As described in the previous embodiment, this application discloses a speculative decoding method based on a large language model. During the speculative decoding process, each lexical unit in the draft sequence is verified based on a preset large language model and the verification tolerance threshold. The specific verification process will be described in detail below.

[0100] See Figure 3 As shown, this embodiment of the invention discloses a specific inference decoding method based on a large language model, including:

[0101] Step S21: Calculate the first word with the highest output probability distribution corresponding to the current word position using a preset large language model, and determine the log probability of the first word. Determine the second word corresponding to the current word position in the draft sequence, and determine the log probability of the second word.

[0102] In this embodiment, the drafting model is based on the length determined by the controller. After generating the draft sequence, the validation model (the original large language model) validates the draft sequence. For the i-th word in the draft... (Generated by the drafting model), which is then evaluated by the validation model. First, the validation model calculates its own "best predicted" term at position j+i. (That is, the word whose probability the validation model considers highest). Then the validation model obtains this "best predicted" word. The logarithmic probability is denoted as The validation model simultaneously calculates the "draft lexical" generated by the drafting model. The logarithmic probability is denoted as .

[0103] Step S22: Calculate the target difference between the log probability of the first word and the log probability of the second word, and determine the relationship between the target difference and the verification tolerance threshold. If the target difference is less than or equal to the verification tolerance threshold, then the second word corresponding to the current word position in the draft sequence is determined to have been successfully verified.

[0104] In this embodiment, after calculating the log probabilities corresponding to two lexical units, the difference between the two log probabilities is calculated. Then, determine the relationship between this gap—that is, the target difference between the logarithmic probability of the first word and the logarithmic probability of the second word—and the validation tolerance threshold. If this gap... If the value is less than or equal to the verification tolerance threshold dynamically determined for that position in step S13, then the draft term is judged to be... Validation is successful. The validation process continues until the first word fails validation, or until all draft words have been successfully validated. Then, all validated words are accepted as part of the final generated sequence.

[0105] In this way, this application significantly improves decoding efficiency and robustness, optimizes resource utilization, and achieves system-level intelligent inference optimization, enhancing flexibility and applicability in complex application scenarios while ensuring high-quality output.

[0106] In a preferred embodiment of this application, the confidence score can be determined using various measurement methods, including but not limited to:

[0107] Information entropy measure: Calculate the entropy value of the probability distribution output by the drafting model. The lower the entropy value, the higher the confidence level.

[0108] Marginal metric: Calculate the probability difference or logit difference between the highest probability word and the second highest probability word. The larger the difference, the higher the confidence level.

[0109] Multi-head attention entropy metric: This analyzes the dispersion of attention weights within the Transformer attention mechanism of the drafting model. The more concentrated the attention weights are on a few historical lexical units, the more certain the model's understanding of the current context is, and the higher its confidence level.

[0110] External knowledge base validation: For knowledge-intensive generation tasks, the candidate nomenclature generated by the drafting model is compared with the external knowledge base. If the candidate content is consistent with factual knowledge, its confidence score is increased.

[0111] Integrated confidence score: The above-mentioned multiple metrics are combined into a more robust unified confidence score through weighted summation or other integration methods.

[0112] In this way, by calculating the confidence score, this application uses the confidence score to determine the draft length in subsequent processes, avoiding excessive speculative calculations in uncertain areas, and more effectively allocating computing resources to high-success-rate decoding steps, thereby reducing the average energy consumption of generating tokens per unit time.

[0113] See Figure 4 As shown, this embodiment of the invention discloses a speculative decoding device based on a large language model, comprising:

[0114] The confidence vector determination module 11 is used to determine the output probability distribution of several word positions in the currently generated sequence based on the initial input sequence, the currently generated sequence and the preset decoding time step using the target deep learning model, determine the confidence score of each word position based on the output probability distribution, and determine the confidence vector according to each confidence score.

[0115] The target draft length determination module 12 is used to determine the target control parameters based on the target status signal, perform aggregation calculation on the confidence vector to obtain the aggregate confidence, and determine the target draft length through the aggregate confidence and the target control parameters; the target status signal includes hardware resource signal, quality of service signal and the remaining delay budget signal allocated to the current speculative decoding request;

[0116] Verification module 13 is used to determine the verification tolerance threshold based on the target control parameters and the confidence score, generate a draft sequence based on the target draft length using the target deep learning model, and verify each word in the draft sequence based on the preset large language model and the verification tolerance threshold.

[0117] The jump module 14 is used to add the successfully verified target word to the end of the currently generated sequence to obtain the target sequence, determine the target sequence as the currently generated sequence, and jump back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence and the preset decoding time step to perform the next round of speculative decoding.

[0118] In some specific embodiments, the confidence vector determination module 11 can be used to determine the confidence score of each word position based on the output probability distribution using a first confidence score determination formula; the first confidence score determination formula is:

[0119] ;

[0120] Where C is the confidence score; P is the output probability distribution at the current word position; H(P) is the information entropy of the output probability distribution at the current word position; The size of the predefined lexical table; The maximum possible entropy value of the lexical table;

[0121] Alternatively, the confidence score for each term position can be determined based on the output probability distribution using a second confidence score determination formula; the second confidence score determination formula is:

[0122] ;

[0123] Wherein, C is the confidence score; The probability of the word with the highest output probability distribution value in the output probability distribution at the current word position; This represents the probability of the word at the current word position that is the second highest output probability distribution value in the output probability distribution.

[0124] In some specific embodiments, the target draft length determination module 12 can be used to calculate the average value of the confidence vector to obtain the aggregate confidence.

[0125] In some specific embodiments, the target draft length determination module 12 can be used to determine the target control parameters corresponding to the target status signal according to a predefined target mapping table; the target control parameters include a draft length scaling factor and a verification tolerance scaling factor; wherein, the hardware resource signal includes the graphics processor load, temperature, and available video memory; the target mapping table is a mapping table that records the correspondence between the status signal and the control parameters.

[0126] In some specific embodiments, the target draft length determination module 12 can be used to determine the target draft length based on the aggregate confidence level and the target control parameters using a target draft length determination formula; the target draft length determination formula is:

[0127] ;

[0128] in, The target draft length; and These are the preset maximum draft length and the preset minimum draft length, respectively; clip() is the clipping function; The draft length scaling factor; The aggregation confidence level; This is the floor operator.

[0129] In some specific embodiments, the verification module 13 can be used to determine the target product between the confidence score and the verification tolerance scaling factor; and to determine the sum of the target product and the preset minimum tolerance as the verification tolerance threshold.

[0130] In some specific embodiments, the verification module 13 can be used to calculate the first word with the highest output probability distribution corresponding to the current word position using a preset large language model, and determine the log probability of the first word; determine the second word corresponding to the current word position in the draft sequence, and determine the log probability of the second word; calculate the target difference between the log probability of the first word and the log probability of the second word; determine the relationship between the target difference and the verification tolerance threshold; if the target difference is less than or equal to the verification tolerance threshold, then the verification of the second word corresponding to the current word position in the draft sequence is successful.

[0131] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0132] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the large language model-based speculative decoding method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0133] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0134] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0135] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the large language model-based speculative decoding method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0136] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned speculative decoding method based on a large language model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0138] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0140] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0141] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / bos>

Claims

1. A speculative decoding method based on a large language model, characterized in that, include: Using a target deep learning model, based on the initial input sequence, the currently generated sequence, and a preset decoding time step, the output probability distribution of several word positions in the currently generated sequence is determined. Based on the output probability distribution, the confidence score of each word position is determined, and a confidence vector is determined according to each confidence score. The target control parameters are determined based on the target state signal, the confidence vector is aggregated to obtain the aggregate confidence, and the target draft length is determined by the aggregate confidence and the target control parameters. The target status signal includes hardware resource signal, quality of service signal, and the remaining latency budget signal allocated to the current speculative decoding request; The verification tolerance threshold is determined based on the target control parameters and the confidence score. A draft sequence is generated using the target deep learning model based on the target draft length. Each word in the draft sequence is verified based on the preset large language model and the verification tolerance threshold. The successfully verified target word is added to the end of the currently generated sequence to obtain the target sequence. The target sequence is then determined as the currently generated sequence, and the process jumps back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence, and the preset decoding time step to perform the next round of speculative decoding. The step of determining the target control parameters based on the target state signal includes: The target control parameters corresponding to the target state signal are determined according to a predefined target mapping table; the target control parameters include a draft length scaling factor and a verification tolerance scaling factor; wherein, the hardware resource signals include the graphics processor load, temperature, and available video memory; the target mapping table is a mapping table that records the correspondence between the state signals and the control parameters.

2. The inference decoding method based on a large language model according to claim 1, characterized in that, The determination of the confidence score for each lexical position based on the output probability distribution includes: The confidence score for each term position is determined based on the output probability distribution using a first confidence score determination formula; the first confidence score determination formula is as follows: ； Where C is the confidence score; P is the output probability distribution at the current word position; H(P) is the information entropy of the output probability distribution at the current word position; The size of the predefined lexical table; The maximum possible entropy value of the lexical table; Alternatively, the confidence score for each term position can be determined based on the output probability distribution using a second confidence score determination formula; the second confidence score determination formula is: ； Wherein, C is the confidence score; The probability of the word with the highest output probability distribution value in the output probability distribution at the current word position; This represents the probability of the word at the current word position that is the second highest output probability distribution value in the output probability distribution.

3. The inference decoding method based on a large language model according to claim 1, characterized in that, The aggregation calculation of the confidence vector to obtain the aggregated confidence includes: Calculate the average value of the confidence vector to obtain the aggregate confidence score.

4. The inference decoding method based on a large language model according to claim 1, characterized in that, The process of determining the target draft length using the aggregated confidence level and the target control parameters includes: The target draft length is determined based on the aggregate confidence level and the target control parameters using a formula; the target draft length determination formula is as follows: ； in, The target draft length; and These are the preset maximum draft length and the preset minimum draft length, respectively; clip() is the clipping function; The draft length scaling factor; The aggregation confidence level; This is the floor operator.

5. The inference decoding method based on a large language model according to claim 1, characterized in that, The step of determining the verification tolerance threshold based on the target control parameters and the confidence score includes: Determine the target product between the confidence score and the validation tolerance scaling factor; The sum between the target product and the preset minimum tolerance is determined as the verification tolerance threshold.

6. The inference decoding method based on a large language model according to any one of claims 1 to 5, characterized in that, The validation of each lexical unit in the draft sequence based on the preset large language model and the validation tolerance threshold includes: The first word with the highest output probability distribution corresponding to the current word position is calculated using a pre-defined large language model, and the log probability of the first word is determined. Determine the second word corresponding to the current word position in the draft sequence, and determine the logarithmic probability of the second word; Calculate the target difference between the log probability of the first word and the log probability of the second word; Determine the relationship between the target difference and the verification tolerance threshold; If the target difference is less than or equal to the verification tolerance threshold, then the second word corresponding to the current word position in the draft sequence is determined to have been successfully verified.

7. A speculative decoding device based on a large language model, characterized in that, include: The confidence vector determination module is used to determine the output probability distribution of several word positions in the currently generated sequence based on the initial input sequence, the currently generated sequence and the preset decoding time step using the target deep learning model, determine the confidence score of each word position based on the output probability distribution, and determine the confidence vector according to each confidence score. The target draft length determination module is used to determine the target control parameters based on the target state signal, perform aggregation calculation on the confidence vector to obtain the aggregate confidence, and determine the target draft length through the aggregate confidence and the target control parameters. The target status signal includes hardware resource signal, quality of service signal, and the remaining latency budget signal allocated to the current speculative decoding request; The verification module is used to determine the verification tolerance threshold based on the target control parameters and the confidence score, generate a draft sequence based on the target draft length using the target deep learning model, and verify each word in the draft sequence based on the preset large language model and the verification tolerance threshold. The jump module is used to add the successfully verified target word to the end of the currently generated sequence to obtain the target sequence, determine the target sequence as the currently generated sequence, and jump back to the step of using the target deep learning model based on the initial input sequence, the currently generated sequence and the preset decoding time step to perform the next round of speculative decoding; The target draft length determination module is used to determine the target control parameters corresponding to the target status signal according to a predefined target mapping table; the target control parameters include a draft length scaling factor and a verification tolerance scaling factor; wherein, the hardware resource signals include the graphics processor load, temperature, and available video memory; the target mapping table is a mapping table that records the correspondence between status signals and control parameters.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the speculative decoding method based on a large language model as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, which, when executed by a processor, implements the speculative decoding method based on a large language model as described in any one of claims 1 to 6.

Images