A speculative decoding method and apparatus, an electronic device, and a storage medium

Abstract

The application discloses a kind of speculative decoding method, device, electronic equipment and storage medium, the method is executed by verification model, comprising: determining each candidate word element of draft model output, and the strict greed word element obtained by verification model to candidate word element forward calculation;According to each candidate word element and the strict greed word element corresponding to each candidate word element, determine the local risk and unit value risk corresponding to each candidate word element, local risk is used to indicate the risk of inconsistency between candidate word element and strict greed word element, unit value risk is obtained by normalizing the local risk through suffix value after candidate word element;If it is determined that target candidate word element does not satisfy threshold constraint condition according to local risk and unit value risk of target candidate word element, then each candidate word element before target candidate word element is output as continuous submission prefix.The present application does not need to train model, can be inserted, can be audited to relax low-risk candidate word element, to extend continuous submission prefix.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model technology, and in particular to a speculative decoding method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence models, natural language models have been widely applied in scenarios such as online question answering, mathematical reasoning, code generation, knowledge-based question answering, intelligent agents, and search-enhanced generation. Among these, the speculative decoding system consists of a draft model and a validation model. The lightweight draft model quickly generates multiple candidate tokens (lexical units) at once. The validation model then computes a strictly greedy token forward, validating candidate tokens and the strictly greedy token sequentially from left to right. Upon encountering the first mismatch, all subsequent candidate tokens are discarded, and the token before the mismatch position is output.

[0003] To improve the speed and accuracy of speculative decoding, current technologies typically involve retraining the draft and validation models. However, this approach is costly and difficult to migrate quickly to existing inference services. Furthermore, strict greedy validation discards all tokens after the first mismatched token, failing to fully utilize the differences between the strictly greedy token and candidate tokens already calculated by the validation model. Summary of the Invention

[0004] This invention provides a speculative decoding method, apparatus, electronic device, and storage medium to enable auditable relaxation of low-risk candidate tokens without the need for training a model and with pluggable capabilities, thereby extending the consecutive submission prefix.

[0005] In a first aspect, embodiments of the present invention provide a speculative decoding method, which is executed by a verification model, and the method includes: Determine each candidate lexical unit output by the draft model, and verify the strictly greedy lexical units obtained by the model through forward computation of the candidate lexical units; Based on each candidate word and the strictly greedy word corresponding to each candidate word, determine the local risk and unit value risk corresponding to each candidate word. Among them, local risk is used to represent the inconsistency risk between candidate lexical units and strictly greedy lexical units, and unit value risk is obtained by normalizing the local risk through the suffix value after the candidate lexical unit; If the threshold constraint is not met based on the local risk and unit value risk of the target candidate word, then all candidate words preceding the target candidate word will be output as consecutive submission prefixes.

[0006] Secondly, embodiments of the present invention also provide a speculative decoding device, which is deployed in a verification model execution, the device comprising: The lexical determination module is used to determine each candidate lexical output by the draft model, as well as the strictly greedy lexical obtained by the verification model through forward calculation of the candidate lexical. The risk calculation module is used to determine the local risk and unit value risk of each candidate word based on each candidate word and the strictly greedy word corresponding to each candidate word. Among them, local risk is used to represent the inconsistency risk between candidate lexical units and strictly greedy lexical units, and unit value risk is obtained by normalizing the local risk through the suffix value after the candidate lexical unit; The continuous submission prefix determination module is used to output each candidate word preceding the target candidate word as a continuous submission prefix if the threshold constraint is not met based on the local risk and unit value risk of the target candidate word.

[0007] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the speculative decoding method as described in any of the embodiments of the present invention.

[0008] Fourthly, embodiments of the present invention also provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the speculative decoding method as described in any of the embodiments of the present invention.

[0009] The technical solution of this invention involves performing forward computation on each candidate lexical unit output by the draft model using a validation model to obtain strictly greedy lexical units. The local risk and unit value risk between each candidate lexical unit and its corresponding strictly greedy lexical unit are calculated. Local risk represents the inconsistency risk between the candidate lexical unit and the strictly greedy lexical unit, while unit value risk is obtained by normalizing the local risk using the suffix value following the candidate lexical unit. Each candidate lexical unit is evaluated individually. If the local risk and unit value risk of the target candidate lexical unit do not meet the threshold constraints, all candidate lexical units preceding the target candidate lexical unit are output as consecutive submission prefixes. This technical solution eliminates the need to train both the draft and validation models, is pluggable and auditable, and can relax restrictions on low-risk candidate lexical units, thereby extending the consecutive submission prefix.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a speculative decoding method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a speculative decoding method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a speculative decoding device provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices. In the embodiments of this application, certain software, components, models, and other existing industry solutions may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0015] The acquisition, transmission, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0016] Example 1 Figure 1 The flowchart of a speculative decoding method is provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of speculative decoding. The method can be executed by a speculative decoding device, which can be implemented in hardware and / or software. The speculative decoding device can be configured in the verification model and used in conjunction with the draft model.

[0017] like Figure 1 As shown, the method includes: S110. Determine each candidate lexical unit output by the draft model, and the strictly greedy lexical unit obtained by the verification model through forward calculation of the candidate lexical units.

[0018] It should be noted that this embodiment is applicable to speculative decoding systems, which can include a draft model and a verification model, or a draft module and a verification module. This embodiment does not impose any restrictions on this.

[0019] Specifically, the input to the speculative decoding system is a prompt word. The user inputs the prompt word into the speculative decoding system. After encoding, the prompt word is used as the current context input to the draft model. The draft model processes the current context and outputs a candidate token sequence of length K: x1, x2, ..., x K The draft model structure can be a linear token block, a candidate tree, or a multi-branch graph. For a tree-structured draft model, subsequent calculations can be performed along the candidate paths to select the paths that satisfy the gating constraints and the consecutive submission prefixes.

[0020] The validation model performs forward computation on candidate lemmas. Specifically, the validation model can execute teacher-forced once and generate the logit vector z for all K positions in parallel. i (v), for K positions, the strictly greedy lexical unit corresponding to the i-th position, i.e., the i-th candidate lexical unit, is defined as: That is, on the logit distribution of the i-th position obtained by teacher-forced forward, the token with the highest logit score (original score) is the strictly greedy token.

[0021] The above process can be carried out using conventional methods in the existing technology, and this embodiment is not limited thereto.

[0022] S120. Based on each candidate lexical unit and the strictly greedy lexical unit corresponding to each candidate lexical unit, determine the local risk and unit value risk corresponding to each candidate lexical unit.

[0023] Among them, local risk is used to represent the inconsistency risk between candidate lexical units and strictly greedy lexical units, and unit value risk is obtained by normalizing the local risk through the suffix value after the candidate lexical unit.

[0024] In an optional embodiment, determining the local risk corresponding to each candidate word and the strictly greedy word corresponding to each candidate word may include: taking the difference between the original score of the candidate word and the original score of the strictly greedy word corresponding to the candidate word as the local risk corresponding to the candidate word.

[0025] Specifically, local risk can be calculated using the following formula: , This represents the local risk at position i. This is the logit score for strictly greedy lexical units, which is also the highest raw score. The logit score is the score of the candidate term.

[0026] According to the above formula, in If the candidate word is strictly consistent, then the candidate word is considered a strictly consistent word; otherwise, further risk assessment is required to determine whether the requirements should be relaxed.

[0027] In another optional embodiment, determining the local risk corresponding to each candidate lexical and its corresponding strictly greedy lexical may further include: using the difference between the logarithmic probability ratio of the candidate lexical and the equivalent parameter of the logit score of the strictly greedy lexical corresponding to the candidate lexical as the local risk corresponding to the candidate lexical. The equivalent parameter of the logit score includes, but is not limited to: the logarithmic probability ratio, the score difference transformed by a temperature or calibration function, margin, quantile risk, or other additive parameters that monotonically increase with deviation from the target.

[0028] In this embodiment, the unit value risk is obtained by dividing the local risk by the suffix value. Specifically, it can be expressed by the following formula: ,in, This represents the unit value risk of the i-th position, i.e., the i-th candidate word element. This represents the suffix value of the i-th position, which is also the i-th candidate word.

[0029] The suffix value indicates the length of the subsequent candidate lexical units that might be retained if the current position is accepted.

[0030] In an optional embodiment, the suffix value can be represented by the number of subsequent candidate lexics following the candidate lexicographical unit. Specifically, At this point, the suffix value is the length of the remaining candidate lexical units after the current position.

[0031] In another alternative embodiment, the suffix value can also be expressed by the following formula: ,in, Let K represent the suffix value of the i-th candidate lexical unit, and K represent the number of candidate lexical units. This represents the probability of accepting the condition. That is to say This indicates the probability that the next candidate word after the current position can still be accepted. This represents the probability of accepting two consecutive candidate words following the current position, and so on. For example, if K=4 and i=2, =0.8, =0.5, then =2.2.

[0032] In another alternative embodiment, the suffix value can also be represented by a pre-set expected acceptance length, or it can be calculated by combining the pre-set expected acceptance length, the pre-set predicted latency savings, and the pre-set communication savings.

[0033] In this embodiment, local risk is used to quantify the consistency between candidate words at the current position and strictly greedy words. Local risk can also objectively quantify the risk level of candidate words at the current position, transforming the traditional binary event of whether a match exists into a measurable and comparable risk. Unit value risk quantifies how much risk is required for each unit of potential acceleration value. This quantification also allows for macroscopic quantification of the risk the model bears if it accepts an additional candidate word, thus prioritizing lower regret positions that may retain more candidate word computational value.

[0034] S130. If, based on the local risk and unit value risk of the target candidate word, it is determined that the threshold constraint is not met, then all candidate words preceding the target candidate word will be output as consecutive submission prefixes.

[0035] Understandably, strict greedy validation treats all mismatched candidate words equally. For a low-risk candidate word whose logit score is only slightly lower than that of the strict greedy word, and a high-risk candidate word that significantly deviates from the validation model's preference, both will result in the truncation of consecutive prefixes, leading to underutilization of the logit scores already calculated by the validation model. However, if, in order to improve the acceptance rate of candidate words, a threshold comparison of the logit score difference between the strict greedy word and the candidate word is performed position by position to determine whether to accept the candidate word, then small errors will accumulate in long requests, leading to a decrease in accuracy.

[0036] Therefore, in this embodiment, threshold constraints are set for local risk and unit value risk. Each candidate word is judged sequentially, and the position of the first candidate word that does not meet the threshold constraints is found. The continuous submission prefix before that position is returned. The continuous submission prefix refers to the candidate word sequence recognized and accepted by the validated model.

[0037] Specifically, determining whether the threshold constraint is not met based on the local risk and unit value risk of the target candidate word can include: if the cumulative value of the local risk of the target candidate word and all candidate words before the target candidate word is less than or equal to the cumulative risk budget, the unit value risk of the target candidate word is less than or equal to the unit value threshold, and the number of candidate words with non-zero local risk in the draft block containing the target candidate word is less than or equal to the upper limit of word words in a single draft block, then the threshold constraint is met; otherwise, the threshold constraint is not met.

[0038] This embodiment sets up a triple threshold constraint condition. The target candidate word refers to the candidate word currently being judged. By accumulating the local risks of the target candidate word and all preceding candidate words, it is determined whether the accumulated local risk value exceeds the accumulated risk budget. The accumulated risk budget can be dynamically set according to request type, prompt characteristics, user level, task risk, target length, or online quality monitoring. It can also be set in layers, such as setting an accumulated risk budget for the entire request, a phased accumulated risk budget, and a sliding window accumulated risk budget. By judging the accumulated risk budget, small deviations can be prevented from being accumulated unbounded in long requests.

[0039] By determining whether the unit value risk of the target candidate word at the current position exceeds the unit value threshold, the cumulative risk budget can be prevented from being consumed by a single low-value, high-risk candidate word.

[0040] The number of candidate words with non-zero local risk within the draft block containing the target candidate word refers to the number of candidate words in the draft block that have been relaxed and accepted by the validated model up to the current position. The upper limit for the number of words in a single draft block can be defined by a pre-set fixed value, a pre-set proportion multiplied by the total number of words in the single draft block, or risk density. By determining the number of candidate words and the upper limit for the number of words in a single draft block, excessive density of candidate words relaxed and accepted by the validated model within the same draft block can be avoided.

[0041] In this embodiment, for candidate tokens that simultaneously meet the aforementioned triple threshold constraints, the verification model accepts them as relaxed tokens. Simultaneously, it updates status data such as the cumulative local risk value and the number of candidate tokens accepted as relaxed tokens within the same draft block. Furthermore, for relaxed tokens, log data is recorded, including the processing round of the draft block containing the relaxed token, its position within the current draft block, candidate token identifier, strictly greedy token identifier, local risk, suffix value, unit value risk, and the cumulative local risk value before and after accepting the relaxed token. This log data can serve as the data foundation for offline quality analysis, online circuit breaking, configuration auditing, and reproduction experiments.

[0042] To avoid state crosstalk caused by request migration, batch processing rearrangement, or distributed execution, state data updates are released when a request ends, is canceled, or encounters an exception. When a request is migrated across processes, the compact state is synchronized. The compact state is the compact state obtained by low-rank projection compression after forward computation of the verification model. It is a hidden state representation and does not require the synchronization of the complete historical logit vector sequence.

[0043] For the first candidate word that does not meet the above triple gating constraints, stop judging subsequent candidate words and return to the consecutive submitted prefixes before that position.

[0044] Furthermore, if it is determined that the target candidate word is located within the draft block, then the acceptance of subsequent candidate words is stopped, and the target candidate word is replaced by a strictly greedy word corresponding to the target candidate word as a recovery word. The draft block containing the target candidate word is then output.

[0045] In this embodiment, if the first mismatched candidate token is encountered within the draft block, i.e., the first candidate word that does not satisfy the triple gating constraint, subsequent candidate words are no longer accepted, and a recovery token is submitted at the current position. Specifically, the target candidate word is replaced by the strictly greedy word corresponding to the target candidate word at the current position. For example, if the candidate words corresponding to the current draft block include [x1, x2, x3, x4], and x4 does not satisfy the triple gating constraint, then x4 is replaced by its corresponding strictly greedy word, and the current draft block is output in the form [x1, x2, x3, argmax4].

[0046] Furthermore, if all candidate words in the draft block meet the threshold constraints, then all candidate words in the draft block and the predicted words will be output together.

[0047] If all candidate words in the current draft block satisfy the triple threshold constraint, then the entire draft block and the predicted words are output together. The predicted words, also known as the bonus token, are generated by sampling the draft block when the model verifies that all candidate words in the entire draft block are accepted without mismatch. Taking the example above, the candidate words corresponding to the current draft block include [x1, x2, x3, x4]. If all of them pass the triple threshold constraint verification, then [x1, x2, x3, x4, argmax5] is submitted.

[0048] After verifying all candidate lexical units in the draft block with triple threshold constraints, the candidate lexical units of the draft model, the strictly greedy lexical units of the validation model, and the current consecutive submission prefixes are updated, and the next round of speculative decoding verification of lexical units continues.

[0049] It should be noted that the auditable relaxed acceptance of candidate lexical units by the verification model in this embodiment is based on the premise that the cumulative risk budget, unit value threshold, and upper limit of lexical units per draft block are all non-zero. When any of the above three threshold constraints is zero, the verification model degenerates into a strict greedy prefix verification. The verification model degradation mode is used for consistency baseline and fault protection in this embodiment.

[0050] In this embodiment, the difference in raw scores between candidate tokens calculated by the validation model and strictly greedy tokens is fully utilized to distinguish between low-risk and high-risk candidate tokens, rather than performing binarized truncation. Through complementary triple gating constraints, risk can be controlled at the request level, token level, and draft block level, and trajectory-by-trajectory auditing is achieved through state data recording. This embodiment only adds a decision layer to the validation model, without requiring training or modification of the original draft model and validation model, thus making it applicable to various types of speculative decoding systems. It supports a validation model degradation mode for easy fault protection and consistency comparison. In a parallel environment, only compact scores and decision submissions are communicated, avoiding the need to replicate the raw scores of the complete token sequence. Experiments have verified that this embodiment achieves significant workload throughput gain for various types of speculative decoding systems and validation models. Workload throughput gain measures the percentage / fold increase in throughput of the speculative decoding system relative to the baseline autoregressive decoding system under the same hardware resources and real business load. Throughput refers to the total number of tokens produced per second by the speculative decoding system.

[0051] The technical solution of this invention involves performing forward computation on each candidate lexical unit output by the draft model using a validation model to obtain strictly greedy lexical units. The local risk and unit value risk between each candidate lexical unit and its corresponding strictly greedy lexical unit are calculated. Local risk represents the inconsistency risk between the candidate lexical unit and the strictly greedy lexical unit, while unit value risk is obtained by normalizing the local risk using the suffix value following the candidate lexical unit. Each candidate lexical unit is evaluated individually. If the local risk and unit value risk of the target candidate lexical unit do not meet the threshold constraints, all candidate lexical units preceding the target candidate lexical unit are output as consecutive submission prefixes. This technical solution eliminates the need to train both the draft and validation models, is pluggable and auditable, and can relax restrictions on low-risk candidate lexical units, thereby extending the consecutive submission prefix.

[0052] Example 2 Figure 2 This is a flowchart of a speculative decoding method provided in Embodiment 2 of the present invention. Based on the above embodiments, the present invention adds a verification model screening step based on random parameter perturbation and a multi-process parallel execution step.

[0053] like Figure 2 As shown, the method includes: S210. After inputting the sample data into the draft model, determine the sample candidate words output by the draft model, and the sample strictly greedy words obtained by forward calculation of the sample candidate words by at least two candidate verification models.

[0054] Among them, at least one parameter differs among the candidate validation models.

[0055] Sample data refers to the context data after lightweight samples are input into the speculative decoding system. In this embodiment, the purpose of setting each candidate verification model is to select the optimal parameter configuration of the verification model. Therefore, in order to improve the optimization speed, lightweight samples are selected to test each candidate verification model.

[0056] The candidate validation model is obtained by applying a slight perturbation to the specified model parameters of the original validation model. It should be noted that each candidate validation model must include a candidate validation model with a perturbation intensity of 0 (at which point the candidate validation model is equal to the original validation model), so as to ensure that the original validation model can be reverted when the validation model parameters cannot be optimized.

[0057] The number of model parameters, the type of model parameters, and the intensity of the perturbation can be flexibly set according to the needs of speculative decoding, and this embodiment does not impose any restrictions.

[0058] The process by which each candidate validation model performs forward computation on the candidate lexical units to obtain the strictly greedy lexical units is the same as the process by which the validation model performs forward computation on the candidate lexical units to obtain the strictly greedy lexical units in the above embodiment, and will not be repeated in this embodiment.

[0059] S220. Determine the candidate consecutive submission prefixes obtained after each candidate verification model performs speculative decoding verification on the sample candidate words and the sample strictly greedy words.

[0060] Similarly, the process of the candidate validation model performing speculative decoding validation on the sample candidate words and sample strictly greedy words is the same as the process in the above embodiment where the validation model calculates the local risk and unit value risk of each candidate word based on the candidate words and strictly greedy words, and judges the triple threshold constraint conditions based on the local risk and unit value risk. This embodiment will not repeat the details.

[0061] S230. Based on the candidate consecutive submission prefix and / or verification speed corresponding to each candidate verification model, determine the target verification model among the candidate verification models.

[0062] In this embodiment, during the candidate validation model verification process, the throughput, consecutive submission prefix, accuracy, and risk events corresponding to each candidate validation model are recorded. For candidate validation models whose accuracy and risk events are within the required range, the target validation model is determined from among the candidate validation models based on their corresponding candidate consecutive submission prefix and / or validation speed.

[0063] Specifically, based on the actual needs of the speculative decoding request, the following can be selected: prioritizing consecutive submission prefix length and choosing the candidate verification model with the largest average consecutive submission prefix length as the target verification model; or prioritizing verification speed and choosing the candidate verification model with the largest average verification speed as the target verification model; or normalizing and weighting the average consecutive submission prefix length and the average verification speed to obtain a comprehensive value, and choosing the candidate verification model with the largest comprehensive value as the target verification model.

[0064] S240. Determine each candidate lexical unit output by the draft model, and the strictly greedy lexical unit obtained by the target validation model through forward calculation of the candidate lexical units.

[0065] S250. Based on each candidate lexical unit and the strictly greedy lexical unit corresponding to each candidate lexical unit, determine the local risk and unit value risk corresponding to each candidate lexical unit.

[0066] S260. If, based on the local risk and unit value risk of the target candidate word, it is determined that the threshold constraint is not met, then all candidate words preceding the target candidate word will be output as consecutive submission prefixes.

[0067] The processes S240-S260 have been described in the above embodiments and will not be repeated here.

[0068] The technical solution in this embodiment does not require retraining the validation model or draft model, nor does it modify existing model checkpoints. Its primary deployment location is after the validation model completes teacher-forced forward computation and before candidate lexical submission and cache update. Therefore, it can be used as a modular validation decision layer, supporting single-card, multi-card, single-machine multi-card, or multi-machine tensor parallel inference engines.

[0069] In a specific application scenario, when the technical solution of this embodiment is used as the single-machine verification model side of the modular verification decision layer accessing the confidence scheduling speculative decoding framework (semi-autoregressive speculative decoding), the strict greedy prefix verification of the original verification model is replaced with the auditable relaxed verification of this embodiment. Using the technical solution of this embodiment, with a cumulative risk budget of 8, a unit value threshold of 0.25, and a single draft block term limit of 2, multiple speculative decoding experiments were conducted, all of which yielded significant workload throughput gains (average +4.525%).

[0070] In another specific application scenario, when the technical solution of this embodiment is used as the fusion verification model side of the modular verification decision layer connected to the efficient adaptive latent variable speculative decoding system, the native rejection sampler is wrapped, and the original scores of the strictly greedy tokens of the verification model are directly reused without increasing the forward pass count of the verification model. The cumulative risk budget is stored on the device side, and the fusion GPU (Graphics Processing Unit) kernel can be used to complete the max, gather, risk accumulation constraints, and continuous submission prefix selection of the efficient adaptive latent variable speculative decoding system. Using the technical solution of this embodiment, the cumulative risk budget is set to 12, the unit value threshold is set to 0.5, and the single draft block term limit is set to 2. Multiple speculative decoding experiments were conducted, and significant workload throughput gains (average +4.555%) were obtained.

[0071] In another specific application scenario, this embodiment also supports multi-process parallel execution. Specifically, each process determines its local maximum original score, the candidate lexical identifier corresponding to the local maximum original score, and the global draft original score. Through the coordination process, based on the local maximum original score of each process, the candidate lexical identifier corresponding to the local maximum original score, and the global draft original score, it is determined whether the threshold constraint condition is met. Through the coordination process, the continuous submission prefix length, recovered lexical or predicted lexical, the local risk increment of this round, and the remaining risk budget are broadcast until it is determined that the threshold constraint condition is not met.

[0072] Each process (rank) holds a logit (raw score) layer shard. For its local logit layer shard, it calculates the local maximum raw score for each position and its corresponding candidate lemma identifier. Through distributed maximum value calculation, multiple processes collaboratively solve for the global draft raw score maximum value. A deterministic tie-breaking decision is used to handle raw score ties.

[0073] For each process, a triplet is generated consisting of the local maximum raw score, the candidate lexical identifier corresponding to the local maximum raw score, and the global draft raw score. A coordination process synchronizes these triplets across processes and verifies threshold constraints based on the triplets from each process. Simultaneously, the coordination process is responsible for broadcasting the continuous submission prefix length, recovered or predicted lexical terms, local risk increment for this round, and remaining risk budget among the processes.

[0074] The technical solution of this embodiment is compatible with pipelined parallel or multi-machine deployment, and does not generate additional video memory and communication overhead.

[0075] Example 3 Figure 3 This is a schematic diagram of a speculative decoding device provided in Embodiment 3 of the present invention. The device is deployed in a verification model, such as... Figure 3 As shown, the device includes: The lexical determination module 310 is used to determine each candidate lexical output by the draft model, as well as the strictly greedy lexical obtained by the verification model through forward calculation of the candidate lexical. The risk calculation module 320 is used to determine the local risk and unit value risk corresponding to each candidate word based on each candidate word and the strictly greedy word corresponding to each candidate word. Among them, local risk is used to represent the inconsistency risk between candidate lexical units and strictly greedy lexical units, and unit value risk is obtained by normalizing the local risk through the suffix value after the candidate lexical unit; The continuous submission prefix determination module 330 is used to output each candidate word preceding the target candidate word as a continuous submission prefix if it is determined that the threshold constraint condition is not met based on the local risk and unit value risk of the target candidate word.

[0076] The technical solution of this invention involves performing forward computation on each candidate lexical unit output by the draft model using a validation model to obtain strictly greedy lexical units. The local risk and unit value risk between each candidate lexical unit and its corresponding strictly greedy lexical unit are calculated. Local risk represents the inconsistency risk between the candidate lexical unit and the strictly greedy lexical unit, while unit value risk is obtained by normalizing the local risk using the suffix value following the candidate lexical unit. Each candidate lexical unit is evaluated individually. If the local risk and unit value risk of the target candidate lexical unit do not meet the threshold constraints, all candidate lexical units preceding the target candidate lexical unit are output as consecutive submission prefixes. This technical solution eliminates the need to train both the draft and validation models, is pluggable and auditable, and can relax restrictions on low-risk candidate lexical units, thereby extending the consecutive submission prefix.

[0077] Based on the above embodiments, optionally, the risk calculation module 320 includes: The local risk calculation unit is used to take the difference between the original score of the candidate word and the original score of the strictly greedy word corresponding to the candidate word as the local risk of the candidate word.

[0078] Based on the above embodiments, optionally, the suffix value is represented by the number of subsequent candidate lexical units following the candidate lexical unit; Alternatively, it can be expressed using the following formula: ,in, Let K represent the suffix value of the i-th candidate lexical unit, and K represent the number of candidate lexical units. This represents the probability of accepting the condition.

[0079] Based on the above embodiments, optionally, the continuous submission prefix determination module 330 includes: The threshold constraint judgment unit is used to determine whether the threshold constraint is satisfied if the cumulative value of the local risk of the target candidate word and all candidate words before the target candidate word is less than or equal to the cumulative risk budget, the unit value risk of the target candidate word is less than or equal to the unit value threshold, and the number of candidate words with non-zero local risk in the draft block where the target candidate word is located is less than or equal to the upper limit of word words in a single draft block; otherwise, the threshold constraint is not satisfied.

[0080] Optionally, based on the above embodiments, the apparatus further includes: The first draft block output unit is used to stop accepting subsequent candidate words if it is determined that the target candidate word is located in the draft block, and replace the target candidate word with a strictly greedy word corresponding to the target candidate word as a recovery word, and output the draft block where the target candidate word is located. The second draft block output unit is used to output all candidate words and predicted words in the draft block if all candidate words in the draft block meet the threshold constraint conditions.

[0081] Optionally, based on the above embodiments, the apparatus further includes: The sample word determination module is used to determine each sample candidate word output by the draft model after the sample data is input into the draft model, as well as the sample strictly greedy word obtained by forward calculation of the sample candidate words by at least two candidate verification models. Among them, at least one parameter differs among the candidate validation models; The candidate consecutive submission prefix determination module is used to determine the candidate consecutive submission prefixes obtained by each candidate verification model after speculative decoding verification of sample candidate words and sample strictly greedy words. The target validation model determination module is used to determine the target validation model from among the candidate validation models based on the candidate consecutive submission prefix and / or validation speed corresponding to each candidate validation model.

[0082] Based on the above embodiments, optionally, when multiple processes are executed in parallel, the apparatus further includes: The triplet determination module is used by each process to determine the local maximum raw score, the candidate lexical identifier corresponding to the local maximum raw score, and the global draft raw score. The threshold constraint judgment module is used to determine whether the threshold constraint is met by coordinating the process, based on the local maximum original score of each process, the candidate word identifier corresponding to the local maximum original score, and the global draft original score. The broadcast module is used to broadcast the consecutive submission prefix length, recovered or predicted terms, local risk increment for the current round, and remaining risk budget through the coordination process until it is determined that the threshold constraint is not met.

[0083] The speculative decoding device provided in this embodiment of the invention can execute the speculative decoding method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0084] Example 4 Figure 4A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0085] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as speculative decoding methods.

[0088] In some embodiments, the speculative decoding method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the speculative decoding method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the speculative decoding method by any other suitable means (e.g., by means of firmware).

[0089] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0090] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable speculative decoding device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0091] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0093] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0094] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0095] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0096] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A speculative decoding method, characterized in that, The method is executed by a validation model, and the method includes: Determine each candidate lexical unit output by the draft model, and verify the strictly greedy lexical units obtained by the model through forward computation of the candidate lexical units; Based on each candidate word and the strictly greedy word corresponding to each candidate word, determine the local risk and unit value risk corresponding to each candidate word. Among them, local risk is used to represent the inconsistency risk between candidate lexical units and strictly greedy lexical units, and unit value risk is obtained by normalizing the local risk through the suffix value after the candidate lexical unit; If the threshold constraint is not met based on the local risk and unit value risk of the target candidate word, then all candidate words preceding the target candidate word will be output as consecutive submission prefixes.

2. The method according to claim 1, characterized in that, Based on each candidate lexical unit and its corresponding strictly greedy lexical unit, determine the local risk associated with each candidate lexical unit, including: The difference between the original score of the candidate word and the original score of the corresponding strictly greedy word is taken as the local risk of the candidate word.

3. The method according to claim 1, characterized in that, The suffix value is represented by the number of subsequent candidate lexical units following the candidate lexical unit; Alternatively, it can be expressed using the following formula: ,in, Let K represent the suffix value of the i-th candidate lexical unit, and K represent the number of candidate lexical units. This represents the probability of accepting the condition.

4. The method according to claim 1, characterized in that, Based on the local risk and unit value risk of the target candidate lexical units, the conditions that do not meet the threshold constraints are determined, including: If the cumulative value of the local risk of the target candidate word and all candidate words before the target candidate word is less than or equal to the cumulative risk budget, the unit value risk of the target candidate word is less than or equal to the unit value threshold, and the number of candidate words with non-zero local risk in the draft block where the target candidate word is located is less than or equal to the upper limit of word words in a single draft block, then the threshold constraint condition is satisfied. Otherwise, it is determined that the threshold constraint condition is not met.

5. The method according to claim 1, characterized in that, After determining that the threshold constraint is not met, the following steps are also included: If it is determined that the target candidate word is located in the draft block, then the acceptance of subsequent candidate words is stopped. Furthermore, the target candidate word is replaced by the strictly greedy word corresponding to the target candidate word as the recovery word, and the draft block containing the target candidate word is output. The method further includes: If all candidate words in the draft block meet the threshold constraints, then all candidate words in the draft block and the predicted words will be output together.

6. The method according to claim 1, characterized in that, The method further includes: After inputting the sample data into the draft model, the draft model outputs each sample candidate word, and at least two candidate validation models perform forward calculations on the sample candidate words to obtain the sample strictly greedy words. Among them, at least one parameter differs among the candidate validation models; The candidate consecutive submission prefixes are determined after each candidate validation model performs speculative decoding validation on the sample candidate words and the sample strictly greedy words. The target validation model is determined from among the candidate validation models based on the candidate consecutive submission prefix and / or validation speed corresponding to each candidate validation model.

7. The method according to claim 5, characterized in that, When the method is executed in parallel by multiple processes, the method further includes: Each process determines its local maximum raw score, the candidate lexical identifier corresponding to its local maximum raw score, and the global draft raw score. By coordinating the process, based on the local maximum original score of each process, the candidate lexical identifier corresponding to the local maximum original score, and the global draft original score, it is determined whether the threshold constraint conditions are met. The coordination process broadcasts the consecutive submission prefix length, recovered or predicted terms, local risk increment for the current round, and remaining risk budget until it is determined that the threshold constraint is not met.

8. A speculative decoding device, characterized in that, The device is deployed on the verification model, and the device includes: The lexical determination module is used to determine each candidate lexical output by the draft model, as well as the strictly greedy lexical obtained by the verification model through forward calculation of the candidate lexical. The risk calculation module is used to determine the local risk and unit value risk of each candidate word based on each candidate word and the strictly greedy word corresponding to each candidate word. Among them, local risk is used to represent the inconsistency risk between candidate lexical units and strictly greedy lexical units, and unit value risk is obtained by normalizing the local risk through the suffix value after the candidate lexical unit; The continuous submission prefix determination module is used to output each candidate word preceding the target candidate word as a continuous submission prefix if the threshold constraint is not met based on the local risk and unit value risk of the target candidate word.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speculative decoding method as described in any one of claims 1-7.

10. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the speculative decoding method as described in any one of claims 1-7.

Images