Selective speculative decoding

By combining selective speculative decoding logic with a draft model, the problems of latency and cost in the inference process of machine learning models are solved, and more efficient and accurate output generation is achieved. In particular, when the accuracy of the draft model decreases, unnecessary speculative decoding calculations are avoided.

CN123125604APending Publication Date: 2026-11-17MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202580028779.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-12-02
Filing Date
2025-03-13
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing machine learning models suffer from inference costs and latency issues during the inference process, especially when using larger models. Inference decoding methods may fail to accurately estimate the output, leading to increased latency and higher costs.

Method used

A selective speculative decoding logic is adopted. By combining a draft model and a main machine learning model, speculative decoding is selectively used to generate the output, reducing unnecessary speculative decoding computations. The first part of the output is generated using the draft model, and the second part is generated on the main machine learning model. The process is optimized through parallel verification and similarity checks.

Benefits of technology

It effectively reduces inference latency and cost, and improves the accuracy and efficiency of output generation. In particular, it avoids unnecessary speculative decoding calculations and saves resources when the accuracy of the draft model decreases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125604A_ABST
    Figure CN123125604A_ABST
Patent Text Reader

Abstract

The computing system (10) includes one or more processing devices (12) configured to receive a cue word (20). The one or more processing devices lexically represent the cue word to obtain a lexically represented cue word (24) including an input lexical (26). Based at least in part on the input lexical, the one or more processing devices compute an output (50) including an output lexical over a plurality of autoregressive generation iterations (46). The computed output includes: performing selective speculative decoding logic, at least in part on the context (30) including the lexically represented cue word and a previous output lexical sequence (32), to select a first part (54) and a second part (56) of the output. The computed output also includes computed the first part via speculative decoding using one or more draft models (42) and computed the second part at the main machine learning model (44) without speculative decoding. The one or more processing devices transmit the output to an additional computation process (60).
Need to check novelty before this filing date? Find Prior Art

Description

Background Technology

[0001] In recent years, machine learning models such as Large Language Models (LLMs) and Large Multimodal Models (LMMs) have included increasingly more layers and parameters to allow them to perform more complex tasks. When performing inference at a machine learning model, the total number of computations tends to increase with the number of parameters. Furthermore, processing time tends to increase with the number of layers. Therefore, the increase in model size in recent years has brought a trade-off: larger models with more advanced capabilities also tend to have higher inference costs and latency. Summary of the Invention

[0002] According to one aspect of this disclosure, a computational system is provided, including one or more processing devices configured to receive a cue word. The one or more processing devices are further configured to lexically represent the cue word to obtain a lexically represented cue word comprising a plurality of input lexical units. Based at least in part on the input lexical units, the one or more processing devices are further configured to compute an output comprising a plurality of output lexical units over a plurality of autoregressive generation iterations. Computing the output includes: in one or more autoregressive generation iterations, performing selective speculative decoding logic, at least in part on a context comprising the lexically represented cue word and a previous sequence of output lexical units, to select a first portion and a second portion of the output. The computational output also includes computing the first portion of the output via speculative decoding using one or more draft models. The computational output also includes computing the second portion of the output at a main machine learning model without using speculative decoding. The one or more processing devices are further configured to transmit the output to an additional computational process.

[0003] This synopsis is provided to introduce some concepts in a simplified form, which will be further described in the detailed embodiments below. This synopsis is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to the implementation of solutions to any or all the shortcomings mentioned in any part of this disclosure. Attached Figure Description

[0004] Figure 1 A computing system according to an example embodiment is illustrated, in which one or more processing devices are configured to perform selective speculative decoding.

[0005] Figure 2 It schematically shows the following based on Figure 1 An example of a computational system when processing context at the selective speculative decoding logic.

[0006] Figure 3 It schematically shows the following based on Figure 1An example computing system in which one or more processing devices are configured to execute multiple draft machine learning models.

[0007] Figure 4 It schematically shows the following based on Figure 1 An example of a computational system when performing parallel verification during speculative decoding.

[0008] Figure 5 It schematically shows the following based on Figure 1 Examples of computational systems, in which one or more draft models include one or more deterministic policies.

[0009] Figure 6A It schematically shows the following based on Figure 1 An example computing system in which one or more processing devices are further configured to estimate the expected value of performing speculative decoding.

[0010] Figure 6B It schematically shows the following based on Figure 6A Examples of computational systems during the training of complexity classification machine learning models.

[0011] Figure 7 It schematically shows the following based on Figure 1 Example computing systems in which the user inputs a speculative decoding selection of the user input via a graphical user interface.

[0012] Figure 8A It shows according to Figure 1 The flowchart illustrates an example of a method for a computational system to generate a response to a cue word using selectively activated speculative decoding.

[0013] Figure 8B-8H This shows what can be executed in some examples. Figure 8A Additional steps to the method.

[0014] Figure 9 It shows where it can be instantiated Figure 1 A schematic diagram of an example computing environment for a computing system. Detailed Implementation

[0015] In some existing machine learning systems, speculative decoding is used to reduce latency in processing some inputs. In existing methods of speculative decoding, a smaller machine learning model (in terms of the number of layers) is used to generate draft terms that approximate the output of a larger machine learning model. The smaller model has lower latency than the larger machine learning model. After computing the draft terms, the larger machine learning model is used to check the accuracy of the approximation. This verification can be performed on the draft terms in parallel. Therefore, the verification latency can be reduced compared to autoregressive generation of output terms at the larger machine learning model, where output terms are computed sequentially. In examples where the smaller model accurately approximates the output of the larger model, speculative decoding can correspondingly reduce inference latency.

[0016] In existing methods, speculative decoding is performed during the generation of the entire output of the machine learning model. However, during some tasks, a smaller machine learning model may fail to accurately estimate the output of a larger machine learning model. When the smaller machine learning model's estimate is inaccurate, the larger machine learning model is used instead to autoregressively generate output terms. Therefore, in such examples, speculative decoding may increase inference costs and latency rather than reduce them due to the additional computation performed at the smaller model.

[0017] To address the aforementioned shortcomings of current speculative decoding methods, a computational system 10 is provided, such as... Figure 1 This is an illustrative example. (Using...) Figure 1 The computational system 10 uses selective speculative decoding logic 40 to determine that speculative decoding is selectively applied to the cue word 20. Therefore, the computational system 10 can reduce the latency associated with speculative decoding while avoiding speculative decoding in scenarios unlikely to result in inference acceleration.

[0018] The computing system 10 includes one or more processing devices 12 and one or more memory devices 14. The one or more processing devices 12 may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), neural processing units (NPUs), and / or other types of hardware accelerators. The one or more memory devices 14 may include, for example, one or more volatile memory devices and one or more non-volatile memory devices.

[0019] In some examples, one or more processing devices 12 and / or one or more memory devices 14 may include multiple physical components distributed among multiple different physical computing devices. For example, one or more processing devices 12 and / or one or more memory devices 14 may be included in a networked system of multiple physical computing devices located in a data center. Part of the functionality of one or more processing devices 12 and / or one or more memory devices 14 may additionally or alternatively be performed at one or more client computing devices.

[0020] One or more processing devices 12 are configured to receive the prompt 20. For example, the prompt 20 may be received in natural language. In some examples, the prompt 20 may be entered by a user at a user interface. In other examples, the prompt 20 may be generated programmatically at another computational process and may be received from that other computational process via an application programming interface (API).

[0021] One or more processing devices 12 are also configured to execute a lexer 22 to compute a lexicalized cue word 24 based at least in part on the cue word 20. The lexicalized cue word 24 includes a plurality of input lexical units 26, which may, for example, indicate a word, a part of a word, or other characters such as numbers or punctuation marks. The lexer 22 is thus configured to encode the cue word 20 in a form that can be used as input to a machine learning model.

[0022] Based at least in part on the input lexicon 26, one or more processing devices 12 are also configured to compute an output 50 comprising a plurality of output lexicons 52. The output 50 is computed over a plurality of autoregressive generation iterations 46, in which corresponding output lexicons 52 are generated. Figure 1 As shown in the example, the lexicalized cue word 24 is included in the context 30 along with the previously output lexical sequence 32, which includes each previously output lexical 34 generated at the corresponding previously executed autoregressive generation iteration 46. At each autoregressive generation iteration 46, the current context 30 is used as input.

[0023] When output 50 is generated, one or more processing devices 12 are also configured to execute selective speculative decoding logic 40. Selective speculative decoding logic 40 programmatically determines when speculative decoding is used. At selective speculative decoding logic 40, one or more processing devices 12 are configured to select a first portion 54 and a second portion 56 of output 50 during output generation. For each output term in output term 52, one or more processing devices 12 are configured to determine whether the output term 52 is included in the first portion 54 or the second portion 56 of output 50 before generating that output term 52.

[0024] One or more processing devices 12 are also configured to compute a first portion 54 of the output 50 via speculative decoding and a second portion 56 of the output 50 without using speculative decoding. The first portion 54 is computed using one or more draft models 42, while the second portion 56 is computed at the main machine learning model 44. The main machine learning model 44 may be, for example, an LLM or LMM. The one or more draft models 42 may also be machine learning models. Additionally or alternatively, as discussed in further detail below, the one or more draft models 42 may be deterministic policies.

[0025] After generating output 50, one or more processing devices 12 are further configured to transmit output 50 to an additional computation process 60. For example, the additional computation process 60 may be a graphical user interface (GUI) where one or more processing devices 12 are configured to display output 50 to a user. As another example, one or more processing devices 12 may be configured to transmit output 50 to a compiler where output 50 is compiled into assembly-level instructions.

[0026] Figure 2 The computational system 10 is schematically illustrated when context 30 is processed at selective speculative decoding logic 40. Figure 2 In the example, context 30 includes multiple lexical batches 36, each lexical batch including one or more lexical units. The lexical units included in batch 36 can be either input lexical units 26 included in lexicalization prompt 24 or previously output lexical units 34 included in previously output lexical unit sequence 32. Each lexical batch 36 can each include the same number of lexical units.

[0027] exist Figure 2 In one example, one or more processing devices 12 are configured to perform selective speculative decoding logic 40 for each lexical batch 36. Therefore, one or more processing devices 12 are configured to determine whether to activate or deactivate speculative decoding at predefined intervals (based on the number of lexical units). In other examples, one or more processing devices 12 may be configured to perform selective speculative decoding logic 40 at intervals with other predefined batch numbers (such as every two lexical batches 36 or every three lexical batches 36).

[0028] like Figure 2 As shown in the example, the first part 54 and the second part 56 may include a set of output terms 52 that are at least partially discontinuous within the output 50. Figure 2 In the example, one or more processing devices 12 start using speculative decoding to generate output token 52, switch to generating output token 52 at the main machine learning model 44 without speculative decoding, switch back to using speculative decoding, and switch back to not using speculative decoding. Therefore, in Figure 2 In the example, the first part 54 and the second part 56 are both discontinuous.

[0029] In some examples, such as Figure 2 As shown, the selective speculative decoding logic 40 is also configured to select the number of draft models 42 used in speculative decoding and to select whether to use speculative decoding. Figure 2 In the example, one or more processing devices 12 begin using a first number of draft models 42 to generate output 50. When one or more processing devices 12 switch back to generating output 50 via speculative decoding after using the main machine learning model 44, the one or more processing devices 12 are configured to generate output tokens 52 using a different number of draft models 42. Therefore, one or more processing devices 12 are configured to compute a first part 54 at multiple draft models 42, and at the selective speculative decoding logic 40, during the generation of the first part 54, modify the number of draft models 42 used to compute the first part 54.

[0030] For example, the number of draft models 42 can be changed to dynamically adjust for changes in the complexity of the lexical generation task. For instance, the number of draft models 42 can be increased when the selective inference decoding logic 40 estimates that accurately generating subsequent parts of output 50 might utilize a model capability level between that of a single draft model 42 and that of the main machine learning model 44. Conversely, the selective inference decoding logic 40 can reduce the number of draft models 42 when the estimation complexity of subsequent parts of output 50 decreases.

[0031] As another example, different draft models 42 can be specialized for different tasks. In such an example, selective speculative decoding logic 40 can identify the subject area of ​​context 30 and select those one or more draft models 42 based on the subject area associated with them. For example, selective speculative decoding logic 40 can identify a specific programming language in context 30 and select draft models 42 trained to generate code in that programming language. In some examples, instead of increasing or decreasing the number of draft models 42 used in speculative decoding, selective speculative decoding logic 40 can replace one or more other draft models 42 with one or more draft models 42 without changing the total number.

[0032] Figure 3 The computational system 10 is illustrated schematically in an example of one or more draft models 42, including multiple draft machine learning models. Figure 3 The draft models 42 shown in the examples each include multiple parameters 43. Furthermore, Figure 3The main machine learning model 44 is shown, which includes multiple parameters 45. Draft models 42 have a corresponding number of draft model parameters 47, which is less than the number of main model parameters 48 of the main machine learning model 44. Therefore, each draft model 42 can have a lower corresponding inference cost than the main machine learning model 44.

[0033] One or more processing devices 12 are also configured to compute the first part 54 at least in part by generating corresponding draft lexical units 70 at the draft model 42. The draft machine learning model is configured to generate a corresponding draft lexical sequence 72 of draft lexical units 70 as a suggested continuation of the context 30. When the second part 56 is generated, one or more processing devices 12 are alternatively configured to compute one or more main model output lexical units 78 at the main machine learning model 44 at least in part based on the context 30.

[0034] During the speculative decoding process, one or more processing devices 12 are also configured to calculate one or more similarity values ​​74 between draft lemmas 70. The one or more similarity values ​​74 can be calculated lemmas-by-lemmas between the draft lemmas 70 generated at each draft model 42. Alternatively, the similarity values ​​74 can be calculated between the draft lemma sequences 72. The similarity value 74 can be, for example, a cosine similarity value, or alternatively, some other similarity function can be used to calculate it.

[0035] exist Figure 3 In some examples, one or more processing devices 12 are also configured to determine that one or more similarity values ​​74 are below a predefined similarity threshold 76. In response to determining that one or more similarity values ​​74 are below the predefined similarity threshold 76, one or more processing devices 12 are also configured to switch from generating output words 52 at multiple draft models 42 to generating output words 52 at the main machine learning model 44. When one or more processing devices 12 determine that the predictions of the draft models 42 deviate from each other, as indicated by the similarity value 74 falling below the predefined similarity threshold 76, the selective speculative decoding logic 40 accordingly disables speculative decoding. In some examples, where multiple similarity values ​​74 are computed at the selective speculative decoding logic 40, one or more processing devices 12 may be configured to disable speculative decoding in response to determining that any of the similarity values ​​74 are below the predefined similarity threshold 76. In other examples, one or more processing devices 12 may be configured to disable speculative decoding in response to determining that all similarity values ​​74 or some other number of similarity values ​​74 (e.g., more than half) are below the predefined similarity threshold 76.

[0036] In some examples, during the computation of the second part 56, one or more processing devices 12 are configured to execute selective speculative decoding logic 40 to determine whether speculative decoding is activated. In such examples, one or more processing devices 12 may be configured to generate draft lexical sequences 72 at a predefined interval 77 (e.g., every five lexical batches 36) at the draft model 42. One or more processing devices 12 may also be configured to compute similarity values ​​74 for those draft lexical sequences 72 and determine whether the similarity value 74 is higher than a predefined similarity threshold 76. In response to determining that the similarity value 74 is higher than the predefined similarity threshold 76, one or more processing devices 12 may also be configured to reactivate speculative decoding. One or more processing devices 12 may be configured accordingly to check at predefined intervals 77 whether to switch to using speculative decoding.

[0037] Figure 4 The computational system 10 is schematically illustrated when parallel verification is performed during speculative decoding. Figure 4 In the example, such as in Figure 3 In the example, one or more processing devices 12 are configured to compute the first part 54 at least in part by generating the corresponding draft lexicon 70 at the draft model 42. Figure 4 The calculation of the corresponding draft term 70 at draft model 42 is shown during three autoregressive generation iterations 46. During those autoregressive generation iterations 46, draft term 70 is sampled from the draft probability distribution 71 calculated at draft model 42.

[0038] At the main machine learning model 44, one or more processing devices 12 are also configured to perform a parallel validation check 80 on the draft lexicon 70. During the parallel validation check 80, one or more processing devices 12 are configured to generate a corresponding main probability distribution 82 associated with the draft lexicon 70 at the main machine learning model 44. Furthermore, during the parallel validation check 80, one or more processing devices 12 are also configured to compare the main probability distribution 82 with the draft probability distribution 71 and / or the draft lexicon 70. One or more processing devices 12 may be configured, for example, to perform greedy decoding, approximate greedy decoding, or kernel sampling. In the example of generating multiple draft lexicon sequences 72, one or more processing devices 12 may be configured to perform lexicon tree validation while performing the parallel validation check 80.

[0039] exist Figure 4In the example, one or more processing devices 12 are also configured to determine that one or more draft lexical units 70 fail the parallel validation check 80. In response to determining that one or more draft lexical units 70 fail the parallel validation check 80, one or more processing devices 12 are also configured to disable speculative decoding and switch from generating output lexical units 52 at multiple draft models 42 to generating output lexical units 52 at the main machine learning model 44. One or more processing devices 12 can be configured to disable speculative decoding in the computation of one or more subsequent output lexical units 52, thereby avoiding the cost associated with executing draft models 42, instead of using the main machine learning model 44 to replace the failing draft lexical units 70 at various levels as in conventional parallel validation methods.

[0040] In some examples, such as Figure 5 As shown, in addition to or as an alternative to one or more draft machine learning models, one or more draft models 42 may include one or more deterministic policies. Each of these deterministic policies includes one or more deterministic rules 90 that specify the corresponding draft lexicons 70 to be generated when different types of input are received in the context 30. In some examples, one or more deterministic rules 90 may directly output one or more draft lexicons 70.

[0041] In other examples, one or more processing devices 12 are configured to execute one or more deterministic strategies to compute the first part 54 at least in part by performing a database lookup operation 92. When performing the database lookup operation 92, one or more processing devices 12 are configured to retrieve one or more database records 96 from database 94. In some examples, the one or more database records 96 may be one or more draft lexical units 70. In other examples, one or more processing devices 12 may also be configured to post-process the one or more database records 96, such as lexicalizing the one or more database records 96 at a lexer 22 to obtain one or more draft lexical units 70. Parallel validation can then be performed on the draft lexical units 70 to generate the first part 54, as described above.

[0042] One or more processing devices 12 are configured to generate a first part 54 of output 50 by retrieving one or more database records 96 from database 94 and using those database records 96 to compute draft lexicons 70, while still using the main machine learning model 44 to check the consistency of draft lexicons 70 with context 30. For example, a database lookup operation 92 can be used when selective inference decoding logic 40 identifies a predefined pattern with deterministic completion in context 30. In such an example, by performing the database lookup operation 92, one or more processing devices 12 can avoid the costs associated with executing one or more draft machine learning models.

[0043] Figure 6A A computational system 10 in one example is schematically illustrated, wherein at selective speculative decoding logic 40, one or more processing devices 12 are further configured to estimate an expected value 104 for performing speculative decoding. The one or more processing devices 12 are further configured to determine whether to use speculative decoding based at least in part on the expected value 104. The expected value 104 is computed at an expected value module 100, which is included in the selective speculative decoding logic 40 and includes a predefined value function 102. Furthermore, the one or more processing devices 12 can be configured to compute an expected value 106 for not using speculative decoding but instead computed at the main machine learning model 44 for the output term 52. By determining whether the expected value 104 or the expected value 106 is higher, the selective speculative decoding logic 40 can determine whether to use speculative decoding.

[0044] A predefined value function 102 can encode an estimate of computing resource utilization when generating output 50. For example, a weighted estimate of latency, processing device usage, memory bandwidth usage, and / or energy consumption can be encoded in the predefined value function 102. In some examples, the predefined value function 102 specifies the expected information value (EVI) associated with draft lexicon 70.

[0045] In some examples, one or more processing devices 12 are configured to use hardware attribute data 108 of computing system 10 as input to a predefined value function 102. Hardware attribute data 108 may indicate attributes of one or more processing devices 12 and / or one or more memory devices 14 used to execute the main machine learning model 44 and one or more draft models 42. In some examples, network topology data of the hardware devices included in computing system 10 may be indicated in the hardware attribute data 108. Therefore, at the predefined value function 102, one or more processing devices 12 may be configured to use hardware attribute data 108 to calculate quantities such as latency and processing device usage.

[0046] At the expectation module 100, one or more processing devices 12 may also be configured to compute a task complexity estimate 110 at least in part based on context 30. The task complexity estimate 110 may be a classification at least in part based on context 30 of whether speculative decoding will produce an estimate of a continuous context 30 that has passed parallel validation. One or more processing devices 12 may be configured to compute the task complexity estimate 110 at a complexity classification model 112, which may be a deterministic model or a machine learning model.

[0047] Figure 6B An example of training complexity classification model 112 when it is a machine learning model is shown. Figure 6B As illustrated in the example, the complexity classification model 112 can be trained via supervised learning using training data 114, which includes training context 116 and an indication 118 of whether the completions of those training contexts 116 computed at the draft model 42 have passed parallel validation. The complexity classification model 112 is thus trained to classify context 30 based on whether parallel validation is likely to succeed. The complexity classification model 112 can be a lightweight machine learning model with lower latency and processing costs (e.g., in terms of processing power and energy usage) than the main machine learning model 44 and one or more draft models 42. Therefore, performing the complexity classification model 112 while computes the expected value 104 using speculative decoding can result in processing cost savings.

[0048] In some examples, at least a portion of the selective speculative decoding logic 40 can be specified by user input. Figure 7 The illustration schematically shows a computing system 10 in which the user inputs a guessing decoding selection user input 120 via GUI 130. The user can also use GUI 130 to input prompts 20. Figure 7 In the example, prompt 20 is "Generate the chess notation of the example game in the RioGambitAccepted variant of RuyLopez's opening".

[0049] exist Figure 7 The example input for speculative decoding selection user input 120 includes model specification 122 and speculative decoding activation rule 124. In model specification 122, the user specifies which models are used as the master machine learning model 44 and one or more draft models 42, as well as the number of draft models 42. Figure 7 In the example, the user instructs the computing system 10 to use GPT-4 as the master machine learning model 44, use GPT-3.5 as the draft model 42, and use both draft models 42 when performing speculative decoding.

[0050] The speculative decoding selection user input 120 indicates one or more speculative decoding activation rules 124 associated with the prompt word 20. At the selective speculative decoding logic 40, one or more processing devices 12 are also configured to select the first part 54 and the second part 56 at least partially by applying one or more speculative decoding activation rules 124. In the speculative decoding activation rule 124, the user-indicated selective speculative decoding logic 40 has a low threshold for switching to using the master machine learning model 44 without speculative decoding. This low threshold may correspond to a high value of a predefined similarity threshold 76 below which one or more processing devices 12 are configured to disable speculative decoding, as referenced above. Figure 3 As discussed above. In other examples, the user can directly specify a predefined similarity threshold of 76 in the speculative decoding activation rule 124. Figure 7 In the example, speculative decoding activation rule 124 also includes instructions for checking whether speculative decoding is activated or deactivated at each lexical batch 36.

[0051] In some examples, users can enter custom-written code that defines at least a portion of the selective speculative decoding logic 40, instead of selecting a predefined option when specifying the selective decoding activation rule 124. For example, a user can write code that specifies a predefined value function 102 and enter that code into the selective speculative decoding logic 40.

[0052] exist Figure 7 In the example, when output 50 is generated, one or more processing devices 12 are also configured to assign corresponding output lexical metadata 126 to output lexical 52. The output lexical metadata 126 of each output lexical 52 indicates the main machine learning model 44 or one or more draft models 42 used to generate that output lexical 52. Thus, the output lexical metadata 126 indicates the first part 54 and the second part 56 of output 50.

[0053] Figure 7 The example shown, output 50, begins with the output word sequence “1. e4e52. Nf3Nc63. Bb5a64.Ba4Nf65. O-ONxe46.”. This sequence is generated at the draft model 42 as part of the first part 54 of output 50. This output word sequence appears multiple times in the training data of both the draft model 42 and the main machine learning model 44, serving as the move sequence defining the RioGambitAccepted variant of RuyLopez. Therefore, the above output word sequence is accurately generated by the draft model 42 via speculative decoding.

[0054] Output 50 continues to output the word sequence "Re1d5", which is alternatively generated at the main machine learning model 44 as part of the second part 56. When these output words 52 are generated, one or more processing devices 12 have moved away from the portion of output 50 accurately predicted by the draft model 42, as indicated, for example, by a similarity value 74 falling below a predefined similarity threshold 76. Alternatively, one or more processing devices 12 are configured to compute these words at the main machine learning model 44. Thus, one or more processing devices 12 utilize the higher capabilities of the main machine learning model 44 to produce output words 52 that are more likely to represent valid and reasonable chess moves.

[0055] exist Figure 7 In the example, one or more processing devices 12 are also configured to reactivate speculative decoding to compute the term "7.". Since "7." and other round number indicators follow a consistent pattern, these round number indicators can be accurately generated via speculative decoding. Following this term, one or more processing devices 12 are also configured to compute "Nxe5Be7" at the main machine learning model 44. Therefore, the output 50 interleaves the output term 52 computed at the draft model 42 and the output term 52 computed at the main machine learning model 44.

[0056] One or more processing devices 12 are also configured to present a graphical representation 132 of the output 50 and the output lexical metadata 126 to a user at a GUI 130. Figure 7 In the example, graphical representation 132 includes highlighting applied to the first portion 54 and the second portion 56, making the first portion 54 and the second portion 56 visually distinguishable. For example, the first portion 54 and the second portion 56 may be highlighted in different colors. Text labels may be additionally or alternatively included in graphical representation 132. In an example where the group of draft models 42 changes during the generation of output 50, as a result of adding, removing, or replacing one or more draft models 42, these changes in the group of draft models 42 may also be graphically represented at GUI 130, such as assigning different colored highlighting to areas of the first portion 54 generated at one or more draft models 42 in different groups. Graphical representation 132 accordingly indicates the source of the different portions of output 50.

[0057] Figure 8A A flowchart is shown of a method 200 for use with a computing system to generate a response to a cue word using selectively activated speculative decoding. At step 202, method 200 includes receiving a cue word. For example, the cue word may be in the form of natural language. At step 204, method 200 further includes lexicalizing the cue word to obtain a lexicalized cue word comprising a plurality of input lexicals.

[0058] At step 206, at least in part based on the input lexical units, method 200 further includes computing an output comprising a plurality of output lexical units. The output is computed in a plurality of autoregressive generation iterations, wherein at each autoregressive generation iteration, the output lexical units are computed and appended to a context comprising the lexicalized input and the previous sequence of output lexical units.

[0059] When the output is computed, steps 208, 210, and 212 are performed during step 206. In step 208, method 200 further includes performing selective speculative decoding logic to select a first portion and a second portion of the output. The selective speculative decoding logic is performed in one or more autoregressive generation iterations of the autoregressive generation iteration, based at least in part on the context including lexicalized cue words and the sequence of previously output lexical terms. For each output lexical term, the selective speculative decoding logic determines whether the output lexical term should be included in the first portion or the second portion before the output lexical term is generated. For example, the first and second portions can be selected on a batch-by-batch basis for each of multiple batches of lexical terms included in the context.

[0060] At step 210, step 206 further includes computing a first portion of the output using one or more draft models via speculative decoding. For example, the one or more draft models may include multiple draft machine learning models. At step 212, step 206 further includes computing a second portion of the output at the main machine learning model without using speculative decoding. In the example where the one or more draft models include multiple draft machine learning models, the draft models may have a corresponding number of draft model parameters that is less than the number of main model parameters of the main machine learning model. Therefore, the processing cost associated with computing the output tokens at the draft models may be lower than the processing cost associated with computing the output tokens at the main machine learning model.

[0061] At step 214, method 200 further includes transferring the output to an additional computation process. For example, the additional computation process could be a GUI or a compiler.

[0062] Figure 8B-8H Additional steps of method 200 that can be performed in some examples are shown. Figure 8B The steps that can be performed when selective speculative decoding logic is executed and the output is computed at step 206 are illustrated. At step 216, method 200 may further include computed a first part, at least in part, by generating corresponding draft lexical units at multiple draft models. The draft models that generate the draft lexical units may be draft machine learning models.

[0063] At step 218, method 200 may further include calculating one or more similarity values ​​between draft lexics. For example, the similarity value could be a cosine similarity value. Similarity values ​​can be calculated between individual lexics or sequences of lexics. In some examples, similarity values ​​are calculated between the output probability distributions of the draft model sampled to calculate draft lexics, rather than between the draft lexics themselves.

[0064] At step 220, method 200 may further include determining that one or more similarity values ​​are below a predefined similarity threshold. At step 222, method 200 may further include disabling speculative decoding in response to determining that one or more similarity values ​​are below the predefined similarity threshold. In some examples, when generating the second part, the selective speculative decoding logic also periodically checks whether the draft lexical units computed at the draft model exceed the predefined similarity value, and if so, activates speculative decoding.

[0065] Figure 8C Additional steps that may be performed during step 206 are shown in some examples. At step 224, method 200 may further include computing the first part at least in part by generating corresponding draft lexics at one or more draft models. At step 226, method 200 may further include performing a parallel validation check on the draft lexics at the main machine learning model. For example, the parallel validation check may be performed via greedy decoding, approximate greedy decoding, or kernel sampling.

[0066] In step 228, method 200 may further include determining that one or more draft lexical units in the draft lexical unit have failed the parallel validation check. In step 230, method 200 may further include deactivating speculative decoding in response to determining that one or more draft lexical units have failed the parallel validation check. In an example where the selective speculative decoding logic checks whether speculative decoding is reactivated at a predefined interval, the check may include performing parallel validation and reactivating speculative decoding if the draft lexical unit passes the parallel validation check.

[0067] Figure 8D Additional steps that may be performed during step 206 are shown in some examples. At step 232, method 200 may further include computing the first part at multiple draft models. At step 234, method 200 may further include modifying the number of draft models used to compute the first part during the generation of the first part at the selective speculative decoding logic. In some examples, the selective speculative decoding logic may replace one or more draft models without changing the total number of draft models. The set of draft models used in speculative decoding may be modified, for example, to take into account the task complexity and / or subject matter differences between the different parts of the output.

[0068] Figure 8EThe steps of method 200, which may be performed at the selective speculative decoding logic during step 208 in some examples, are illustrated. At step 236, method 200 may further include estimating the expected value for performing speculative decoding based on a predefined value function. The expected value for performing speculative decoding may be an expected value based on information provided by draft lexical units. The expected value may, for example, be encoded as an approximation of latency, power consumption, the use of one or more processing devices, and / or the use of one or more memory devices. In some examples, the expected value is calculated at least in part based on hardware attribute data of the computing system. A task complexity estimate based at least in part on context computation may also be used to calculate the expected value.

[0069] At step 238, method 200 may further include determining whether to use speculative decoding based at least in part on the expected value. For example, the selective speculative decoding logic may also compute the expected value without using speculative decoding, and may determine whether to use speculative decoding based on which expected value is higher.

[0070] Figure 8F Additional steps of method 200 that may be performed in some examples when the first portion of the output is computed at step 210 are illustrated. At step 240, method 200 may further include executing one or more deterministic policies included in one or more draft models. The deterministic policies may be included in the set of draft models along with or instead of one or more draft machine learning models. Step 240 may include, at step 242, performing a database lookup operation while executing one or more deterministic policies. The database records retrieved via the database lookup operation may be draft lexics or sequences of draft lexics. Alternatively, the database records may be post-processed to obtain one or more draft lexics.

[0071] Figure 8G Additional steps of method 200, which can be performed in some examples, are shown. At step 244, method 200 may further include receiving speculative decoding selection user input, which indicates one or more speculative decoding activation rules associated with a prompt word. For example, a user may input speculative decoding selection user input as code defining at least a portion of selective speculative decoding logic. Model selection of the main machine learning model and / or one or more draft models may also be included in the speculative decoding selection user input. At step 246, method 200 may further include: at the selective speculative decoding logic, selecting a first portion and a second portion, at least in part, by applying one or more speculative decoding activation rules. In examples where model selection is included in the speculative decoding selection user input, the selected model may be used to generate the first portion and the second portion.

[0072] Figure 8HAdditional steps of method 200 that can be performed when generating output and transferring it to an additional computation process are shown. At step 248, method 200 may further include assigning corresponding output lexical metadata to the output lexical. The output lexical metadata for each output lexical may indicate the main machine learning model or one or more draft models used to generate that output lexical. At step 250, method 200 may further include presenting the output along with a graphical representation of the output lexical metadata to the user at a GUI. For example, the output lexical metadata may be represented via highlighting or via text labels.

[0073] Using the systems and methods discussed above, machine learning model outputs are generated in a manner that utilizes speculative decoding to reduce latency. However, instead of using speculative decoding throughout the entire output generation process, it is activated selectively to save the computational costs associated with using speculative decoding on parts of the output where draft lexical units may be inaccurate. The selective speculative decoding logic can be specified in a flexible manner, adaptable to different use case scenarios with different attributes.

[0074] The methods and processes described herein are tied to a computing system of one or more computing devices. In particular, such methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0075] Figure 9 A non-limiting embodiment of a computing system 300 capable of performing one or more of the methods and processes described above is schematically illustrated. The computing system 300 is shown in a simplified form. The computing system 300 can embody the above description and... Figure 1 The computing system 10 is shown in the figure. The components of the computing system 300 may be included in one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, video game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices (such as smartwatches and head-mounted augmented reality devices).

[0076] The computing system 300 includes processing circuitry 302, volatile memory 304, and non-volatile storage device 306. The computing system 300 may optionally include a display subsystem 308, an input subsystem 310, a communication subsystem 312, and / or... Figure 9 Other components not shown.

[0077] Processing circuitry 302 typically includes one or more logic processors, which are physical devices configured to execute instructions. For example, a logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, change the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0078] The logic processor may include one or more physical processors configured to execute software instructions. Additionally or alternatively, the logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of processing circuitry 302 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Components of processing circuitry 302 may optionally be distributed across two or more separate devices that may be remotely located and / or configured for coordinated processing. For example, aspects of the computing system 300 disclosed herein may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration. In this case, it will be understood that these virtualized aspects run on different physical logic processors on various different machines. These different physical logic processors on different machines will be understood to be collectively covered by processing circuitry 302.

[0079] The non-volatile storage device 306 includes one or more physical devices configured to store instructions executable by processing circuitry to implement the methods and processes described herein. When implementing such methods and processes, the state of the non-volatile storage device 306 can be changed—for example, to store different data.

[0080] Non-volatile storage device 306 may include removable and / or built-in physical devices. Non-volatile storage device 306 may include optical memory, semiconductor memory and / or magnetic memory, or other mass storage technologies. Non-volatile storage device 306 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 306 is configured to retain instructions even when the non-volatile storage device 306 is powered off.

[0081] Volatile memory 304 may include a physical device containing random access memory. Volatile memory 304 is typically used by processing circuitry 302 to temporarily store information during the processing of software instructions. It should be understood that when volatile memory 304 is powered off, it typically does not continue storing instructions.

[0082] The processing circuitry 302, the volatile memory 304, and the non-volatile storage device 306 can be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SoCs), and complex programmable logic devices (CPLDs).

[0083] The terms "module," "program," and "engine" can be used to describe aspects of a computing system 300 typically implemented in software by a processor to perform specific functions using portions of volatile memory, involving transformation processing specifically configured to perform those functions. Therefore, a module, program, or engine can be instantiated via processing circuitry 302 that executes instructions held by non-volatile storage device 306 using portions of volatile memory 304. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass a single file or a group of files such as executable files, data files, libraries, drivers, scripts, database records, etc. When included, the display subsystem 308 can be used to present a visual representation of the data stored by the non-volatile storage device 306. The visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device 306, and thus change the state of the non-volatile storage device 306, the state of the display subsystem 308 can also be changed to visually represent changes in the underlying data. The display subsystem 308 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the processing circuitry 302, the volatile memory 304, and / or the non-volatile storage device 306 in a shared housing, or such display devices may be peripheral display devices.

[0084] When included, the input subsystem 310 may include or interface with one or more user input devices, such as a keyboard, mouse, touchscreen, camera, or microphone.

[0085] When included, the communication subsystem 312 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 312 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem 312 can be configured to communicate via wired or wireless local area networks or wide area networks, broadband cellular networks, etc. In some embodiments, the communication subsystem 312 may allow the computing system 300 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0086] The following paragraphs discuss several aspects of this disclosure. According to one aspect of this disclosure, a computational system is provided, including one or more processing devices configured to receive a cue word. The one or more processing devices are further configured to lexically represent the cue word to obtain a lexically represented cue word comprising a plurality of input lexical units. Based at least in part on the input lexical units, the one or more processing devices are further configured to compute an output comprising a plurality of output lexical units over a plurality of autoregressive generation iterations. Computing the output includes: in one or more autoregressive generation iterations, performing selective speculative decoding logic, at least in part on a context comprising the lexically represented cue word and a previous sequence of output lexical units, to select a first portion and a second portion of the output. Computing the output also includes computing the first portion of the output via speculative decoding using one or more draft models. Computing the output also includes computing the second portion of the output at a main machine learning model without using speculative decoding. The one or more processing devices are further configured to transmit the output to an additional computational process. The above features can have the technical effect of generating output in a manner that reduces latency by using speculative decoding, while also avoiding the computational costs associated with executing a draft model when the draft lexical units of the generated output may be inaccurate.

[0087] Accordingly, one or more draft models may include multiple draft machine learning models with a corresponding number of draft model parameters, which is lower than the number of parameters in the main machine learning model. Compared to computing output lexical units at the main machine learning model, the above features can have the technical advantage of computing draft lexical units with lower latency and cost.

[0088] According to this aspect, one or more processing devices can also be configured to compute the first part at least in part by generating corresponding draft lexical units at multiple draft models. One or more processing devices can also be configured to compute one or more similarity values ​​between draft lexical units. One or more processing devices can also be configured to determine that one or more similarity values ​​are below a predefined similarity threshold. One or more processing devices can also be configured to disable speculative decoding in response to determining that one or more similarity values ​​are below the predefined similarity threshold. The above features can have the technical effect of disabling speculative decoding when the outputs of the draft models are inconsistent with each other.

[0089] According to this aspect, one or more processing devices may also be configured to compute the first part at least partially by generating corresponding draft lexical units at one or more draft models. At the main machine learning model, one or more processing devices may also be configured to perform parallel validation checks on the draft lexical units. One or more processing devices may also be configured to determine that one or more draft lexical units in the draft lexical units have failed the parallel validation check. One or more processing devices may also be configured to disable speculative decoding in response to determining that one or more draft lexical units have failed the parallel validation check. The above features can have the following technical effects: checking the accuracy of draft lexical units in a low-latency manner, and disabling speculative decoding when the draft lexical units are inconsistent with the output of the main machine learning model.

[0090] Accordingly, at the selective speculative decoding logic, one or more processing devices may also be configured to estimate the expected value for performing speculative decoding based on a predefined value function. One or more processing devices may also be configured to determine whether to use speculative decoding based at least in part on the expected value. The above features can have the following technical effect: speculative decoding is selectively activated when the expected value for using speculative decoding is positive according to the predefined value function.

[0091] According to this aspect, the context may include multiple lexical batches. One or more processing devices can be configured to perform selective speculative decoding logic for each lexical batch. The above features can have the technical effect of determining whether to activate or deactivate speculative decoding at predefined intervals.

[0092] According to this aspect, one or more draft models include one or more deterministic policies. One or more processing devices can be configured to execute one or more deterministic policies to compute the first part at least in part by performing a database lookup operation. The aforementioned feature can have the technical effect of allowing autoregressive lexical generation at the machine learning model to be selectively replaced by the database lookup operation.

[0093] Accordingly, one or more processing devices can also be configured to assign corresponding output lexical metadata to the output lexical. The output lexical metadata for each output lexical can indicate the main machine learning model or one or more draft models used to generate that output lexical. One or more processing devices can also be configured to present the output along with a graphical representation of the output lexical metadata to a user at a graphical user interface (GUI). These features can have the technical effect of displaying the origin of different parts of the output.

[0094] According to this aspect, one or more processing devices are further configured to compute the first part at multiple draft models. At the selective speculative decoding logic, during the generation of the first part, one or more processing devices can also be configured to modify the number of draft models used to compute the first part. The above features can have the technical effect of adding or removing draft models based on output attributes such as estimated complexity.

[0095] According to this aspect, one or more processing devices may also be configured to: receive speculative decoding selection user input, which indicates one or more speculative decoding activation rules associated with a prompt word. At the selective speculative decoding logic, the one or more processing devices may also be configured to select a first part and a second part at least in part by applying one or more speculative decoding activation rules. The above features may have the technical effect of allowing a user to specify at least a portion of the selective speculative decoding logic in a prompt word-specific manner.

[0096] According to another aspect of this disclosure, a method for use with a computing system is provided. The method includes receiving a cue word. The method further includes lexicalizing the cue word to obtain a lexicalized cue word comprising a plurality of input lexicals. Based at least in part on the input lexicals, the method further includes computing an output comprising a plurality of output lexicals over a plurality of autoregressive generation iterations. Computing the output includes: in one or more autoregressive generation iterations, performing selective speculative decoding logic to select a first portion and a second portion of the output based at least in part on a context comprising the lexicalized cue word and a sequence of previously output lexicals. Computing the output further includes computing the first portion of the output via speculative decoding using one or more draft models. Computing the output further includes computing the second portion of the output at a main machine learning model without using speculative decoding. The method further includes transferring the output to an additional computational process. The above features can have the technical effect of generating output in a manner that reduces latency by using speculative decoding, while also avoiding the computational costs associated with executing a draft model when the draft lexicals of the generated output may be inaccurate.

[0097] Accordingly, one or more draft models may include multiple draft machine learning models with a corresponding number of draft model parameters, which is lower than the number of parameters in the main machine learning model. Compared to computing output lexical units at the main machine learning model, the above features can have the technical advantage of computing draft lexical units with lower latency and cost.

[0098] According to this aspect, the method may further include computing the first part at least in part by generating corresponding draft lexical units at multiple draft models. The method may also include computing one or more similarity values ​​between draft lexical units. The method may further include determining that one or more similarity values ​​are below a predefined similarity threshold. The method may further include disabling speculative decoding in response to determining that one or more similarity values ​​are below the predefined similarity threshold. The above features can have the technical effect of disabling speculative decoding when the outputs of the draft models are inconsistent with each other.

[0099] According to this aspect, the method may further include computing the first part at least partially by generating corresponding draft lexics at one or more draft models. At the main machine learning model, the method may further include performing parallel validation checks on the draft lexics. The method may further include determining one or more draft lexics that have failed the parallel validation checks. The method may further include disabling speculative decoding in response to determining that one or more draft lexics have failed the parallel validation checks. The above features can have the following technical effects: checking the accuracy of draft lexics in a low-latency manner, and disabling speculative decoding when draft lexics are inconsistent with the output of the main machine learning model.

[0100] According to this aspect, the method may further include: at the selective speculative decoding logic, estimating the expected value of performing speculative decoding based on a predefined value function. The method may also include determining whether to use speculative decoding based at least in part on the expected value. The above features can have the following technical effect: when the expected value of using speculative decoding is positive according to the predefined value function, speculative decoding is selectively activated.

[0101] According to this aspect, the method may further include executing one or more deterministic policies included in one or more draft models to compute the first part. When executing one or more deterministic policies, the method may further include performing a database lookup operation. The above features may have the technical effect of allowing autoregressive lexical generation at the machine learning model to be selectively replaced by the database lookup operation.

[0102] According to this aspect, the method may further include assigning corresponding output lexical metadata to the output lexical. The output lexical metadata for each output lexical may indicate the main machine learning model or one or more draft models used to generate that output lexical. The method may also include presenting the output along with a graphical representation of the output lexical metadata to the user at a graphical user interface (GUI). These features can have the technical effect of displaying the origin of different parts of the output.

[0103] According to this aspect, the method may further include computing the first part at multiple draft models. The method may also include, at the selective speculative decoding logic, modifying the number of draft models used to compute the first part during its generation. These features can have the technical effect of adding or removing draft models based on output properties such as estimated complexity.

[0104] According to this aspect, the method may further include: receiving speculative decoding selection user input, the speculative decoding selection user input indicating one or more speculative decoding activation rules associated with a prompt word. At the selective speculative decoding logic, the method may further include selecting a first part and a second part at least in part by applying one or more speculative decoding activation rules. The above features may have the technical effect of allowing a user to specify at least a portion of the selective speculative decoding logic in a prompt word-specific manner.

[0105] According to another aspect of this disclosure, a computing system is provided, including one or more processing devices configured to receive a cue word. The one or more processing devices are further configured to lexicalize the cue word to obtain a lexicalized cue word comprising a plurality of input lexicals. Based at least in part on the input lexicals, the one or more processing devices are further configured to compute an output comprising a plurality of output lexicals in a plurality of autoregressive generation iterations. Computing the output includes: in one or more autoregressive generation iterations, performing selective speculative decoding logic to select a first part and a second part of the output based at least in part on a context comprising the lexicalized cue word and a previous sequence of output lexicals. Computing the output also includes computing the first part of the output via speculative decoding using a set of one or more draft models. At the selective speculative decoding logic, the one or more processing devices are further configured to modify the set of one or more draft models used to generate the first part during the generation of the output. Computing the output also includes computing the second part of the output at a main machine learning model without using speculative decoding. The one or more processing devices are further configured to assign corresponding output lexical metadata to the output lexicals. The output lexical metadata for each output lexical indicates the main machine learning model or one or more draft models used to generate the output lexical. One or more processing devices are also configured to transmit the output for display at a graphical user interface (GUI) along with a graphical representation of the output lexical metadata. These features can have the following technical effects: generating output in a manner that reduces latency using speculative decoding, while also avoiding the computational costs associated with executing a draft model when the draft lexical units of the generated output may be inaccurate.

[0106] As used in this document, "and / or" is defined as inclusive or ∨, as specified in the following truth table:

[0107] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The particular routines or methods described herein may represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described may be performed in the order shown and / or described, in another order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0108] The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system (10), comprising: One or more processing devices (12) are configured to: Receive prompt word (20); The prompt word is lexicalized to obtain a lexicalized prompt word (24) that includes multiple input lexical units (26); An output (50) comprising multiple output terms (52) is computed over multiple autoregressive generation iterations (46), at least in part, based on the input terms, wherein computed said output comprises: In one or more autoregressive generation iterations of the autoregressive generation iterations, selective speculative decoding logic (40) is performed at least in part based on the context (30) including the lexicalized cue words and the previously output lexical sequence (32) to select a first part (54) and a second part (56) of the output; The first portion of the output is computed using one or more draft models (42) via speculative decoding; and The second portion of the output is computed at the main machine learning model (44) without using speculative decoding; and The output is then transmitted to an additional computation process (60).

2. The computing system according to claim 1, wherein the one or more draft models include a plurality of draft machine learning models having a corresponding number of draft model parameters, wherein the corresponding number of draft model parameters is lower than the number of main model parameters of the main machine learning model.

3. The computing system according to claim 1 or 2, wherein the one or more processing devices are further configured to: The first part is computed at least in part by generating corresponding draft lexical units at multiple draft models; Calculate one or more similarity values ​​between the draft lexical units; Determine that one or more similarity values ​​are lower than a predefined similarity threshold; as well as In response to determining that one or more similarity values ​​are below the predefined similarity threshold, speculative decoding is disabled.

4. The computing system according to any one of claims 1-3, wherein the one or more processing devices are further configured to: The first part is computed at least in part by generating corresponding draft lexical units at the one or more draft models; Parallel validation checks are performed on the draft lexical units at the main machine learning model. One or more draft lexical units in the draft lexical units were determined to have failed the parallel validation check; as well as In response to determining that one or more draft lexical units have failed the parallel verification check, speculative decoding is disabled.

5. The computing system according to any one of claims 1-4, wherein at the selective speculative decoding logic, the one or more processing devices are further configured to: Estimate the expected value of the speculative decoding based on a predefined value function; and Whether to use speculative decoding is determined at least in part based on the expected value.

6. The computing system according to any one of claims 1-5, wherein: The context includes multiple lexical batches; and The one or more processing devices are configured to perform the selective inference decoding logic for each lexical batch in the lexical batch.

7. The computing system according to any one of claims 1-6, wherein: The one or more draft models include one or more deterministic strategies; as well as The one or more processing devices are configured to execute the one or more deterministic strategies to compute the first portion at least in part by performing a database lookup operation.

8. The computing system according to any one of claims 1-7, wherein the one or more processing devices are further configured to: Assigning corresponding output lexical metadata to the output lexical, wherein the output lexical metadata for each output lexical indicates the main machine learning model or the one or more draft models used to generate the output lexical; and The output, along with a graphical representation of the output lexical metadata, is presented to the user at a graphical user interface (GUI).

9. The computing system according to any one of claims 1-8, wherein the one or more processing devices are further configured to: Calculate the first part at multiple draft models; and At the selective speculative decoding logic, during the generation of the first part, the number of draft models used to calculate the first part is modified.

10. The computing system according to any one of claims 1-9, wherein the one or more processing devices are further configured to: Receive speculative decoding selection user input, wherein the speculative decoding selection user input indicates one or more speculative decoding activation rules associated with the prompt word; and At the selective speculative decoding logic, the first part and the second part are selected at least in part by applying one or more speculative decoding activation rules.

11. A method (200) for use with a computing system, the method comprising: Receive (202) prompt word; The prompt word is lexicalized (204) to obtain a lexicalized prompt word that includes multiple input lexical units; At least in part based on the input lexical units, an output comprising multiple output lexical units is computed over multiple autoregressive generation iterations (206), wherein computed said output comprises: In one or more autoregressive generation iterations of the autoregressive generation iterations, at least in part based on the context including the lexicalized cue words and the previously output lexical sequence, (208) selective speculative decoding logic is performed to select a first part and a second part of the output; The first portion of the output (210) is computed using one or more draft models via speculative decoding; and The second part of the output described in (212) is computed at the main machine learning model without using speculative decoding; and The output is transmitted (214) to an additional computation process.

12. The method of claim 11, wherein the one or more draft models comprise a plurality of draft machine learning models having a corresponding number of draft model parameters, the corresponding number of draft model parameters being lower than the number of main model parameters of the main machine learning model.

13. The method according to claim 11 or 12, further comprising: The first part is computed at least in part by generating corresponding draft lexical units at multiple draft models; Calculate one or more similarity values ​​between the draft lexical units; Determine that one or more similarity values ​​are lower than a predefined similarity threshold; as well as In response to determining that one or more similarity values ​​are below the predefined similarity threshold, speculative decoding is disabled.

14. The method according to any one of claims 11-13, further comprising: The first part is computed at least in part by generating corresponding draft lexical units at the one or more draft models; Parallel validation checks are performed on the draft lexical units at the main machine learning model. One or more draft lexical units in the draft lexical units were determined to have failed the parallel validation check; as well as In response to determining that one or more draft lexical units have failed the parallel verification check, speculative decoding is disabled.

15. The method according to any one of claims 11-14, further comprising, at the selective speculative decoding logic: Estimate the expected value of the speculative decoding based on a predefined value function; and Whether to use speculative decoding is determined at least in part based on the expected value.

16. The method according to any one of claims 11-15, further comprising: Execute one or more deterministic strategies included in the one or more draft models to compute the first part; as well as When one or more deterministic strategies are executed, a database lookup operation is performed.

17. The method according to any one of claims 11-16, further comprising: The corresponding output lexical metadata is assigned to the output lexical, wherein the output lexical metadata of each output lexical indicates the main machine learning model or the one or more draft models used to generate the output lexical; as well as The output, along with a graphical representation of the output lexical metadata, is presented to the user at a graphical user interface (GUI).

18. The method according to any one of claims 11-17, further comprising: The first part is calculated at multiple draft models; as well as At the selective speculative decoding logic, during the generation of the first part, the number of draft models used to calculate the first part is modified.

19. The method according to any one of claims 11-18, further comprising: Receive speculative decoding selection user input, wherein the speculative decoding selection user input indicates one or more speculative decoding activation rules associated with the prompt word; as well as At the selective speculative decoding logic, the first part and the second part are selected at least in part by applying one or more speculative decoding activation rules.

20. A computing system (10), comprising: One or more processing devices (12) are configured to: Receive prompt word (20); The prompt word is lexicalized to obtain a lexicalized prompt word (24) that includes multiple input lexical units (26); An output (50) comprising multiple output terms (52) is computed over multiple autoregressive generation iterations (46), at least in part, based on the input terms, wherein computed said output comprises: In one or more autoregressive generation iterations of the autoregressive generation iterations, selective speculative decoding logic (40) is performed at least in part based on the context (30) including the lexicalized cue words and the previously output lexical sequence (32) to select a first part (54) and a second part (56) of the output; The first part of the output is computed using a set of one or more draft models (42) via speculative decoding. Wherein, at the selective speculative decoding logic, the one or more processing devices are further configured to modify the set of one or more draft models used to generate the first part during the generation of the output; and The second part of the output is computed at the main machine learning model (44) without using speculative decoding; The corresponding output lexical metadata (126) is assigned to the output lexical, wherein the output lexical metadata of each output lexical indicates the main machine learning model or the one or more draft models used to generate the output lexical; and The output is transmitted to be displayed at a graphical user interface (GUI) (130) together with a graphical representation (132) of the output lexical metadata.