Speculative decoding in autoregressive generative artificial intelligence models

Abstract

Some aspects of this disclosure provide techniques and apparatus for generating responses to input queries using a generative model. The method generally includes generating a first set of tokens based on an input query and a first generative model. The first set of tokens is output to a second generative model for validation. A second set of tokens is speculatively generated while waiting to receive instructions for a selected set of tokens from the first set of tokens. Instructions for a selected set of tokens from the first set of tokens are received. Tokens from the second set of tokens associated with the selected set of tokens are output to the second generative model for validation, and the selected set of tokens is output as a response to the input query.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application claims priority and benefits of U.S. Provisional Patent Application No. 63 / 454,605, filed on 24 March 2023, entitled "Speculative Decoding in Autoregressive Generative Artificial Intelligence Models," and U.S. Patent Application No. 18 / 479,672, filed on 2 October 2023, entitled "Speculative Decoding in Autoregressive Generative Artificial Intelligence Models," both of which have been assigned to the assignee of this application and are incorporated herein by reference in their entirety. [Background technology]

[0002] The aspects of this disclosure relate to generative artificial intelligence models, and more specifically, to speculative decoding in generative artificial intelligence models.

[0003] Generative artificial intelligence models can be used in a variety of environments to generate responses to input queries. For example, generative AI models can be used in chatbot applications where large language models (LLMs) are used to generate answers or at least responses to input queries. Other examples of applications for generative AI models include stable diffusion, where a model generates an image from an input text description of the content of a desired image, and decision transformers, where future actions are predicted based on a sequence of previous actions in a given environment.

[0004] In general, generating responses to queries using generative artificial intelligence models can be computationally expensive. For example, in a chatbot deployment where a large language model is used to generate responses to queries formatted as text queries, the response to the query may be generated using a path through the large language model for each token (e.g., a word or part of a word) generated as part of the response. The output of each path may be a probability distribution of the set of tokens (e.g., a word or part of a word), from which the next token (e.g., a word or part of a word) may be selected, for example, by sampling or based on maximum likelihood. Since the path through the large language model is used to generate each word (or token(s)) in the response to the query, the computational cost can be modeled as the product of the number of words in the response and the computational resource cost (e.g., in terms of processing power, memory bandwidth, and / or other computational resources used) of running the path through the large language model, which generally increases as the number of parameters in the large language model increases. [Overview of the Initiative]

[0005] Some aspects of this disclosure provide a method for generating a response to an input query using a generative artificial intelligence model. The method generally includes generating multiple sets of tokens, each set of tokens, based on an input query and a first generative model, wherein each set of tokens corresponds to a candidate response to the input query. The multiple sets of tokens are output to a second generative model for validation. Instructions for a selected set of tokens from the multiple sets of tokens are received from the second generative model based on the input query and the multiple sets of tokens. The selected set of tokens is output as a response to the input query.

[0006] Some aspects of this disclosure provide a method for validating a response to an input query generated using a generative artificial intelligence model. The method generally includes receiving an input query and multiple sets of tokens generated by a first generative model, where each set of tokens corresponds to a candidate response to the input query. The probability distribution associated with each set of tokens is compared with a corresponding probability distribution generated by a second generative model for each set of tokens. A set of tokens is selected based on the comparison, and the selected set of tokens is output to the first generative model.

[0007] Some aspects of this disclosure provide a method for parallel speculative generation of responses to input queries using a generative artificial intelligence model. The method generally includes generating a first plurality of sets of tokens, where each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to the input query, based on an input query and a first generative model. The first plurality of sets of tokens are output to a second generative model for verification. A second plurality of sets of tokens are speculatively generated, where each set of tokens in the second plurality of sets of tokens corresponds to a second portion of a candidate response to the input query, while waiting to receive instructions from the second generative model for a selected set of tokens from the first plurality of sets of tokens. Instructions for a selected set of tokens from the first plurality of sets of tokens have been received from the second generative model. Tokens from the second plurality of sets of tokens associated with the selected set of tokens are output to the second generative model for verification, and the selected set of tokens is output as a response to the input query.

[0008] Another embodiment provides a processing system configured to perform the aforementioned method and the methods described herein; a non-temporary computer-readable medium containing instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the aforementioned method and the methods described herein; a computer program product embodied on a computer-readable storage medium containing code for performing the aforementioned method and the methods further described herein; and a processing system comprising means for performing the aforementioned method and the methods further described herein.

[0009] The following description and related drawings provide a detailed representation of specific exemplary features of one or more embodiments.

[0010] The attached figures illustrate only a few aspects of this disclosure and should therefore not be considered as limiting the scope of this disclosure. [Brief explanation of the drawing]

[0011] [Figure 1] An embodiment of group speculative decoding in a generative model according to aspects of this disclosure is shown. [Figure 2] This disclosure illustrates an exemplary relationship between a draft model and a target model used in group speculative decoding in a generative model. [Figure 3] An embodiment of parallel speculative decoding in a generative model according to an aspect of this disclosure is shown. [Figure 4] This disclosure provides an exemplary pipeline for hybrid speculative decoding in a generative model according to aspects of this disclosure. [Figure 5] An embodiment of hybrid speculative decoding in a generative model according to aspects of this disclosure is shown. [Figure 6] An embodiment of hybrid parallel speculative decoding in a generative model according to aspects of this disclosure is shown. [Figure 7] An embodiment of hybrid group speculative decoding in a generative model according to aspects of this disclosure is shown. [Figure 8] Illustrative operations for generating a response to an input query using a generative artificial intelligence model, according to an aspect of the present disclosure. [Figure 9] Illustrative operations for verifying a response to an input query using a generative artificial intelligence model, according to an aspect of the present disclosure. [Figure 10] Illustrative operations for speculatively generating a response to an input query and verifying the response to the input query using a generative artificial intelligence model, according to an aspect of the present disclosure. [Figure 11] An illustrative processing system configured to execute various aspects of the present disclosure is shown.

[0012] For ease of understanding, where possible, the same reference numbers are used to designate identical elements common to the drawings. It is contemplated that elements and features of one aspect may be beneficially incorporated into other aspects without further elaboration.

Best Mode for Carrying Out the Invention

[0013] Aspects of the present disclosure provide an apparatus, a method, a processing system, and a computer-readable medium for efficiently generating a response to an input query using a generative artificial intelligence model.

[0014] Generally, generative artificial intelligence models generate responses to queries input into the model. For example, a large-scale language model (LLM) deployed within a chatbot can generate responses to queries using multiple paths through the LLM, each successive path based on the query and tokens (or words or parts thereof) generated using previous paths through the LLM. Typically, these LLMs can contain millions or even billions of weights or parameters. Due to the size of these models, and the actions performed on each token to predict what the next token should be based on the query and previously generated tokens, deploying LLMs on various devices that may have limited memory, storage, and / or processing power compared to cloud compute instances on which LLMs typically operate can be impractical or even impossible. Furthermore, the computational complexity involved in generating responses to queries provided as input to the model can result in significant energy consumption, processing time, memory usage, and / or other resource utilization, which can prevent computing resources from being used for other tasks.

[0015] To improve the efficiency and throughput of large-scale language models, speculative decoding techniques allow smaller language models, sometimes known as draft large-scale language models (also called draft models), to run alongside larger language models, sometimes known as target large-scale language models (also called target models). In such cases, the draft model can speculatively generate additional tokens and probabilities used to sample these additional tokens based on the current set of accepted tokens. The target model can generate tokens based on the tokens generated by the draft model. To produce results, the target model can perform rejection sampling on a per-to-token basis to accept or reject tokens generated by the draft model, such that the draft model and the target model have similar probability distributions.

[0016] In some embodiments, the draft model can be a pruned version of a target model selected such that the draft model and the target model have similar probability distributions. In other embodiments, the draft model can be a smaller version of the target model (e.g., trained with millions of tokens instead of hundreds of millions or even billions of tokens).

[0017] Aspects of the present disclosure provide techniques for using speculative decoding techniques to generate responses to queries input to a large language model on a group basis. Generally, the draft model can generate one or more sets of tokens as candidate responses to a query. The target model can then perform sampling rejection for each set. Tokens within a group can be selected based on the conditional probabilities of the tokens within the group. By performing rejection sampling for each set, aspects of the present disclosure can increase the throughput (e.g., the number of tokens generated per second) of the draft model and the target model configured to generate responses token by token while maintaining a close relationship between the probability distributions within the draft model and the target model.

[0018] Aspects of the present disclosure provide techniques for generating responses to queries input to a large language model using a speculative decoding technique in which a draft model and a target model operate in parallel, also referred to herein as "hybrid speculative decoding." In the hybrid speculative decoding technique, while previously speculatively generated tokens are being verified by the target model, the draft model can speculatively generate one or more tokens. The draft model can be executed on an edge device, and the target model can be executed on a server (e.g., hosted in a cloud computing environment that is locally accessible by the edge device). By executing both the draft model and the target model in parallel, the rate at which tokens are generated can be maximized or at least increased. Further, in situations where the edge device is disconnected from the server, the edge device can continue to generate tokens to generate a response to the received query.

[0019] Speculative Decoding in Generative AI Models Generally, autoregressive token generation (e.g., in a large language model) can take the history tokens as input to generate an output. That is, autoregressive token generation can be represented by the following formula. x t ~p(x│x0,x1,...,x t-1 )→x t+1 ~p(x|x0,x1,...,x t-1 ,x t ) Where x t represents a sequence of tokens generated at time t having a conditional probability p conditioned on the selection of tokens x0 to x t-1 , and x t+1 represents tokens from x0 to x tThis represents a sequence of tokens generated at time t+1, with a conditional probability p given to the selection of a token. In general, a single token can be generated each time the autoregressive model is run, meaning that N inferences can be performed to generate a sequence of N tokens. As mentioned above, speculative decoding techniques can be used to accelerate token generation by using a draft model that is smaller in size than the target model, which speculatively generates tokens, and the target model is used to validate the tokens generated (speculatively) by the draft model.

[0020] In a speculative decryption pipeline, the draft model can speculatively generate n tokens in an autoregressive manner according to the following equation.

[0021]

number

[0022]

number

[0023] The target model takes the n generated tokens, processes the n tokens in parallel, and generates a probability distribution for each of the n tokens according to the following formula.

[0024]

number

[0025] Next, the target model can validate the tokens generated by the draft model by comparing the distribution from the draft model and the target model to determine whether the tokens are accepted or rejected. For some function f and some threshold α (also known as the acceptance rate)

[0026]

number

[0027]

number

[0028]

number

[0029] Speculative decryption with an acceptance rate of α can result in cost savings compared to iteratively generating tokens using a single autoregressive model. The inference cost savings for iterative token generation can be expressed by the following equation:

[0030]

number

[0031] N=1000,C target =10,C draft Consider the example where =1, n=4, α=3, in the formula, N corresponds to the number of tokens, and C AR This corresponds to the computational cost using the acceptance rate of α, and C target This corresponds to the computational cost of generating a set of tokens using the target model, and C draftn corresponds to the computational cost of generating a set of tokens using the draft model, and n corresponds to the number of speculatively generated tokens generated through a single pass through the autoregressive model. In such an example, speculative decryption can result in a 35% reduction in computational cost compared to autoregressive iterative token generation alone.

[0032] However, per-token speculative decryption can impose limitations on the rate at which tokens are generated, as explained, because the first token may be individually sampled by the draft model and then validated by the target model before the next token is sampled by the draft model and validated by the target model. In other words, generating a response to an input query using per-token speculative decryption techniques may involve running the draft model and the target model for each token generated as part of the response to the input query, which can use a considerable amount of computational resources (e.g., processor time, memory, memory bandwidth, etc.) to generate the response.

[0033] Exemplary group speculative decoding in generative artificial intelligence models Figure 1 shows Example 100 of group speculative decoding (GSD) in a generative artificial intelligence model according to an aspect of this disclosure.

[0034] As illustrated, the draft model and the target model can be used relative to (or together with) each other to perform group speculative decryption of tokens to generate responses to incoming queries for processing by one or more generative artificial intelligence models. As will be explained in more detail below, group speculative decryption of tokens can allow multiple sets of tokens to be speculatively generated by the draft model for verification by the target model. Since multiple sets of tokens can be generated by the draft model for verification by the target model, group speculative decryption can increase the token generation rate of the generative artificial intelligence model by generating multiple sets of tokens that can be accepted as a correct response. This is because the more sets of tokens there are, the higher the probability that at least one set contains one or more tokens that can be accepted as a response.

[0035] A draft model generally takes into account the input of an received query and forms sampled groups of tokens that include one or more groups of high-probability nodes from a probability distribution of outputs across a set of potential tokens. Groups of high-probability nodes may include groups selected based on various techniques, such as the top k selection (e.g., the selection of the k tokens with the highest probabilities in the probability distribution) or kernel-based selection (e.g., selection based on the sum of probabilities satisfying a threshold probability). Tokens not included in one or more groups may be considered singleton set groups (e.g., groups containing a single member). By selecting groups of candidate tokens, a draft model can sample tokens based on a probability distribution. However, each group may be treated as a single selection with aggregated group probabilities (e.g., as the sum of individual token probabilities), so the draft model matches, or at least approaches, the target model in terms of the calculated probability that each group is a sufficient response to the received query. While individual token probabilities may not match well, a set of high-probability tokens may have probabilities that are likely to be similar in the draft and target models, so grouping can achieve this match, or at least approximation, between the probabilities calculated by the draft and target models.

[0036] In the next iteration of running the draft model, when a sampled group of tokens is input to the draft model, the tokens within the sampled group are input to the sample positions and treated independently. The result may be a tree data structure 110 with a prompt as the root node 111 of the tree data structure, where subsequent levels in the tree data structure 110 represent different tokens (or groups of tokens) combined with each of the previously selected combinations of tokens. At some point (for example, after generating a tree with a defined depth corresponding to the maximum length of the sequence generated by the draft model), the draft model may output the generated tree data structure 110 to the target model for further processing. In some embodiments, the tree data structure 110 may be output to the target model along with the groupings and selection probabilities generated by the draft model.

[0037] In some embodiments, the number of nodes at each level of the tree data structure 110 and the depth of the tree data structure 110 can be defined a priori. The number of nodes at each level of the tree can be defined globally, per level, or in some other way. For example, the number of nodes at any level of the tree data structure 110 can be defined based on the branching factor at the immediately preceding level of the tree data structure 110. For example, a branching factor of 2 for a node at the nth level of the tree data structure 110 may result in the generation of two nodes (tokens) at the (n+1)th level of the tree data structure 110 for each node (token) at the nth level of the tree. On the other hand, the depth of the tree data structure 110 can be defined based on the maximum number of tokens (e.g., words) that can be generated using any path through the draft model. For example, if the draft model is configured to generate a sequence of up to 5 tokens in any instance of speculative generation, the depth of the tree may be 6 (to include the root node at the first level of the tree data structure 110 that corresponds to the input to the draft model).

[0038] A draft model can be configured to trigger token generation by a target model based on various complexity and / or performance criteria. For example, a draft model may trigger token generation by a target model based on a complexity or performance criterion associated with the size of the generated tree. In another example, a draft model may be configured to trigger token generation by a target model based on a time criterion associated with the expected time it takes for the target model to generate a set of tokens that can be compared to the generated tree. In general, these complexity and / or performance criteria may set an upper limit on the number of tokens generated by the draft model for validation by the target model. This upper limit may, in some embodiments, be based on the number of nodes in the tree data structure and may be influenced by, for example, a branching factor defined for different levels of the tree data structure in which the sampled tokens are organized, the depth of the tree data structure, etc. The worst-case computational load in the final round of speculative token generation may be configured to be limited by the memory bandwidth on the device on which the draft model is run.

[0039] Based on the generated tree data structure 110, grouping, and selection probabilities received from the draft model, the target model generates an output (probability) distribution for each subpath in the generated tree data structure 110. Each subpath can generally be a continuous path through the generated tree data structure 110, starting from the root node and ending at any node in the generated tree data structure 110. The target model can generate an output distribution for each subpath using a single path through the target model, for example, by including all tree nodes in the generated tree data structure 110 as token inputs and performing masked self-attention and position coding on each subpath in the tree data structure 110. Based on the output distribution for each node in the generated tree data structure 110, the target model can select a set of tokens 112 by traversing from the root node of the generated tree data structure 110. If the target model contains multiple tokens from a selected set 112 of tokens, the tokens can be selected from within the group using normalized group token probabilities (e.g., token probabilities conditional on the selection of a group containing the tokens), and the selected tokens can be redefined as the root node of the generated tree data structure 110. The process described herein can be repeated to generate subsequent tokens to be included in the selected set 112 of tokens by defining the selected tokens as the root node of the tree data structure 110.

[0040] If groups of tokens are rejected based on probability distributions generated by the target model, the final group of tokens may be selected based on a group-level rejection sampling method. The final token may be selected from the selected final group of tokens based on normalized group token probabilities. If all groups of tokens are accepted by the target model, the final token may be sampled from the final tree-leaf distribution (e.g., the probability distribution generated across leaf nodes in the tree data structure 110 corresponding to the last accepted or validated token for any subpath through the tree data structure 110), which may allow the target model to generate additional tokens for the token(s) speculatively generated by the draft model. The selected set of tokens 112 may then be communicated back to the draft model to be used as input for subsequent rounds of token speculation and validation by the target model, using the techniques described herein.

[0041] Figure 2 shows an example of the relationship between tokens generated by a draft model and a target model used for group speculative decryption in a generative model, according to an aspect of this disclosure.

[0042] The tree generated by the draft model generally allows for diverse speculation, providing many candidate sets of tokens for validation by the target model, thereby potentially improving the acceptance rate of tokens generated by the target model. As illustrated, the acceptance of token number 4 in the group of tokens generated by the draft model at time t=1 210 leads to the acceptance of token number 1 in the group of tokens generated by the draft model at time t=2 220. Subsequently, at time t=3 230, there may be two options: token number 3 or token number 5. As illustrated, token number 3 may be accepted as a sampled token, and token number 5 may be rejected as a sampled token. At time t=4 240, token 4 from the speculatively generated set of tokens based on the selection of token 3 at time t=3 230 may be accepted as a sampled token based on the calculated probabilities of tokens 3 and 4, conditional on the acceptance of token 3 at time t=3 230. On the other hand, both tokens 4 and 5 from a set of tokens speculatively generated based on the selection of token 5 at time t=3 230 may be rejected based on the rejection of token 5 at time t=3 230.

[0043] The target model may be supplied with a token tree as input instead of a sequence of tokens. For example, the target model may be supplied with a tree containing the illustrated options for token 3 or 5, which are candidate tokens at time t=3 230, the options for token 3 or 4 at time t=4 240 if token 3 is selected at time t=3 230, and the options for token 4 or 5 at time t=4 240 if token 5 is selected at time t=3 230. As illustrated, by inputting a token tree data structure (for example, as shown by the sequence of tokens shown from time t=1 to t=4) into the target model at time t=5 250, the target model may be able to produce an output containing five tokens, one more than the sequence of four tokens (single or multiple) contained in the tree generated by the draft model, thereby providing an increased acceptance rate for the token tree and enabling the generation of additional tokens that will be included in the responses generated by the draft model and the target model. In the example shown in Figure 2, the selected set of five tokens includes token 4 at time t=1 210, token 1 at time t=2 220, token 3 at time t=3 230, token 4 at time t=4 240 (conditional on the selection of token 3 at time t=3 230), and token 2 at time t=5 250 (conditional on the selection of token 4 at time t=4 240).

[0044] In general, a new selection of token groups may be performed each time the draft model generates a new distribution of one or more subsequent tokens that will be included in the response to the received query processed by the draft model and the target model. As described, the use of token group selection can minimize or at least reduce the likelihood of group rejection by enabling a closer match between the draft model probability distribution and the target model probability distribution. In some embodiments, the draft model may choose not to form a group when there is a low level of uncertainty (e.g., when the next token selection has low entropy) in order to minimize or at least reduce computational resource consumption.

[0045] In general, forming groups of tokens when the use of group speculative decoding (to generate responses to input queries using a generative artificial intelligence model) is likely to have a high level of effectiveness can reduce the computational complexity of both the draft and target models while maintaining a high level of throughput (e.g., the number of tokens per second generated by the draft and target models). In general, triggering the target model can be influenced by various considerations, such as the maximum tree size to keep the model computational performance metrics within limits, estimations of when longer speculations will have diminishing returns (e.g., when the computational cost of generating a larger tree of speculated tokens is unlikely to be worthwhile, considering the likelihood that the target model will reach later tokens in the tree), and other performance considerations.

[0046] In some embodiments, a draft model may match a target model (e.g., with respect to probability distributions) but may have faster inference performance than the target model on the same hardware. Generally, smaller models can generate more speculative tokens, but the likelihood of those generated tokens being rejected by the target model may increase. Group speculative decoding can address this increased likelihood of token rejection at the cost of increased computational cost for longer sequences, as described above.

[0047] In some embodiments, parameters in the draft model that affect the likelihood of the draft model selecting a token with a lower probability of being the exact token included in the set of tokens, such as a temperature parameter, may be adjusted or selected to improve the performance of group speculative decoding.

[0048] In some embodiments, the draft model may be fine-tuned to match (or at least approximate) the target model and to maximize (or at least increase) the probability that the speculatively generated tokens produced by the draft model will be accepted as valid tokens by the target model.

[0049] In general, token generation performance for group speculative decoding can be improved compared to per-token speculative decoding. That is, given a Kullback-Leibler (KL) divergence between a draft model and a target model, if we measure how the probability distribution of the draft model differs from that of the target model (treating the target model as the baseline), the number of tokens generated for each target model run may be greater for group speculative decoding than for per-token speculative decoding. Different grouping strategies (e.g., group size, additional tokens) may have different computational complexity characteristics. Therefore, the choice of grouping strategy may be based on a trade-off between computational complexity and performance, taking into account the boundary parameters of the read bandwidth of the draft and target models, as well as hardware performance.

[0050] Exemplary parallel speculative decoding in generative artificial intelligence models As described above, group speculative decryption, by combining speculative token generation in a draft model with token acceptance in a target model, can enable the generation of a final set of tokens that match the distribution by the target model while increasing the rate at which tokens are generated. Furthermore, aspects of this disclosure provide parallel speculative decryption in which the draft model speculatively generates multiple sets of tokens in parallel, thereby providing an increased token generation rate while improving the computational efficiency associated with generating responses to queries using a generative artificial intelligence model.

[0051] Figure 3 shows an example 300 of parallel speculative decoding (PSD) in a generative artificial intelligence model according to an aspect of this disclosure.

[0052] As illustrated, the draft model can generate multiple candidate token sequences by running multiple instances of the speculative decryption process. Each iteration may be run with different parameters or seeds for token sampling. Running N iterations with S tokens generated in each iteration may result in the generation of a total of N*S tokens, which can be provided to the target model for processing.

[0053] For example, as shown in Example 300, the draft model can first run multiple instances of the speculative decoding process to generate a first candidate sequence 302 and a second candidate sequence 304 for prompt input to the draft model. As shown in the figure, N=2 and S=4. Thus, the draft model can generate a total of eight tokens, which are two sequences, each containing four tokens. It should be noted that this is just one example, and any values ​​for N and S can be used to define the number of candidate sequences generated by the draft model and the number of tokens contained in each sequence, respectively.

[0054] The target model is N * Using S tokens as input, the target model processes N candidate token sequences in a single run. *The result of processing S tokens can be a set of N potentially acceptable sequences of varying lengths, depending on where the target model rejected tokens in each of the N candidate token sequences. Various techniques can be used to select the final sequence based on the N potentially acceptable sequences. In a non-restrictive example, a "greedy" technique may result in the selection of the longest sequence with arbitrarily broken connections (e.g., between sequences of the same length), which may result in the largest number of tokens being selected each time the target model is run. In some cases, longest-length selection may bias the effective final sampling distribution, thereby causing the probability distribution of the target model to converge, at least to some extent, to the probability distribution of the draft model.

[0055] For example, as shown in the figure, the target model accepts two of the four tokens in the first candidate sequence 302 and four of the four tokens in the second candidate sequence 304. Since more tokens are accepted from the second candidate sequence 304 than from the first candidate sequence 302, a "greedy" technique may result in the target model selecting the second candidate sequence 304 as the sequence on which the draft model generates subsequent candidate sequences 312 and 314. In this case, there is no need to perform speculative token generation based on the first candidate sequence 302, since the first candidate sequence 302 had fewer accepted tokens than the second candidate sequence 304. However, in other cases, other techniques may be implemented to select one of the candidate sequences that contains the sequence that cannot have the most accepted tokens.

[0056] Figure 3 illustrates parallel speculative decryption based on per-token speculative decryption, but it should be noted that N candidate token sequences can be generated similarly or alternatively based on the group speculative decryption technique described above. For example, using the group speculative decryption technique described above, multiple sets of tokens generated by multiple instances of a draft model can be generated for verification by the target model. In some embodiments, the sets of tokens generated by instances of the draft model may be organized into a tree data structure (as described above), resulting in the generation of multiple tree data structures, each individual tree data structure corresponding to a set of tokens generated by a particular instance of the draft model executed using instance-specific parameters. The target model can attempt to verify the tokens in each set of tokens to determine the selected set of tokens based on which subsequent speculative decryption will be performed. The longest path through each tree data structure may be identified by the target model as a candidate sequence. Using a "greedy" technique, tokens in candidate sequences with the longest path can be selected as a set of tokens based on which subsequent speculative decryption will be performed, and can be output to the draft model by the target model for use as the root node of the subsequent tree data structure (e.g., a tree data structure corresponding to subsequent tokens generated as part of the response to an input query).

[0057] Exemplary hybrid speculative decoding In some cases, it may be possible to run generative artificial intelligence models using autoregressive inference in hybrid systems. Examples of these hybrid systems may include systems in which these generative models run on edge devices (e.g., smartphones, tablet computers, laptop computers, desktop computers, etc.) and cloud servers, on edge devices and home servers, on different home servers, or similar configurations. Thus, in some embodiments, a draft model may run on a local device and a target model may run on a remote device. Running generative artificial intelligence models in hybrid systems can accelerate the rate at which responses to queries are generated and reduce the computational load on servers (e.g., by offloading some computational processes to other devices). Furthermore, the accuracy of the target model can be maintained when the server on which the target model is running is available, and when an edge device is disconnected from the server, a large-scale language model (or other generative artificial intelligence model) may remain available on the edge device (albeit with lower accuracy).

[0058] However, in hybrid systems, cost savings may be limited if both the draft and target models are hosted on cloud systems. Furthermore, the sequential nature of inference (e.g., involving speculatively generating tokens using the draft model and then validating them using the target model) may limit the rate at which tokens are generated.

[0059] To further reduce the computational cost associated with generating responses to queries using generative artificial intelligence models, aspects of this disclosure provide various techniques for speculative decryption in a hybrid environment. In various aspects, speculative decryption in a hybrid environment may be performed in parallel, with candidate token set generation using a draft model (e.g., on an edge device) and token validation using a target model (e.g., on a server). The draft model may continue to generate subsequent sets of draft tokens, assuming fluctuations in the number of tokens accepted, and padding may be added to account for the difference in the number of tokens accepted from the previous round of speculative token generation. The draft and target models may be executed sequentially and substantially in parallel, thereby accelerating the generation of responses to queries using generative artificial intelligence models.

[0060] Figure 4 shows an exemplary pipeline 400 for hybrid speculative decoding (HSD) in a generative model according to an aspect of the present disclosure.

[0061] As illustrated, in order to maximize, or at least increase, throughput (e.g., the number of tokens generated per second), a draft model 402 running on an edge device 403 (which may be called a local device or first device) may continuously generate batches of candidate tokens (or sets of tokens corresponding to different responses of varying lengths to an input prompt, conditional on any previously accepted tokens) and output these tokens to a target model 404 running on a server 405 (which may be called a remote device or second device) (e.g., in a cloud computing environment). The edge device 403 may receive a set of accepted tokens from the target model 404 via the server 405. The draft model 402 may prune a sample tree to fit the set of accepted tokens. In some embodiments, when the set of accepted tokens is empty (e.g., when the target model does not accept any tokens), the draft model 402 may backtrack to the last token accepted by the target model 404 and resume speculative token generation from the last accepted token. For example, to backtrack, Draft Model 402 can prune the generated sample token tree back to the last accepted token and then resume speculative generation based on the pruned tree.

[0062] In an exemplary pipeline 400, a draft model 402 running on an edge device 403 generates a first set 410 of speculative tokens and may output this first set of speculative tokens to a target model 404 running on a server 405 for verification. In some embodiments, the first set 410 of speculative tokens may include multiple subsets of speculative tokens, each subset corresponding to one or more sequences of tokens generated by different instances of the draft model 402 running with different operating parameters (e.g., according to group speculative decryption, parallel speculative decryption, or other techniques).

[0063] In some embodiments, while the target model 404 is validating a first set 410 of speculative tokens, the draft model 402 can generate a second set 414 of speculative tokens. The second set 414 of speculative tokens may be generated based on the assumption that the target model 404 has accepted 1 to S tokens from the first set 410 of speculative tokens, where S corresponds to the number of tokens per sequence generated by the draft model 402. The draft model 402 can then receive a selected set 412 of tokens from the target model 404. Based on the selected set 412 of tokens, the draft model 402 can identify a subset of the second set 414 of speculative tokens for analysis (e.g., tokens whose generation is conditioned on accepting the correct number of tokens from the first set 410 of speculative tokens) and provide this subset to the target model 404 for validation. This process may continue, for example, by generating a third set of speculative tokens 418 while the draft model 402 generates a third set of speculative tokens 418 while the target model 404 validates a second set of speculative tokens 414 and returns a selection 416 from the second set of speculative tokens 414 to the draft model 402, and by generating a fourth set of speculative tokens 422 while the target model 404 validates the third set of speculative tokens 418 and returns a selection 420 from the third set of speculative tokens 418 to the draft model 402, and so on. In other embodiments, the draft model 402 may wait to generate a second set of speculative tokens 414 until the draft model 402 receives a selected set of tokens 412 from the target model 404.

[0064] Some of the processes described herein for speculatively generating tokens using a draft model may be performed in parallel with, or substantially in parallel with, the validation of a set of speculatively generated tokens using a target model, while minimizing or at least reducing the likelihood that draft model 402 will backtrack and resume speculative token generation from the last accepted token (e.g., to minimize or at least reduce the likelihood that target model 404 will reject each of the speculatively generated tokens generated by draft model 402), according to group selection and target model trigger conditions selected to achieve sequential, or at least nearly sequential, operation of draft model 402 and target model 404. Furthermore, group selection and target model trigger conditions may be selected to avoid, or at least reduce the likelihood of, placing an excessive burden on the computing power of either the edge device or the server (e.g., consequently, the edge device generates tokens and the server validates the tokens within defined performance metrics that minimize or at least reduce the likelihood of encountering processing system bottlenecks such as memory thrashing where data is repeatedly swapped between on-processor and off-processor memory, or network bandwidth bottlenecks).

[0065] In some embodiments, hybrid speculative decryption may be performed per token. In such cases, the draft model 402 may operate sequentially and generate various branching paths based on the generation of new speculative tokens. In some embodiments, the edge device 403 may execute multiple original speculative decryption instances (e.g., using different parameters input to the draft model 402) so that the draft model 402 generates multiple paths in a tree representing options that the target model 404 can verify. Based on some defined trigger criteria, the edge device 403 may provide the generated tree to the target model 404 running on a server 405 (e.g., in a cloud computing environment), the target model 404 may select a sequence of tokens corresponding to a likely valid response to the input query, as described above, and return the selected sequence to the draft model 402 running on the edge device. The draft model 402 may then prune the generated tree to fit the selected sequence returned by the target model 404. In this embodiment, the token generation rate may correspond to the rate at which the draft model 402 generates tokens, and tree pruning is based on target selection. Smaller draft models may generate tokens at a higher rate, but the likelihood of the target model 404 accepting the speculatively generated tokens by the draft model 402 is lower, so inference accuracy and speed may have an inverse relationship in this embodiment. On the other hand, larger draft models may generate tokens at a lower rate, but the likelihood of the target model 404 accepting the speculatively generated tokens by the draft model 402 is higher.

[0066] Figure 5 shows an embodiment 500 of hybrid speculative decryption in a generative model according to an aspect of the present disclosure. In hybrid speculative decryption, a first set 502 of speculatively generated tokens (also called “Round 1 Draft Tokens”), generated by a draft model (e.g., Draft Model 402 shown in Figure 4) in a first round of inference, may be provided to a target model (e.g., Target Model 404 shown in Figure 4) for verification. As described above, in various embodiments, the draft model may run on a first device and the target model may run on a second device.

[0067] In various embodiments, while a first set of speculatively generated tokens (502) is being processed by the target model, the draft model makes assumptions about a different number of accepted tokens to generate a second set of draft tokens (also called "round 2 draft tokens") in a second round of inference. For example, as shown in the figure, the second set of draft tokens may include: (1) a first subset 504 which may include a speculatively generated set of tokens based on the assumption that the first draft token from the first set is accepted; (2) a second subset 506 which includes a speculatively generated set of tokens based on the assumption that the first and second draft tokens from the first set are accepted; (3) a third subset 508 which includes a speculatively generated set of tokens based on the assumption that the first to third draft tokens from the first set are accepted; and (4) a fourth subset 510 which includes a speculatively generated set of tokens based on the assumption that all four tokens from the first set are accepted. If it is assumed that fewer tokens will be accepted than the number of tokens in the first set, padding (e.g., a null value, a default constant, etc.) may be added so that each assumption is of the same length. As shown in the diagram, a second set of draft tokens may be generated in a batch process, and as a result, a suitable selection of draft tokens, conditional on a specific set of accepted tokens from the first set of speculatively generated tokens 502, can be provided to the target model for verification.

[0068] Figure 6 shows an example 600 of hybrid speculative decoding in a generative model according to an aspect of the present disclosure. In this example 600 (also known as an example of hybrid parallel speculative decoding (HPSD)), the parallel speculative decoding aspect described above may be used to generate various candidate sets of tokens for the target model to validate. The target model may operate sequentially, as in the case of speculative decoding performed token by token. However, instead of generating tokens token by token, the draft model may speculatively generate groups of tokens based on a probability distribution that approximates, but does not need to be equal to, the probability distribution associated with the target model, and generate different tree data structures, as shown in Figure 6, based on the input query (or prompt) and the generated groups of tokens. Based on several defined trigger criteria, an edge device (also called a local device or first device) may provide the generated tree data structure to a target model running on a server (also called a remote device or second device) (for example, in a cloud computing environment). The server, as described above, selects a sequence of tokens corresponding to the most likely valid responses to the input query and returns the selected sequence to a draft model running on the edge device. Generally, the selected sequence of tokens may include speculatively generated tokens that the target model can validate, plus one additional token generated based on the speculatively generated tokens. The draft model can then prune the generated tree to fit the tokens validated by the target model.

[0069] As illustrated, different instances of the draft model using different parameters can independently generate a first speculative set of tokens 602 and a second speculative set of tokens 604 based on input prompts. The draft model (e.g., running locally or on a first device) can provide the first speculative set of tokens 602 and the second speculative set of tokens 604 to a target model (e.g., running remotely or on a second device) for verification. In parallel with, or at least substantially parallel to, the target model verifying the first speculative set of tokens 602 and the second speculative set of tokens 604, the draft model can generate a first subsequent set of tokens 612 based on the assumption that the target model has accepted the first speculative set of tokens 602, and generate a second subsequent set of tokens 614 based on the assumption that the target model has accepted the second speculative set of tokens 604. In this embodiment, as indicated by the ellipse, the target model accepts the second speculative set of tokens 604 rather than the first speculative set of tokens 602 (in the same manner as described above with respect to Figure 3, for example). Since the target model accepts the second speculative set of tokens 604, the draft model can discard the first subsequent set of tokens 612 and provide the second subsequent set of tokens 614 for verification by the target model. Furthermore, although not shown, while the target model is verifying the second subsequent set of tokens 614, the draft model may generate additional sets of tokens based on the second subsequent set of tokens 614.

[0070] The processes of speculative token generation, pruning (e.g., by discarding unselected sets of tokens), and target model validation may be repeated until a termination condition is reached for the input (e.g., no further tokens are needed to generate a suitable response to the input query, the maximum number of tokens has been generated, or the maximum amount of computational resources has been used to generate the response).

[0071] In this embodiment, the overall token generation rate may correspond to the rate at which the draft model generates tokens. However, since the draft model can resume speculative token generation using tokens up to the last token validated as input to the draft model, previously generated tokens may be discarded when the target model determines that a speculatively generated token in the generated tree is not a candidate to be included in response to a query. Similar to per-token-based hybrid speculative decryption, smaller draft models may generate tokens at a higher rate, but the likelihood of the target model accepting tokens speculatively generated by the draft model is lower, so inference accuracy and speed may have an inverse relationship in this embodiment.

[0072] Figure 7 shows an embodiment 700 of hybrid group speculative decoding (HGSD) in a generative model according to an aspect of the present disclosure. In this embodiment 700, the aspect of group speculative decoding described above may be used to generate various candidate sets of tokens for a target model to validate. The target model (e.g., running remotely or on a second device) may operate sequentially, as in the case of speculative decoding performed per token. However, instead of generating tokens per token, a draft model (e.g., running locally or on a first device) may speculatively generate groups of tokens based on a probability distribution equivalent to the probability distribution associated with the target model, and generate a tree (e.g., a tree data structure) based on an input query (or prompt) and the generated groups of tokens. For example, based on some defined trigger criteria, an edge device may provide the generated tree to a target model running on a server (e.g., in a cloud computing environment), which, as described above, may select a sequence of tokens corresponding to a likely valid response to the input query and return the selected sequence to the draft model running on the edge device. Generally, a selected sequence of tokens may include one additional token generated based on the speculatively generated tokens, in addition to the speculatively generated tokens that the target model can validate. The draft model can then prune the generated tree to fit the tokens validated by the target model.

[0073] For example, as illustrated, a tree data structure can be generated using an input prompt that serves as the root node of the tree data structure. A first set of tokens 702, containing multiple groups (represented as different paths through the tree), can be speculatively generated by the draft model as described above and provided to the target model for validation. While the first set of tokens 702 is being validated by the target model, the draft model can speculatively generate a second set of tokens 704, and different subsets of tokens in the second set of tokens 704 are generated based on the assumption that different sets of tokens from the first set of tokens 702 are accepted by the target model. In Example 700, a selected set of tokens 706 may correspond to a set of tokens validated by the target model while the draft model is generating subsequent sets of tokens. In general, the tree data structure can be pruned based on the validation of various groups of speculatively generated tokens by the target model as described above, and as a result, computational resources are not wasted speculatively generating further groups of tokens based on previous sets of tokens that were not accepted by the target model.

[0074] Similar to Example 600 shown in Figure 6, the overall token generation rate in the hybrid group speculative decryption shown by Example 700 in Figure 7 may correspond to the rate at which the draft model generates tokens. However, since the draft model can resume speculative token generation using tokens up to the last token validated as input to the draft model, previously generated tokens may be discarded when the target model determines that a speculatively generated token in the generated tree is not a candidate to be included in response to a query. Similar to token-based hybrid speculative decryption, smaller draft models may generate tokens at a higher rate, but the likelihood of the target model accepting tokens speculatively generated by the draft model is lower, so inference accuracy and speed may have an inverse relationship in this example.

[0075] Exemplary behavior for speculative decoding in generative artificial intelligence models Figure 8 shows an exemplary operation 800 that may be performed by a computing device to generate a response to an input query using a generative artificial intelligence model, according to aspects of this disclosure. The computing device for performing operation 800 may be a device capable of deploying at least a draft model, such as a smartphone, tablet computer, laptop computer, desktop, server, or cloud compute instance hosted in a distributed computing environment.

[0076] As shown in the figure, operation 800 begins in block 810, in which the computing device generates multiple sets of tokens based on an input query and a first generation model. Generally, each set of tokens in the multiple sets of tokens may correspond to a candidate answer to the input query. The input query may be received from the user of the computing device as a prompt in which a text query can be entered, for example, by converting a natural language utterance into a text representation of the input query.

[0077] In block 820, operation 800 proceeds to the computing device outputting multiple sets of tokens to a second generative model for verification (e.g., by another computing device).

[0078] In block 830, operation 800 proceeds to the computing device receiving instructions from a second generative model for a selected set of tokens from multiple sets of tokens, based on an input query and multiple sets of tokens.

[0079] In block 840, operation 800 proceeds for the computing device to output a selected set of tokens (for example, to a user or application, to another model, etc.) as a response to an input query.

[0080] In some embodiments, each set of tokens among multiple sets of tokens includes a group of tokens that have the highest probability in a probability distribution associated with the first generative model across a population of tokens on which the first generative model is trained.

[0081] In some embodiments, each set of tokens among multiple sets of tokens includes a group of tokens selected based on the sum of probabilities associated with the tokens within the group, where the sum exceeds a threshold probability.

[0082] In some embodiments, multiple sets of tokens are represented as a tree data structure. Within the tree data structure, the root node may generally correspond to an input query, and different paths through the tree data structure may generally correspond to different sets of tokens from multiple sets of tokens. In some embodiments, the depth of the tree data structure may correspond to the maximum number of tokens generated by a single path through a first generative model, and the maximum size of the tree data structure may be set based on a computational complexity metric associated with generating a target set of tokens by a second generative model.

[0083] In some embodiments, operation 800 may further include pruning a tree data structure based on a selected set of tokens. Subsequent sets of tokens are generated based on the pruned tree data structure and the input query and output to a second generative model for validation. Based on the input query, the pruned tree data structure, and the subsequent sets of tokens, the second generative model receives instructions for a selected set of subsequent tokens from the subsequent sets of tokens. Outputting a selected set of tokens in response to an input query may include outputting a selected set of tokens and a selected set of subsequent tokens in response to an input query.

[0084] In some embodiments, each set of tokens among multiple sets of tokens may be generated using a unique instance of a first generative model and unique parameters as input to that unique instance of the first generative model.

[0085] In some embodiments, operation 800 may further include generating multiple subsequent sets of tokens based on the input query and multiple sets of tokens while a second generative model validates multiple sets of tokens. Based on the multiple subsequent sets of tokens and the selected sets of tokens, refined subsequent sets of tokens may be generated, and these refined subsequent sets of tokens may be output to the second generative model for validation.

[0086] In some embodiments, a set of tokens among a succession of multiple sets of tokens may include padding that takes into account the number of tokens in a selected set of tokens, which is less than the maximum number of tokens (or a pre-configured number of tokens).

[0087] In some embodiments, operation 800 further includes receiving tokens generated by a second generative model based on a selected set of tokens. The received tokens may be output as subsequent additional tokens to the selected set of tokens.

[0088] In some embodiments, the first generative model may correspond to a draft model in a speculative decoding pipeline, and the second generative model may correspond to a target model in a speculative decoding pipeline.

[0089] In some embodiments, the first generative model and the second generative model may have equivalent probability distributions.

[0090] In some embodiments, the first generative model may have a probability distribution that approximates the probability distribution associated with the second generative model. The approximation of the probability distribution can be a probability distribution that falls within a threshold difference from the probability distribution associated with the second generative model, since the probability distribution associated with the first generative model does not need to exactly match the probability distribution associated with the second generative model.

[0091] In some embodiments, the first generative model may run locally (e.g., on a local device or local system), and the second generative model may be a model hosted remotely (e.g., on a remote system or remote device).

[0092] Figure 9 shows an exemplary operation 900 for validating a response to an input query using a generative artificial intelligence model, according to aspects of the present disclosure. Operation 900 may be performed by a device (or a system having multiple devices) capable of deploying at least the target model, such as a server (e.g., server 405 in Figure 4) or a cloud compute instance hosted in a distributed computing environment.

[0093] As shown in the figure, operation 900 starts in block 910 and receives multiple sets of input queries and tokens from the device on which the first generative model is running, with each set of tokens corresponding to a candidate response to the input query.

[0094] In block 920, operation 900 proceeds to compare the probability distribution associated with each set of tokens among multiple sets of tokens with the corresponding probability distribution generated by the second generative model for each set of tokens.

[0095] In block 930, operation 900 proceeds to select a set of tokens from multiple sets of tokens based on the comparison.

[0096] In block 940, operation 900 proceeds to output instructions for a selected set of tokens to the first generative model.

[0097] In some embodiments, multiple sets of tokens are represented as a tree data structure. Within the tree data structure, the root node may generally correspond to an input query, and different paths through the tree data structure may generally correspond to different sets of tokens from multiple sets of tokens. In some embodiments, the depth of the tree data structure may correspond to the maximum number of tokens generated by a single path through a first generative model, and the maximum size of the tree data structure may be set based on a computational complexity metric associated with generating a target set of tokens by a second generative model.

[0098] In some embodiments, comparing the probability distribution associated with each set of tokens among multiple sets of tokens with the corresponding probability distribution generated by a second generative model for each set of tokens involves generating the probability distribution for each set of tokens based on a single pass through the second generative model. This single pass through the second generative model may be performed based on masked self-attention and position coding in a tree data structure.

[0099] In some embodiments, operation 900 further includes using a second generative model to generate additional tokens based on a selected set of tokens. The additional tokens (or instructions thereof) may be output to the first generative model along with the selected set of tokens (or as part of instructions for the selected set of tokens).

[0100] In some embodiments, the first generative model may correspond to a draft model in a speculative decoding pipeline, and the second generative model may correspond to a target model in a speculative decoding pipeline.

[0101] Figure 10 shows an exemplary operation 1000 that may be performed by a computing device to speculatively generate a response to an input query and validate the response to the input query using a generative artificial intelligence model according to an aspect of the present disclosure. The computing device for performing operation 800 may be a device capable of deploying at least a draft model, such as a smartphone, tablet computer, laptop computer, desktop, server, or cloud compute instance hosted in a distributed computing environment.

[0102] As shown in the figure, operation 1000 begins in block 1010, in which the computing device generates a first set of tokens based on an input query and a first generation model. Generally, each set of tokens in the first set of tokens may correspond to a candidate answer to the input query. The input query may be received from the user of the computing device as a prompt in which a text query can be entered, for example, by converting a natural language utterance into a text representation of the input query.

[0103] In block 1020, operation 1000 proceeds to the computing device output a first set of tokens to a second generative model for verification (e.g., by another computing device).

[0104] In block 1030, operation 1000 proceeds to speculatively generate a second set of tokens while the computing device awaits instructions from a second generative model for a selected set of tokens from a first set of tokens. Each set of tokens in the second set of tokens generally corresponds to a second portion of a candidate response to an input query.

[0105] In block 1040, operation 1000 proceeds to receive instructions from a second generative model for a selected set of tokens from a set of multiple tokens.

[0106] In block 1050, operation 1000 proceeds to output tokens from a second set of tokens associated with a selected set of tokens to a second generative model for verification.

[0107] In block 1060, operation 1000 proceeds to output a selected set of tokens (for example, to a user or application, to another model, etc.) as a response to the input query.

[0108] In some embodiments, operation 1000 further includes receiving instructions for a second selected set of tokens from a second set of tokens associated with a selected set of tokens. The second selected set of tokens may be output as another part of the response to the input query (e.g., to a user or application, to another model, etc.).

[0109] In some embodiments, each set of tokens among a first set of multiple sets of tokens includes a group of tokens that have the highest probability in a probability distribution associated with a first generative model across the population of tokens.

[0110] In some embodiments, each set of tokens among a first set of multiple sets of tokens includes a group of tokens selected based on the sum of probabilities associated with the tokens within the group, the sum exceeding a threshold probability.

[0111] In some embodiments, a first set of tokens may be represented as a tree data structure. Within the tree data structure, the root node may correspond to an input query. Each path through the tree data structure may correspond to a set of tokens from the first set of tokens.

[0112] In some embodiments, each set of tokens among a first set of multiple sets of tokens is generated using a unique instance of a first generative model and unique parameters as input to that unique instance of the first generative model.

[0113] In some embodiments, operation 1000 further includes generating refined subsequent sets of tokens based on a selected set of tokens and a second set of tokens. The refined subsequent sets of tokens for verification may be output to a second generative model. A third set of tokens may be speculatively generated while waiting to receive instructions for a second selected set of tokens from the refined subsequent sets of tokens. In some embodiments, the sets of tokens among the subsequent sets of tokens include padding that takes into account the number of tokens in the selected set of tokens, which is less than the maximum number of tokens.

[0114] In some embodiments, the first generative model may correspond to a draft model in a speculative decoding pipeline. The second generative model may correspond to a target model in a speculative decoding pipeline.

[0115] In some embodiments, the first generative model may have a probability distribution that approximates the probability distribution associated with the second generative model. The approximation of the probability distribution can be a probability distribution that falls within a threshold difference from the probability distribution associated with the second generative model, since the probability distribution associated with the first generative model does not need to exactly match the probability distribution associated with the second generative model.

[0116] In some embodiments, the first generative model may run locally (e.g., on a local device or local system), and the second generative model may be a model hosted remotely (e.g., on a remote system or remote device).

[0117] Exemplary Processing System for Speculative Decoding in Generative Artificial Intelligence Models Figure 11 shows an exemplary processing system 1100 for generating responses to query inputs to a generative artificial intelligence model based on group speculative decoding, as described herein with respect to Figures 8 to 10, for example.

[0118] The processing system 1100 includes a central processing unit (CPU) 1102, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 1102 may be loaded, for example, from program memory associated with the CPU 1102, or from a memory partition (for example, memory 1124).

[0119] The processing system 1100 also includes additional processing components tailored to specific functions, such as a graphics processing unit (GPU) 1104, a digital signal processor (DSP) 1106, a neural processing unit (NPU) 1108, and a connectivity component 1112.

[0120] NPUs, such as the NPU1108, are generally special circuits configured to implement control and arithmetic logic for executing machine learning algorithms, including algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), and random forests (RFs). NPUs are sometimes also called neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0121] NPUs such as the NPU1108 are configured to accelerate the execution of common machine learning tasks, including image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other embodiments, such NPUs may be part of a dedicated neural network accelerator.

[0122] NPUs can be optimized for training or inference, or in some cases, configured to balance performance between both. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0123] NPUs, designed to accelerate training, are generally configured to accelerate the optimization of new models, a highly computationally intensive operation that involves inputting existing datasets (often labeled or tagged), iterating through those datasets, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0124] NPUs designed to accelerate inference are generally configured to operate on complete models. Therefore, such NPUs can be configured to take new data as input and process this new data rapidly through a model that has already been trained to generate model outputs (e.g., inferences).

[0125] In some implementations, the NPU1108 is part of one or more of the CPU1102, GPU1104, and / or DSP1106. These may be located on user equipment (UE) of a wireless communication system or on another computing device.

[0126] In some embodiments, the connection component 1112 may include subcomponents for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long-Term Evolution, LTE), fifth-generation (5G) connectivity (e.g., New Radio, NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The connection component 1112 may be further coupled to one or more antennas 1114.

[0127] The processing system 1100 may also include one or more sensor processing units 1116 associated with any type of sensor, one or more image signal processors (ISPs) 1118 associated with any type of image sensor, and / or a navigation processor 1120 which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0128] The processing system 1100 may also include one or more input and / or output devices 1122, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0129] In some embodiments, one or more of the processors in the processing system 1100 may be based on an ARM or RISC-V instruction set.

[0130] The processing system 1100 also includes memory 1124, which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 1124 includes computer executable components that can be executed by one or more of the aforementioned processors of the processing system 1100.

[0131] In particular, in this embodiment, the memory 1124 includes a token set generation component 1124A, a token selection receiving component 1124B, a token selection component 1026C, and an output generation component 1124D, a generation model 1124E, and optionally a comparison component 1124F. The components shown, and other components not shown, may be configured to perform various aspects of the methods described herein.

[0132] In general, the processing system 1100 and / or its components may be configured to perform the methods described herein.

[0133] Exemplary clause Details of the various implementations of this disclosure are described in the following numbered clauses.

[0134] Clause 1: A method performed by a processor, comprising: generating a plurality of sets of tokens, each set of tokens in the plurality of sets of tokens corresponding to a candidate response to the input query, based on an input query and a first generative model; outputting the plurality of sets of tokens to a second generative model for validation; receiving instructions from the second generative model for a selected set of tokens from the plurality of sets of tokens, based on the input query and the plurality of sets of tokens; and outputting the selected set of tokens as a response to the input query.

[0135] Clause 2: The method according to Clause 1, wherein each set of tokens among multiple sets of tokens includes a group of tokens that have the highest probability in a probability distribution associated with a first generative model across the population of tokens.

[0136] Clause 3: The method according to Clause 1 or 2, wherein each set of tokens among multiple sets of tokens includes a group of tokens selected based on the sum of probabilities associated with the tokens within the group of tokens, and the sum exceeds a threshold probability.

[0137] Clause 4: The method according to any one of Clauses 1 to 3, wherein multiple sets of tokens are represented as a tree data structure, the root node of the tree data structure corresponds to an input query, and each path through the tree data structure corresponds to a set of tokens from the multiple sets of tokens.

[0138] Clause 5: The method according to Clause 4, wherein the depth of the tree data structure corresponds to the maximum number of tokens generated by a single pass through the first generative model.

[0139] Clause 6: The method according to Clause 4 or 5, wherein the maximum size of the tree data structure is set based on a computational complexity metric associated with generating a target set of tokens by a second generative model.

[0140] Clause 7: The method of any one of Clauses 4 to 6, further comprising pruning a tree data structure based on a selected set of tokens, generating a plurality of subsequent sets of tokens based on the pruned tree data structure and an input query, outputting a plurality of subsequent sets of tokens to a second generative model for validation, and receiving instructions from the second generative model for a selected set of subsequent tokens from a plurality of subsequent sets of tokens based on the input query, the pruned tree data structure, and a plurality of subsequent sets of tokens, wherein outputting the selected set of tokens as a response to the input query includes outputting the selected set of tokens and the selected set of subsequent tokens as a response to the input query.

[0141] Clause 8: The method according to any one of Clauses 1 to 7, wherein each set of tokens among multiple sets of tokens is generated using a unique instance of the first generating model and unique parameters as input to the unique instance of the first generating model.

[0142] Clause 9: The method according to any one of Clauses 1 to 8, further comprising: generating multiple subsequent sets of tokens based on an input query and multiple sets of tokens while a second generating model validates multiple sets of tokens; generating a refined subsequent set of tokens based on the multiple subsequent sets of tokens and a selected set of tokens; and outputting the refined subsequent set of tokens to the second generating model for validation.

[0143] Clause 10: The method according to Clause 9, including padding that takes into account the number of tokens in a selected set of tokens, such that one set of tokens in a subsequent set of tokens is less than the maximum number of tokens.

[0144] Clause 11: The method according to any one of Clauses 1 to 10, further comprising receiving tokens generated by a second generating model based on a selected set of tokens, and outputting the received tokens as subsequent additional tokens to the selected set of tokens.

[0145] Clause 12: The method according to any one of Clauses 1 to 11, wherein the first generative model corresponds to a draft model in a speculative decoding pipeline, and the second generative model corresponds to a target model in a speculative decoding pipeline.

[0146] Clause 13: The method according to Clause 12, wherein the draft model includes a model trained to have a probability distribution that approximates the corresponding probability distribution of the target model.

[0147] Clause 14: The method described in any one of Clauses 1 to 13, wherein the first generation model includes a model that runs on a local system, and the second generation model includes a model that runs on a remote system.

[0148] Clause 15: A method performed by a processor, comprising: receiving an input query and a plurality of sets of tokens generated by a first generative model, wherein each set of tokens in the plurality of sets of tokens corresponds to each candidate response to the input query; comparing the probability distribution associated with each set of tokens in the plurality of sets of tokens with a corresponding probability distribution generated by a second generative model for each set of tokens; selecting a set of tokens from the plurality of sets of tokens based on the comparison; and outputting instructions for the selected set of tokens to the first generative model.

[0149] Clause 16: The method according to Clause 15, wherein multiple sets of tokens are represented as a tree data structure, the root node of the tree data structure corresponds to an input query, and each path through the tree data structure corresponds to a set of tokens from the multiple sets of tokens.

[0150] Clause 17: The method according to Clause 15 or 16, wherein the probability distribution associated with each set of tokens among multiple sets of tokens is compared with the corresponding probability distribution generated by the second generative model for each set of tokens, which is generated for each set of tokens based on a single path through the second generative model.

[0151] Clause 18: The method according to Clause 17, wherein a single pass through the second generative model is performed based on masked self-attention and positional coding in the tree data structure.

[0152] Clause 19: The method described in any one of Clauses 15 to 18, further comprising using a second generating model to generate additional tokens based on a selected set of tokens, and outputting the additional tokens to the first generating model.

[0153] Clause 20: The method according to any one of Clauses 15 to 19, wherein the first generative model corresponds to a draft model in a speculative decoding pipeline, and the second generative model corresponds to a target model in a speculative decoding pipeline.

[0154] Clause 21: A method performed by a processor, comprising: generating a first plurality of sets of tokens based on an input query and a first generative model, wherein each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to an input query; outputting the plurality of tokens to a second generative model for validation; speculatively generating a second plurality of sets of tokens, wherein each set of tokens in the second plurality of sets of tokens corresponds to a second portion of a candidate response to an input query, while waiting to receive instructions from the second generative model for a selected set of tokens from the first plurality of sets of tokens; receiving instructions from the second generative model for a selected set of tokens from the first plurality of sets of tokens; outputting tokens from the second plurality of sets of tokens associated with the selected set of tokens to the second generative model for validation; and outputting the selected set of tokens as a response to an input query.

[0155] Clause 22: The method according to claim 21, further comprising receiving instructions for a second selected set of tokens from a second plurality of sets of tokens associated with a selected set of tokens, and outputting the second selected set of tokens as another part of the response to an input query.

[0156] Clause 23: The method according to Clause 21 or 22, wherein each set of tokens among a first set of tokens includes a group of tokens that have the highest probability in a probability distribution associated with a first generative model across the population of tokens.

[0157] Clause 24: The method described in any one of Clauses 21 to 23, wherein each set of tokens in a first set of tokens includes a group of tokens selected based on the sum of probabilities associated with the tokens in the group of tokens, the sum of which exceeds a threshold probability.

[0158] Clause 25: The method according to any one of Clauses 21 to 24, wherein a first set of tokens is represented as a tree data structure, the root node of the tree data structure corresponds to an input query, and each path through the tree data structure corresponds to a set of tokens from the first set of tokens.

[0159] Clause 26: The method according to any one of Clauses 21 to 25, wherein each set of tokens in a first set of tokens is generated using a unique instance of the first generating model and unique parameters as input to the unique instance of the first generating model.

[0160] Clause 27: The method of any one of Clauses 21 to 26, further comprising: generating a refined successive set of tokens based on a selected set of tokens and a second set of tokens; outputting the refined successive set of tokens to a second generating model for verification; and speculatively generating a third set of tokens while waiting to receive instructions from the second generating model for a second selected set of tokens from the refined successive set of tokens.

[0161] Clause 28: The method according to Clause 27, including padding that takes into account the number of tokens in a selected set of tokens, such that one set of tokens in a subsequent set of tokens is less than the maximum number of tokens.

[0162] Clause 29: The method according to any one of Clauses 21 to 28, wherein the first generative model corresponds to a draft model in a speculative decoding pipeline, and the second generative model corresponds to a target model in a speculative decoding pipeline.

[0163] Clause 30: The method according to Clause 29, wherein the draft model includes a model trained to have a probability distribution that approximates the corresponding probability distribution of the target model.

[0164] Clause 31: The method described in any one of Clauses 21 to 30, wherein the first generating model includes a model that runs on a local system, and the second generating model includes a model that runs on a remote system.

[0165] Clause 32: A processing system comprising a memory storing executable instructions and one or more processors, wherein one or more processors are configured to execute executable instructions to cause the processing system to perform the method described in any one of Clauses 1 to 31.

[0166] Clause 33: A processing system comprising means for performing the method described in any one of Clauses 1 to 31.

[0167] Clause 34: A computer-readable medium storing instructions that, when executed by one or more processors, cause one or more processors to perform the method described in any one of Clauses 1 through 31.

[0168] Additional considerations The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may apply to other embodiments. For example, changes may be made to the function and arrangement of the elements described without departing from the scope of this disclosure. Various examples may omit, replace, or add various procedures or components as needed. For example, the methods described may be performed in an order different from the order described herein, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined with some other embodiments. For example, an apparatus may be implemented or a method may be practiced using any number of embodiments described herein. In addition, the scope of this disclosure is intended to encompass apparatus or methods that are practiced using other structures, functions, or structures and functions in addition to, or other than, the various embodiments of this disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0169] As used herein, the term “exemplary” means “serving as an example, case, or illustration.” No embodiment described herein as “exemplary” should be construed as necessarily preferable or advantageous to any other embodiment.

[0170] Where used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other sequence of a, b, and c).

[0171] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0172] The methods disclosed herein include one or more steps or actions for achieving the method. The steps and / or actions of the method may be interchanged with one another without departing from the claims. In other words, unless a specific order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, may be modified without departing from the claims. Furthermore, the various operations of the methods described above may be performed by any preferred means capable of performing the corresponding functions. These means may include, but are not limited to, various hardware and / or software components, and / or various hardware and / or software modules, including, but not limited to, circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations are shown in the figures, those operations may have corresponding equivalent means-plus-function components with similar numbering.

[0173] The following claims are not intended to be limited to the embodiments shown herein, but rather the full scope consistent with the language of the claims should be recognized. Within the claims, a singular reference to an element is intended to mean "one or more" rather than "only one" unless otherwise explicitly stated. Unless otherwise explicitly stated, the term "several" refers to one or more. No element of a claim should be construed under Section 112(f) of the U.S. Patent Act unless it is explicitly enumerated using the phrase "means to..." or, in the case of a method claim, enumerated using the phrase "steps to...". All structural and functional equivalents of the elements of various embodiments described throughout this disclosure, whether known to those skilled in the art or to become known thereafter, are expressly incorporated by reference herein and intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether such disclosure is expressly stated in the claims or not.

Claims

1. A processing system, Memory that stores executable instructions, A system comprising one or more processors, wherein the one or more processors execute the executable instructions and the system performs the following: Based on the input query and the first generation model, a first plurality of sets of tokens are generated, wherein each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to the input query. For verification purposes, the aforementioned sets of tokens are output to the second generative model. While waiting to receive instructions from the second generation model for a selected set of tokens from the first set of tokens, speculatively generate a second set of tokens, wherein each set of tokens in the second set of tokens corresponds to a second portion of the candidate response to the input query. The second generation model receives the instruction for the selected set of tokens from the first set of multiple sets of tokens, For verification purposes, the second generation model outputs tokens from the second set of tokens associated with the selected set of tokens. The selected set of tokens is output as a response to the input query. A processing system configured in such a way.

2. The one or more processors in the processing system To receive instructions for a second selected set of tokens from a second plurality of sets of tokens associated with the selected set of tokens, The second selected set of tokens is output as another part of the response to the input query. It is further structured in such a way. The processing system according to claim 1.

3. The processing system according to claim 1, wherein each set of tokens among the first plurality of sets of tokens includes a group of tokens having the highest probability in a probability distribution associated with the first generative model across the population of tokens.

4. The processing system according to claim 1, wherein each set of tokens among the first plurality of sets of tokens includes the group of tokens selected based on the sum of probabilities associated with tokens within the group of tokens, the sum exceeding a threshold probability.

5. The aforementioned first set of tokens is represented as a tree data structure, The root node of the aforementioned tree data structure corresponds to the input query, Each path through the aforementioned tree data structure corresponds to a set of tokens from the first set of tokens, The processing system according to claim 1.

6. The processing system according to claim 1, wherein each set of tokens among the first plurality of sets of tokens is generated using a unique instance of the first generative model and unique parameters as input to the unique instance of the first generative model.

7. The one or more processors in the processing system Based on the selected set of tokens and the second set of tokens, a refined subsequent set of tokens is generated. The second generative model is then prompted to output the refined subsequent set of tokens for verification. The second generation model speculatively generates a third set of tokens while waiting to receive instructions for a second selected set of tokens from the refined subsequent set of tokens. It is further structured in such a way. The processing system according to claim 1.

8. The processing system according to claim 7, comprising padding that takes into account the number of tokens in the selected set of tokens such that one set of tokens among the subsequent sets of tokens is less than the maximum number of tokens.

9. The first generative model corresponds to the draft model in the speculative decoding pipeline, The second generation model corresponds to the target model in the speculative decoding pipeline. The processing system according to claim 1.

10. The processing system according to claim 9, wherein the draft model includes a model trained to have a probability distribution that approximates the corresponding probability distribution of the target model.

11. The first generation model includes a model that runs on a local system, The second generative model includes a model that is executed on a remote system. The processing system according to claim 1.

12. A method that is executed by a processor, Based on the input query and the first generation model, a first plurality of sets of tokens are generated, wherein each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to the input query. To the second generation model, output the aforementioned sets of tokens for verification, The second generation model speculatively generates a second set of tokens, where, while waiting to receive instructions for a selected set of tokens from the first set of tokens, each set of tokens in the second set of tokens corresponds to a second portion of the candidate response to the input query. From the second generation model, the instructions for the selected set of tokens from the first set of multiple sets of tokens are received, To the second generation model, for verification purposes, tokens from the second set of tokens associated with the selected set of tokens are output, Outputting the selected set of tokens as a response to the input query, Methods that include...

13. Receiving instructions for a second selected set of tokens from a second plurality of sets of tokens associated with the selected set of tokens, Outputting the second selected set of tokens as another part of the response to the input query, The method according to claim 12, further comprising:

14. The method according to claim 12, wherein each set of tokens among the first plurality of sets of tokens includes a group of tokens having the highest probability in a probability distribution associated with the first generative model across the population of tokens.

15. The method according to claim 12, wherein each set of tokens among the first plurality of sets of tokens includes the group of tokens selected based on the sum of probabilities associated with tokens in the group of tokens, the sum exceeding a threshold probability.

16. The aforementioned first set of tokens is represented as a tree data structure, The root node of the aforementioned tree data structure corresponds to the input query, Each path through the aforementioned tree data structure corresponds to a set of tokens from the first set of tokens, The method according to claim 12.

17. The method according to claim 12, wherein each set of tokens among the first plurality of sets of tokens is generated using a unique instance of the first generative model and unique parameters as input to the unique instance of the first generative model.

18. To generate a refined subsequent set of tokens based on the selected set of tokens and the second set of tokens, The second generation model outputs the refined subsequent set of tokens for verification, The second generation model speculatively generates a third set of tokens while waiting to receive instructions for a second selected set of tokens from the refined subsequent set of tokens, The method according to claim 12, further comprising:

19. The method according to claim 18, comprising padding that takes into account the number of tokens in the selected set of tokens such that one set of tokens among the subsequent sets of tokens is less than the maximum number of tokens.

20. The first generative model corresponds to the draft model in the speculative decoding pipeline, The second generation model corresponds to the target model in the speculative decoding pipeline. The method according to claim 12.

21. The method according to claim 20, wherein the draft model includes a model trained to have a probability distribution that approximates the corresponding probability distribution of the target model.

22. The first generation model includes a model that runs on a local system, The second generative model includes a model that is executed on a remote system. The method according to claim 12.

23. A processing system, Means for generating a first plurality of sets of tokens based on an input query and a first generation model, wherein each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to the input query, The second generation model includes means for outputting the aforementioned multiple sets of tokens for verification, Means for speculatively generating a second set of tokens, wherein, while waiting to receive instructions from the second generation model for a selected set of tokens from the first set of tokens, each set of tokens in the second set of tokens corresponds to a second portion of the candidate response to the input query, The second generation model includes means for receiving the instruction for the selected set of tokens from the first set of multiple sets of tokens, The second generation model includes means for outputting tokens from the second set of tokens associated with the selected set of tokens for verification, Means for outputting the selected set of tokens as a response to the input query, A processing system equipped with the following features.

24. Means for receiving instructions for a second selected set of tokens from a second plurality of sets of tokens associated with the selected set of tokens, Means for outputting the second selected set of tokens as another part of the response to the input query, The processing system according to claim 23, further comprising the following:

25. The aforementioned first set of tokens is represented as a tree data structure, The root node of the aforementioned tree data structure corresponds to the input query, Each path through the aforementioned tree data structure corresponds to a set of tokens from the first set of tokens, The processing system according to claim 23.

26. The processing system according to claim 23, wherein each set of tokens among the first plurality of sets of tokens is generated using a unique instance of the first generative model and unique parameters as input to the unique instance of the first generative model.

27. Means for generating a refined subsequent set of tokens based on the selected set of tokens and the second set of tokens, The second generation model includes means for outputting the refined subsequent set of tokens for verification, Means for speculatively generating a third set of tokens while waiting to receive instructions for a second selected set of tokens from the refined subsequent set of tokens from the second generation model, The processing system according to claim 23, further comprising the following:

28. The first generative model corresponds to the draft model in the speculative decoding pipeline, The second generation model corresponds to the target model in the speculative decoding pipeline. The processing system according to claim 23.

29. The first generation model includes a model that runs on a local system, The second generative model includes a model that is executed on a remote system. The processing system according to claim 23.

30. A computer-readable medium storing executable instructions, wherein when the instructions are executed by one or more processors, Based on the input query and the first generation model, a first plurality of sets of tokens are generated, wherein each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to the input query. To the second generation model, output the aforementioned sets of tokens for verification, The second generation model speculatively generates a second set of tokens, where, while waiting to receive instructions for a selected set of tokens from the first set of tokens, each set of tokens in the second set of tokens corresponds to a second portion of the candidate response to the input query. From the second generation model, the instructions for the selected set of tokens from the first set of multiple sets of tokens are received, To the second generation model, for verification purposes, tokens from the second set of tokens associated with the selected set of tokens are output, Outputting the selected set of tokens as a response to the input query, Perform an action that includes Computer-readable media.

Images