Speculative decoding in autoregressive generative artificial intelligence models

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for generating a response to query input in a generative artificial intelligence model. An example method generally includes receiving a plurality of token sets generated based on an input cue and a first generative artificial intelligence model, each token set of the plurality of token sets corresponding to a candidate response to the input cue; selecting a set of tokens from the plurality of sets of tokens using a second generative artificial intelligence model and a recursive adjustment to the target distribution associated with the received plurality of sets of tokens; and outputting the selected set of tokens as a response to the input cue.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application S / N. 18 / 538,912, filed December 13, 2023, entitled “Speculative Decoding in Autoregressive Generative Artificial Intelligence Models,” which claims the benefit and priority of U.S. Provisional Patent Application S / N. 63 / 460,850, filed April 20, 2023, entitled “Speculative Decoding in Autoregressive Generative Artificial Intelligence Models,” both of which are assigned to the assignee of this application and are incorporated herein by reference in their entirety.

[0003] introduction

[0004] This disclosure relates in various aspects to generative artificial intelligence models, and more specifically to speculative decoding in generative artificial intelligence models (also referred to herein as "generative artificial intelligence models").

[0005] Generative AI models can be used in a variety of environments to generate responses to input prompts (also known as queries or inputs). For example, generative AI models can be used in chatbot applications, where large language models (LLMs) are used to generate answers to input prompts, or at least responses to those prompts. Other examples of using generative AI models include implicit diffusion models where the model generates an image based on an input text description of the content of the desired image, decision transformers that predict future actions based on a sequence of previous actions in a given environment, and so on.

[0006] Generally, using generative AI models to generate responses to queries can be computationally expensive. For example, in a chatbot deployment that uses a large language model to generate responses to queries formatted as text, the response to the query may be generated for each token (e.g., a word or a portion of a word) generated as part of the response using a one-round pass through the large language model. The output of each round pass may be a probability distribution of the token set (e.g., individual words or portions of multiple words), which can be selected from the token set, for example, by sampling or based on maximum likelihood. Since a one-round pass through the large language model is used to generate each word (or tokens) in the query response, the computational overhead can be modeled as the product of the number of words included in the response and the computational resource overhead of performing a one-round pass through the large language model (e.g., in terms of processing power, memory bandwidth, and / or other computational resources used), which generally increases with the number of parameters within the large language model.

[0007] Brief Overview

[0008] Certain aspects of this disclosure provide a method for generating a response to an input prompt using a generative artificial intelligence model. The method generally includes: receiving a plurality of token sets generated based on the input prompt and a first generative artificial intelligence model, each of the plurality of token sets corresponding to a candidate response to the input prompt; selecting a token set from the plurality of token sets using a second generative artificial intelligence model and a recursive adjustment to a target distribution associated with the received plurality of token sets; and outputting the selected token set as a response to the input prompt.

[0009] Certain aspects of this disclosure provide a method for generating a response to an input prompt using a generative artificial intelligence model. The method generally includes: generating a first plurality of token sets based on the input prompt and the generative artificial intelligence model, each token set in the first plurality of token sets corresponding to a first portion of a candidate response to the input prompt; speculatively generating a second plurality of token sets using the generative artificial intelligence model, each token set in the second plurality of token sets generally based on a second portion of a candidate response to the input prompt corresponding to the first plurality of token sets; selecting a token set from the first plurality of token sets while speculatively generating the second plurality of token sets; and outputting the selected token set from the first plurality of token sets along with associated token sets from the second plurality of token sets as a response to the input prompt.

[0010] Other aspects include: a processing system configured to perform the foregoing methods and those methods described herein; a non-transient computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product implemented on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including means for performing the foregoing methods and those methods further described herein.

[0011] The following description and related figures illustrate certain illustrative features of one or more aspects. Brief description of the attached diagram

[0013] The accompanying drawings depict only certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0014] Figure 1 Examples of speculative decoding in generative artificial intelligence models based on various aspects of this disclosure are explained.

[0015] Figure 2A and 2B Examples of recursive inference decoding in generative artificial intelligence models based on various aspects of this disclosure are explained.

[0016] Figure 3 An example token tree generated using recursive speculative decoding in a generative artificial intelligence model, based on various aspects of this disclosure, is explained.

[0017] Figure 4 Example operations for generating responses to input prompts in generative artificial intelligence models using recursive speculative decoding, according to various aspects of this disclosure, are explained.

[0018] Figure 5 An example pipeline for self-speculatory decoding in generative artificial intelligence models, based on various aspects of this disclosure, is described.

[0019] Figure 6 An example architecture for self-inferential decoding in generative artificial intelligence models, based on various aspects of this disclosure, is explained.

[0020] Figure 7 Example operations for generating responses to input prompts using generative artificial intelligence models and self-predictive decoding, according to various aspects of this disclosure, are explained.

[0021] Figure 8 An example processing system configured to perform various aspects of this disclosure is described.

[0022] Figure 9An example processing system configured to perform various aspects of this disclosure is described.

[0023] To facilitate understanding, the same reference numerals are used wherever possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration.

[0024] Detailed description

[0025] This disclosure provides apparatus, methods, processing systems, and computer-readable media for efficiently generating responses to input queries using generative artificial intelligence models.

[0026] Generally, generative AI models generate responses to queries input to the model. For example, a large language model (LLM) deployed within a chatbot can use multiple rounds of passage from the large language model to generate responses to queries, where each successive round is based on the query (which can be tokenized for processing) and tokens (or words) generated using previous rounds of passage from the large language model. Typically, these large language models can include a large number (e.g., billions or even trillions) of weights or parameters within the model. Due to the size of these models and the operations performed on each token to predict what the next token should be in response to the query and previously generated tokens, deploying large language models on various devices with limited memory, storage, and / or processing power relative to the cloud computing instances on which they typically run may be impractical or even impossible. Furthermore, in some cases, the memory bandwidth involved in generating responses to queries provided as input to the model can prevent computational resources from being used for other tasks.

[0027] To improve the efficiency and throughput of large language models, speculative decoding techniques allow smaller language models (sometimes called draft large language models, or draft models or approximate models) to operate alongside larger language models (sometimes called target large language models, or target models) (e.g., sequentially or in parallel). In such cases, the draft model can sequentially speculatively generate additional tokens along with probabilities for sampling these additional tokens based on the currently accepted set of tokens. The target model can then generate tokens based on those generated by the draft model. To produce the final result, the target model can perform rejection sampling on a per-token basis to accept or reject individual tokens generated by the draft model, ensuring that the draft and target models have similar probability distributions.

[0028] In some respects, the draft model can be a pruned version of the selected target model so that the draft model and the target model have similar probability distributions. In other respects, the draft model can be a smaller version of the target model (e.g., trained on millions of tokens rather than on hundreds of millions or even billions of tokens).

[0029] Certain aspects of this disclosure provide techniques and apparatus for generating responses to query inputs up to large language models using recursive speculative decoding techniques. Generally, a draft model can generate one or more sets of tokens as candidate responses to the query, which can be constructed as multiple branches (e.g., in a tree data structure). The target model can then recursively perform sample rejection on the tokens provided by the draft model. Generally, recursive sample rejection allows the probability distribution used by the target model when sampling tokens generated by the draft model to be updated to remove tokens rejected by the target model, and the updated probability distribution is subsequently used to sample subsequent tokens generated by the draft model. By recursively performing rejection sampling, certain aspects of this disclosure can preserve a close relationship between the probability distributions within the draft model and the target model, while increasing the throughput (e.g., the number of tokens generated per second) of the draft and target models compared to draft and target models configured to generate responses on a per-token basis.

[0030] Other aspects of this disclosure provide techniques and apparatus for generating responses to query inputs of large-scale language models using speculative decoding techniques, wherein a single model speculatively generates tokens in response to the query input and verifies previously generated tokens, also referred to herein as "self-speculative decoding". In self-speculative decoding, a model can speculatively generate one or more tokens and speculatively generate additional tokens based on a different number of speculatively generated tokens verified by the model. By using the same model to speculatively generate tokens in response to a query and performing verification (e.g., rejection sampling) on ​​the speculatively generated tokens, aspects of this disclosure reduce the computational cost involved in training and using generative artificial intelligence models compared to using multiple separately trained models to speculatively generate tokens and perform verification on the speculatively generated tokens. Furthermore, compared to other speculative decoding techniques, the rate of token generation can be maximized or at least increased when using self-speculative decoding.

[0031] Speculative Decoding in Generative Artificial Intelligence Models

[0032] Generally speaking (e.g., in large language models), autoregressive token generation takes historical tokens as input to generate output. That is, autoregressive token generation can be represented by the following expression:

[0033]

[0034] Where x t Let p represent the sequence of tokens generated at time t, with conditional probability p for each token x0 to x1. t-1 The choice is a condition, and x t+1 Let p represent the sequence of tokens generated at time t + 1, with conditional probability p for each token x0 to x1. t The choice is a condition. Generally, a single token can be generated each time the autoregressive model is executed, which means that N inferences can be performed to generate a sequence of N tokens. As discussed above, speculative decoding techniques can be used to accelerate token generation by using a draft model that is smaller than the target model and speculatively generates tokens faster than the target model, where the target model is used to verify the tokens generated (speculously) by the draft model.

[0035] In the speculative decoding pipeline, the draft model can autoregressively and speculatively generate n tokens based on the following expression:

[0036] , …,

[0037] Where t corresponds to the time point. Corresponding to the pair of tokens x0 to x0 associated with the token x selected at time t. t-1 The choice of is a conditional probability distribution, and Let x represent the token x that is speculatively generated by the draft model at time t.

[0038] The target model takes the generated n tokens and processes them in parallel according to the following expression to generate a probability distribution for each of the n tokens:

[0039]

[0040] Where k corresponds to the token index of the n generated tokens, and This corresponds to the probability distribution generated by the target model at time t for the token x generated by the draft model.

[0041] The target model can then validate the token generated by the draft model by comparing the distributions from the draft model and the target model to determine whether the token is accepted or rejected. For a given function f and a certain threshold α (also known as the acceptance rate), when... At that time, given a token The token can be accepted. Otherwise, it can be rejected. The final token can then be determined based on a certain function. It is generated at the first rejection position or the last position n.

[0042] Compared to iteratively generating tokens using a single autoregressive model, speculative decoding with an acceptance rate of α may result in lower costs. The cost savings from inference compared to iterative token generation can be expressed by the following formula:

[0043]

[0044] Where N corresponds to the number of tokens. Corresponding to the computational cost of using the acceptance rate α, Corresponding to the computational cost of generating the token set using the target model, Corresponding to the computational cost of generating the token set using the draft model, This corresponds to the computational cost of speculatively generating the token set using the draft model, and n corresponds to the number of tokens speculatively generated in a single round of passage using the autoregressive model. Consider... , , , ,and Examples of this approach exist. In such examples, speculative decoding can potentially reduce computational overhead by 35% compared to standalone autoregressive iterative token generation.

[0045] However, as discussed, per-token speculative decoding can impose a limitation on the rate of token generation, since the first token may be sampled individually by the draft model and subsequently verified by the target model, and then the next token may be sampled by the draft model and verified by the target model. In other words, generating a response to an input prompt using per-token speculative decoding can involve executing both the draft and target models for each token generated as a part of the response to that input prompt, which can utilize considerable computational resources (e.g., processor time, memory, memory bandwidth, etc.) to generate the response.

[0046] Example recursive speculative decoding in generative artificial intelligence models

[0047] Figure 1 Example 100 of recursive inference decoding in a generative artificial intelligence model based on various aspects of this disclosure is explained.

[0048] As explained, the draft model and the target model can be used in combination (or otherwise together) to perform recursive speculative decoding of tokens to generate responses to queries received for processing by one or more generative AI models. As discussed further in detail below, recursive speculative decoding of tokens allows the draft model to speculatively generate multiple sets (or sequences) of tokens for verification by the target model. Because multiple sets (or sequences) of tokens can be generated by the draft model for verification by the target model, recursive speculative decoding increases the token generation rate of the generative AI model by generating multiple sets (or sequences) of tokens that can be accepted as correct responses, as a larger number of token sets increases the probability that at least one set includes a sequence of one or more tokens that will be accepted as a response.

[0049] Given the input of the received query, the draft model selects multiple high-probability nodes (tokens) based on the probability distribution of the output over the potential token set. Each high-probability node (token) can be selected based on various techniques, such as optimal k-selection (e.g., selecting the k tokens with the highest probability within the probability distribution), kernel-based selection (e.g., selection based on the sum of probabilities satisfying a threshold probability), etc. By selecting many candidate tokens, the draft model can sample tokens based on the probability distribution and organize a tree structure that can be recursively traversed to identify the set of tokens that are suitable outputs for a given input, as discussed in further detail below.

[0050] When the sampled token group is input into the draft model in the next iteration of the draft model, the tokens in the sampled token group are input at the sampling positions and processed independently. The result may be a tree data structure 110, where the hint serves as the root node 111 of the tree data structure, and subsequent layers within the tree data structure 110 represent different tokens (or token groups), combined with each of the previously selected token combinations. At some point in time (e.g., after generating a tree with a defined depth (corresponding to the maximum length of the sequence generated by the draft model), the draft model may output the generated tree data structure 110 to the target model for further processing. In some aspects, the tree data structure 110 may be output to the target model along with the respective groupings and selection probabilities generated by the draft model.

[0051] In some aspects, the draft model can be configured to trigger the target model to generate tokens (and subsequently speculatively decode) based on various criteria. These criteria may include complexity or performance criteria, such as those associated with the size of the generated tree data structure 110. In some aspects, these criteria may include time criteria associated with the expected amount of time it takes for the target model to generate the set of tokens, against which the generated tree data structure 110 can be compared. Generally, these complexity and / or performance criteria may set an upper limit on the number of tokens generated by the draft model for validation by the target model. In some aspects, this upper limit may be based on the number of nodes in the tree data structure and may be influenced, for example, by branching factors defined for different layers of the tree data structure 110 to which the sampled tokens are organized, the depth of the tree data structure 110, etc. The worst-case computational load of the final round of speculative token generation may be configured to be defined by the memory bandwidth at the device on which the draft model executes.

[0052] In some respects, the number of nodes at each level of the tree data structure 110 (e.g., where...) Figure 1 The token n shown corresponds to the nth level in the tree data structure 110, and level 0 corresponds to the root node 111 in the tree data structure 110. The depth of the tree data structure 110 can be defined a priori. The number of nodes at each level of the tree can be defined globally, on a per-level basis, or in some other way. For example, the number of nodes at any level of the tree data structure 110 can be defined based on the branch factor of the tree data structure 110 at the immediately preceding level. For example, a branch factor of 2 for a node at the nth level of the tree data structure 110 can result in 2 nodes (tokens) being generated at the (n+1)th level of the tree data structure 110 for each node (token) at the nth level of the tree. Meanwhile, the depth of the tree data structure 110 can be defined based on the maximum number of tokens (e.g., words) generated using any round of the draft model. For example, if the draft model is configured to generate a sequence of tokens with a maximum length of 5 during any instance of speculative generation, the depth of the tree can be 6 (to include the root node 111 corresponding to the input to the draft model in the first level of the tree data structure 110).

[0053] The target model recursively performs rejection sampling on (1) tokens generated by the draft model and included in the generated tree data structure 110 and (2) probability distribution q provided as input to the target model. Rejection sampling can be performed recursively at each node in the generated tree, wherein the termination condition for token selection by the target model at a given layer of the tree data structure 110 is modeled as a recursive problem, where the termination condition of the recursive problem is to accept a token or reject all tokens. During recursive rejection sampling, the target model may accept or reject tokens and adjust the probability distribution used to validate subsequent tokens in the generated tree. If a token is rejected, the probability distribution is updated. This can be generated to evaluate subsequent tokens in the tree, where p represents the probability associated with the rejected token from the original probability distribution q. Subsequently, the probability distribution is updated. This can be used to evaluate the next token in the tree. The resulting set of selected tokens 112 can be recursively identified by traversing from the root node 111 of the generated tree data structure 110 based on an updated probability distribution generated for each node in the generated tree data structure, as discussed in further detail below.

[0054] In one example, recursive rejection sampling can be performed using a "greedy" technique, where the first accepted token is returned as a valid part of the response to the input query (or prompt). In another example, recursive rejection sampling can be performed to determine whether to accept each token at a given layer in the tree data structure 110. The token selected at a given layer in the tree data structure 110 can, for example, be the token with the highest probability of being included as a valid token in the response to the input prompt. In yet another example, a cumulative probability distribution can be generated for each sequence of accepted tokens from the generated tree data structure, and the sequence with the largest cumulative probability distribution can be selected as the response to the input prompt. Of course, it should be recognized that these are merely examples of techniques upon which the selection of token sequences from the tree data structure 110 is based, and other techniques for selecting token sequences based on recursive rejection sampling (and corresponding adjustments to the probability distribution q when a token is rejected) can be considered.

[0055] In some respects, the draft model can match the probability distribution of the target model, but can have faster inference performance on the same hardware. Generally, a smaller model can generate many speculative tokens, but the probability that these tokens will be rejected by the target model increases. The speculative decoding techniques discussed in this paper address this increase in the probability of token rejection, at the cost of increased computational overhead for longer sequences. Finally, at the draft model, a temperature parameter (generally used in this paper to refer to a parameter affecting the draft model's likelihood of selecting tokens (words) with lower probabilities) can be tuned to improve the performance of recursive speculative decoding. In some respects, the draft model can be fine-tuned to match or at least approximate the probability distribution of the target model, maximizing or at least increasing the probability that speculatively generated tokens (or token sequences) generated by the draft model will be accepted as valid tokens by the target model.

[0056] Generally, recursive speculative decoding can improve token generation performance compared to per-token speculative decoding. That is, for any given Kullback-Leibler (KL) divergence between the draft and target models, the number of tokens generated per target model may be greater for recursive speculative decoding than per-token speculative decoding. KL divergence typically measures the difference between the probability distribution of the draft model and the probability distribution of the target model (which is considered the reference distribution). Different selection strategies (e.g., group size, additional tokens, etc.) can have different computational complexity characteristics. Accordingly, given the bounds of read bandwidth for the draft and target models and hardware performance, the choice of strategy for selecting tokens for acceptance or rejection using recursive speculative decoding can be based on a trade-off between computational complexity and performance.

[0057] Figure 2A and 2B Examples 200A and 200B of recursive inference decoding in generative artificial intelligence models based on various aspects of this disclosure are explained.

[0058] Figure 2AExample 200A, as explained, depicts an instance where one of several tokens generated by a draft model is accepted to be included in the response to prompt 210. As explained, the draft model generates four proposed tokens X1 220A, X2 220B, X3 220C, and X4 220D, each with associated probabilities p1, p2, p3, and p4 from the original target distribution q. The target model can sequentially examine these tokens to determine whether to accept or reject each token. As explained, the target model first examines token X1 220A to determine whether token X1 220A should be accepted or rejected as part of the response to a given input (e.g., prompt 210). In this example, the target distribution q1 222A can be set as the original target distribution q, and the target model can determine whether to accept or reject token X1 220A based on selection criterion 224A:

[0059]

[0060] Where U1 represents the generated random number between [0,1].

[0061] If token X1 220A is accepted (not shown), token X1 220A can be output, and the target model analysis of the proposed token generated from the draft model can proceed to the traversal tree to analyze the nodes (tokens) connected to the node represented by token X1 220A in the generated tree.

[0062] Otherwise, if token X1 220A is rejected, as explained, the objective model can proceed to determine whether token X2 220B should be accepted or rejected. In doing so, the objective model can use a new objective distribution q2 222B, which can be the result of subtracting the probability p1 associated with the rejected token X1 220A from the objective distribution q1 222A, such that q2 = (q1 - p1). + The target model can then determine whether to accept or reject the token X2 220B based on selection criterion 224B:

[0063]

[0064] U2 also represents the generated random number between [0,1].

[0065] Similar to token X1 220A discussed above, the target model can determine that token X2 220B satisfies the acceptance criterion and thus return token X2 220B as the selected token (not shown). Otherwise, if token X2 220B is rejected, as explained, the target model proceeds to determine whether token X3 220C should be accepted or rejected using the updated target distribution q3 222C and acceptance criterion U3 224C, by removing p2 from q2 (e.g., so that q3 = (q2 - p2)). + As explained, the process can continue until a token is determined to be accepted (in this example, token X4 220D is determined to be accepted using the updated target distribution q4 222D as explained in Example 200A) (e.g., based on acceptance criterion U4 224D) and the output is the selected token Y 230.

[0066] Figure 2B Example 200B, as explained, illustrates an instance where the target model rejects each of the tokens 220A-220D generated by the draft model. In this example, the target model can generate a final target distribution 222E, sample tokens from this final target distribution 222E, and terminate traversal of the generated tree. As explained, the final target distribution q5 222E can be the result of subtracting the probability associated with the final token in the generated token set (e.g., the probability p4 associated with token X4 220D). The final target distribution q5 222E can be obtained from the equation q5=(q4-p4). + The tokens 240 sampled from the final target distribution q5 222E can be returned as the output Y of performing recursive rejection sampling on the generated tokens.

[0067] Figure 3 An example tree data structure 300 for generating tokens using recursive speculative decoding in a generative artificial intelligence model, based on various aspects of this disclosure, is explained.

[0068] As explained, the tree data structure 300 comprises multiple layers, starting from the root node 310, which represents the input to the generative artificial intelligence model (which can be tokenized), as well as tokens previously generated by the draft model and accepted by the target model using recursive rejection sampling, as discussed above. Given some previously generated set of tokens, each layer 320, 330, and 340 (and / or) in the tree data structure 300 Figure 3(Other layers not explained in the text) can be the result of speculatively generating several tokens from the draft model. The number of tokens generated for a node at level n of tree data structure 300 in level n+1 of tree data structure 300 can be determined based on a branching factor defined for a node at level n of tree data structure 300. As discussed, the branching factor can be the same across all layers of tree data structure 300, or the branching factor can be defined for each layer of tree data structure 300. In some aspects, the branching factor can be defined such that nodes at lower levels of tree data structure 300 generate a larger number of branches and nodes at higher levels of tree data structure 300 generate a smaller number of branches, or vice versa.

[0069] To traverse the tree data structure 300 and select a set of tokens from the speculatively generated token set, the target model can recursively accept or reject tokens (e.g., tokens represented by nodes that are direct children of the root node 310) starting from the first set of tokens generated by the draft model at the first level 320 of the tree data structure 300. Generally, while traversing the tree, the target model can determine whether a token should be accepted or rejected and adjust the target distribution based on this determination. At the first level 320 of the tree data structure 300, if the first token is rejected, the target model can remove the first token from the target distribution (as discussed above with respect to Figure 2) and proceed to determine whether the second token at the first level of the tree should be accepted or rejected based on the updated target distribution. When rejecting the first token at the first level 320 of the tree data structure 300, the target model does not need to analyze tokens represented by nodes that are directly or indirectly children of the node representing the rejected first token in the tree data structure 300.

[0070] Generally, when traversing the tree data structure 300, the target distribution can be recursively modified until a termination condition is met. The termination condition could be, for example, a sequence of tokens identifying an acceptable response output accepted as input (represented by the root node 310 of the tree data structure 300), plus additional nodes sampled from the modified target distribution after the token sequence is accepted by the target model. In another example, the termination condition could be determining that a token sequence speculatively generated by the draft model has not yet been accepted by the target model. In such cases, as described above... Figure 2B The additional tokens sampled from the final target distribution discussed herein can be output as a response to the input corresponding to the root node 310 in the tree data structure 300. As discussed above, the target distribution can be a target distribution generated by removing the probabilities associated with the generated tokens from the original target distribution generated by the draft model.

[0071] exist Figure 3In the example explained, at the first level 320 of the tree data structure 300, the first token 322A can be rejected. Because the first token 322A is rejected as one of the candidate tokens to be included in the response to the input corresponding to the root node 310, tokens that are child tokens of the first token 322A in the tree data structure 300 do not need to be checked for selection as candidate tokens. Therefore, the direct child tokens of the first token 322A in the second level 330 of the tree data structure 300 and the grandchild tokens of the first token 322A in the third level 340 of the tree data structure 300 can be discarded or ignored.

[0072] When rejecting the first token 322A from the set of candidate tokens in the first layer 320 of the tree data structure 300, as discussed, the target probability distribution associated with the tokens in the first layer 320 of the tree data structure 300 can be modified to remove the probability value associated with the first token 322A. The updated target distribution can then be used to examine the second token 322B in the first layer 320 of the tree data structure 300. As explained, the second token 322B can be accepted as a candidate token to be included in the response to the input corresponding to the root node 310, and the sub-tokens (e.g., 332A, 332B, and 332C) of the second token 322B that can proceed to the second layer 330 of the tree data structure 300 are analyzed.

[0073] As explained, tokens 332A and 332B can be rejected by the target model. When rejecting tokens 332A and 332B, the target probability distribution used to accept or reject tokens in the second layer 330 of the tree data structure 300 can be adjusted to remove probabilities associated with tokens 332A and 332B. This adjusted target probability distribution can then be used to determine whether to accept or reject token 332C in the second layer 330 of the tree data structure 300. As explained, the target model can accept token 332C and analyze the sub-tokens (e.g., 342A, 342B) of token 332C that can proceed to the third layer 340 of the tree data structure 300. In the third layer 340 of the tree data structure 300, the target model can use the techniques discussed above to examine tokens 342A and 342B, and can accept token 342A as a candidate to be included in the response to the input corresponding to the root node 310.

[0074] In some respects, accepting the second token 322B as a candidate token to be included in the response to the input corresponding to the root node 310 does not preclude analyzing the third token 322C to include or reject it as a candidate token to be included in the response to the input corresponding to the root node 310. In such cases, accepting or rejecting sub-tokens (and other tokens) in layers 320 and 330 of the tree data structure 300 Figure 3(Not explained in the text) can be performed as discussed above regarding the acceptance or rejection of the sub-tokens of the second token 322B. In some aspects, such as when the probability of the second token 322B being a candidate token for the response to the input corresponding to the root node 310 exceeds a threshold, the third token 322C can be considered rejected, and the analysis of the sub-tokens of the third token 322C can be bypassed.

[0075] Compared to per-token generation and acceptance, the recursive speculative decoding technique discussed in this paper can lead to a significant gain in token generation performance. In general, compared to per-token speculative decoding, recursive speculative decoding allows for an increase in token acceptance performance (e.g., the number of tokens accepted within a given draft length) per execution of the target model. The amount of performance improvement scales with increasing branches generated for any token and increasing draft length. Furthermore, compared to per-token speculative decoding, the recursive speculative decoding technique discussed in this paper provides an increasing rate of increase in token generation (over time).

[0076] Example operations for recursive speculative decoding in generative artificial intelligence models.

[0077] Figure 4 The present disclosure describes aspects of generative artificial intelligence models that can be executed by computing devices to use (e.g., as described herein). Figures 1 to 3 Example operation 400 (discussed) to generate a response to input prompts. Operation 400 can be performed by a computing device on which at least the target model can be deployed, such as a laptop computer, desktop computer, server, cloud computing instance hosted in a distributed computing environment, etc.

[0078] As explained, operation 400 at box 410 begins by receiving multiple sets of tokens. Generally, the multiple sets of tokens can be tokens generated based on input prompts and a first generative artificial intelligence model (e.g., by a draft model deployed at a client device from which the multiple sets of tokens are received). Input prompts can be received at the client device, for example via text input prompts, audio capture prompts, or other technologies through which input can be received from the user. Generally, each of the multiple sets of tokens can correspond to a candidate answer for the input prompt.

[0079] In some respects, multiple token sets can be organized into a tree data structure (e.g., Figure 1 The tree data structure explained in 110 or Figure 3The tree data structure described in section 300 may have a root node and one or more leaf nodes. An input cue may correspond to the root node of the tree data structure. Each path through the tree data structure may correspond to a different candidate response to the input cue. It should be understood that a path through the tree data structure does not need to terminate at a node in the tree data structure that does not have a leaf node; that is, a candidate response may correspond to a path that partially traverses the tree data structure (e.g., to reflect the rejected token in the response).

[0080] In some respects, the number of tokens at any given level of a tree data structure can be based on the branching factor associated with the preceding layer in the tree data structure. A tree data structure can have different branching factors assigned to different levels within the tree data structure. Generally, a higher branching factor allows for the generation of a larger number of tokens at any given level of the tree data structure and generally increases the total number of tokens included in the tree data structure. Conversely, a lower branching factor restricts the generation of tokens at any given level of the tree data structure and generally reduces the total number of tokens included in the tree data structure.

[0081] In some respects, the size of each token set can be based on a metric of computational complexity associated with the generation of the target token set by the second generative artificial intelligence model.

[0082] In box 420, operation 400 proceeds to using a second generative artificial intelligence model and recursive adjustments to the probability distribution to select a set of tokens from multiple sets of tokens.

[0083] In some respects, recursive adjustments to the probability distribution involve rejecting tokens from multiple sets of tokens as candidates for inclusion as input prompts. Generally, a token may be rejected if a defined acceptance criterion based on the target distribution and the probabilities associated with tokens in the original target distribution is not met. The current distribution used by the target model to determine whether a token should be accepted or rejected can be modified by removing (or subtracting) the probabilities associated with tokens in the original target distribution, resulting in an updated target distribution that can be used to determine whether the next token should be accepted or rejected as a candidate for inclusion as an input prompt.

[0084] In some aspects, selecting a token set from multiple token sets may include rejecting a first token at the first level of a tree data structure representing the multiple token sets. An adjusted probability distribution may be generated based on the rejection of the first token. Within the tree data structure, sub-tokens of the first token at levels deeper than the first level of the tree data structure may be discarded. Second tokens at the first level of the tree data structure may be accepted or rejected based on the adjusted probability distribution.

[0085] In some aspects, selecting a token set from multiple token sets may include rejecting each token set generated by a first generative AI model. Tokens may be sampled using a second generative AI model based on a target distribution of probabilities that exclude each token set generated by the first generative AI model.

[0086] In box 430, operation 400 proceeds to output the selected set of tokens as a response to the input prompt.

[0087] In some respects, the first generative AI model may correspond to a draft model in the speculative decoding pipeline, and the second generative AI model may correspond to a target model in the speculative decoding pipeline. The first generative AI model can be executed on a client device, and the second generative AI model can be executed on a remote system (such as a server computer, cloud computing instance, etc.) of the client device. In some respects, the first and second generative AI models can be executed on the same device.

[0088] In some respects, the first generative AI model and the second generative AI model can have equivalent probability distributions.

[0089] In some respects, the first generative AI model may have a probability distribution that approximates the probability distribution associated with the second generative AI model.

[0090] Example self-predictive decoding in generative artificial intelligence models

[0091] In some respects, various types of speculative decoding (including group speculative decoding and recursive speculative decoding) can be implemented using a single generative AI model that combines the functionality of the draft and target models discussed above. In doing so, draft token generation, target token generation, and token acceptance can be parallelized within a single generative AI model. For example, using a single generative AI model reduces the computational overhead involved in generating both the target and draft models, improves the performance of generative tasks by performing token verification and speculative generation in a single pass through the single generative AI model, reduces the amount of memory used to store the model for speculative decoding of generative tasks, and so on.

[0092] Figure 5 An example pipeline 500 for self-predictive decoding in a generative artificial intelligence model, according to various aspects of this disclosure, is described. In some aspects, pipeline 500 can be used to generate and analyze a tree data structure of tokens associated with candidate responses to query inputs in a generative intelligent model, such as... Figure 3 The tree data structure explained in the text is 300, etc.

[0093] As explained, pipeline 500 uses a single generative AI model to speculatively generate tokens and verify the speculatively generated tokens. During the first round of inference in pipeline 500, a first token set 502 is speculatively generated. As explained, for example, the first token set 502 may include tokens 1 through 4 and may be provided as input during the second round of inference in pipeline 500 to speculatively generate the next token set as a batch process for generating multiple token sets. While the speculatively generated first token set is processed by the single generative AI model, the single generative AI model continues to speculatively generate multiple second draft token sets 504, 506, 508, and 510 in the second round of inference in pipeline 500.

[0094] When generating the second draft token sets 504, 506, 508, and 510, assumptions can be made about different numbers of accepted tokens from the first token set 502. For example, as explained, the second draft token set 504 may assume acceptance of the first draft token from the first token set 502 and may include a token set speculatively generated based on the acceptance of the first token. The second draft token set 506 may assume acceptance of both the first and second draft tokens from the first token set 502 and may include a token set speculatively generated based on the acceptance of the first and second tokens. The second draft token set 508 may assume acceptance of the first through third draft tokens from the first token set 502 and may include a token set speculatively generated based on the acceptance of the first through third tokens. Finally, the second draft token set 510 may assume acceptance of all four tokens from the first token set 502 and may include a token set speculatively generated based on the acceptance of all four tokens. In each respect, for cases where fewer tokens than those included in the first token set 502 are assumed to be accepted, padding 503 (e.g., null values, predefined constants, etc.) may be added so that each assumption has the same length.

[0095] Once a single generative AI model completes its rejection sampling of the speculatively generated token set, it selects from the first set the speculatively generated token set associated with the accepted token set as input to the single generative AI model for another round of inference using that single generative AI model to speculatively generate tokens. In this example, it can be seen that all four tokens in the first token set 502 have been accepted by the single generative AI model as draft verification 512, and thus, the token set 510 can be used to further speculatively generate tokens using the single generative AI model.

[0096] The above process can continue until a termination event occurs. Successive rounds of speculative generation can be based on the assumption that several tokens generated from the previous round of speculative generation are accepted by a single generative AI model. For example, such as... Figure 5 As explained, draft token sets 522, 524, 526, and 528 can be generated in the (k+1)th round of inference, where the tokens included in these draft token sets are based on several tokens speculatively generated in addition to the N accepted tokens generated in the (k-1)th round of inference. In this example, it can be seen that the four speculatively generated tokens generated during the kth round of inference have been accepted as draft verification 520, and tokens N+5 to N+8 can be used to further speculatively generate tokens using a single generative artificial intelligence model.

[0097] In some respects, a termination event may include generating special tokens used to mark the end of the response (e.g., because the probabilities associated with these tokens have fallen below an acceptable threshold probability value, so no further tokens can reasonably be included in the response). In some respects, a termination event may be achieved when a threshold number of tokens have been generated.

[0098] In some respects, when all tokens generated from the previous round of speculative token generation are rejected by a single generative AI model, the process can be restarted by providing the last accepted set of tokens plus tokens sampled from the final distribution as input to the single generative AI model (e.g., as discussed above).

[0099] Figure 6 Example architectures 600A and 600B for self-predictive decoding in generative artificial intelligence models, according to various aspects of this disclosure, are described. Both example architectures 600A and 600B allow for self-predictive decoding in any round of the model (such as in...). Figure 1 , 2A (In the token generation process explained in 2B, 3, 4 and 5) multiple tokens are generated, as discussed above.

[0100] In example architecture 600A, a generative AI model 610 can be trained to generate multiple predictive cue embeddings 612 attached to a set of input tokens, allowing multiple output tokens 614 to be generated in parallel. These predictive cue embeddings can be embeddings corresponding to tokens included in the response to the input cue (including any previously generated and accepted tokens). The generative AI model 610 can be a generative AI model updated using various fine-tuning techniques, such as a pre-trained large language model or other pre-trained generative AI models. For example, a generative AI model used to generate text responses to text input (also known as a large language model) can be updated using techniques such as low-rank adaptation (LoRA) for large language models.

[0101] In example architecture 600B, the generative AI model can be implemented as a partially autoregressive model. The inference operation used to speculatively generate tokens can be performed using a subset of layers in the partially autoregressive model (e.g., the top n layers or the bottom n layers of the model). In doing so, the layers used to speculatively generate tokens can create a context that allows modeling causal and / or other relationships for the speculatively generated tokens, which can be fed as input to the part of the model that validates these tokens as valid responses to input cues.

[0102] Architecture 600B can be implemented in various ways such that autoregressive inference and the generation of multiple token sets for acceptance and / or rejection can be generated using a small number of autoregressive layers in the generative AI model. In example implementation 620, the generative AI model may include multiple non-autoregressive layers 622A-622C and an autoregressive layer 624. The layers in the generative AI model may be organized into a stack, where the lowest layer in the stack corresponds to the layer that receives input for processing, and the highest layer in the stack corresponds to the layer that generates output. In implementation 620, non-autoregressive layers 622A-622C may be placed at the bottom of the stack, and autoregressive layer 624 may be placed at the top of the stack. Conversely, in example implementation 630, the layers of the generative AI model may be organized such that autoregressive layer 632 is placed at the bottom of the stack, and non-autoregressive layers 632A-632C are placed at the top of the stack. These autoregressive layers 624 and 632 can, for example, operate in a loop to continuously generate and receive tokens to be output as a response to input prompts (and, in some respects, previously generated tokens are included as part of the response to input prompts).

[0103] Example operations for generating responses to input queries using self-predictive decoding in generative artificial intelligence models.

[0104] Figure 7 Example operation 700, which can be performed by a computing device according to various aspects of this disclosure to generate a response to an input prompt using a generative artificial intelligence model, is described. Operation 700 can be performed by a device on which a generative artificial intelligence model, which can at least serve as a draft model and a target model, is deployed, such as a smartphone, tablet computer, laptop computer, desktop computer, server, cloud computing instance hosted in a distributed computing environment, etc.

[0105] As explained, operation 700 at box 710 begins by generating a first plurality of token sets based on input prompts and a generative artificial intelligence model. In some aspects, these token sets may include a sequence of tokens that can be partially or fully accepted as candidate responses to input prompts.

[0106] In some respects, the first plurality of token sets can be represented as a tree data structure. Within this tree data structure, the input hint can correspond to the root node. Each token set in the first plurality of token sets can be represented by a navigable path through the tree data structure (e.g., as mentioned above regarding...). Figure 3 (As discussed).

[0107] In some respects, the depth of this tree data structure corresponds to the maximum number of tokens generated in a single pass through a generative artificial intelligence model.

[0108] In some respects, the depth of this tree data structure corresponds to the maximum number of tokens generated in a single pass through a generative artificial intelligence model.

[0109] In some respects, the maximum size of this tree data structure can be set based on a metric of computational complexity associated with the set of tokens generated by the generative artificial intelligence model.

[0110] In box 720, operation 700 proceeds to generate a second set of tokens speculatively using a generative artificial intelligence model.

[0111] In box 730, operation 700 proceeds to select a token set from the first multiple token set while speculatively generating a second multiple token set.

[0112] In some respects, selecting a token set from a first plurality of token sets may include selecting the longest sequence of accepted tokens from the first plurality of token sets.

[0113] In some aspects, the token sets in the second plurality of token sets may include padding to account for the number of tokens selected from the token sets in the first plurality of token sets. For example, for the number N of tokens included in each token set from the first plurality of token sets, the selected plurality of tokens may include 0 to N tokens. In such cases, the padding included in the token sets in the second plurality of token sets may include no padding (corresponding to accepting all N tokens from the token sets in the first plurality of token sets), one padding token (corresponding to accepting N-1 tokens), two padding tokens (corresponding to accepting N-2 tokens), and so on.

[0114] In some respects, to select a set of tokens from a first plurality of tokens, the first token at the first level of the tree data structure can be rejected. Based on the rejection of the first token, n adjusted probability distributions can be generated, and the tree data structure can be pruned by discarding child tokens of the first token from the tree data structure. Child tokens of the first token can be discarded because rejecting the first token generally implies that child tokens of the first token are unlikely to be valid responses to input prompts. The second token at the first level of the tree data structure can then be determined, based on the adjusted probability distributions, whether to accept or reject it.

[0115] In some aspects, selecting a token set from a first plurality of token sets may include rejecting each token set in the first plurality of token sets generated by the generative artificial intelligence model. Tokens may be sampled using the generative artificial intelligence model based on a target distribution of probabilities that exclude each token set in the first plurality of token sets. The token set selected from the first plurality of token sets may include the sampled tokens.

[0116] In box 740, operation 700 proceeds to output the set of tokens selected from the first plurality of tokens and the associated set of tokens from the second plurality of tokens as a response to the input prompt.

[0117] In some respects, generative AI models include those trained to generate multiple tokens based on predicted cue embeddings in response to input cues.

[0118] In some aspects, generative artificial intelligence models include models comprising one or more non-autoregressive layers and one or more autoregressive layers. In some aspects, one or more autoregressive layers may be included at the top of a layer stack representing a generative artificial intelligence model. In some aspects, one or more autoregressive layers may be included at the bottom of a layer stack representing a generative artificial intelligence model.

[0119] Example processing system for recursive speculative decoding in generative artificial intelligence models

[0120] Figure 8 An example processing system 800 for generating responses to query inputs to generative artificial intelligence models based on recursive speculative decoding is described, such as those described herein. Figure 4 As described.

[0121] The processing system 800 includes a central processing unit (CPU) 802, which in some examples may be a multi-core CPU. Instructions executed at the CPU 802 may be loaded, for example, from program memory associated with the CPU 802 or from a memory partition (e.g., memory 824).

[0122] The processing system 800 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, and a connectivity component 812.

[0123] NPUs (such as the NPU 808) are generally configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0124] NPUs (such as the NPU 808) are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, such NPUs may be part of a dedicated neural network accelerator.

[0125] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0126] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0127] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and quickly process those segments through an already trained model to generate model outputs (e.g., inference).

[0128] In some implementations, the NPU 808 is part of one or more of a CPU 802, a GPU 804, and / or a DSP 806. These can reside in the user equipment (UE) of a wireless communication system or on another computing device.

[0129] In some examples, connectivity component 812 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Connectivity component 812 may be further coupled to one or more antennas 814.

[0130] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or a navigation processor 820, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0131] The processing system 800 may also include one or more input and / or output devices 822, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0132] In some examples, one or more processors of the processing system 800 may be based on the ARM or RISC-V instruction set.

[0133] The processing system 800 also includes a memory 824, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 824 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 800.

[0134] Specifically, in this example, memory 824 includes a token set receiving component 824A, a token selection component 824B, an output generation component 824C, and a generative artificial intelligence model 824D. The depicted components, as well as other undepicted components, can be configured to perform various aspects of the methods described herein.

[0135] Generally, the processing system 800 and / or its components can be configured to perform the methods described herein.

[0136] Example processing system for self-speculating decoding in generative artificial intelligence models

[0137] Figure 9 An example processing system 900 for generating responses to query inputs to generative artificial intelligence models based on self-inferential decoding is described, such as those described herein. Figure 7 As described.

[0138] The processing system 900 includes a central processing unit (CPU) 902, which in some examples may be a multi-core CPU. Instructions executed at the CPU 902 may be loaded, for example, from program memory associated with the CPU 902 or from a memory partition (e.g., memory 924).

[0139] The processing system 900 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, and a connectivity component 912.

[0140] NPUs (such as the NPU 908) are generally configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0141] NPUs (such as the NPU 908) are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip (such as a system-on-a-chip (SoC)), while in other examples, such NPUs may be part of a dedicated neural network accelerator.

[0142] An NPU can be optimized for training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0143] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0144] NPUs designed to accelerate inference are typically configured to operate on the full model. Such NPUs can thus be configured to take new data segments as input and quickly process those segments through an already trained model to generate model outputs (e.g., inference).

[0145] In some implementations, the NPU 908 is part of one or more of the CPU 902, GPU 904, and / or DSP 906. These can reside in the user equipment (UE) of a wireless communication system or on another computing device.

[0146] In some examples, connectivity component 912 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., LTE), fifth-generation connectivity (e.g., NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Connectivity component 912 may be further coupled to one or more antennas 914.

[0147] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0148] The processing system 900 may also include one or more input and / or output devices 922, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0149] In some examples, one or more processors of the processing system 900 may be based on the ARM or RISC-V instruction set.

[0150] The processing system 900 also includes a memory 924, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 924 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 900.

[0151] Specifically, in this example, memory 924 includes a token set generation component 924A, a token selection component 924B, an output generation component 924C, and a generative artificial intelligence model 924D. The depicted components, as well as other undepicted components, can be configured to perform various aspects of the methods described herein.

[0152] Generally, the processing system 900 and / or its components can be configured to perform the methods described herein.

[0153] Example Terms

[0154] Implementation details of each aspect of this disclosure are described in the following numbered clauses.

[0155] Item 1: A processor-implemented method comprising: receiving a plurality of token sets generated based on an input prompt and a first generative artificial intelligence model, each of the plurality of token sets corresponding to a candidate response to the input prompt; selecting a token set from the plurality of token sets using a second generative artificial intelligence model and a recursive adjustment of a target distribution associated with the received plurality of token sets; and outputting the selected token set as a response to the input prompt.

[0156] Clause 2: The method of Clause 1, wherein the multiple token sets are organized into a tree data structure.

[0157] Clause 3: The method of Clause 2, wherein the input prompt corresponds to the root node of the tree data structure, and each path through the tree data structure corresponds to a different candidate response to the input prompt.

[0158] Clause 4: The method of Clause 2 or 3, wherein the number of tokens at a layer in the tree data structure is based on the branching factor associated with the previous layer immediately adjacent to that layer in the tree data structure.

[0159] Clause 5: The method of any of Clauses 2 to 4, wherein the depth of the tree data structure corresponds to the maximum number of tokens generated by a single round of passage through the first generative artificial intelligence model.

[0160] Clause 6: The method of any of Clauses 1 to 5, wherein the size of each token set is based on a computational complexity metric associated with the generation of the target token set by the second generative artificial intelligence model.

[0161] Clause 7: The method of any of Clauses 1 to 6, wherein the recursive adjustment of the target distribution comprises: determining whether to accept or reject a first token from one of the plurality of token sets; and adjusting the probability distribution used to verify a second token in the token set after the first token based on the determination to accept or reject the first token.

[0162] Clause 8: The method of Clause 7, wherein adjusting the probability distribution includes subtracting the probability value associated with the first token from the probability distribution based on determining that the first token should be rejected.

[0163] Clause 9: The method of any of Clauses 1 to 8, wherein selecting the token set from the plurality of token sets comprises: rejecting a first token at a first level of a tree data structure representing the plurality of token sets; generating an adjusted probability distribution based on rejecting the first token; discarding a sub-token of the first token at a level deeper than the first level of the tree data structure from the tree data structure; and determining whether to accept or reject a second token at the first level of the tree data structure based on the adjusted probability distribution.

[0164] Clause 10: The method of any of Clauses 1 to 9, wherein selecting the token set from the plurality of token sets comprises: rejecting each token set generated by the first generative artificial intelligence model; and sampling the tokens using a second generative artificial intelligence model based on a target distribution of probabilities that exclude each token set generated by the first generative artificial intelligence model, wherein the selected token set includes the sampled tokens.

[0165] Clause 11: The method of any of Clauses 1 to 10, wherein: a first generative artificial intelligence model corresponds to a draft model in the speculative decoding pipeline, and a second generative artificial intelligence model corresponds to a target model in the speculative decoding pipeline.

[0166] Item 12: A processor-implemented method comprising: generating a first plurality of token sets based on an input prompt and a generative artificial intelligence model, each of the first plurality of token sets corresponding to a first portion of a candidate response to the input prompt; speculatively generating a second plurality of token sets using the generative artificial intelligence model, each of the second plurality of token sets based on a second portion of the first plurality of token sets corresponding to the candidate response to the input prompt; selecting a token set from the first plurality of token sets while speculatively generating the second plurality of token sets; and outputting the token set selected from the first plurality of token sets and the associated token set from the second plurality of token sets as a response to the input prompt.

[0167] Clause 13: The method of Clause 12, wherein selecting a token set from a first plurality of token sets comprises: selecting the longest sequence of accepted tokens from the first plurality of token sets.

[0168] Clause 14: The method of Clause 12 or 13, wherein the token set in the second plurality of token sets includes: padding to account for the number of tokens in the token set selected from the first plurality of token sets.

[0169] Clause 15: The method of any of Clauses 12 to 14, wherein the first plurality of token sets is represented as a tree data structure, and the root node of the tree data structure corresponds to the input prompt.

[0170] Clause 16: The method of Clause 15, wherein the depth of the tree data structure corresponds to the maximum number of tokens generated by a single round of passage through the generative artificial intelligence model.

[0171] Clause 17: The method of Clause 15 or 16, wherein the maximum size of the tree data structure is set based on a computational complexity metric associated with the set of tokens generated by the generative artificial intelligence model.

[0172] Clause 18: The method of any of Clauses 15 to 17, wherein selecting the first plurality of tokens comprises: rejecting a first token at a first level of a tree data structure representing the first plurality of tokens; generating an adjusted probability distribution based on rejecting the first token; discarding sub-tokens of the first token from the tree data structure; and determining whether to accept or reject a second token at the first level of the tree data structure based on the adjusted probability distribution.

[0173] Clause 19: The method of any of Clauses 12 to 18, wherein selecting the token set from the first plurality of token sets comprises: rejecting each of the first plurality of token sets generated by the generative artificial intelligence model; and sampling tokens using the generative artificial intelligence model based on a target distribution of probabilities that exclude each of the first plurality of token sets, wherein the token set selected from the first plurality of token sets includes the sampled tokens.

[0174] Clause 20: The method of any of Clauses 12 to 19, wherein the generative artificial intelligence model includes a generative artificial intelligence model trained to generate multiple tokens based on predictive prompt embeddings in response to input prompts.

[0175] Clause 21: The method of any of Clauses 12 to 20, wherein the generative artificial intelligence model includes a model comprising one or more non-autoregressive layers and one or more autoregressive layers.

[0176] Item 22: The method of Item 21, wherein the one or more autoregressive layers comprise one or more layers on top of a layer stack representing a generative artificial intelligence model.

[0177] Clause 23: The method of Clause 21 or 22, wherein the one or more autoregressive layers are included at the bottom of one or more layers representing a layer stack of generative artificial intelligence models.

[0178] Clause 24: A processing system comprising: at least one memory having executable instructions stored thereon; and one or more processors configured to execute the executable instructions to cause the processing system to perform any of the operations of Clauses 1 to 23.

[0179] Clause 25: A processing system comprising: means for performing operations as described in any of Clauses 1 to 23.

[0180] Clause 26: A non-transient computer-readable medium having on it executable instructions that, when executed by one or more processors, perform any of the operations of Clauses 1 to 23.

[0181] Additional considerations

[0182] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein are not intended to limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, changes may be made to the function and arrangement of the elements in discussion without departing from the scope of this disclosure. Various procedures or components may be appropriately omitted, substituted, or added to various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Moreover, features described with reference to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice a method. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that supplement or differ from the aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be implemented by one or more elements of the claims.

[0183] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” should not be construed as superior to or better than the others.

[0184] As used in this article, the phrase “at least one of” a list of items refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0185] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertaining, and similar actions. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and similar actions. Likewise, "determine" can also include parsing, selecting, choosing, building, and similar actions.

[0186] The methods disclosed herein include one or more steps or actions for implementing the method. These method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, the various operations of the above methods can be performed by any suitable means capable of performing the corresponding functions. These means may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations illustrated in the drawings are present, these operations may have corresponding paired means with similar numbers plus functional components.

[0187] The following claims are not intended to be limited to the aspects shown herein, but should be granted the full scope consistent with the language of the claims. Within the claims, references to singular elements are not intended to mean “one and only one” (unless specifically stated so), but rather “one or more.” Unless specifically stated otherwise, the term “some / a” refers to one or more. No element of the claims should be interpreted in accordance with the provisions of 35 USC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “steps for…”. Elements of all aspects described throughout this disclosure that are now or hereafter known to a person skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be donated to the public, whether or not such disclosure is expressly stated in the claims.

Claims

1. A processing system, comprising: It has at least one memory storing executable instructions; as well as One or more processors, the one or more processors being configured to execute the executable instructions to cause the processing system to: Receive multiple sets of tokens generated based on input prompts and a first generative artificial intelligence model, each of the multiple sets of tokens including a sequence of tokens corresponding to candidate responses to the input prompts; A second generative artificial intelligence model is used, along with a recursive adjustment of the target distribution associated with the multiple received token sets, to select a token set from the multiple token sets. as well as Output the selected set of tokens as a response to the input prompt.

2. The processing system of claim 1, wherein the plurality of token sets are organized into a tree data structure.

3. The processing system of claim 2, wherein the root node of the tree data structure corresponds to the input prompt, and wherein each path through the tree data structure corresponds to a different token sequence corresponding to the candidate response to the input prompt.

4. The processing system as described in claim 2, wherein: The tree data structure includes multiple layers, each layer corresponding to a token in the token sequence, and The number of tokens at a specific layer in the tree data structure is based on the branching factor associated with the previous layer immediately adjacent to that specific layer in the tree data structure.

5. The processing system of claim 2, wherein the depth of the tree data structure corresponds to a parameter defining the maximum number of tokens generated by a single round of passage through the first generative artificial intelligence model.

6. The processing system of claim 1, wherein the size of each token set is based on a computational complexity metric associated with the generation of the target token set by the second generative artificial intelligence model.

7. The processing system of claim 1, wherein the recursive adjustment of the target distribution comprises: Determine whether to accept or reject the first token from the token set among the plurality of token sets; as well as The probability distribution of the second token in the token set used to verify the first token is adjusted based on whether to accept or reject the first token.

8. The processing system of claim 7, wherein, in order to adjust the probability distribution, the one or more processors are configured to cause the processing system to subtract the probability value associated with the first token from the probability distribution based on determining that the first token should be rejected.

9. The processing system of claim 1, wherein, in order to select the token set from the plurality of token sets, the one or more processors are configured to cause the processing system to: Reject the first token representing the first level of the tree data structure containing the multiple token sets; An adjusted probability distribution is generated based on rejecting the first token; Discard or ignore sub-tokens of the first token located at a level deeper than the first level of the tree data structure from the tree data structure; as well as The decision to accept or reject the second token at the first level of the tree data structure is based on the adjusted probability distribution.

10. The processing system of claim 1, wherein, in order to select the token set from the plurality of token sets, the one or more processors are configured to cause the processing system to: Reject each set of tokens generated by the first generative artificial intelligence model; and The second generative artificial intelligence model samples tokens based on a target distribution that excludes the probability associated with each set of tokens generated by the first generative artificial intelligence model, wherein the selected set of tokens includes the sampled tokens.

11. The processing system of claim 1, wherein: The first generative artificial intelligence model corresponds to the draft model in the speculative decoding pipeline, and The second generative artificial intelligence model corresponds to the target model in the speculative decoding pipeline.

12. A processor-implemented method, comprising: Receive multiple sets of tokens generated based on input prompts and a first generative artificial intelligence model, each of the multiple sets of tokens including a sequence of tokens corresponding to candidate responses to the input prompts; A second generative artificial intelligence model is used, along with a recursive adjustment of the target distribution associated with the multiple received token sets, to select a token set from the multiple token sets. as well as Output the selected set of tokens as a response to the input prompt.

13. The method of claim 12, wherein the plurality of token sets are organized into a tree data structure.

14. The method of claim 13, wherein the root node of the tree data structure corresponds to the input prompt, and wherein each path through the tree data structure corresponds to a different token sequence corresponding to the candidate response to the input prompt.

15. The method of claim 13, wherein: The tree data structure includes multiple layers, each layer corresponding to a token in the token sequence, and The number of tokens at a specific layer in the tree data structure is based on the branching factor associated with the previous layer immediately adjacent to that specific layer in the tree data structure.

16. The method of claim 13, wherein the depth of the tree data structure corresponds to a parameter defining the maximum number of tokens generated by a single round of passage through the first generative artificial intelligence model.

17. The method of claim 12, wherein the size of each token set is based on a computational complexity metric associated with the generation of the target token set by the second generative artificial intelligence model.

18. The method of claim 12, wherein the recursive adjustment of the target distribution comprises: Determine whether to accept or reject the first token from the token set among the plurality of token sets; as well as The probability distribution of the second token in the token set used to verify the first token is adjusted based on whether to accept or reject the first token.

19. The method of claim 18, wherein adjusting the probability distribution comprises: The probability value associated with the first token is subtracted from the probability distribution based on the determination to reject the first token.

20. The method of claim 12, wherein selecting the token set from the plurality of token sets comprises: Reject the first token representing the first level of the tree data structure containing the multiple token sets; An adjusted probability distribution is generated based on rejecting the first token; Discard or ignore sub-tokens of the first token located at a level deeper than the first level of the tree data structure from the tree data structure; as well as The decision to accept or reject the second token at the first level of the tree data structure is based on the adjusted probability distribution.

21. The method of claim 12, wherein selecting the token set from the plurality of token sets comprises: Reject each set of tokens generated by the first generative artificial intelligence model; as well as The second generative artificial intelligence model samples tokens based on a target distribution that excludes the probability associated with each set of tokens generated by the first generative artificial intelligence model, wherein the selected set of tokens includes the sampled tokens.

22. The method of claim 12, wherein: The first generative artificial intelligence model corresponds to the draft model in the speculative decoding pipeline, and The second generative artificial intelligence model corresponds to the target model in the speculative decoding pipeline.

23. A processing system, comprising: A means for receiving multiple sets of tokens generated based on an input prompt and a first generative artificial intelligence model, each of the multiple sets of tokens including a sequence of tokens corresponding to candidate responses to the input prompt; A means for selecting a set of tokens from the plurality of token sets using a second generative artificial intelligence model and a recursive adjustment of a target distribution associated with the received plurality of token sets; as well as A means for outputting a selected set of tokens as a response to the input prompt.

24. The processing system of claim 23, wherein the plurality of token sets are organized into a tree data structure.

25. The processing system of claim 24, wherein the root node of the tree data structure corresponds to the input prompt, and wherein each path through the tree data structure corresponds to a different token sequence corresponding to the candidate response to the input prompt.

26. The processing system of claim 24, wherein: The tree data structure includes multiple layers, each layer corresponding to a token in the token sequence, and The number of tokens at a specific layer in the tree data structure is based on the branching factor associated with the previous layer immediately adjacent to that specific layer in the tree data structure.

27. The processing system of claim 24, wherein the depth of the tree data structure corresponds to a parameter defining the maximum number of tokens generated by a single round of passage through the first generative artificial intelligence model.

28. The processing system of claim 23, wherein the size of each token set is based on a computational complexity metric associated with the generation of the target token set by the second generative artificial intelligence model.

29. The processing system of claim 23, wherein the recursive adjustment of the target distribution comprises: Determine whether to accept or reject the first token from the token set among the plurality of token sets; as well as The probability distribution of the second token in the token set used to verify the first token is adjusted based on whether to accept or reject the first token.

30. The processing system of claim 29, wherein adjusting the probability distribution comprises: The probability value associated with the first token is subtracted from the probability distribution based on the determination to reject the first token.

31. The processing system of claim 23, wherein the means for selecting the token set from the plurality of token sets comprises: A means for rejecting a first token at the first level of a tree data structure representing the plurality of token sets; A means for generating an adjusted probability distribution based on the rejection of the first token; A means for discarding or ignoring a sub-token of the first token located at a level deeper than the first level of the tree data structure; as well as A means for determining whether to accept or reject a second token at the first layer of the tree data structure based on the adjusted probability distribution.

32. The processing system of claim 23, wherein the means for selecting the token set from the plurality of token sets comprises: A means for rejecting each set of tokens generated by the first generative artificial intelligence model; as well as A means for sampling tokens using the second generative artificial intelligence model based on a target distribution that excludes probabilities associated with each set of tokens generated by the first generative artificial intelligence model, wherein the selected set of tokens includes the sampled tokens.

33. The processing system of claim 23, wherein: The first generative artificial intelligence model corresponds to the draft model in the speculative decoding pipeline, and The second generative artificial intelligence model corresponds to the target model in the speculative decoding pipeline.

34. A non-transitory computer-readable medium having executable instructions stored thereon, the executable instructions performing operations including the following when executed by a processor: Receive multiple sets of tokens generated based on input prompts and a first generative artificial intelligence model, each of the multiple sets of tokens including a sequence of tokens corresponding to candidate responses to the input prompts; A second generative artificial intelligence model is used, along with a recursive adjustment of the target distribution associated with the multiple received token sets, to select a token set from the multiple token sets. as well as Output the selected set of tokens as a response to the input prompt.

Images