Speculative decoding in autoregressive generative artificial intelligence models

Abstract

Some aspects of this disclosure provide techniques and apparatus for generating responses to query inputs in a generative artificial intelligence model. An exemplary method generally includes receiving an input prompt and a plurality of sets of tokens generated based on a first generative artificial intelligence model, where each set of tokens in the plurality of sets of tokens corresponds to a candidate response to the input prompt; selecting one set of tokens from the plurality of sets of tokens using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens; and outputting the selected set of tokens as a response to the input prompt.

Description

[Technical Field]

[0001] (Cross-reference of related applications)

[0001] This application claims priority and benefits of U.S. Provisional Patent Application No. 63 / 460,850, filed on 20 April 2023, and U.S. Patent Application No. 18 / 538,912, filed on 13 December 2023, also entitled "Speculative Decoding in Autoregressive Generative Artificial Intelligence Models," both of which have been assigned to the assignee of this application and are incorporated herein by reference in their entirety. [Background technology]

[0002]

[0002] Aspects of the present disclosure relate to generative artificial intelligence models, and more particularly to speculative decoding in generative artificial intelligence models (also referred to herein as “generative artificial intelligence models”).

[0003]

[0003] Generative artificial intelligence models can be used in a variety of environments to generate responses to input prompts (also called queries or inputs). For example, generative artificial intelligence models can be used in chatbot applications where large language models (LLMs) are used to generate answers or at least responses to input prompts. Other examples of where generative artificial intelligence models can be used include latent diffusion models in which the model generates an image from an input text description of the content of a desired image, and decision transformers in which future actions are predicted based on a sequence of previous actions in a given environment.

[0004]

[0004] In general, generating responses to queries using generative artificial intelligence models can be computationally expensive. For example, in a chatbot deployment where a large language model is used to generate responses to queries formatted as text queries, the response to the query may be generated using a path through the large language model for each token (e.g., a word or part of a word) generated as part of the response. The output of each path may be a probability distribution for one set of tokens (e.g., a word or part of a word), from which the next token (e.g., a word or part of a word) may be selected, for example, by sampling or based on maximum likelihood. Since the path through the large language model is used to generate each word (or token(s)) in the response to the query, the computational cost can be modeled as the product of the number of words in the response and the computational resource cost (e.g., in terms of processing power, memory bandwidth, and / or other computational resources used) of implementing the path through the large language model, which generally increases as the number of parameters in the large language model increases. [Overview of the Initiative]

[0005]

[0005] Some aspects of the present disclosure provide a method for generating a response to an input prompt using a generative artificial intelligence model. The method generally includes receiving an input prompt and a plurality of sets of tokens generated based on a first generative artificial intelligence model, wherein each set of tokens in the plurality of sets of tokens corresponds to a candidate response to the input prompt; selecting one set of tokens from the plurality of sets of tokens using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens; and outputting the selected set of tokens as a response to the input prompt.

[0006]

[0006] Some aspects of the present disclosure provide a method for generating a response to an input prompt using a generative artificial intelligence model. The method generally includes generating a first plurality of sets of tokens, where each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to an input prompt, based on an input prompt and a generative artificial intelligence model. A second plurality of sets of tokens is speculatively generated using the generative artificial intelligence model. Each set of tokens in the second plurality of sets of tokens generally corresponds to a second portion of a candidate response to an input prompt based on the first plurality of sets of tokens. While speculatively generating the second plurality of sets of tokens, one set of tokens is selected from the first plurality of sets of tokens, and the selected set of tokens from the first plurality of tokens and the associated set of tokens in the second plurality of tokens are output as a response to an input prompt.

[0007]

[0007] Another embodiment provides a processing system configured to carry out the above-described method and the method described herein; a non-temporary computer-readable medium containing instructions that, when executed by one or more processors of the processing system, cause the processing system to carry out the above-described method and the method described herein; a computer program product embodied on a computer-readable storage medium containing code for carrying out the above-described method and the method described herein; and a processing system comprising means for carrying out the above-described method and the method described herein.

[0008]

[0008] The following description and related drawings describe in detail certain exemplary features of one or more embodiments. [Brief explanation of the drawing]

[0009]

[0009] The attached figures illustrate only some aspects of the present disclosure and should therefore not be considered as limiting the scope of the present disclosure. [Figure 1]

[0010] An example of speculative decoding in a generative artificial intelligence model according to the aspects of this disclosure is shown. [Figure 2A]

[0011] An example of recursive speculative decoding in a generative artificial intelligence model according to the aspects of this disclosure is shown. [Figure 2B] An example of recursive speculative decoding in a generative artificial intelligence model according to the aspects of this disclosure is shown. [Figure 3]

[0012] An exemplary tree of tokens generated using recursive speculative decryption in a generative artificial intelligence model according to aspects of this disclosure is shown. [Figure 4]

[0013] This disclosure provides exemplary behavior for generating a response to an input prompt using recursive speculative decoding in a generative artificial intelligence model. [Figure 5]

[0014] This disclosure provides an exemplary pipeline for self-speculative decoding in a generative artificial intelligence model. [Figure 6]

[0015] This disclosure provides an exemplary architecture for self-speculative decoding in a generative artificial intelligence model. [Figure 7]

[0016] This figure shows an exemplary operation for generating a response to an input prompt using a generative artificial intelligence model and self-speculative decoding according to an aspect of the present disclosure. [Figure 8]

[0017] An exemplary processing system configured to implement various aspects of this disclosure is shown. [Figure 9]

[0018] An exemplary processing system configured to implement various aspects of this disclosure is shown.

[0010]

[0019] For ease of understanding, where possible, the same reference numbers are used to designate the same elements common to the drawings. It is contemplated that the elements and features of one embodiment can be beneficially incorporated into other embodiments without further elaboration.

Mode for Carrying Out the Invention

[0011]

[0020] Aspects of the present disclosure provide an apparatus, a method, a processing system, and a computer-readable medium for efficiently generating a response to an input query using a generative artificial intelligence model.

[0012]

[0021] Generally, a generative artificial intelligence model generates a response to a query input to the model. For example, a large language model (LLM) deployed within a chatbot can generate a response to a query using multiple paths through the large language model, where each successive path is based on the query (which may be tokenized for processing) and tokens (or words) generated using previous paths through the large language model. Generally, these large language models can include a large number (e.g., billions or even trillions) of weights or parameters within the model. Due to the size of these models, as well as the operations performed on each token to predict what the next token should be depending on the query and previously generated tokens, deploying the large language model on various devices with limited memory, storage, and / or processing capabilities relative to the cloud computing instances on which the large language model typically operates is not practical or may not even be possible. Further, in some cases, the memory bandwidth associated with generating a response to a query provided as input to the model can prevent computational resources from being used for other tasks.

[0013]

[0022] To improve the efficiency and throughput of large-scale language models, speculative decoding techniques allow smaller language models, sometimes known as draft large-scale language models (or draft models or approximate models), to run (e.g., sequentially or in parallel) alongside larger language models, sometimes known as target large-scale language models (or target models). In such cases, the draft model can speculatively generate additional tokens in a sequence and probabilities used to sample these additional tokens based on the current set of accepted tokens. The target model can generate tokens based on the tokens generated by the draft model. To produce results, the target model can perform rejection sampling on a per-to-token basis to accept or reject individual tokens generated by the draft model, such that the draft model and the target model have similar probability distributions.

[0014]

[0023] In some embodiments, the draft model may be a pruned version of the target model, selected such that the draft model and the target model have similar probability distributions. In other embodiments, the draft model may be a smaller version of the target model (e.g., trained with millions of tokens instead of hundreds of millions or even billions of tokens).

[0015]

[0024] Some aspects of this disclosure provide techniques and apparatus for generating responses to query inputs to large-scale language models using recursive speculative decoding techniques. Generally, a draft model can generate one or more sets of tokens as candidate responses to a query, which may be structured as multiple branches (e.g., in a tree data structure). A target model can recursively perform sampling rejection on tokens provided by the draft model. Generally, recursive sampling rejection allows the probability distribution used by the target model when sampling tokens generated by the draft model to be updated to remove tokens rejected by the target model, and the updated probability distribution is then used to sample subsequent tokens generated by the draft model. By performing recursive rejection sampling, certain aspects of this disclosure can increase the throughput of the draft and target models (e.g., the number of tokens generated per second) for draft and target models configured to generate responses token by token, while maintaining close relationships between the probability distributions within the draft and target models.

[0016]

[0025] Other aspects of the present disclosure provide techniques and apparatus for generating responses to query inputs to large language models using a speculative decoding technique, also referred to herein as "self-speculative decoding", in which a single model speculatively generates tokens in response to a query input and validates previously generated tokens. In the self-speculative decoding technique, the model can speculatively generate additional tokens based on varying the number of speculatively generated tokens that are validated by the model. By using the same model to speculatively generate tokens in response to a query and perform validation of the speculatively generated tokens (e.g., rejection sampling on the tokens), aspects of the present disclosure can reduce the computational expense associated with training and using multiple separately trained models for speculatively generating tokens and performing validation of the speculatively generated tokens. Further, the rate at which tokens are generated can be maximized or at least increased using self-speculative decoding compared to other speculative decoding techniques.

[0017]

[0026] Speculative Decoding in a Generative AI Model Generally, autoregressive token generation (e.g., in a large language model) can take the history tokens as input to generate an output. That is, autoregressive token generation can be represented by the following formula. x t ~p(x│x0,x1,...,x t-1 [[ID=1`3]])→x t+1 ~p(x|x0,x1,...,x t-1 ,x t ) In the formula, x t represents a sequence of tokens generated at time t having a conditional probability p conditioned on the selection of tokens x0~x t-1 , and x t+1 represents tokens x0~x tThis represents a sequence of tokens generated at time t+1, with a conditional probability p given to the selection of a token. Generally, a single token can be generated each time the autoregressive model is run, meaning that N inferences can be performed to generate a sequence of N tokens. As described above, speculative decoding techniques can be used to accelerate token generation by using a smaller draft model than the target model, which speculatively generates tokens faster than the target model, and the target model is used to validate the tokens generated (speculatively) by the draft model.

[0018]

[0027] In a speculative decryption pipeline, the draft model can speculatively generate n tokens autoregressively according to the following equation:

number

[0019] In the formula, t corresponds to the time point,

[0020]

number

[0021] , token x0~x t-1 This corresponds to the conditional probability distribution associated with the selected token x at time t, given the selection of ,

[0022]

number

[0023] This represents a token x that was speculatively generated at time t by the draft model.

[0024]

[0028] The target model takes the n generated tokens, processes the n tokens in parallel, and generates a probability distribution for each of the n tokens according to the following formula.

[0025]

number

[0026] In the formula, k corresponds to the token index for the n tokens generated,

[0027]

number

[0028] This corresponds to the probability distribution generated by the target model at time t of token x generated by the draft model.

[0029]

[0029] Next, the target model can verify the tokens generated by the draft model by comparing the distribution from the draft model and the target model to determine whether the tokens are accepted or rejected. For some function f and some threshold α (also known as the acceptance rate)

[0030]

number

[0031] If so, given token

[0032]

number

[0033] If it is acceptable, the token may be rejected. Then the final token is a function

[0034]

number

[0035] Based on this, it may be generated at the first rejection position or the last position n.

[0036]

[0030] Speculative decryption can result in cost savings compared to using a single autoregressive model to iteratively generate tokens at an acceptance rate of α. The inference cost savings for iterative token generation can be expressed by the following equation:

[0037]

number

[0038] In the formula, N corresponds to the number of tokens, and C AR This corresponds to the computational cost using the acceptance rate of α, and C target This corresponds to the computational cost of generating one set of tokens using the target model, C draft This corresponds to the computational cost of generating one set of tokens using the draft model, C SD n corresponds to the computational cost of speculatively generating one set of tokens using the draft model, and n corresponds to the number of tokens speculatively generated in a single pass through the autoregressive model. N=1000, C target =10, C draft Consider an example where =1, n=4, and α=3. In such an example, speculative decryption can result in a 35% reduction in computational cost compared to autoregressive iterative token generation alone.

[0039]

[0031] However, per-token speculative decoding can impose limitations on the rate at which tokens are generated, as described above, because the first token may be individually sampled by the draft model and then validated by the target model before the next token is sampled by the draft model and validated by the target model. That is, generating a response to an input prompt using a per-token speculative decoding technique may involve running the draft model and the target model for each token generated as part of the response to the input prompt, which can use a considerable amount of computational resources (e.g., processor time, memory, memory bandwidth, etc.) to generate the response.

[0040] Exemplary recursive speculative decoding in generative artificial intelligence models

[0032] Figure 1 shows an example 100 of recursive speculative decoding in a generative artificial intelligence model according to an aspect of the present disclosure.

[0041]

[0033] As shown in the figure, the draft model and the target model can be used together (or otherwise together) to perform recursive speculative decryption of tokens to generate responses to queries received for processing by one or more generative artificial intelligence models. As will be described in more detail below, recursive speculative decryption of tokens can allow multiple sets (or sequences of tokens) of tokens to be speculatively generated by the draft model for verification by the target model. Since multiple sets (or sequences) of tokens can be generated by the draft model for verification by the target model, recursive speculative decryption can increase the token generation rate of the generative artificial intelligence model by generating multiple sets (sequences) of tokens that can be accepted as a correct response, and a larger number of sets of tokens can increase the probability that at least one set contains a sequence of one or more tokens that will be accepted as a response.

[0042]

[0034] The draft model generally takes into account the input of an incoming query and selects a number of high-probability nodes (tokens) from a probability distribution of outputs across a set of potential tokens. The high-probability nodes (tokens) may be selected based on various techniques such as the top k selection (e.g., the selection of the k tokens with the highest probability in the probability distribution) or kernel-based selection (e.g., selection based on the sum of probabilities that satisfy a threshold probability). By selecting many candidate tokens, the draft model can organize a tree structure that can be recursively traversed to sample tokens based on the probability distribution and identify one set of tokens that is a suitable output for a given input, as will be described in more detail below.

[0043]

[0035] In the next iteration of running the draft model, when a sampled group of tokens is input to the draft model, the tokens within the sampled group of tokens are input to the sample positions and treated independently. The result may be a tree data structure 110 with a prompt as the root node 111 of the tree data structure, where subsequent levels in the tree data structure 110 represent different tokens (or groups of tokens) combined with each of the previously selected combinations of tokens. At some point (for example, after generating a tree with a defined depth corresponding to the maximum length of the sequence generated by the draft model), the draft model may output the generated tree data structure 110 to the target model for further processing. In some embodiments, the tree data structure 110 may be output to the target model along with the groupings and selection probabilities generated by the draft model.

[0044]

[0036] In some embodiments, the draft model may be configured to trigger token generation (and subsequent speculative decryption) by the target model based on various criteria. These criteria may include complexity criteria or performance criteria, such as complexity or performance criteria associated with the size of the generated tree data structure 110. In some embodiments, these criteria may include time criteria associated with the expected amount of time for the target model to generate one set of tokens that the generated tree data structure 110 can compare. Generally, these complexity and / or performance criteria may set an upper limit on the number of tokens generated by the draft model for verification by the target model. In some embodiments, this upper limit may be obtained based on the number of nodes in the tree data structure and may be influenced by, for example, a branching factor defined for multiple different levels of the tree data structure 110 in which the sampled tokens are organized, the depth of the tree data structure 110, etc. The worst-case computational load in the final round of speculative token generation may be configured to be limited by the memory bandwidth on the device on which the draft model runs.

[0045]

[0037] In some embodiments, the number of nodes at each level of the tree data structure 110 (for example, token n shown in Figure 1 corresponds to the nth level in the tree data structure 110, and level 0 corresponds to the root node 111 in the tree data structure 110) and the depth of the tree data structure 110 can be defined a priori. The number of nodes at each level of the tree can be defined globally, per level, or in some other way. For example, the number of nodes at any level of the tree data structure 110 can be defined based on the branching factor at the immediately preceding level of the tree data structure 110. For example, a branching factor 2 for a node at the nth level of the tree data structure 110 may result in the generation of two nodes (tokens) at the (n+1)th level of the tree data structure 110 for each node (token) at the nth level of the tree. On the other hand, the depth of the tree data structure 110 can be defined based on the maximum number of tokens (e.g., words) that can be generated using any path through the draft model. For example, if the draft model is configured to generate a sequence of five tokens of a maximum length in any instance of speculative generation, the tree depth could be 6 (in order to include the root node 111 corresponding to the input to the draft model at the first level of the tree data structure 110).

[0046]

[0038] The target model recursively performs rejection sampling on (1) the tokens generated by the draft model, which are contained in the generated tree data structure 110, and (2) the probability distribution q provided as input to the target model. Rejection sampling may be performed recursively at each node in the generated tree, and the termination condition of token selection by the target model at a given layer of the tree data structure 110 is modeled as a recursive problem where the termination condition of the recursive problem is either acceptance of the token or rejection of all tokens. When performing rejection sampling recursively, the target model can adjust the probability distribution used to accept or reject tokens and to validate subsequent tokens in the generated tree. If a token is rejected, an updated probability distribution q'=(qp) may be generated for use when evaluating subsequent tokens in the tree, where p represents the probability associated with the rejected token from the original probability distribution q. The updated probability distribution q' may then be used to evaluate the next token in the tree. The resulting selected set of tokens 112 can be recursively identified by traversing from the root node 111 of the generated tree data structure 110, based on the updated probability distribution generated for each node in the generated tree data structure, as will be explained in more detail below.

[0047]

[0039] In one example, recursive rejection sampling may be performed using a “greedy” technique, where the first token to be accepted is returned as a valid part of the response to the input query (or prompt). In another example, recursive rejection sampling may be performed to determine whether to accept each token at a given level in the tree data structure 110. The selected token at that given level in the tree data structure 110 may be, for example, the token with the highest probability of being a valid token to include in the response to the input prompt. In yet another example, a cumulative probability distribution may be generated from the generated tree data structure for each accepted sequence of tokens, and the sequence with the largest cumulative probability distribution may be selected as the response to the input prompt. Of course, these are just examples of techniques based on the fact that a sequence of tokens can be selected from the tree data structure 110, and it should be recognized that other techniques for selecting a sequence of tokens based on recursive rejection sampling (and corresponding adjustments to the probability distribution q if a token is rejected) are conceivable.

[0048]

[0040] In some embodiments, a draft model may match the probability distribution of a target model but have faster inference performance than the target model on the same hardware. Generally, smaller models can generate more speculative tokens, but the likelihood of the generated tokens being rejected by the target model may increase. The speculative decoding techniques described herein can address this increased likelihood of token rejection at the expense of increased computational cost for longer sequences. Finally, in a draft model, a temperature parameter (generally used herein to refer to a parameter that affects the likelihood of the draft model selecting tokens (words) with a lower likelihood) may be tuned to improve the performance of recursive speculative decoding. In some embodiments, a draft model may match, or at least approximate, the probability distribution of a target model and be fine-tuned to maximize, or at least increase, the probability that speculatively generated tokens (or sequences of tokens) generated by the draft model are accepted as valid tokens by the target model.

[0049]

[0041] In general, token generation performance for recursive speculative decoding can be improved compared to per-token speculative decoding. That is, for any given Kullback-Leibler (KL) divergence between the draft model and the target model, the number of tokens generated for each target model run may be greater for recursive speculative decoding than for per-token speculative decoding. The KL divergence generally measures how the probability distribution of the draft model differs from the probability distribution of the target model (treating the target model as the baseline distribution). Different selection strategies (e.g., group size, additional tokens) may have different computational complexity characteristics. Therefore, the selection of a strategy for selecting tokens to accept rejection using recursive speculative decoding may be based on a trade-off between computational complexity and performance, taking into account the boundary parameters of the read bandwidth of the draft and target models as well as hardware performance.

[0050]

[0042] Figures 2A and 2B show examples 200A and 200B of recursive speculative decoding in a generative artificial intelligence model according to an aspect of the present disclosure.

[0051]

[0043] Example 200A shown in Figure 2A illustrates an example in which one of several tokens generated by the draft model is accepted for inclusion in the response to prompt 210. As shown, the draft model generates four proposed tokens X1220A, X2220B, X3220C, and X4220D, each having associated probabilities p1, p2, p3, and p4 from the original target distribution q. The target model can sequentially examine these tokens to determine whether to accept or reject each token. As shown, the target model first examines token X1220A to determine whether token X1220A should be accepted or rejected as a token to be returned as part of the response to a given input (e.g., prompt 210). In this embodiment, the target distribution q1222A may be set to the original target distribution q, and the target model can determine whether to accept or reject token X1220A based on selection criteria 224A:

[0052]

number

[0053] In the formula, U1 represents a randomly generated number between [0,1].

[0054]

[0044] If token X1220A is accepted (not shown), token X1220A may be output, and the target model analysis of the proposed token generated by the draft model may proceed to traverse the tree and analyze the nodes (tokens) connected to the nodes represented by token X1220A in the generated tree.

[0055]

[0045] Instead, as shown in the figure, if token X1220A is rejected, the target model can proceed to determine whether token X2220B should be accepted or rejected. In doing so, the target model can use a new target distribution q2222B, which is q2=(q1-p1) + This could be the result of subtracting the probability p1 associated with the rejected token X1220A from the target distribution q1222A. The target model can then determine whether to accept or reject token X2220B based on the selection criterion 224B:

[0056]

number

[0057] In the formula, U2 also represents a randomly generated number between [0,1].

[0058]

[0046] Similar to token X1220A described above, the target model can determine that token X2220B meets the acceptance criteria and therefore return token X2220B as the selected token (not shown). Otherwise, if token X2220B is rejected as shown, the target model can determine q2 (for example, q3 = (q2 - p2) + The process then proceeds to determine whether token X3220C should be accepted or rejected, using the updated target distribution q3222C (which removes p2 from the updated target distribution q4222D) and the acceptance criterion U3224C. This process may continue until it is determined that the token (in this embodiment, token X4220D as shown in Example 200A using the updated target distribution q4222D) is accepted (e.g., based on acceptance criterion U4224D) and output as the selected token Y230, as shown in the figure.

[0059]

[0047] Example 200B shown in Figure 2B illustrates an example in which the target model rejects each of the tokens 220A to 220D generated by the draft model. In this embodiment, the target model can generate a final target distribution 222E, sample tokens from this final target distribution 222E, and terminate the traverse of the generated tree. As shown in the figure, the final target distribution q5222E may be the result of subtracting the probability associated with the final token in the set of generated tokens (e.g., the probability p4 associated with token X4220D). The final target distribution q5222E is given by the equation q5 = (q4 - p4) + This can be represented as follows. The token 240 sampled from the final target distribution q5222E can be returned as output Y, which performs recursive rejection sampling on the generated tokens.

[0060]

[0048] Figure 3 shows an exemplary tree data structure 300 of tokens generated using recursive speculative decryption in a generative artificial intelligence model according to an aspect of the present disclosure.

[0061]

[0049] As shown in the figure, the tree data structure 300 includes, as described above, several levels starting from the root node 310 representing the input (which may be tokenized) to the generative artificial intelligence model, and tokens previously generated by the draft model and accepted by the target model using recursive rejection sampling. Each level 320, 330, and 340 (and / or other levels not shown in Figure 3) in the tree data structure 300 may be the result of the draft model speculatively generating some tokens, considering some previous sets of generated tokens. The number of tokens generated at level n+1 of the tree data structure 300 for a node at level n of the tree data structure 300 can be determined based on a branching factor defined for the node at level n of the tree data structure 300. As described above, the branching factor may be the same across all levels of the tree data structure 300, or the branching factor may be defined for each level of the tree data structure 300. In some embodiments, the branching factor may be defined using more branches generated for nodes at lower levels of the tree data structure 300 and fewer branches for nodes at higher levels of the tree data structure 300, or vice versa.

[0062]

[0050] In order to traverse the tree data structure 300 and select one set of tokens from a group of speculatively generated tokens, the target model can recursively accept or reject tokens (for example, tokens represented by nodes that are direct child nodes of the root node 310) starting from a first set of tokens generated by the draft model at the first level 320 of the tree data structure 300. Generally, when traversing the tree, the target model can determine whether a token should be accepted or rejected, and adjust the target distribution based on the determination of whether a token should be accepted or rejected. If the first token is rejected at the first level 320 of the tree data structure 300, the target model can remove the first token from the target distribution (as described above with respect to Figure 2) and proceed to determine whether a second token should be accepted or rejected at the first level of the tree based on the updated target distribution. When rejecting a first token at the first level 320 of the tree data structure 300, the target model does not need to analyze tokens represented by nodes that are direct or indirect child nodes of the node in the tree data structure 300 that represents the rejected first token.

[0063]

[0051] Generally, when traversing the tree data structure 300, the target distribution can be recursively modified until a termination condition is reached. The termination condition may be, for example, the identification of a sequence of tokens that is accepted as an acceptable response output for the input (represented by the root node 310 of the tree data structure 300), plus additional nodes sampled from the modified target distribution after the sequence of tokens has been accepted by the target model. In another example, the termination condition may be the determination that a sequence of tokens speculatively generated by the draft model was not accepted by the target model. In such a case, as described above with respect to Figure 2B, the additional tokens sampled from the final target distribution may be output as a response to the input corresponding to the root node 310 in the tree data structure 300. The target distribution may be, as described above, the target distribution generated by removing the probabilities associated with the generated tokens from the original target distribution generated by the draft model.

[0064]

[0052] In the example shown in Figure 3, the first token 322A may be rejected at the first level 320 of the tree data structure 300. Since the first token 322A is rejected as one of the candidate tokens to be included in the response to the input corresponding to the root node 310, the tokens in the tree data structure 300 that are child tokens of the first token 322A do not need to be examined for selection as candidate tokens. Thus, the direct child tokens of the first token 322A at the second level 330 of the tree data structure 300, and the grandchild tokens of the first token 322A at the third level 340 of the tree data structure 300, may be discarded or ignored.

[0065]

[0053] When rejecting the first token 322A from the set of candidate tokens at the first level 320 of the tree data structure 300, as described, the target probability distribution associated with the token at the first level 320 of the tree data structure 300 may be modified to remove the probability value associated with the first token 322A. The updated target distribution may then be used when examining the second token 322B at the first level 320 of the tree data structure 300. As shown in the figure, the second token 322B may be accepted as a candidate token to be included in the response to the input corresponding to the root node 310, and the analysis may proceed to the child tokens of the second token 322B at the second level 330 of the tree data structure 300 (e.g., 332A, 332B, and 332C).

[0066]

[0054] As shown in the figure, tokens 332A and 332B can be rejected by the target model. When rejecting tokens 332A and 332B, the target probability distribution used to accept or reject tokens in the second level 330 of the tree data structure 300 can be adjusted to remove the probabilities associated with tokens 332A and 332B. This adjusted target probability distribution can then be used to determine whether to accept or reject token 332C in the second level 330 of the tree data structure 300. As shown in the figure, the target model can accept token 332C, and the analysis can proceed to the child tokens of token 332C (e.g., 342A, 342B) in the third level 340 of the tree data structure 300. In the third level 340 of the tree data structure 300, the target model can examine tokens 342A and 342B using the technique described above and may accept token 342A as a candidate to be included in the response to the input corresponding to the root node 310.

[0067]

[0055] In some embodiments, the acceptance of a second token 322B as a candidate token to be included in the response to an input corresponding to the root node 310 may not preclude the analysis of a third token 322C to be included or rejected as a candidate token to be included in the response to an input corresponding to the root node 310. In such cases, the acceptance or rejection of child tokens at levels 320 and 330 of the tree data structure 300 (not shown in Figure 3, in particular) may be carried out as described above with respect to the acceptance or rejection of child tokens of the second token 322B. In some embodiments, the third token 322C may be considered rejected, such as when the probability that the second token 322B is a candidate token in the response to an input corresponding to the root node 310 exceeds a threshold, and the analysis of child tokens of the third token 322C may be bypassed.

[0068]

[0056] The recursive speculative decryption techniques described herein can lead to a significant gain in token generation performance compared to token generation and acceptance. Generally, recursive speculative decryption techniques can enable an increase in token acceptance performance (e.g., the number of tokens accepted for a given draft length) each time the target model is executed, compared to token-by-token speculative decryption techniques. The amount of performance improvement can increase as the number of branches generated for any given token increases and as the draft length increases. Furthermore, the recursive speculative decryption techniques described herein can enable an (over time) increase in the rate at which tokens are generated compared to token-by-token speculative decryption techniques.

[0069] Exemplary behavior of recursive speculative decoding in generative artificial intelligence models

[0057] Figure 4 shows an exemplary operation 400 that may be performed by a computing device to generate a response to an input prompt using a generative artificial intelligence model (for example, as described herein in relation to Figures 1 to 3) according to aspects of the present disclosure. Operation 400 may be performed by a computing device capable of deploying at least a target model, such as a laptop computer, a desktop computer, a server, or a cloud compute instance hosted in a distributed computing environment.

[0070]

[0058] As shown in the figure, operation 400 begins in block 410 with receiving multiple sets of tokens. Generally, multiple sets of tokens may be tokens generated based on an input prompt and a first generating artificial intelligence model (for example, by a draft model deployed on the client device where the multiple sets of tokens are received). The input prompt may be received on the client device from a user via, for example, a text input prompt, an audio capture prompt, or other techniques on which input can be provided to the client device. Generally, each set of tokens in the multiple sets of tokens may correspond to a candidate answer to an input prompt.

[0071]

[0059] In some embodiments, multiple sets of tokens may be organized into a tree data structure (e.g., tree data structure 110 shown in Figure 1 or tree data structure 300 shown in Figure 3). The tree data structure may have a root node and one or more leaf nodes. An input prompt may correspond to the root node of the tree data structure. Each path through the tree data structure may correspond to a different candidate response to an input prompt. It should be understood that a path through the tree data structure does not have to terminate at a node in the tree data structure that does not have leaf nodes; that is, a candidate response may correspond to a path that partially traverses the tree data structure (e.g., to reflect tokens rejected in the response).

[0072]

[0060] In some embodiments, the number of tokens at any given level of a tree data structure may be based on a branching factor associated with the level immediately preceding that level in the tree data structure. A tree data structure may have multiple different branching factors assigned to multiple different levels within the tree data structure. Generally, higher branching factors allow for the generation of more tokens at any given level of the tree data structure, and generally can increase the total number of tokens contained in the tree data structure. On the other hand, lower branching factors may restrict the generation of tokens at any given level of the tree data structure, and generally can decrease the total number of tokens contained in the tree data structure.

[0073]

[0061] In some embodiments, the size of each set of tokens may be based on a computational complexity metric associated with generating a target set of tokens by a second generative artificial intelligence model.

[0074]

[0062] In block 420, operation 400 proceeds to select one set of tokens from multiple sets of tokens using a second generative artificial intelligence model and recursive adjustment of the probability distribution.

[0075]

[0063] In some embodiments, the recursive adjustment of the probability distribution includes rejecting one token from a set of tokens as a candidate to be included as a response to an input prompt. Generally, a token may be rejected if it does not meet some defined acceptance criteria, which may be based on the target distribution and the probabilities associated with the token in the original target distribution. The current distribution used by the target model to determine whether a token should be accepted or rejected may be modified by removing (or subtracting) the probabilities associated with the token in the original target distribution from the current distribution, thereby generating an updated target distribution that may be used to determine whether the next token should be accepted or rejected as a candidate to be included as a response to an input prompt.

[0076]

[0064] In some embodiments, selecting one set of tokens from multiple sets of tokens may include rejecting a first token at the first level of a tree data structure representing multiple sets of tokens. A tuned probability distribution may be generated based on the rejection of the first token. Within the tree data structure, child tokens of the first token at levels deeper than the first level of the tree data structure may be discarded. A second token at the first level of the tree data structure may be accepted or rejected based on a tuned probability distribution.

[0077]

[0065] In some embodiments, selecting a set of tokens from multiple sets of tokens may include rejecting each set of tokens generated by the first generative artificial intelligence model. Tokens may be sampled using a second generative artificial intelligence model based on a target distribution that excludes the probabilities associated with each set of tokens generated by the first generative artificial intelligence model.

[0078]

[0066] In block 430, operation 400 proceeds to output a selected set of tokens in response to an input prompt.

[0079]

[0067] In some embodiments, the first generative artificial intelligence model may correspond to a draft model in a speculative decoding pipeline, and the second generative artificial intelligence model may correspond to a target model in a speculative decoding pipeline. The first generative artificial intelligence model may run on a client device, and the second generative artificial intelligence model may run on a remote system from the client device, such as a server computer or a cloud computing instance. In some embodiments, the first and second generative artificial intelligence models may run on the same device.

[0080]

[0068] In some embodiments, the first generative artificial intelligence model and the second generative artificial intelligence model may have equivalent probability distributions.

[0081]

[0069] In some embodiments, the first generative artificial intelligence model may have a probability distribution that approximates the probability distribution associated with the second generative artificial intelligence model.

[0082] Exemplary self-speculative decoding in generative artificial intelligence models

[0070] In some embodiments, various types of speculative decoding, including group speculative decoding and recursive speculative decoding, can be achieved using a single generative artificial intelligence model that combines the functionality of the draft model and target model described above. In doing so, draft token generation, target token generation, and token acceptance can be parallelized in a single generative artificial intelligence model. Using a single generative artificial intelligence model can, for example, reduce the computational cost associated with generating both the target model and the draft model, improve the performance of the generation task by performing token verification and speculative generation in a single pass through the single generative artificial intelligence model, and reduce the amount of memory used to store the models used for speculative decoding in the generation task.

[0083]

[0071] Figure 5 shows an exemplary pipeline 500 for self-speculative decoding in a generative artificial intelligence model according to an aspect of the present disclosure. In some aspects, pipeline 500 may be used to generate and analyze a tree data structure of tokens associated with candidate responses to query inputs to a generative intelligence model, such as the tree data structure 300 shown in Figure 3.

[0084]

[0072] As shown in the figure, pipeline 500 speculatively generates tokens and validates the speculatively generated tokens using a single generative artificial intelligence model. During the first inference round in pipeline 500, a first set of tokens 502 is speculatively generated. As shown in the figure, for example, the first set of tokens 502 may include tokens 1 to 4 and may be provided as input during a second round in pipeline 500 to speculatively generate the next set of tokens as a batch process in which multiple sets of tokens are generated. While the first set of speculatively generated tokens is processed by the single generative artificial intelligence model, the single generative artificial intelligence model continues to speculatively generate multiple second sets of draft tokens 504, 506, 508, and 510 during the second inference round in pipeline 500.

[0085]

[0073] When generating the second sets of draft tokens 504, 506, 508, and 510, assumptions may be made about a different number of accepted tokens from the first set of tokens 502. For example, as shown in the figure, the second set of draft tokens 504 may include a set of tokens speculatively generated based on the acceptance of the first draft token, assuming the acceptance of the first draft token from the first set of tokens 502. The second set of draft tokens 506 may include a set of tokens speculatively generated based on the acceptance of the first and second draft tokens, assuming the acceptance of the first and second draft tokens from the first set of tokens 502. The second set of draft tokens 508 may include a set of tokens speculatively generated based on the acceptance of the first to third draft tokens, assuming the acceptance of the first to third draft tokens from the first set of tokens 502. Finally, the second set of draft tokens 510 may include a set of tokens speculatively generated based on the acceptance of all four tokens, assuming the acceptance of all four tokens from the first set of tokens 502. In various embodiments, if it is assumed that fewer tokens are accepted than the number of tokens contained in the first set 502 of tokens, padding 503 (e.g., a null value, a default constant, etc.) can be added so that each assumption is of the same length.

[0086]

[0074] Once the single generative artificial intelligence model has completed rejection sampling for the speculatively generated set of tokens, the single generative artificial intelligence model selects a set of speculatively generated tokens associated with the accepted set of tokens from the first set as input to the single generative artificial intelligence model for another inference round in which tokens are speculatively generated using the single generative artificial intelligence model. In this embodiment, all four tokens in the first set of tokens 502 are accepted by the single generative artificial intelligence model as draft validation 512, and thus the set of tokens 510 can be used for further speculative generation of tokens using the single generative artificial intelligence model.

[0087]

[0075] The above process may continue until a termination event occurs. Successive rounds of speculative generation may be based on an assumption of the number of tokens from previous rounds of speculative generation that are accepted by a single generative artificial intelligence model. For example, as shown in Figure 5, the draft token sets 522, 524, 526, and 528 may be generated in the (k+1)th round of inference, and the tokens included in the draft token set are based on the number of speculatively generated tokens that exceed the N accepted tokens generated in the (k-1)th round of inference. In this embodiment, it can be seen that four speculatively generated tokens generated during the kth round of inference are accepted as draft validation 520, and tokens N+5 to N+8 may be used for further speculative generation of tokens using a single generative artificial intelligence model.

[0088]

[0076] In some embodiments, the termination event may involve the generation of special tokens used to indicate the end of a response (for example, the response cannot reasonably contain any further tokens because the probabilities associated with these tokens fall below a threshold probability value for acceptance). In some embodiments, the termination event may be reached when a threshold number of tokens have been generated.

[0089]

[0077] In some embodiments, if all tokens from previous rounds of speculative token generation are rejected by a single generative artificial intelligence model, the process can be restarted with the last set of accepted tokens, plus tokens sampled from the final distribution (as described above, for example), which are provided as input to the single generative artificial intelligence model.

[0090]

[0078] Figure 6 shows exemplary architectures 600A and 600B for self-speculative decoding in a generative artificial intelligence model according to aspects of the present disclosure. Both exemplary architectures 600A and 600B can enable the generation of multiple tokens in any path through the model, such as the generation of tokens shown in Figures 1, 2A, 2B, 3, 4, and 5, as described above.

[0091]

[0079] In an exemplary architecture 600A, the generative artificial intelligence model 610 may be trained to generate a plurality of predictive prompt embeddings 612 attached to an input set of tokens in order to enable the parallel generation of a plurality of output tokens 614. These predictive prompt embeddings may be embeddings corresponding to tokens (including any previously generated and accepted tokens) that are included in the response to an input prompt. The generative artificial intelligence model 610 may be a generative artificial intelligence model such as a pre-trained large language model or other pre-trained generative artificial intelligence model that is updated using a variety of fine-tuning techniques. For example, a generative artificial intelligence model (also known as a large language model) used to generate text responses to text inputs may be updated using techniques such as low-rank adaptation (LoRA) of a large language model.

[0092]

[0080] In exemplary architecture 600B, the generative artificial intelligence model may be implemented as a partial autoregressive model. The inference operations used to speculatively generate tokens may be implemented using a subset of layers within the partial autoregressive model (e.g., the top n layers of the model or the bottom n layers of the model). In doing so, the layers used to speculatively generate tokens may create a context that allows causal and / or other relationships to be modeled with respect to the speculatively generated tokens, which may be supplied as input to a part of the model that validates the tokens as valid responses to input prompts.

[0093]

[0081] Architecture 600B can be implemented in various ways such that autoregressive inference, and the generation of multiple sets of tokens for acceptance and / or rejection, can be generated using a small number of autoregressive layers in the generative artificial intelligence model. In exemplary implementation form 620, the generative artificial intelligence model may include multiple non-autoregressive layers 622A-622C and an autoregressive layer 624. Multiple layers in the generative artificial intelligence model may be organized into a stack, with the bottom layer of the stack corresponding to a layer that receives input for processing, and the top layer of the stack corresponding to a layer that generates output. In implementation form 620, the non-autoregressive layers 622A-622C may be located at the bottom of the stack, and the autoregressive layer 624 may be located at the top of the stack. In contrast, in exemplary implementation form 630, multiple layers of the generative artificial intelligence model may be organized such that the autoregressive layer 632 is located at the bottom of the stack, and the non-autoregressive layers 632A-632C are located at the top of the stack. These autoregressive layers 624 and 632 may operate in a loop to continuously generate and accept tokens (and, in some embodiments, previously generated tokens included as partial responses to input prompts) that should be output as responses to input prompts.

[0094] Exemplary behavior for generating responses to input queries using self-speculative decoding in generative artificial intelligence models.

[0082] Figure 7 shows an exemplary operation 700 that may be performed by a computing device to generate a response to an input prompt using a generative artificial intelligence model according to an aspect of the present disclosure. Operation 700 may be performed by a device capable of deploying at least a generative artificial intelligence model that can function as both a draft model and a target model, such as a smartphone, tablet computer, laptop computer, desktop, server, or cloud computing instance hosted in a distributed computing environment.

[0095]

[0083] As shown in the figure, operation 700 starts in block 710 and generates a first set of tokens based on an input prompt and a generating artificial intelligence model. In some embodiments, the set of tokens may include a sequence of tokens that can be partially or completely accepted as a candidate response to an input prompt.

[0096]

[0084] In some embodiments, a first set of tokens may be represented as a tree data structure. Within this tree data structure, an input prompt may correspond to the root node. Each set of tokens in the first set of tokens may be represented by a navigable path through the tree data structure (as described above, for example, with respect to Figure 3).

[0097]

[0085] In some embodiments, the depth of the tree data structure may correspond to the maximum number of tokens generated by a single pass through the generative artificial intelligence model.

[0098]

[0086] In some embodiments, the depth of the tree data structure may correspond to the maximum number of tokens generated by a single pass through the generative artificial intelligence model.

[0099]

[0087] In some embodiments, the maximum size of the tree data structure may be set based on a computational complexity metric associated with generating one set of tokens by a generative artificial intelligence model.

[0100]

[0088] In block 720, operation 700 proceeds to speculatively generate a second set of tokens using a generative artificial intelligence model.

[0101]

[0089] In block 730, operation 700 proceeds to select one set of tokens from the first set of tokens while speculatively generating a second set of tokens.

[0102]

[0090] In some embodiments, selecting a set of tokens from a first set of tokens may include selecting the longest sequence of accepted tokens from the first set of tokens.

[0103]

[0091] In some embodiments, a set of tokens in a second set of tokens may include padding that takes into account the number of tokens in a selected set of tokens from the first set of tokens. For example, for the number of tokens N in each set of tokens from the first set of tokens, the selected number of tokens may include 0 to N tokens. In such cases, the padding included in one set of tokens in the second set of tokens may include no padding (corresponding to the acceptance of all N tokens in the set of tokens from the first set of tokens), one padding token (corresponding to the acceptance of N-1 tokens), two padding tokens (corresponding to the acceptance of N-2 tokens), and so on.

[0104]

[0092] In some embodiments, a first token may be rejected at the first level of the tree data structure in order to select a set of tokens from a first set of tokens. Based on the rejection of the first token, n tuned probability distributions may be generated, and the tree data structure may be pruned by discarding the child tokens of the first token from the tree data structure. The rejection of the first token may generally imply that the child tokens of the first token cannot be a valid response to the input prompt, and therefore the child tokens of the first token may be discarded. Then, based on the tuned probability distributions, it may be determined whether to accept or reject the second token at the first level of the tree data structure.

[0105]

[0093] In some embodiments, selecting a set of tokens from a first set of multiple sets of tokens may include rejecting each set of tokens in the first set of multiple sets of tokens generated by a generative artificial intelligence model. Tokens may be sampled using a generative artificial intelligence model based on a target distribution that excludes the probabilities associated with each set of tokens in the first set of multiple sets of tokens. The selected set of tokens from the first set of multiple sets of tokens may include the sampled tokens.

[0106]

[0094] In block 740, operation 700 proceeds to output, in response to an input prompt, a selected set of tokens from a first set of tokens and the associated set of tokens within a second set of tokens.

[0107]

[0095] In some embodiments, the generative artificial intelligence model includes a generative artificial intelligence model trained to generate a plurality of tokens in response to an input prompt based on predictive prompt embeddings.

[0108]

[0096] In some embodiments, the generative artificial intelligence model includes a model comprising one or more non-autoregressive layers and one or more autoregressive layers. In some embodiments, the one or more autoregressive layers may consist of one or more layers at the top of the stack of layers representing the generative artificial intelligence model. In some embodiments, the one or more autoregressive layers may consist of one or more layers at the bottom of the stack of layers representing the generative artificial intelligence model.

[0109] Exemplary Processing System for Recursive Speculative Decoding in Generative Artificial Intelligence Models

[0097] Figure 8 shows an exemplary processing system 800 for generating responses to query inputs to a generative artificial intelligence model based on recursive speculative decoding, for example, as described herein with respect to Figure 4.

[0110]

[0098] The processing system 800 includes a central processing unit (CPU) 802, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 802 may be loaded, for example, from program memory associated with the CPU 802, or from a memory partition (for example, memory 824).

[0111]

[0099] The processing system 800 also includes additional processing components tuned to specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, and a connectivity component 812.

[0112]

[0100] NPUs such as the NPU808 are generally special circuits configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs may also be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0113]

[0101] NPUs such as the NPU808 are configured to accelerate the performance of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs may be instantiated on a single chip, such as a system on a chip (SoC), while in other embodiments, such NPUs may be part of a dedicated neural network accelerator.

[0114]

[0102] The NPU can be optimized for training or inference, or in some cases, it can be configured to balance performance between both. With respect to an NPU capable of performing both training and inference, the two tasks can still generally be performed independently.

[0115]

[0103] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is an extremely computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating through that dataset, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0116]

[0104] NPUs designed to accelerate inference are generally configured to operate on a complete model. Therefore, such an NPU may be configured to take new data as input and process this new data quickly through a model that has already been trained to produce model outputs (e.g., inferences).

[0117]

[0105] In some implementations, the NPU808 is part of one or more of the CPU802, GPU804, and / or DSP806. These may be located on user equipment (UE) of a wireless communication system or on another computing device.

[0118]

[0106] In some embodiments, the connectivity component 812 may include, for example, subcomponents for third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long-Term Evolution, LTE), fifth-generation (5G) connectivity (e.g., New Radio, NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The connectivity component 812 may be further coupled to one or more antennas 814.

[0119]

[0107] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor, and / or a navigation processor 820 which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0120]

[0108] The processing system 800 may also include one or more input and / or output devices 822, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, a microphone, etc.

[0121]

[0109] In some embodiments, one or more of the processors of the processing system 800 may be based on an ARM or RISC-V instruction set.

[0122]

[0110] The processing system 800 also includes memory 824 which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 824 includes a computer executable component which can be executed by one or more of the aforementioned processors of the processing system 800.

[0123]

[0111] In particular, in this embodiment, the memory 824 includes a token set receiving component 824A, a token selection component 824B, an output generation component 824C, and a generating artificial intelligence model 824D. The components shown, and other components not shown, may be configured to carry out various aspects of the methods described herein.

[0124]

[0112] Generally, the processing system 800 and / or its components may be configured to carry out the methods described herein.

[0125] Exemplary Processing System for Self-Speculative Decoding in Generative Artificial Intelligence Models

[0113] Figure 9 shows an exemplary processing system 900 for generating responses to query inputs to a generative artificial intelligence model based on self-speculative decoding, as described herein with respect to Figure 7, for example.

[0126]

[0114] The processing system 900 includes a central processing unit (CPU) 902, which in some embodiments may be a multi-core CPU. Instructions executed in the CPU 902 may be loaded, for example, from program memory associated with the CPU 902, or from a memory partition (for example, memory 924).

[0127]

[0115] The processing system 900 also includes additional processing components tuned to specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908, and connectivity components 912.

[0128]

[0116] NPUs such as the NPU908 are generally special circuits configured to implement control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs may also be referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligence processing units (IPUs), vision processing units (VPUs), or graph processing units.

[0129]

[0117] NPUs such as the NPU908 are configured to accelerate the performance of common machine learning tasks such as image classification, machine translation, object detection, and various other predictive models. In some embodiments, multiple NPUs may be instantiated on a single chip such as a system-on-a-chip (SoC), while in other embodiments, such NPUs may be part of a dedicated neural network accelerator.

[0130]

[0118] The NPU can be optimized for training or inference, or in some cases, it can be configured to balance performance between both. With respect to an NPU capable of performing both training and inference, the two tasks can still generally be performed independently.

[0131]

[0119] NPUs designed to accelerate training are generally configured to accelerate the optimization of new models, which is a highly computationally intensive operation that involves inputting an existing dataset (often labeled or tagged), iterating through that dataset, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on incorrect predictions involves backpropagating through layers of the model to determine gradients to reduce prediction errors.

[0132]

[0120] NPUs designed to accelerate inference are generally configured to operate on a complete model. Therefore, such an NPU may be configured to take new data as input and process this new data quickly through a model that has already been trained to produce model outputs (e.g., inferences).

[0133]

[0121] In some implementations, the NPU908 is part of one or more of the CPU902, GPU904, and / or DSP906. These may be located on the user equipment (UE) of the wireless communication system or on another computing device.

[0134]

[0122] In some embodiments, the wireless connectivity component 912 may include, for example, subcomponents for third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., LTE), fifth-generation (5G) connectivity (e.g., NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The connectivity component 912 may be further coupled to one or more antennas 914.

[0135]

[0123] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor, and / or a navigation processor 920 which may include satellite-based positioning system components (e.g., GPS or GLONASS), and inertial positioning system components.

[0136]

[0124] The processing system 900 may also include one or more input and / or output devices 922, such as a screen, a touch-sensitive surface (including a touch-sensitive display), physical buttons, a speaker, or a microphone.

[0137]

[0125] In some embodiments, one or more of the processors of the processing system 900 may be based on an ARM or RISC-V instruction set.

[0138]

[0126] The processing system 900 also includes memory 924 which represents one or more static and / or dynamic memories, such as dynamic random access memory and flash-based static memory. In this embodiment, memory 924 includes computer executable components which can be executed by one or more of the aforementioned processors of the processing system 900.

[0139]

[0127] In particular, in this embodiment, the memory 924 includes a token set generation component 924A, a token selection component 924B, an output generation component 924C, and a generation artificial intelligence model 924D. The components shown, and other components not shown, may be configured to carry out various aspects of the methods described herein.

[0140]

[0128] Generally, the processing system 900 and / or its components may be configured to carry out the methods described herein.

[0141] Exemplary clause

[0129] Details of the implementation of various aspects of this disclosure are described in the following numbered clauses.

[0142]

[0130] Clause 1: A method performed by a processor, comprising: receiving a processor input prompt and a plurality of sets of tokens generated based on a first generative artificial intelligence model, wherein each set of tokens in the plurality of sets of tokens corresponds to a candidate response to the input prompt; selecting one set of tokens from the plurality of sets of tokens using a second generative artificial intelligence model and recursive adjustment of a target distribution associated with the received plurality of sets of tokens; and outputting the selected set of tokens as a response to the input prompt.

[0143]

[0131] Clause 2: The method according to Clause 1, wherein multiple sets of tokens are organized into a tree data structure.

[0144]

[0132] Clause 3: The method according to Clause 2, wherein an input prompt corresponds to the root node of a tree data structure, and each path through the tree data structure corresponds to a different candidate response to the input prompt.

[0145]

[0133] Clause 4: The method according to Clause 2 or 3, wherein the number of tokens at a level in the tree data structure is based on a branch factor associated with the level immediately preceding that level in the tree data structure.

[0146]

[0134] Clause 5: The method according to any of Clauses 2 to 4, wherein the depth of the tree data structure corresponds to the maximum number of tokens generated by a single pass through the first generative artificial intelligence model.

[0147]

[0135] Clause 6: The method according to any of Clauses 1 to 5, wherein the size of each set of tokens is based on a computational complexity metric associated with generating a target set of tokens by a second generative artificial intelligence model.

[0148]

[0136] Clause 7: The method according to any one of Clauses 1 to 6, wherein the recursive adjustment of the target distribution determines whether to accept or reject a first token in one set of tokens from a plurality of sets of tokens, and adjusts the probability distribution used to validate a second token in the set of tokens following the first token based on the decision to accept or reject the first token.

[0149]

[0137] Clause 8: The method according to Clause 7, comprising subtracting a probability value associated with the first token from the probability distribution based on the determination that adjusting the probability distribution would reject the first token.

[0150]

[0138] Clause 9: The method according to any one of Clauses 1 to 8, wherein selecting a set of tokens from a set of tokens includes rejecting a first token at the first level of a tree data structure representing the set of tokens, generating a tuned probability distribution based on the rejection of the first token, discarding child tokens of the first token from the tree data structure at levels deeper than the first level of the tree data structure, and determining whether to accept or reject a second token at the first level of the tree data structure based on the tuned probability distribution.

[0151]

[0139] Clause 10: The method according to any one of Clauses 1 to 9, wherein selecting a set of tokens from a plurality of sets of tokens is to reject each set of tokens generated by a first generative artificial intelligence model, and to sample tokens using a second generative artificial intelligence model based on a target distribution that excludes the probabilities associated with each set of tokens generated by the first generative artificial intelligence model, wherein the selected set of tokens includes the sampled tokens.

[0152]

[0140] Clause 11: The method according to any one of Clauses 1 to 10, wherein a first generative artificial intelligence model corresponds to a draft model in a speculative decoding pipeline, and a second generative artificial intelligence model corresponds to a target model in a speculative decoding pipeline.

[0153]

[0141] Clause 12: A method implemented by a processor, comprising: generating a first plurality of sets of tokens based on an input prompt and a generative artificial intelligence model, wherein each set of tokens in the first plurality of sets of tokens corresponds to a first portion of a candidate response to an input prompt; speculatively generating a second plurality of sets of tokens using a generative artificial intelligence model, wherein each set of tokens in the second plurality of sets of tokens corresponds to a second portion of a candidate response to an input prompt based on the first plurality of sets of tokens; selecting one set of tokens from the first plurality of sets of tokens while speculatively generating the second plurality of sets of tokens; and outputting the selected set of tokens from the first plurality of tokens and the associated set of tokens in the second plurality of tokens as a response to an input prompt.

[0154]

[0142] Clause 13: The method of Clause 12, wherein selecting a set of tokens from a first set of tokens includes selecting the longest sequence of accepted tokens from a first set of tokens.

[0155]

[0143] Clause 14: The method according to Clause 12 or 13, wherein the set of tokens in a second set of tokens includes padding that takes into account the number of tokens in a selected set of tokens from the first set of tokens.

[0156]

[0144] Clause 15: The method according to any of Clauses 12 to 14, wherein a first set of tokens is represented as a tree data structure, and the root node of the tree data structure corresponds to an input prompt.

[0157]

[0145] Clause 16: The method according to Clause 15, wherein the depth of the tree data structure corresponds to the maximum number of tokens generated by a single pass through the generative artificial intelligence model.

[0158]

[0146] Clause 17: The method according to Clause 15 or 16, wherein the maximum size of the tree data structure is set based on a computational complexity metric associated with generating one set of tokens by a generative artificial intelligence model.

[0159]

[0147] Clause 18: The method according to any one of Clauses 15 to 17, wherein selecting a set of tokens from a first set of tokens includes rejecting a first token at the first level of a tree data structure representing the first set of tokens, generating a adjusted probability distribution based on the rejection of the first token, discarding child tokens of the first token from the tree data structure, and determining whether to accept or reject a second token at the first level of the tree data structure based on the adjusted probability distribution.

[0160]

[0148] Clause 19: The method according to any one of Clauses 12 to 18, wherein selecting a set of tokens from a first set of tokens includes rejecting each set of tokens in a first set of tokens generated by a generative artificial intelligence model, and sampling tokens using the generative artificial intelligence model based on a target distribution that excludes the probabilities associated with each set of tokens in a first set of tokens, wherein the selected set of tokens from the first set of tokens includes the sampled tokens.

[0161]

[0149] Clause 20: The method according to any of Clauses 12 to 19, wherein the generative artificial intelligence model is trained to generate multiple tokens in response to an input prompt based on predictive prompt embedding.

[0162]

[0150] Clause 21: The method according to any one of Clauses 12 to 20, wherein the generative artificial intelligence model includes one or more non-autoregressive layers and a model including one or more autoregressive layers.

[0163]

[0151] Clause 22: The method according to Clause 21, wherein one or more autoregressive layers are located on top of a stack of layers representing a generative artificial intelligence model.

[0164]

[0152] Clause 23: The method according to Clause 21 or 22, wherein one or more autoregressive layers comprise one or more layers at the bottom of a stack of layers representing a generative artificial intelligence model.

[0165]

[0153] Clause 24: A processing system comprising: at least one memory storing executable instructions; and one or more processors configured to execute the executable instructions in order to cause the processing system to perform any of the operations described in Clauses 1 to 23.

[0166]

[0154] Clause 25: A processing system comprising means for performing any of the operations described in Clauses 1 to 23.

[0167]

[0155] Clause 26: A non-temporary computer-readable medium storing executable instructions, wherein when the instructions are executed by one or more processors, they perform the operations described in any of Clauses 1 to 23.

[0168] Additional considerations

[0156] The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The embodiments discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to a person skilled in the art, and the general principles defined herein may also be applied to other embodiments. For example, changes may be made to the function and configuration of the elements discussed without departing from the scope of the disclosure. Various embodiments may omit, substitute, or add various procedures or components as needed. For example, the methods described may be carried out in an order different from the order described, and various steps may be added, omitted, or combined. Also, features described in some embodiments may be combined in some other embodiments. For example, an apparatus may be implemented or a method may be performed using any number of embodiments described herein. Furthermore, the scope of the disclosure is intended to encompass apparatus or methods that are practiced using other structures, functionalities, or structures and functionalities in addition to, or other than, the various embodiments of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0169]

[0157] As used herein, the term “exemplary” means “to serve as an example, case, or illustration.” No embodiment described herein as “exemplary” should be construed as necessarily preferable or advantageous to any other embodiment.

[0170]

[0158] As used herein, the phrase “at least one of” the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to include a, b, c, ab, ac, bc, and abc, as well as any combination having multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other sequence of a, b, and c).

[0171]

[0159] As used herein, the term “determining” encompasses a wide range of actions. For example, “determining” may include calculating, calculating, processing, deriving, investigating, searching (e.g., searching a table, database, or other data structure), and confirming. It may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0172]

[0160] The methods disclosed herein include one or more steps or actions to achieve those methods. The steps and / or actions of those methods can be replaced with one another without departing from the claims. In other words, unless a particular order of steps or actions is specified, the order of any particular steps and / or actions, and / or the use of those steps and / or actions, can be modified without departing from the claims. Furthermore, various operations of the methods described above can be carried out by any preferred means capable of performing the corresponding functions. These means may include, but are not limited to, various hardware components and / or software components, including circuits, application-specific integrated circuits (ASICs), or processors, and / or various hardware modules and / or software modules. Generally, where operations are shown in the figures, those operations may have corresponding equivalent means-plus-function components, similarly numbered.

[0173]

[0161] The following claims are not intended to be limited to the embodiments shown herein, but the full scope consistent with the language of the claims should be recognized. In the claims, a singular reference to an element is intended to mean "one or more" rather than "one and only one" unless otherwise explicitly stated. Unless otherwise explicitly stated, the term "several" refers to one or more. No element of a claim should be construed under Section 112(f) of the U.S. Patent Act unless that element is explicitly enumerated using the phrase "means for..." or, in the case of a method claim, enumerated using the phrase "steps for...". All structural and functional equivalents of elements of various embodiments described throughout this disclosure, which are known to those skilled in the art or will become known later, are expressly incorporated by reference herein and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is explicitly enumerated in the claims.

Claims

1. A processing system, At least one memory location where executable instructions are stored, One or more processors, wherein the processing system includes: The system receives an input prompt and a plurality of sets of tokens generated based on a first generation artificial intelligence model, wherein each set of tokens in the plurality of sets of tokens includes a sequence of tokens corresponding to a candidate response to the input prompt. Using a second generative artificial intelligence model and recursive adjustment of the target distribution associated with the multiple sets of received tokens, one set of tokens is selected from the multiple sets of tokens. In response to the aforementioned input prompt, the selected set of tokens is output. One or more processors configured to execute the aforementioned executable instructions, A processing system equipped with the following features.

2. The processing system according to claim 1, wherein multiple sets of the aforementioned tokens are organized into a tree data structure.

3. The processing system according to claim 2, wherein the root node of the tree data structure corresponds to the input prompt, and each path through the tree data structure corresponds to a different sequence of tokens corresponding to the candidate response to the input prompt.

4. The tree data structure includes multiple levels, each level corresponding to a token in the sequence of tokens. The number of tokens at a particular level in the tree data structure is based on a branching factor associated with the level immediately preceding that particular level in the tree data structure. The processing system according to claim 2.

5. The processing system according to claim 2, wherein the depth of the tree data structure corresponds to a parameter that defines the maximum number of tokens generated by a single pass through the first generative artificial intelligence model.

6. The processing system according to claim 1, wherein the size of each set of tokens is based on a computational complexity metric associated with generating a target set of tokens by the second generative artificial intelligence model.

7. The recursive adjustment of the target distribution is Determining whether to accept or reject a first token from one set of tokens from multiple sets of tokens, Based on the decision to accept or reject the first token, the probability distribution used to verify the second token in the set of tokens following the first token is adjusted. The processing system according to claim 1, including the following:

8. The processing system according to claim 7, wherein, in order to adjust the probability distribution, one or more processors are configured to cause the processing system to subtract a probability value associated with the first token from the probability distribution based on a determination to reject the first token.

9. To select a set of tokens from a plurality of sets of tokens, one or more processors provide the processing system with The first token is rejected at the first level of a tree data structure representing multiple sets of the aforementioned tokens. Based on the rejection of the first token, a modified probability distribution is generated. From the tree data structure, discard or ignore the child tokens of the first token at a level deeper than the first level of the tree data structure. Based on the adjusted probability distribution, the first level of the tree data structure is used to determine whether to accept or reject the second token. The processing system according to claim 1, configured as described above.

10. To select a set of tokens from a plurality of sets of tokens, one or more processors provide the processing system with Each set of tokens generated by the first generative artificial intelligence model is rejected. Using the second generative artificial intelligence model, tokens are sampled based on a target distribution that excludes the probabilities associated with each set of tokens generated by the first generative artificial intelligence model, and the selected set of tokens includes the sampled tokens. The processing system according to claim 1, configured as described above.

11. The first generative artificial intelligence model corresponds to a draft model in a speculative decoding pipeline, The second generative artificial intelligence model corresponds to the target model in the speculative decoding pipeline. The processing system according to claim 1.

12. A method implemented by a processor, The process involves receiving an input prompt and a plurality of sets of tokens generated based on a first generation artificial intelligence model, wherein each set of tokens in the plurality of sets of tokens includes a sequence of tokens corresponding to a candidate response to the input prompt. Selecting one set of tokens from multiple sets of tokens using a second generative artificial intelligence model and recursive adjustment of the target distribution associated with multiple sets of the received tokens, In response to the aforementioned input prompt, the selected set of tokens is output, Methods that include...

13. The method according to claim 12, wherein multiple sets of the aforementioned tokens are organized into a tree data structure.

14. The method according to claim 13, wherein the root node of the tree data structure corresponds to the input prompt, and each path through the tree data structure corresponds to a different sequence of tokens corresponding to the candidate response to the input prompt.

15. The tree data structure includes multiple levels, each level corresponding to a token in the sequence of tokens. The number of tokens at a particular level in the tree data structure is based on a branching factor associated with the level immediately preceding that particular level in the tree data structure. The method according to claim 13.

16. The method according to claim 13, wherein the depth of the tree data structure corresponds to a parameter that defines the maximum number of tokens generated by a single pass through the first generative artificial intelligence model.

17. The method according to claim 12, wherein the size of each set of tokens is based on a computational complexity metric associated with generating a target set of tokens by the second generative artificial intelligence model.

18. The recursive adjustment of the target distribution is Determining whether to accept or reject a first token from one set of tokens from multiple sets of tokens, Based on the decision to accept or reject the first token, the probability distribution used to verify the second token in the set of tokens following the first token is adjusted. The method according to claim 12, including the method described in claim 12.

19. The method according to claim 18, wherein adjusting the probability distribution involves subtracting a probability value associated with the first token from the probability distribution based on the determination that the first token is to be rejected.

20. Selecting a set of tokens from a plurality of sets of tokens is Rejecting the first token at the first level of a tree data structure representing multiple sets of the aforementioned tokens, Based on the rejection of the first token, a modified probability distribution is generated, From the tree data structure, discard or ignore the child tokens of the first token at a level deeper than the first level of the tree data structure, Based on the adjusted probability distribution, determine whether to accept or reject the second token at the first level of the tree data structure. The method according to claim 12, including the method described in claim 12.

21. Selecting a set of tokens from a plurality of sets of tokens is Rejecting each set of tokens generated by the first generative artificial intelligence model, Using the second generative artificial intelligence model, sample tokens based on a target distribution that excludes the probabilities associated with each set of tokens generated by the first generative artificial intelligence model, and the selected set of tokens includes the sampled tokens. The method according to claim 12, including the method described in claim 12.

22. The first generative artificial intelligence model corresponds to a draft model in a speculative decoding pipeline, The second generative artificial intelligence model corresponds to the target model in the speculative decoding pipeline. The method according to claim 12.

23. A processing system, Means for receiving an input prompt and a plurality of sets of tokens generated based on a first generation artificial intelligence model, wherein each set of tokens in the plurality of sets of tokens includes a sequence of tokens corresponding to a candidate response to the input prompt. A means for selecting one set of tokens from multiple sets of tokens, using a second generative artificial intelligence model and recursive adjustment of target distributions associated with multiple sets of received tokens, Means for outputting the selected set of tokens in response to the input prompt, A processing system equipped with the following features.

24. The processing system according to claim 23, wherein multiple sets of the aforementioned tokens are organized into a tree data structure.

25. The processing system according to claim 24, wherein the root node of the tree data structure corresponds to the input prompt, and each path through the tree data structure corresponds to a different sequence of tokens corresponding to the candidate response to the input prompt.

26. The tree data structure includes multiple levels, each level corresponding to a token in the sequence of tokens. The number of tokens at a particular level in the tree data structure is based on a branching factor associated with the level immediately preceding that particular level in the tree data structure. The processing system according to claim 24.

27. The processing system according to claim 24, wherein the depth of the tree data structure corresponds to a parameter that defines the maximum number of tokens generated by a single pass through the first generative artificial intelligence model.

28. The processing system according to claim 23, wherein the size of each set of tokens is based on a computational complexity metric associated with generating a target set of tokens by the second generative artificial intelligence model.

29. The recursive adjustment of the target distribution is Determining whether to accept or reject a first token from one set of tokens from multiple sets of tokens, Based on the decision to accept or reject the first token, the probability distribution used to verify the second token in the set of tokens following the first token is adjusted. The processing system according to claim 23, including the following:

30. The processing system according to claim 29, comprising adjusting the probability distribution by subtracting a probability value associated with the first token from the probability distribution based on the determination that the first token is to be rejected.

31. The means for selecting a set of tokens from a plurality of sets of tokens is Means for rejecting a first token at the first level of a tree data structure representing multiple sets of the aforementioned tokens, Means for generating an adjusted probability distribution based on the rejection of the first token, Means for discarding or ignoring child tokens of the first token at a level deeper than the first level of the tree data structure, Means for determining whether to accept or reject a second token at the first level of the tree data structure based on the adjusted probability distribution, The processing system according to claim 23, including the following:

32. The means for selecting a set of tokens from a plurality of sets of tokens is Means for rejecting each set of tokens generated by the first generative artificial intelligence model, Means for sampling tokens using the second generative artificial intelligence model, based on a target distribution that excludes the probabilities associated with each set of tokens generated by the first generative artificial intelligence model, and the selected set of tokens includes the sampled tokens. The processing system according to claim 23, including the following:

33. The first generative artificial intelligence model corresponds to a draft model in a speculative decoding pipeline, The second generative artificial intelligence model corresponds to the target model in the speculative decoding pipeline. The processing system according to claim 23.

34. A non-temporary computer-readable medium in which executable instructions are stored, wherein when the executable instructions are executed by a processor, The process involves receiving an input prompt and a plurality of sets of tokens generated based on a first generation artificial intelligence model, wherein each set of tokens in the plurality of sets of tokens includes a sequence of tokens corresponding to a candidate response to the input prompt. Selecting one set of tokens from multiple sets of tokens using a second generative artificial intelligence model and recursive adjustment of the target distribution associated with multiple sets of the received tokens, In response to the aforementioned input prompt, the selected set of tokens is output, A non-temporary computer-readable medium that performs operations including [specific actions].

Images