Speculative Decoding in Autoregressive Models of Generative Artificial Intelligence

Description

1 / 46 Speculative Decoding in Autoregressive Models of Generative Artificial Intelligence CROSS-REFERENCE TO RELATED DEPOSIT REQUESTS

[0001] This application claims priority to US patent application serial number US Provisional Patent Application No. 18 / 538,965, entitled Speculative Decoding in Autoregressive Generative Artificial Intelligence Models, filed December 13, 2023, which claims benefit of and priority to US Provisional Patent Application Serial No. 63 / 460,850, entitled Speculative Decoding in Autoregressive Generative Artificial Intelligence Models, filed Thursday, April 20, 2023, both of which are assigned to the assignee of this document and are incorporated herein by reference in their entirety. INTRODUCTION

[0002] Aspects of the present disclosure relate to generative artificial intelligence models and, more specifically, to speculative decoding in generative artificial intelligence models (also referred to in the present invention as generative artificial intelligence models).

[0003] Generative artificial intelligence models can be used in various environments to generate a response to an input prompt (also called a query or input). For example, generative artificial intelligence models can be used in chatbot applications where large language models (LLMs) are used to generate a response, or at least a replica, to an input prompt. Other examples where generative artificial intelligence models can be used include a latent diffusion model, in which a model generates an image from an input text description of the desired image content, and decision transformers, in which future actions are predicted based on sequences of previous actions within a given environment, or similar.

[0004] In general, generating a response to a query using generative artificial intelligence models can have a high computational cost. For example, Petition 870250089725, dated 02 / 10 / 2025, pages 221 / 295 2 / 46 In a chatbot deployment where a large-scale language model is used to generate a response to a query formatted as a text query, a response to the query can be generated using a pass through the large-scale language model for each token (e.g., a word or part of a word) generated as part of the response. The output of each pass can be a probability distribution on a set of tokens (e.g., words or parts of words) from which the next token (e.g., a word or part of a word) can be selected, for example, by sampling or based on maximum likelihood.Since a pass-through of a large-scale language model is used to generate each word (or token(s)) in a query response, the computational cost can be modeled as the product of the number of words included in the response and the cost of computational resources (e.g., in terms of processing power, memory bandwidth, and / or other computing resources used) to perform a pass-through of the large-scale language model, which generally increases as the number of parameters within the large-scale language model increases. BRIEF SUMMARY

[0005] Certain aspects of the present disclosure provide a method for generating a response to an input prompt using a generative artificial intelligence model. In general, the method includes receiving a plurality of token sets generated based on an input prompt and a first generative artificial intelligence model, where each token set in the plurality of token sets corresponds to a candidate response to the input prompt; selecting, using a second generative artificial intelligence model and recursively fitting a target distribution associated with the received plurality of token sets, a token set from among the plurality of token sets; and emitting the selected token set as a response to the input prompt. Petition 870250089725, dated 02 / 10 / 2025, pages 222 / 295 3 / 46

[0006] Certain aspects of this disclosure provide a method for generating a response to an input prompt using a generative artificial intelligence model. In general, the method includes generating, based on an input prompt and a generative artificial intelligence model, a first plurality of token sets, where each set of tokens in the first plurality of token sets corresponds to a first portion of a candidate response to the input prompt. Using the generative artificial intelligence model, a second plurality of token sets is generated speculatively. Each set of tokens in the second plurality of token sets corresponds, in general, to a second portion of the candidate response to the input prompt based on the first plurality of token sets.While the second plurality of token sets is generated speculatively, a set of tokens from the first plurality of token sets is selected, and the selected set of tokens from the first plurality of tokens and an associated set of tokens in the second plurality of tokens are issued as a response to the input prompt.

[0007] Other aspects provide processing systems configured to perform the aforementioned methods, as well as those described in the present invention; non-transient computer-readable media comprising instructions that, when executed by one or more processors of a processing system, cause the processing system to perform the aforementioned methods, as well as those described in the present invention; a computer program product embedded in a computer-readable storage medium comprising code to perform the aforementioned methods, as well as those additionally described in the present invention; and a processing system comprising means to perform the aforementioned methods, as well as those additionally described in the present invention. Petition 870250089725, dated 02 / 10 / 2025, pages 223 / 295 4 / 46

[0008] The following description and related drawings set out in detail certain illustrative attributes of one or more aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The attached figures depict only certain aspects of this disclosure and, therefore, should not be considered limiting the scope of this disclosure.

[0010] Figure 1 illustrates an example of speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0011] Figures 2A and 2B illustrate examples of recursive speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0012] Figure 3 illustrates a tree of example tokens generated using recursive speculative decoding in generative artificial intelligence models, according to aspects of this disclosure.

[0013] Figure 4 illustrates example operations for generating a response to an input prompt using recursive speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0014] Figure 5 illustrates an example pipeline for self-speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0015] Figure 6 illustrates example architectures for self-speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0016] Figure 7 illustrates example operations for generating a response to an input prompt using a generative artificial intelligence model and self-speculative decoding, in accordance with aspects of the present disclosure.

[0017] Figure 8 depicts an example processing system configured to perform various aspects of the present disclosure.

[0018] Figure 9 depicts an example processing system configured to perform various aspects of the present disclosure. Petition 870250089725, dated 02 / 10 / 2025, pages 224 / 295 5 / 46

[0019] To facilitate understanding, identical reference numbers have been used, where possible, to designate identical elements that are common to the drawings. It is contemplated that the elements and attributes of one aspect may be beneficially incorporated into other aspects without further mention. DETAILED DESCRIPTION

[0020] Aspects of the present disclosure provide computer-readable devices, methods, processing systems and means for efficiently generating responses to input queries using generative artificial intelligence models.

[0021] In general, generative artificial intelligence models generate a response to a query provided as input to the model. For example, a large-scale language model (LLM) deployed in a chatbot can generate a response to a query using multiple passes through the large-scale language model, with each successive pass being based on the query (which may be tokenized for processing) and the tokens (or words) generated using previous passes through the large-scale language model. In general, these large-scale language models can include a large number (e.g., billions or even trillions) of weights or parameters in the model.Due to the size of these models and the operations performed on each token to predict which token should be generated next in response to a query and to previously generated tokens, it may not be practical, or even possible, to deploy large-scale language models on a variety of devices that may have limited memory, storage, and / or processing capacity compared to the cloud computing instances on which large-scale language models typically operate. Additionally, in some cases, the memory bandwidth involved in generating a response to a query provided as input to a model may prevent computing resources from being used for other tasks. Petition 870250089725, dated 02 / 10 / 2025, pages 225 / 295 6 / 46

[0022] To improve the efficiency and throughput of large-scale language models, speculative decoding techniques enable a smaller language model, sometimes known as a preliminary large-scale language model (or as a preliminary model, or an approximation model), to be run (e.g., sequentially or in parallel) with a larger language model, sometimes known as a target large-scale language model (or as a target model). In this case, the preliminary model can speculatively generate additional tokens sequentially and probabilities used to sample these additional tokens based on a current set of accepted tokens. The target model can generate tokens based on the tokens generated by the preliminary model.To generate a result, the target model can perform token-by-token rejection sampling in order to accept or reject individual tokens generated by the preliminary model, so that the preliminary model and the target model have similar probability distributions.

[0023] In some respects, the preliminary model may be a pruned version of the target model, chosen so that the preliminary model and the target model have similar probability distributions. In other respects, the preliminary model may be a smaller version of the target model (for example, trained on millions of tokens, rather than hundreds of millions or even billions of tokens).

[0024] Certain aspects of the present disclosure provide techniques and apparatus for generating answers to a query provided as input to a large-scale language model using speculative decoding techniques. In general, the preliminary model can generate one or more sets of tokens as candidate answers to the query, which can be structured as a plurality of branches (e.g., in a tree data structure). The target model, in turn, can perform recursive sample rejection on tokens provided by the preliminary model. In general, the Petition 870250089725, dated 02 / 10 / 2025, pages 226 / 295 7 / 46 Recursive rejection sampling enables the probability distribution used by the target model in sampling tokens generated by the preliminary model to be updated to remove a token rejected by the target model, and the updated probability distribution is then used to sample subsequent tokens generated by the preliminary model. By implementing recursive rejection sampling, certain aspects of the present disclosure can retain a close relationship between the probability distributions within the preliminary and target models, while increasing the throughput of the preliminary and target models (e.g., the number of tokens generated per second) compared to the preliminary and target models configured to generate token-by-token responses.

[0025] Other aspects of the present disclosure provide techniques and apparatus for generating responses to a query provided as input to a large-scale language model using speculative decoding techniques in which a single model speculatively generates tokens in response to the query provided as input and verifies previously generated tokens, also referred to in the present invention as self-speculative decoding. In self-speculative decoding techniques, a model can speculatively generate one or more tokens and speculatively generate additional tokens based on varying numbers of speculatively generated tokens that are verified by the model.By using the same model to speculatively generate tokens in response to a query and to perform verification (e.g., rejection sampling) on ​​the speculatively generated token, aspects of this disclosure can reduce the computational cost involved in training and using generative artificial intelligence models compared to using multiple models trained separately to speculatively generate tokens and perform verification of the speculatively generated tokens. Additionally, the rate at which tokens are generated can be maximized, or at least increased, with self-speculative decoding, compared to other speculative decoding techniques. Petition 870250089725, dated 02 / 10 / 2025, pages 227 / 295 8 / 46 Speculative decoding in generative artificial intelligence models

[0026] In general, autoregressive token generation (for example, in large-scale language models) can take historical tokens as an input in order to generate an output. That is, autoregressive token generation can be represented by the expression: Xt ~ p(x0, x1, ..., x-1) ^ xm ~ p(x0, x1, ..., xt-1'xt) where xt represents a sequence of tokens generated at time t, having a conditional probability p conditioned on the selection of tokens x0 ≠ xt-1, and xt+1 represents a sequence of tokens generated at time t + 1, having a conditional probability p conditioned on the selection of tokens x0 ≠ xt. In general, a single token can be generated each time an autoregressive model is run, meaning that N inferences can be made to generate a sequence of N tokens. As discussed above, speculative decoding techniques can be used to accelerate token generation by using a preliminary model, smaller in size than the target model, which speculatively generates tokens faster than the target model, with the target model being used to verify the tokens generated (speculatively) by the preliminary model.

[0027] In a speculative decoding pipeline, the preliminary model can speculatively generate n tokens autoregressively, according to the expression: χpreliminar ~ ppreliminar _ ppreliminar (χ|χ χ) ^preliminar ~ ppreliminar preliminar preliminarXt+n ~ pt+n-1 where t corresponds to a point in time, pVpreliminar corresponds to the conditional probability distribution associated with a token selected at the instant conditioned to the selection of tokens x0a xt-1, expreliminar represents a token x generated speculatively at time t by the preliminary model. Petition 870250089725, dated 02 / 10 / 2025, pages 228 / 295 9 / 46

[0028] The target model takes the n generated tokens and processes them in parallel to generate probability distributions for each of the n tokens, according to the expression: [pdestination, pdestination.....p^ino] = [pdeDestination (χ|χ0, ...χ^ ^delete, .,χpreliminary)] ? where k corresponds to a token index relative to the n tokens generated and pdestmo corresponds to a probability distribution generated by the target model at time t for the x tokens generated by the preliminary model.

[0029] The target model can then verify the tokens generated by the preliminary model by comparing the distributions of the preliminary model and the target model to determine whether a token is accepted or rejected. A given token x^mnr can be accepted when f(ppeliminar,p <d^inoy) < a, para alguma função f e algum limiar a (também conhecido como taxa de aceitação). Caso contrário, o token pode ser rejeitado. O token final pode, então, ser gerado na primeira posição de rejeição ou na última posição n com base em alguma função g(pPreliminar ,p^est;ino).

[0030] Speculative decoding, with an acceptance rate of a, can result in cost reductions compared to using a single autoregressive model to iteratively generate tokens. The inference cost savings, compared to iterative token generation, can be represented by the expression: N . .. CAR = ^ CSD = nNCpreliminary + cdestination where N corresponds to a number of tokens, CAR corresponds to a computational cost using an acceptance rate of a, cdestination corresponds to the computational cost of generating a set of tokens using the destination model, Cpreliminary corresponds to the computational cost of generating a set of tokens using the preliminary model, CSD corresponds to the computational cost of speculatively generating a set of tokens using the preliminary model, and en corresponds to the number of tokens speculatively generated in a single pass through an autoregressive model. Consider an example in Petition 870250089725, dated 02 / 10 / 2025, pp. 229 / 295 10 / 46 where N = 1000, Cdestination = 10, cpreliminary = 1, n = 4, and a = 3. In such an example, speculative decoding can result in a 35% reduction in computational cost compared to iterative autoregressive token generation alone.

[0031] However, token-by-token speculative decoding, as discussed, can impose limits on the rate at which tokens are generated, as a first token may be sampled individually by a preliminary model, and then verified by a target model before the next token is sampled by the preliminary model and verified by the target model. That is, generating a response to an input prompt using token-by-token speculative decoding techniques may involve executing both the preliminary model and the target model for each token generated as part of a response to the input prompt, which may use significant amounts of computational resources (e.g., processor time, memory, memory bandwidth, etc.) in order to generate the response. Example of recursive speculative decoding in generative artificial intelligence models

[0032] Figure 1 illustrates an example 100 of recursive speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0033] As illustrated, a preliminary model and a target model can be used together (or, in other words, jointly) to perform recursive speculative token decoding to generate a response to a query received for processing by one or more generative artificial intelligence models. As discussed in more detail below, recursive speculative token decoding can enable multiple sets of tokens (or sequences of tokens) to be speculatively generated by a preliminary model for verification by a target model. Because multiple sets (or sequences) of tokens can be generated by a preliminary model for verification by a target model, recursive speculative decoding can increase the token generation rate of artificial intelligence models. Petition 870250089725, dated 02 / 10 / 2025, pages 230 / 295 11 / 46 generative by generating multiple sets (sequences) of tokens that can be accepted as a correct answer, since larger numbers of token sets can increase the probability that at least one set includes a sequence of one or more tokens that will be accepted as an answer.

[0034] In general, the preliminary model selects a plurality of high-probability nodes (tokens) from a probability distribution for an output relative to a set of potential tokens, given an input from the received query. The high-probability nodes (tokens) can be selected based on various techniques, such as top-k selection (e.g., selecting the k tokens having the highest probabilities within a probability distribution), kernel-based selection (e.g., selection based on a sum of probabilities that meet a threshold probability), or similar techniques. By choosing from many candidate tokens, the preliminary model can sample tokens based on the probability distribution and arrange a tree structure that can be traversed recursively, as discussed in more detail below, in order to identify a set of tokens that are a suitable output for the given input.

[0035] When a sampled group of tokens is provided as input to the preliminary model in the next iteration of the preliminary model's execution, the tokens in the sampled group of tokens are provided as input to the sample location and treated independently. The result may be a 110-tree data structure, with a prompt as a root node of the tree data structure, and subsequent levels within the 110-tree data structure representing different tokens (or groups of tokens), matching each of the previously selected token combinations. At some point in time (for example, after generating a tree with a defined depth, corresponding to a maximum length of a sequence generated by the preliminary model), the preliminary model may output the generated 110-tree data structure to the target model for further processing. The 110-tree data structure may, Petition 870250089725, dated 02 / 10 / 2025, pp. 231 / 295 12 / 46 in some respects, be issued to the target model with groupings and selection probabilities generated by the preliminary model.

[0036] In some respects, the preliminary model can be configured to trigger token generation by the target model (and subsequent speculative decoding) based on various criteria. These criteria may include a complexity criterion or a performance criterion, such as a complexity or performance criterion associated with the size of the generated 110-tree data structure. In some respects, these criteria may include a time criterion associated with an expected amount of time for the target model to generate a set of tokens to which the generated 110-tree data structure can be compared. Generally, these complexity and / or performance criteria may establish an upper limit on the number of tokens generated by the preliminary model for verification by the target model.This upper limit can, in some respects, be based on a number of nodes in the tree data structure and can be influenced, for example, by a branching factor defined for different levels of the 110-tree data structure in which the sampled tokens are organized, a depth of the 110-tree data structure, or similar factors. The worst-case computational load in the last round of speculative token generation can be configured to be limited by the memory bandwidth on the device where the preliminary model is running.

[0037] In some respects, the number of nodes at each level of the 110-tree data structure (for example, where the token n illustrated in Figure 1 corresponds to the n-level in the 110-tree data structure, and where level 0 corresponds to the root node 111 in the 110-tree data structure) and the depth of the 110-tree data structure can be defined a priori. The number of nodes at each level of the tree can be defined globally, level by level, or in some other way. For example, the number of nodes at any level of the 110-tree data structure can be defined based on a branching factor at the immediately preceding level of the 110-tree data structure. For example, a Petition 870250089725, dated 02 / 10 / 2025, pp. 232 / 295 A branching factor of 2 for nodes in the nth level of the 110-tree data structure can result in the generation of 2 nodes (tokens) in the n+1th level of the 110-tree data structure for each node (token) in the nth level of the tree. Meanwhile, the depth of the 110-tree data structure can be defined based on a maximum number of tokens (e.g., words) that can be generated using any pass through the preliminary model. For example, if the preliminary model is configured to generate a sequence with a maximum length of 5 tokens during any speculative generation instance, then the tree depth could be 6 (to include, in the first level of the 110-tree data structure, the root node 111 corresponding to the entry in the preliminary model).

[0038] The target model recursively performs rejection sampling (1) on the tokens generated by the preliminary model and included in the generated 110-tree data structure and (2) on a probability distribution q provided as input to the target model. Rejection sampling can be performed recursively at each node in the generated tree, where a termination condition for token selection by the target model at a given layer of the 110-tree data structure is modeled as a recursion problem in which a termination condition of the recursion problem is either the acceptance of a token or the rejection of all tokens. By recursively performing rejection sampling, the target model can accept or reject a token and adjust the probability distribution used to check a subsequent token in the generated tree.If a token is rejected, an updated probability distribution q' = (q - p) can be generated for use in evaluating subsequent tokens in the tree, where p represents the probability associated with the rejected token from the original probability distribution q. Subsequently, the updated probability distribution q' can be used to evaluate the next token in the tree. The resulting set of 112 selected tokens can be recursively identified by traversing from the root node 111 of the generated 110 tree data structure. Petition 870250089725, dated 02 / 10 / 2025, pp. 233 / 295 14 / 46 based on the updated probability distributions generated for each node in the generated tree data structure, as discussed in more detail below.

[0039] In one example, recursive rejection sampling can be performed using greedy techniques, where the first token that is accepted is returned as a valid portion of an input query (or prompt) response. In another example, recursive rejection sampling can be performed to determine whether to accept each token at a given level in the 110-tree data structure. The token selected at that given level in the 110-tree data structure might be, for example, the token having the highest probability of being a valid token for inclusion in an input prompt response.In yet another example, a cumulative probability distribution can be generated for each accepted sequence of tokens from the generated tree data structure, and the sequence with the highest cumulative probability distribution can be selected as the response to the input prompt. It should be recognized, of course, that these are only examples of techniques based on which token sequences can be selected from the tree data structure 110, and other techniques for selecting token sequences based on recursive rejection sampling (and the corresponding adjustment of the probability distribution q when tokens are rejected) can be considered.

[0040] In some respects, the preliminary model may match the probability distribution of the target model, but may have faster inference performance than the target model on the same hardware. In general, smaller models may generate many speculative tokens, but may have an increased probability that the generated tokens will be rejected by the target model. The speculative decoding techniques discussed in the present invention can address this increased probability of token rejection, at the cost of increasing the computational cost for longer sequences. Finally, in the preliminary model, a temperature parameter (generally used in the present invention to refer to a parameter that influences the probability) Petition 870250089725, dated 02 / 10 / 2025, pages 234 / 295 15 / 46 of the preliminary model selecting a token (word) with a lower probability) can be adjusted to improve the performance of recursive speculative decoding. In some respects, the preliminary model can be fine-tuned to match, or at least approximate, the probability distribution of the target model and maximize, or at least increase, the probability that the speculatively generated tokens (or token sequences) generated by the preliminary model will be accepted as valid tokens by the target model.

[0041] In general, the performance of token generation for recursive speculative decoding can be increased compared to token-by-token speculative decoding. That is, for any Kullback-Leibler (KL) divergence between the preliminary model and the target model, the number of tokens generated for each round of the target model can be greater for recursive speculative decoding than for token-by-token speculative decoding. KL divergence, in general, measures how the probability distribution of the preliminary model differs from the probability distribution of the target model (treating the target model as the reference distribution). Different selection strategies (e.g., group size, additional tokens, etc.) may have different computational complexity characteristics.Consequently, the selection of a strategy for selecting tokens for acceptance or rejection using recursive speculative decoding can be based on a trade-off between computational complexity and performance, given the read bandwidth-limiting parameters for the primary and target models and the hardware performance.

[0042] Figures 2A and 2B illustrate examples 200A, 200B of recursive speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure.

[0043] Example 200A illustrated in Figure 2A depicts an example in which one of multiple tokens generated by a preliminary model is accepted for inclusion in Petition 870250089725, dated 02 / 10 / 2025, pages 235 / 295 16 / 46 a response to a prompt 210. As illustrated, the preliminary model generates four proposed tokens X1 220A, X2220B, X3220C, and X4220D, with associated probabilities p1, p2, p3, p4 from an original target distribution q. A target model can sequentially examine these tokens to determine whether to accept or reject each token. As illustrated, the target model first examines the token X1 220A to determine whether the token 220A should be accepted or rejected as a token to be returned as part of a response to a given input (e.g., prompt 210). In this example, the target distribution q1 222A can be defined for the original target distribution q, and the target model can determine whether to accept or reject the token X1 220A based on the selection criteria 224A: " " . C qi(Xi)} <mm]1,1l Wi)) onde Ur representa um número aleatório gerado entre [0,1].

[0044] If the X1 220A token is accepted (not shown), then the X1 220A token can be issued, and the analysis of the target model of the proposed tokens generated by the preliminary model can proceed to traverse the tree in order to analyze nodes (tokens) connected to the node represented by the X1 220A token in the generated tree.

[0045] Otherwise, if, as illustrated, token X1 220A is rejected, then the target model can proceed to determine whether token X2220B should be accepted or rejected. In doing so, the target model can use a new target distribution q2222B, which can be the result of subtracting the probability p1 associated with the rejected token X1 220A from the target distribution q1 222A, such that q2 = (q1 - p1) + . The target model can then determine whether to accept or reject token X2220B based on selection criteria 224B: „ . L q2(X2h < mm 1,2l 'p2(X2)J where U2 also represents a random number generated between [0,1].

[0046] As with the X1 220A token discussed above, the target model may determine that the X2220B token satisfies the acceptance criteria and thus return the X2220B token as the selected token (not shown). Otherwise, Petition 870250089725, dated 02 / 10 / 2025, pages 236 / 295 17 / 46 if, as illustrated, token X2 220B is rejected, then the target model proceeds to determine whether token X3220C should be accepted or rejected, using an updated target distribution q3222C that removes p2 from q2 (for example, so that q3 = (q2 - p2)+) and acceptance criteria U3224C. This process can continue until, as illustrated, it is determined that a token (in this example, token X4220D illustrated in example 200A using an updated target distribution q4222D) is accepted (for example, based on acceptance criteria i / 4224D) and issued as the selected token Y 230.

[0047] Example 200B illustrated in Figure 2B illustrates an example in which the target model rejects each of the tokens 220A to 220D generated by the preliminary model. In this example, the target model can generate a final target distribution 222E, sample a token from this final target distribution 222E, and terminate the generated tree traversal. As illustrated, the final target distribution q5222E can be the result of subtracting the probability associated with the final token from the set of generated tokens (e.g., the probability p4 associated with token X4220D). The final target distribution q5222E can be represented by the equation q5 = (q4 - p4) + . A token 240 sampled from the final target distribution q5222E can be returned as the output Y of performing recursive rejection sampling on the generated tokens.

[0048] Figure 3 illustrates an example 300-tree data structure of tokens generated using recursive speculative decoding in generative artificial intelligence models, according to aspects of this disclosure.

[0049] As illustrated, the 300-tree data structure includes a plurality of levels, starting from a root node 310 that represents the input (which may be tokenized) into a generative artificial intelligence model and tokens previously generated by the preliminary model and accepted by the target model using recursive rejection sampling, as discussed above. Each level 320, 330, and 340 (and / or others not illustrated in Figure 3) in the data structure Petition 870250089725, dated 02 / 10 / 2025, pp. 237 / 295 18 / 46 in a 300-tree structure can be the result of speculative token generation by the preliminary model, given some prior set of generated tokens. The number of tokens generated at level n+1 of the 300-tree data structure for a node at level n of the 300-tree data structure can be determined based on a branching factor defined for the nodes at level n of the 300-tree data structure. As discussed, the branching factor can be the same across all levels of the 300-tree data structure, or the branching factor can be defined for each level of the 300-tree data structure. In some respects, the branching factor can be defined with a larger number of branches generated for nodes at lower levels of the 300-tree data structure and with a smaller number of branches for nodes at higher levels of the 300-tree data structure, or vice versa.

[0050] To traverse the 300-tree data structure and select a set of tokens from speculatively generated token groups, the target model can recursively accept or reject tokens starting from the first set of tokens generated by the preliminary model at the first level 320 of the 300-tree data structure (for example, tokens represented by nodes that are the immediate child nodes of the root node 310). In general, when traversing the tree, the target model can determine whether a token should be accepted or rejected and adjust the target distribution based on the determination of whether the token should be accepted or rejected.In the first 320 level of the 300-tree data structure, if a first token is rejected, then the target model can remove the first token from the target distribution (as discussed above in relation to Figure 2) and proceed to determine whether a second token in the first level of the tree should be accepted or rejected based on an updated target distribution. By rejecting the first token in the first 320 level of the 300-tree data structure, the target model does not need to analyze tokens represented by nodes that are, directly or indirectly, child nodes of the node in the 300-tree data structure representing the first rejected token. Petition 870250089725, dated 02 / 10 / 2025, pages 238 / 295 19 / 46

[0051] In general, when traversing the 300-tree data structure, the target distribution can be recursively modified until a termination condition is reached. A termination condition could be, for example, the identification of a sequence of tokens that are accepted as acceptable response outputs for the input (represented by the root node 310 of the 300-tree data structure), plus an additional node sampled from the modified target distribution after the token sequence has been accepted by the target model. In another example, a termination condition could be a determination that no speculatively generated token sequence from the preliminary model has been accepted by the target model. In this case, as discussed above in relation to Figure 2B, an additional token sampled from the final target distribution can be emitted as a response to the input corresponding to the root node 310 in the 300-tree data structure.The target distribution, as discussed above, can be a target distribution generated by removing probabilities associated with the tokens generated from the original target distribution generated by the preliminary model.

[0052] In the example illustrated in Figure 3, at the first level 320 of the tree data structure 300, a first token 322A can be rejected. Because the first token 322A is rejected as one of the candidate tokens for inclusion in a response to the input corresponding to the root node 310, the tokens in the tree data structure 300 that are child tokens of the first token 322A do not need to be examined for selection as candidate tokens. Thus, the immediate child tokens of the first token 322A at the second level 330 of the tree data structure 300 and the grandchild tokens of the first token 322A at the third level 340 of the tree data structure 300 can be discarded or ignored.

[0053] By rejecting the first token 322A from the set of candidate tokens at the first level 320 of the 300-tree data structure, as discussed, the target probability distribution associated with the tokens at the first level 320 of the 300-tree data structure can be modified to remove the value of Petition 870250089725, dated 02 / 10 / 2025, pp. 239 / 295 The 20 / 46 probability associated with the first token 322A. The updated target distribution can subsequently be used to examine a second token 322B at the first level 320 of the tree data structure 300. As illustrated, the second token 322B can be accepted as a candidate token for inclusion in a response to the input corresponding to the root node 310, and an analysis can proceed to the child tokens (e.g., 332A, 332B, and 332C) of the second token 322B at the second level 330 of the tree data structure 300.

[0054] As illustrated, tokens 332A and 332B can be rejected by the target model. By rejecting tokens 332A and 332B, the target probability distribution used to accept or reject tokens at the second level 330 of the 300 tree data structure can be adjusted to remove the probabilities associated with tokens 332A and 332B. This adjusted target probability distribution can then be used to determine whether to accept or reject token 332C at the second level 330 of the 300 tree data structure. As illustrated, the target model can accept token 332C, and an analysis can proceed to the child tokens (e.g., 342A, 342B) of token 332C at the third level 340 of the 300 tree data structure.At the third level 340 of the tree data structure 300, the target model can examine tokens 342A and 342B using the techniques discussed above and can accept token 342A as a candidate for inclusion in a response to the input corresponding to the root node 310.

[0055] In some respects, the acceptance of the second token 322B as a candidate token for inclusion in a response to the input corresponding to the root node 310 may not prevent the analysis of the third token 322C for inclusion or rejection as a candidate token for inclusion in a response to the input corresponding to the root node 310. In this case, the acceptance or rejection of child tokens at levels 320 and 330 of the tree data structure 300 (among others, not illustrated in Figure 3) can be performed as discussed above regarding the acceptance or rejection of child tokens of the second token 322B above. In some respects, such as when the Petition 870250089725, dated 02 / 10 / 2025, pp. 240 / 295 If the probability of the second token 322B being a candidate token for a response to the input corresponding to the root node 310 exceeds a threshold value, the third token 322C can be considered to have been rejected, and an analysis of the child tokens of the third token 322C can be ignored.

[0056] The recursive speculative decoding techniques discussed in the present invention can lead to significant gains in token generation performance compared to token-by-token generation and acceptance. In general, recursive speculative decoding techniques can enable increases in token acceptance performance (e.g., the number of tokens accepted for a given preliminary length) each time the target model is run compared to token-by-token speculative decoding techniques. The amount by which performance increases can expand as the number of branches generated for any token increases and as the preliminary length increases. Additionally, the recursive speculative decoding techniques discussed in the present invention can provide increased rates (over time) at which tokens are generated compared to token-by-token speculative decoding techniques. Example operations for recursive speculative decoding in generative artificial intelligence models.

[0057] Figure 4 illustrates example 400 operations that can be performed by a computing device to generate a response to an input prompt using generative artificial intelligence models (for example, as discussed in the present invention in relation to Figures 1 to 3), in accordance with aspects of the present disclosure. The 400 operations can be performed by a computing device on which at least one target model can be deployed, such as a laptop computer, a desktop computer, a server, a cloud computing instance hosted in a distributed computing environment, or similar. Petition 870250089725, dated 02 / 10 / 2025, pp. 241 / 295 22 / 46

[0058] As illustrated, operations 400 begin in block 410, with the receipt of a plurality of token sets. In general, the plurality of token sets can be tokens generated (for example, by a preliminary model deployed on a client device from which the plurality of token sets is received) based on an input prompt and a first generative artificial intelligence model. The input prompt can be received on the client device, for example, from a user via a text input prompt, audio capture prompts, or other techniques by which input can be provided to the client device. In general, each set of tokens in the plurality of token sets can correspond to a candidate response to the input prompt.

[0059] In some respects, the plurality of token sets can be organized into a tree data structure (for example, the 110 tree data structure illustrated in Figure 1 or the 300 tree data structure illustrated in Figure 3). The tree data structure can have a root node and one or more leaf nodes. The input prompt can correspond to the root node of the tree data structure. Each traversal through the tree data structure can correspond to a different candidate response for the input prompt. It should be understood that a traversal through the tree data structure does not need to end at a node in the tree data structure that has no leaf nodes; that is, a candidate response can correspond to a traversal that partially traverses the tree data structure (for example, to reflect rejected tokens in a response).

[0060] In some respects, the number of tokens at any level of the tree data structure may be based on a branching factor associated with a level immediately preceding that level in the tree data structure. The tree data structure may have different branching factors assigned to different levels in the tree data structure. Generally, higher branching factors may allow for the generation of a greater number of tokens at any level. Petition 870250089725, dated 02 / 10 / 2025, pages 242 / 295 23 / 46 of the tree data structure and, in general, increase the total number of tokens included in the tree data structure. At the same time, lower branching factors can restrict token generation at any level of the tree data structure and, in general, decrease the total number of tokens included in the tree data structure.

[0061] In some respects, the size of each token set can be based on a computational complexity metric associated with generating a target token set by the second generative artificial intelligence model.

[0062] In block 420, operations 400 proceed with the selection, using a second generative artificial intelligence model and recursive fitting of a probability distribution, of a set of tokens from among the plurality of sets of tokens.

[0063] In some respects, recursive probability distribution fitting includes rejecting a token from among a plurality of token sets as a candidate for inclusion as a response to an input prompt. In general, a token can be rejected if some defined acceptance criteria are not met, which may be based on a target distribution and a probability associated with the token in the original target distribution. The current distribution used by the target model to determine whether a token should be accepted or rejected can be modified by removing (or subtracting) from the current distribution the probability associated with the token in the original target distribution, resulting in the generation of an updated target distribution that can be used to determine whether the next token should be accepted or rejected as a candidate for inclusion as a response to the input prompt.

[0064] In some respects, selecting the token set from among a plurality of token sets may involve rejecting a first token at a first level of a tree data structure representing the plurality of token sets. An adjusted probability distribution can be generated based on the Petition 870250089725, dated 02 / 10 / 2025, pages 243 / 295 24 / 46 rejection of the first token. Within the tree data structure, child tokens of the first token at levels deeper than the first level of the tree data structure can be discarded. A second token at the first level of the tree data structure can be accepted or rejected based on the adjusted probability distribution.

[0065] In some respects, selecting the token set from the plurality of token sets may involve rejecting each token set generated by the first generative artificial intelligence model. A token may be sampled, using the second generative artificial intelligence model, based on a target distribution that excludes probabilities associated with each token set generated by the first generative artificial intelligence model.

[0066] In block 430, operations 400 proceed with the issuance of the selected token set as a response to the input prompt.

[0067] In some respects, the first generative artificial intelligence model may correspond to a preliminary model in a speculative decoding pipeline, and the second generative artificial intelligence model may correspond to a target model in the speculative decoding pipeline. The first generative artificial intelligence model may run on a client device, and the second generative artificial intelligence model may run on a system remote from the client device, such as a server computer, a cloud computing instance, or similar. In some respects, the first generative artificial intelligence model and the second generative artificial intelligence model may run on the same device.

[0068] In some respects, the first generative artificial intelligence model and the second generative artificial intelligence model may have equivalent probability distributions.

[0069] In some respects, the first generative artificial intelligence model may have a probability distribution that approximates a Petition 870250089725, dated 02 / 10 / 2025, pages 244 / 295 25 / 46 probability distribution associated with the second generative artificial intelligence model. Self-speculative decoding of examples in generative artificial intelligence models

[0070] In some respects, several types of speculative decoding, including group speculative decoding and recursive speculative decoding, can be achieved using a single generative artificial intelligence model that combines the functionality of the preliminary model and the target model discussed above. By doing so, preliminary token generation, target token generation, and token acceptance can be parallelized in a single generative artificial intelligence model.The use of a single generative artificial intelligence model can, for example, reduce the computational cost involved in generating both a target model and a preliminary model, increase the performance of generative tasks by performing token verification and speculative generation in a single pass through the single generative artificial intelligence model, reduce the amount of memory used in storing models used for speculative decoding in generative tasks, and so on.

[0071] Figure 5 illustrates an example 500 pipeline for self-speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure. In some aspects, the 500 pipeline can be used to generate and analyze a token tree data structure associated with a candidate query response provided as input to a generative intelligence model, such as the 300 tree data structure illustrated in Figure 3 or similar.

[0072] As illustrated, the 500 pipeline uses a single generative artificial intelligence model to speculatively generate tokens and verify the speculatively generated tokens. During a first round of inference in the 500 pipeline, a first set of 502 tokens is speculatively generated. As Petition 870250089725, dated 02 / 10 / 2025, pages 245 / 295 26 / 46 illustrates, for example, the first set of 502 tokens might include tokens 1 through 4 and could be provided as input during a second round in the 500 pipeline to speculatively generate the next set of tokens as a batch process in which multiple sets of tokens are generated. While the first set of speculatively generated tokens is processed by the single generative artificial intelligence model, the single generative artificial intelligence model continues to speculatively generate a plurality of second sets of preliminary tokens 504, 506, 508, and 510 in a second round of inference in the 500 pipeline.

[0073] When generating the second sets of preliminary tokens 504, 506, 508 and 510, assumptions can be made for different numbers of tokens accepted from the first set of 502 tokens. For example, as illustrated, the second set of preliminary 504 tokens may assume acceptance of the first preliminary token from the first set of 502 tokens and may include a set of tokens generated speculatively based on the acceptance of the first token. The second set of preliminary 506 tokens may assume acceptance of the first and second preliminary tokens from the first set of 502 tokens and include a set of tokens generated speculatively based on the acceptance of the first and second tokens. The second set of preliminary 508 tokens may assume acceptance of the first to third preliminary tokens from the first set of 502 tokens and include a set of tokens generated speculatively based on the acceptance of the first to third tokens.Finally, the second set of preliminary 510 tokens may assume acceptance of all four preliminary tokens from the first set of 502 tokens and include a set of tokens generated speculatively based on the acceptance of all four tokens. In several respects, for cases where a smaller number of tokens than the number of tokens included in the first set of 502 tokens is assumed to be accepted, 503 padding (e.g., null values, predefined constants, etc.) may be added so that each assumption has the same length. Petition 870250089725, dated 02 / 10 / 2025, pages 246 / 295 27 / 46

[0074] Once the single generative AI model completes rejection sampling on the speculatively generated token set, the single generative AI model selects the speculatively generated token set associated with the accepted token set from the first set as input to the single generative AI model for another round of inference in which tokens are speculatively generated using the single generative AI model. In this example, it can be seen that all four tokens in the first token set 502 were accepted by the single generative AI model as a preliminary check 512, and thus the token set 510 can be used for further speculative token generation using the single generative AI model.

[0075] The above process can be continued until a termination event occurs. Successive rounds of speculative generation can be based on assumptions about the number of tokens from a previous round of speculative generation being accepted by the single generative artificial intelligence model. For example, as illustrated in Figure 5, preliminary token sets 522, 524, 526, and 528 can be generated in the k+1th round of inference, with the tokens included in the preliminary token sets being based on a number of speculatively generated tokens beyond the N accepted tokens generated in the k-1st round of inference. In this example, it can be seen that the four speculatively generated tokens, generated during the kth round of inference, were accepted as a preliminary check 520, and tokens N+5 to N+8 can be used for further speculative token generation using the single generative artificial intelligence model.

[0076] In some respects, a termination event may include the generation of a special token used to denote the end of a response (e.g., that no further tokens could plausibly be included in a response due to the probabilities associated with those tokens falling below a value of Petition 870250089725, dated 02 / 10 / 2025, pages 247 / 295 28 / 46 (threshold probability for acceptance). A termination event can, in some respects, be reached when a threshold number of tokens has been generated.

[0077] In some respects, when all tokens from a previous round of speculative token generation are rejected by the single generative artificial intelligence model, the process can restart with the last set of accepted tokens, plus one token sampled from a final distribution (e.g., as discussed above), being provided as input to the single generative artificial intelligence model.

[0078] Figure 6 illustrates example architectures 600A, 600B for self-speculative decoding in generative artificial intelligence models, according to aspects of the present disclosure. Both example architectures 600A and 600B can enable the generation of multiple tokens in any pass through the model, as in the token generation illustrated in Figures 1, 2A, 2B, 3, 4 and 5, as discussed above.

[0079] In the example architecture 600A, a generative artificial intelligence model 610 can be trained to generate multiple prediction prompt embeddings 612, attached to a set of tokens provided as input, to enable the parallel generation of multiple output tokens 614. These prediction prompt embeddings can be embeddings that correspond to tokens that are included in a response to an input prompt (including any previously generated and accepted tokens). The generative artificial intelligence model 610 can be a generative artificial intelligence model, such as a pre-trained large-scale language model or another pre-trained generative artificial intelligence model, updated using various fine-tuning techniques.For example, a generative artificial intelligence model used to generate textual responses to textual inputs (also known as a large-scale language model) can be updated using techniques such as low-rank adaptation (LoRA) of large-scale language models. Petition 870250089725, dated 02 / 10 / 2025, pages 248 / 295 29 / 46

[0080] In the example architecture 600B, generative artificial intelligence models can be implemented as a partial autoregressive model. The inference operations, used to speculatively generate tokens, can be performed using a subset of layers in the partial autoregressive model (e.g., the top n layers of the model or the bottom n layers of the model). By doing so, the layers used to speculatively generate tokens can create a context that can enable causality and / or other relationships to be modeled for the speculatively generated tokens, which can be fed as input to the portion of the model that verifies the tokens as valid responses to the input prompt.

[0081] The 600B architecture can be implemented in several ways, so that autoregressive inference and the generation of multiple sets of tokens for acceptance and / or rejection can be generated using a small number of autoregressive layers in a generative artificial intelligence model. In the example 620 implementation, a generative artificial intelligence model can include a plurality of non-autoregressive layers 622A to 622C and one autoregressive layer 624. The layers in the generative artificial intelligence model can be arranged in a stack, with the lowest layer in the stack corresponding to the layer that receives an input for processing and the highest layer in the stack corresponding to the layer that generates an output. In the 620 implementation, the non-autoregressive layers 622A to 622C can be arranged at the bottom of the stack and the autoregressive layer 624 can be arranged at the top of the stack.In contrast, in the example implementation 630, the layers of the generative artificial intelligence model can be arranged so that an autoregressive layer 632 is placed at the bottom of the stack and the non-autoregressive layers 632A to 632C are placed at the top of the stack. These autoregressive layers 624 and 632 can operate, for example, in a loop to continuously generate and accept tokens to be issued as a response to a request. Petition 870250089725, dated 02 / 10 / 2025, pp. 249 / 295 30 / 46 input prompt (and, in some respects, previously generated tokens included as a partial response to the input prompt). Example operations for generating responses to input queries using self-speculative decoding in generative artificial intelligence models

[0082] Figure 7 illustrates 700 example operations that can be performed by a computing device to generate a response to an input prompt using generative artificial intelligence models, according to aspects of the present disclosure. The 700 operations can be performed by a device on which at least one generative artificial intelligence model that can act as both a preliminary model and a target model can be deployed, such as a smartphone, a tablet computer, a laptop computer, a desktop computer, a server, a cloud computing instance hosted in a distributed computing environment, or similar.

[0083] As illustrated, operations 700 begin in block 710, with the generation, based on an input prompt and a generative artificial intelligence model, of a first plurality of token sets. In some respects, the token sets may include token sequences that may be accepted, in part or in full, as candidates to respond to the input prompt.

[0084] In some respects, the first plurality of token sets can be represented as a tree data structure. Within this tree data structure, the input prompt can correspond to a root node. Each token set in the first plurality of token sets can be represented by a navigable path through the tree data structure (for example, as discussed above in relation to Figure 3).

[0085] In some respects, the depth of the tree data structure may correspond to a maximum number of tokens generated by a single pass through the generative artificial intelligence model. Petition 870250089725, dated 02 / 10 / 2025, pages 250 / 295 31 / 46

[0086] In some respects, the depth of the tree data structure may correspond to a maximum number of tokens generated by a single pass through the generative artificial intelligence model.

[0087] In some respects, the maximum size of the tree data structure can be defined based on a computational complexity metric associated with generating a set of tokens by the generative artificial intelligence model.

[0088] In block 720, operations 700 proceed with the speculative generation, using the generative artificial intelligence model, of a second plurality of token sets.

[0089] In block 730, operations 700 proceed with the selection, while the second plurality of token sets is speculatively generated from a set of tokens among the first plurality of token sets.

[0090] In some respects, selecting the token set from the first plurality of token sets may involve selecting a longer sequence of accepted tokens from the first plurality of token sets.

[0091] In some respects, token sets in the second plurality of token sets may include padding taking into account a number of tokens in the token set selected from the first plurality of token sets. For example, for a number of tokens N included in each token set from the first plurality of token sets, the selected number of tokens may include between 0 and N tokens. In this case, the padding included in the token sets in the second plurality of token sets may include no padding (corresponding to the acceptance of all N tokens in a token set from the first plurality of tokens), one padding token (corresponding to the acceptance of N - 1 tokens), two padding tokens (corresponding to the acceptance of N - 2 tokens), and so on. Petition 870250089725, dated 02 / 10 / 2025, pages 251 / 295 32 / 46

[0092] In some respects, to select the set of tokens from the first plurality of tokens, a first token can be rejected at a first level of the tree data structure. Based on the rejection of the first token, n fitted probability distributions can be generated, and the tree data structure can be pruned by discarding child tokens of the first token in the tree data structure. The child tokens of the first token can be discarded because the rejection of the first token generally implies that the child tokens of the first token cannot be valid responses to the input prompt. It can then be determined whether a second token should be accepted or rejected at the first level of the tree data structure based on the fitted probability distribution.

[0093] In some respects, selecting the token set from the first plurality of token sets may involve rejecting each token set in the first plurality of token sets generated by the generative artificial intelligence model. A token may be sampled using the generative artificial intelligence model based on a target distribution that excludes probabilities associated with each token set in the first plurality of token sets. The token set selected from the first plurality of token sets may include the sampled token.

[0094] In block 740, operations 700 proceed with the issuance of the token set selected from the first token plurality and an associated token set in the second token plurality as a response to the input prompt.

[0095] In some respects, the generative artificial intelligence model comprises a generative artificial intelligence model trained to generate multiple tokens in response to the input prompt based on prediction prompt embeddings.

[0096] In some respects, the generative artificial intelligence model comprises a model including one or more non-autoregressive layers and Petition 870250089725, dated 02 / 10 / 2025, pages 252 / 295 33 / 46 one or more autoregressive layers. In some respects, the one or more autoregressive layers may include one or more layers at the top of a layer stack representing the generative artificial intelligence model. In some respects, the one or more autoregressive layers may include one or more layers at the bottom of a layer stack representing the generative artificial intelligence model. Example processing systems for recursive speculative decoding in generative artificial intelligence models

[0097] Figure 8 depicts an example 800 processing system for generating a response to a query provided as input to a generative artificial intelligence model based on recursive speculative decoding, as described in the present invention, for example, in relation to Figure 4.

[0098] The 800 processing system includes a central processing unit (CPU) 802, which, in some examples, may be a multi-core CPU. Instructions executed on CPU 802 may be loaded, for example, from a program memory associated with CPU 802 or may be loaded from a memory partition (for example, from memory 824).

[0099] The 800 processing system also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 804, a digital signal processor (DSP) 806, a neural processing unit (NPU) 808, and a connectivity component 812.

[0100] An NPU, such as the NPU 808, is generally a specialized circuit configured to implement the control and arithmetic logic to execute machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), and the like. An NPU may sometimes be alternatively called a signal processor. Petition 870250089725, dated 02 / 10 / 2025, pages 253 / 295 34 / 46 neural (NSP - neural signal processor), a tensor processing unit (TPU - tensor processing unit), a neural network processor (NNP - neural network processor), an intelligence processing unit (IPU - intelligence processing unit), a vision processing unit (VPU - vision processing unit) or a graphics processing unit.

[0101] NPUs, such as the NPU 808, are configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, a plurality of NPUs can be instantiated on a single chip, as a system on a chip (SoC), while in other examples, such NPUs can be part of a dedicated neural network accelerator.

[0102] NPUs can be optimized for training or inference or, in some cases, configured to balance performance between the two. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0103] NPUs designed to accelerate training are generally configured to speed up the optimization of new models, which is a computationally intensive operation involving inputting data into an existing (often identified or tagged) dataset, iterating through the dataset, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on a wrong prediction involves backpropagation through the model layers and gradient determination to reduce the prediction error.

[0104] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs can therefore be configured to provide a new data fragment as input and quickly process this new fragment through an already trained model to generate a model output (e.g., an inference). Petition 870250089725, dated 02 / 10 / 2025, pages 254 / 295 35 / 46

[0105] In some implementations, the NPU 808 is part of one or more of the CPU 802, the GPU 804, and / or the DSP 806. These may be located in user equipment (UE), in a wireless communication system, or other computing device.

[0106] In some examples, the 812 connectivity component may include subcomponents, for example, for third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., long-term evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The 812 connectivity component may be additionally coupled to one or more 814 antennas.

[0107] The processing system 800 may also include one or more sensor processing units 816 associated with any type of sensor, one or more image signal processors (ISPs) 818 associated with any type of image sensor and / or a navigation processor 820, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0108] The 800 processing system may also include one or more 822 input and / or output devices, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones and the like.

[0109] In some instances, one or more of the processors in the 800 processing system may be based on an ARM or RISC-V instruction set.

[0110] The 800 processing system also includes 824 memory, which is representative of one or more static and / or dynamic memories, such as dynamic random access memory, static flash-based memory and Petition 870250089725, dated 02 / 10 / 2025, pages 255 / 295 36 / 46 related. In this example, memory 824 includes computer-executable components, which can be executed by one or more of the previously mentioned processors of the 800 processing system.

[0111] In particular, in this example, the memory 824 includes a token set receiving component 824A, a token selection component 824B, an output generation component 824C, and generative artificial intelligence models 824D. The depicted components, and others not depicted, can be configured to perform various aspects of the methods described in the present invention.

[0112] In general, the processing system 800 and / or its components can be configured to perform the methods described in the present invention. Example processing systems for self-speculative decoding in generative artificial intelligence models

[0113] Figure 9 depicts an example processing system 900 for generating a response to a query provided as input to a generative artificial intelligence model based on self-speculative decoding, as described in the present invention, for example, in relation to Figure 7.

[0114] The 900 processing system includes a central processing unit (CPU) 902, which, in some examples, may be a multi-core CPU. Instructions executed on CPU 902 may be loaded, for example, from a program memory associated with CPU 902 or may be loaded from a memory partition (for example, from memory 924).

[0115] The 900 processing system also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 904, a digital signal processor (DSP) 906, a neural processing unit (NPU) 908 and a connectivity component 912. Petition 870250089725, dated 02 / 10 / 2025, pages 256 / 295 37 / 46

[0116] An NPU, such as the NPU 908, is generally a specialized circuit configured to implement the control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), and the like. An NPU may sometimes be alternatively called a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graphics processing unit.

[0117] NPUs, such as the NPU 908, are configured to accelerate the performance of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, a plurality of NPUs can be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples such NPUs can be part of a dedicated neural network accelerator.

[0118] NPUs can be optimized for training or inference or, in some cases, configured to balance performance between the two. For NPUs capable of performing both training and inference, the two tasks can still generally be performed independently.

[0119] NPUs designed to accelerate training are generally configured to speed up the optimization of new models, which is a computationally intensive operation involving inputting data into an existing (often identified or tagged) dataset, iterating through the dataset, and then adjusting model parameters such as weights and biases to improve model performance. Generally, optimization based on a wrong prediction involves backpropagation through the model layers and gradient determination to reduce the prediction error.

[0120] NPUs designed to accelerate inference are generally configured to operate on complete models. Such NPUs can therefore be Petition 870250089725, dated 02 / 10 / 2025, pages 257 / 295 38 / 46 configured to provide a new data fragment as input and quickly process this new fragment through a pre-trained model to generate a model output (e.g., an inference).

[0121] In some implementations, the NPU 908 is part of one or more of the CPU 902, the GPU 904, and / or the DSP 906. These may be located in a user appliance (UE) in a wireless communication system or other computing device.

[0122] In some examples, the 912 connectivity component may include subcomponents, for example, for third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., LTE), fifth-generation (5G) connectivity (e.g., NR), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The 912 connectivity component may be additionally coupled to one or more 914 antennas.

[0123] The processing system 900 may also include one or more sensor processing units 916 associated with any type of sensor, one or more image signal processors (ISPs) 918 associated with any type of image sensor and / or a navigation processor 920, which may include satellite-based positioning system components (e.g., GPS or GLONASS) as well as inertial positioning system components.

[0124] The processing system 900 may also include one or more input and / or output devices 922, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones and the like.

[0125] In some instances, one or more of the processors in the 900 processing system may be based on an ARM or RISC-V instruction set.

[0126] The 900 processing system also includes 924 memory, which is representative of one or more static and / or dynamic memories, such as a Petition 870250089725, dated 02 / 10 / 2025, pages 258 / 295 39 / 46 dynamic random access memory, a static memory based on flash and the like. In this example, memory 924 includes computer executable components, which can be executed by one or more of the previously mentioned processors of the 900 processing system.

[0127] In particular, in this example, memory 924 includes a token set generation component 924A, a token selection component 924B, an output generation component 924C, and a generative artificial intelligence model 924D. The depicted components, and others not depicted, can be configured to perform various aspects of the methods described in the present invention.

[0128] In general, the 900 processing system and / or its components can be configured to perform the methods described in the present invention. Example clauses

[0129] Implementation details of various aspects of this disclosure are described in the following numbered clauses.

[0130] Clause 1: A processor-implemented method comprising: receiving a plurality of token sets generated based on an input prompt and a first generative artificial intelligence model, wherein each token set in the plurality of token sets corresponds to a candidate response to the input prompt; selecting, using a second generative artificial intelligence model and recursive fitting of a target distribution associated with the received plurality of token sets, a token set from among the plurality of token sets; and issuing the selected token set as a response to the input prompt.

[0131] Clause 2: The method of clause 1, in which the plurality of token sets is organized into a tree data structure.

[0132] Clause 3: The method of clause 2, where the input prompt corresponds to a root node of the tree data structure, and each path through Petition 870250089725, dated 02 / 10 / 2025, pp. 259 / 295 40 / 46 of the tree data structure corresponds to a different candidate answer to the input prompt.

[0133] Clause 4: The method of clause 2 or 3, wherein a number of tokens at a level in the tree data structure is based on a branching factor associated with a level immediately preceding that level in the tree data structure.

[0134] Clause 5: The method of any of clauses 2 to 4, where a depth of the tree data structure corresponds to a maximum number of tokens generated by a single pass through the first generative artificial intelligence model.

[0135] Clause 6: The method of any of clauses 1 to 5, wherein the size of each token set is based on a computational complexity metric associated with the generation of a target token set by the second generative artificial intelligence model.

[0136] Clause 7: The method of any of clauses 1 to 6, wherein the recursive adjustment of the target distribution comprises: determining whether to accept or reject a first token in a set of tokens from among a plurality of sets of tokens; and adjusting a probability distribution used to check a second token in the set of tokens subsequent to the first token based on the determination of whether to accept or reject the first token.

[0137] Clause 8: The method of clause 7, wherein the adjustment of the probability distribution comprises subtracting a probability value associated with the first token from the probability distribution based on the determination to reject the first token.

[0138] Clause 9: The method of any of clauses 1 to 8, wherein the selection of the token set from the plurality of token sets comprises: rejecting a first token at a first level of a tree data structure representing the plurality of token sets; generating a probability distribution fitted based on the rejection of the first token; Petition 870250089725, dated 02 / 10 / 2025, pages 260 / 295 41 / 46 Discard, from the tree data structure, child tokens of the first token at levels deeper than the first level of the tree data structure; and determine whether to accept or reject a second token at the first level of the tree data structure based on the fitted probability distribution.

[0139] Clause 10: The method of any of clauses 1 to 9, wherein the selection of the token set from the plurality of token sets comprises: rejecting each token set generated by the first generative artificial intelligence model; and obtaining, using the second generative artificial intelligence model, a sample of a token based on a target distribution that excludes probabilities associated with each token set generated by the first generative artificial intelligence model, wherein the selected token set comprises the sampled token.

[0140] Clause 11: The method of any of clauses 1 to 10, wherein: the first generative artificial intelligence model corresponds to a preliminary model in a speculative decoding pipeline, and the second generative artificial intelligence model corresponds to a target model in the speculative decoding pipeline.

[0141] Clause 12: A processor-implemented method comprising: generating, based on an input prompt and a generative artificial intelligence model, a first plurality of token sets, wherein each token set in the first plurality of token sets corresponds to a first portion of a candidate response to the input prompt; speculatively generating, using the generative artificial intelligence model, a second plurality of token sets, wherein each token set in the second plurality of token sets corresponds to a second portion of the candidate response to the input prompt based on the first plurality of token sets; while speculatively generating the second plurality of token sets, selecting a token set from the first plurality of token sets; and emitting the selected token set from the first plurality of token sets. Petition 870250089725, dated 02 / 10 / 2025, pp. 261 / 295 42 / 46 token plurality and a set of tokens associated with the second token plurality as a response to the input prompt.

[0142] Clause 13: The method of clause 12, wherein selecting the token set from the first plurality of token sets comprises selecting a longer sequence of accepted tokens from the first plurality of token sets.

[0143] Clause 14: The method of clause 12 or 13, wherein token sets in the second plurality of token sets include padding taking into account a number of tokens in the token set selected from among the first plurality of token sets.

[0144] Clause 15: The method of any of clauses 12 to 14, where the first plurality of token sets is represented as a tree data structure and a root node of the tree data structure corresponds to the input prompt.

[0145] Clause 16: The method of clause 15, where a depth of the tree data structure corresponds to a maximum number of tokens generated by a single pass through the generative artificial intelligence model.

[0146] Clause 17: The method of clause 15 or 16, in which a maximum size of the tree data structure is defined based on a computational complexity metric associated with the generation of a set of tokens by the generative artificial intelligence model.

[0147] Clause 18: The method of any of clauses 15 to 17, wherein the selection of the token set from the first plurality of token sets comprises: rejecting a first token at a first level of the tree data structure representing the first plurality of token sets; generating a fitted probability distribution based on the rejection of the first token; discarding child tokens of the first token from the tree data structure; and determining whether to accept or reject a second token at the first level of the tree data structure based on the fitted probability distribution. Petition 870250089725, dated 02 / 10 / 2025, pages 262 / 295 43 / 46

[0148] Clause 19: The method of any of clauses 12 to 18, wherein the selection of the token set from the first plurality of token sets comprises: rejecting each token set in the first plurality of token sets generated by the generative artificial intelligence model; and obtaining, using the generative artificial intelligence model, a sample of a token based on a target distribution that excludes probabilities associated with each token set in the first plurality of token sets, wherein the token set selected from the first plurality of token sets comprises the sampled token.

[0149] Clause 20: The method of any of clauses 12 to 19, wherein the generative artificial intelligence model comprises a generative artificial intelligence model trained to generate multiple tokens in response to the input prompt based on prediction prompt embeddings.

[0150] Clause 21: The method of any of clauses 12 to 20, wherein the generative artificial intelligence model comprises a model including one or more non-autoregressive layers and one or more autoregressive layers.

[0151] Clause 22: The method of clause 21, wherein the one or more autoregressive layers comprise one or more layers on top of a stack of layers representing the generative artificial intelligence model.

[0152] Clause 23: The method of clause 21 or 22, wherein the one or more autoregressive layers comprise one or more layers in a background of a stack of layers representing the generative artificial intelligence model.

[0153] Clause 24: A processing system comprising: at least one memory having executable instructions stored therein; and one or more processors configured to execute the executable instructions in order to make the processing system perform the operations of any of clauses 1 to 23.

[0154] Clause 25: A processing system, comprising: means for carrying out the operations of any of clauses 1 to 23. Petition 870250089725, dated 02 / 10 / 2025, pages 263 / 295 44 / 46

[0155] Clause 26: A non-transient, computer-readable medium containing instructions which, when executed by one or more processors, perform the operations of any of clauses 1 to 23. Additional considerations

[0156] The preceding description is provided to enable any person skilled in the art to practice the various aspects described in the present invention. The examples discussed in the present invention are not limiting to the scope, applicability, or aspects set forth in the claims. Various modifications of these aspects will be readily apparent to those skilled in the art, and the generic principles defined in the present invention may be applied to other aspects. For example, changes may be made to the function and arrangement of the elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the methods described may be performed in a different order than that described, and various steps may be added, omitted, or combined. Furthermore, attributes described in relation to some examples may be combined in some other examples.For example, an apparatus may be implemented, or a method may be practiced, using any number of the aspects set forth in the present invention. Furthermore, the scope of the disclosure is intended to cover such an apparatus or method practiced with the use of another structure, functionality, or structure and functionality in addition to or different from the various aspects of the disclosure set forth in the present invention. It should be understood that any aspect of the disclosure disclosed in this invention may be incorporated by one or more elements of a claim.

[0157] As used in the present invention, the term example means that it serves as an example, instance, or illustration. Any aspect described in the present invention as exemplary should not necessarily be interpreted as preferential or advantageous in relation to other aspects. Petition 870250089725, dated 02 / 10 / 2025, pp. 264 / 295 45 / 46

[0158] As used in the present invention, an expression referring to at least one of a list of items refers to any combination of those items, including single members. For example, at least one of: a, b, or c is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (for example, aa, aaa, aab, aac, abb, a-cc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0159] As used in the present invention, the term determine encompasses a wide variety of actions. For example, determine may include calculate, compute, process, derive, investigate, search (e.g., search in a table, a database, or other data structure), verify, and the like. In addition, determine may include receive (e.g., receive information), access (e.g., access data in a memory), and the like. Furthermore, determine may include solve, select, choose, establish, and the like.

[0160] The methods disclosed in the present invention comprise one or more steps or actions to perform the methods. The steps and / or actions of the method can be interchanged without departing from the scope of the claims. In other words, unless a particular order of steps or actions is specified, the order and / or use of specific steps and / or actions can be modified without departing from the scope of the claims. Additionally, the various operations of the methods described above can be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including, but not limited to, a circuit, an application-specific integrated circuit (ASIC), or a processor.In general, when operations are illustrated in the figures, these operations may have equivalent components in a more functional way, corresponding to similar numbering.

[0161] The following claims are not intended to be limited to the aspects shown in the present invention, but shall have the full scope consistent Petition 870250089725, dated 02 / 10 / 2025, pages 265 / 295 46 / 46 with the language of the claims. In a claim, reference to an element in the singular is not intended to mean one and only one, unless specifically stated otherwise, but instead means one or more. Unless specifically stated otherwise, the term any refers to one or more. No element of a claim shall be interpreted in accordance with the provisions of Title 35 of the USC Code § 112(f), unless the element is expressly mentioned using the expression means to or, in the case of a method claim, the element is mentioned using the expression step to. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or may become known to those skilled in the art are expressly incorporated into the present invention by way of reference and are intended to be encompassed by the claims.Furthermore, nothing disclosed in this invention is intended to be exclusive to the public, regardless of whether such disclosure is explicitly mentioned in the claims. Petition 870250089725, dated 02 / 10 / 2025, pp. 266 / 295

Claims

1 / 9 CLAIMS 1. A processing system characterized by comprising: at least one memory having executable instructions stored therein; and one or more processors configured to execute the executable instructions in order to make the processing system: generate, based on an input prompt and a generative artificial intelligence model, a first plurality of token sets, wherein each token set in the first plurality of token sets corresponds to a first portion of a candidate response to the input prompt; speculatively generate, using the generative artificial intelligence model, a second plurality of token sets, wherein each token set in the second plurality of token sets corresponds to a second portion of the candidate response to the input prompt based on the first plurality of token sets;While speculatively generating the second plurality of token sets, select a token set from the first plurality of token sets; and emit the selected token set from the first plurality of tokens and an associated token set from the second plurality of tokens as a response to the input prompt.

2. Processing system, according to claim 1, characterized in that, to select the set of tokens from the first plurality of sets of tokens, one or more processors are configured to make the processing system select a longer sequence of accepted tokens from the first plurality of sets of tokens.

3. Processing system, according to claim 1, characterized in that token sets in the second plurality of token sets include padding taking into account a number of tokens in the token set selected from the first plurality of token sets.

4. Processing system, according to claim 1, characterized in that the first plurality of token sets is represented as a tree data structure, and in which a root node of the tree data structure corresponds to the input prompt.

5. Processing system, according to claim 4, characterized in that the depth of the tree data structure corresponds to a maximum number of tokens generated by a single pass through the generative artificial intelligence model.

6. Processing system, according to claim 4, characterized in that a maximum size of the tree data structure is defined based on a computational complexity metric associated with the generation of a set of tokens by the generative artificial intelligence model.

7. Processing system, according to claim 4, characterized in that, to select the token set from the first plurality of token sets, the one or more processors are configured to make the processing system: reject a first token at a first level of the tree data structure representing the first plurality of token sets; generate a fitted probability distribution based on the rejection of the first token; discard or ignore child tokens of the first token in the tree data structure; and determine whether to accept or reject a second token at the first level of the tree data structure based on the fitted probability distribution.

8. Processing system, according to claim 1, characterized in that, to select the set of tokens from the first plurality of token sets, one or more processors are configured to make the processing system: reject each set of tokens in the first plurality of token sets generated by the generative artificial intelligence model; and obtain, using the generative artificial intelligence model, a sample of a token based on a target distribution that excludes probabilities associated with each set of tokens in the first plurality of token sets, wherein the set of tokens selected from the first plurality of token sets comprises the sampled token.

9. Processing system, according to claim 1, characterized in that the generative artificial intelligence model comprises a generative artificial intelligence model trained to generate multiple tokens in response to the input prompt based on prediction prompt embeddings.

10. Processing system, according to claim 1, characterized in that the generative artificial intelligence model comprises a model including one or more non-autoregressive layers and one or more autoregressive layers.

11. Processing system, according to claim 10, characterized in that one or more autoregressive layers comprise one or more layers on top of a stack of layers representing the generative artificial intelligence model.

12. Processing system, according to claim 10, characterized in that one or more autoregressive layers comprise one or more layers in a background of a stack of layers representing the generative artificial intelligence model.

13. Method implemented by a processor characterized by understanding: generating, based on an input prompt and a generative artificial intelligence model, a first plurality of token sets, in which each Petition 870250089725, of 02 / 10 / 2025, page.289 / 295 4 / 9 The set of tokens in the first plurality of token sets corresponds to a first portion of a candidate response to the input prompt; speculatively generate, using the generative artificial intelligence model, a second plurality of token sets, where each set of tokens in the second plurality of token sets corresponds to a second portion of the candidate response to the input prompt based on the first plurality of token sets; while speculatively generating the second plurality of token sets, select a set of tokens from the first plurality of token sets; and emit the selected set of tokens from the first plurality of tokens and an associated set of tokens in the second plurality of tokens as a response to the input prompt.

14. Method according to claim 13, characterized in that the selection of the token set from the first plurality of token sets comprises selecting a longer sequence of accepted tokens from the first plurality of token sets.

15. A method according to claim 13, characterized in that sets of tokens in the second plurality of sets of tokens include padding taking into account a number of tokens in the set of tokens selected from the first plurality of sets of tokens.

16. A method according to claim 13, characterized in that the first plurality of token sets is represented as a tree data structure, and wherein a root node of the tree data structure corresponds to the input prompt.

17. Method, according to claim 16, characterized by a depth of the tree data structure corresponding to a maximum number of tokens generated by a single pass through the generative artificial intelligence model. Petition 870250089725, dated 10 / 02 / 2025, pp. 290 / 295 5 / 9 18. Method, according to claim 16, characterized by a maximum size of the tree data structure being defined based on a computational complexity metric associated with the generation of a set of tokens by the generative artificial intelligence model.

19. A method according to claim 16, characterized in that the selection of the token set from the first plurality of token sets comprises: rejecting a first token at a first level of the tree data structure representing the first plurality of token sets; generating a fitted probability distribution based on the rejection of the first token; discarding or ignoring child tokens of the first token in the tree data structure; and determining whether to accept or reject a second token at the first level of the tree data structure based on the fitted probability distribution.

20. A method according to claim 13, characterized in that the selection of the token set from the first plurality of token sets comprises: rejecting each token set in the first plurality of token sets generated by the generative artificial intelligence model; and obtaining, using the generative artificial intelligence model, a sample of a token based on a target distribution that excludes probabilities associated with each token set in the first plurality of token sets, wherein the token set selected from the first plurality of token sets comprises the sampled token.

21. Method, according to claim 13, characterized in that the generative artificial intelligence model comprises a generative artificial intelligence model trained to generate multiple tokens in response to the input prompt based on prediction prompt embeddings. Petition 870250089725, dated 10 / 02 / 2025, pp. 291 / 295 6 / 9 22. Method, according to claim 13, characterized in that the generative artificial intelligence model comprises a model including one or more non-autoregressive layers and one or more autoregressive layers.

23. Method, according to claim 22, characterized in that the one or more autoregressive layers comprise one or more layers on top of a stack of layers representing the generative artificial intelligence model.

24. Method, according to claim 22, characterized in that one or more autoregressive layers comprise one or more layers in a background of a stack of layers representing the generative artificial intelligence model.

25. A processing system characterized by comprising: means for generating, based on an input prompt and a generative artificial intelligence model, a first plurality of token sets, wherein each token set in the first plurality of token sets corresponds to a first portion of a candidate response to the input prompt; means for speculatively generating, using the generative artificial intelligence model, a second plurality of token sets, wherein each token set in the second plurality of token sets corresponds to a second portion of the candidate response to the input prompt based on the first plurality of token sets; means for, while speculatively generating the second plurality of token sets, selecting a token set from among the first plurality of token sets;and means for issuing the selected token set from the first token plurality and an associated token set in the second token plurality as a response to the input prompt. Petition 870250089725, dated 10 / 02 / 2025, pp. 292 / 295 7 / 9; 26. Processing system according to claim 25, characterized in that the means for selecting the set of tokens from the first plurality of sets of tokens comprise means for selecting a longer sequence of accepted tokens from the first plurality of sets of tokens.

27. Processing system, according to claim 25, characterized in that token sets in the second plurality of token sets include padding taking into account a number of tokens in the token set selected from the first plurality of token sets.

28. Processing system, according to claim 25, characterized in that the first plurality of token sets is represented as a tree data structure, and in which a root node of the tree data structure corresponds to the input prompt.

29. Processing system, according to claim 28, characterized in that the depth of the tree data structure corresponds to a maximum number of tokens generated by a single pass through the generative artificial intelligence model.

30. Processing system, according to claim 28, characterized in that a maximum size of the tree data structure is defined based on a computational complexity metric associated with the generation of a set of tokens by the generative artificial intelligence model.

31. Processing system, according to claim 28, characterized in that the means for selecting the token set from among the first plurality of token sets comprise: means for rejecting a first token at a first level of the tree data structure representing the first plurality of token sets; means for generating an adjusted probability distribution based on the rejection of the first token; Petition 870250089725, dated 10 / 02 / 2025, pp. 293 / 295 8 / 9 means for discarding or ignoring child tokens of the first token in the tree data structure; and means for determining whether to accept or reject a second token at the first level of the tree data structure based on the adjusted probability distribution.

32. Processing system, according to claim 25, characterized in that the means for selecting the set of tokens from the first plurality of sets of tokens comprise: means for rejecting each set of tokens in the first plurality of sets of tokens generated by the generative artificial intelligence model; and means for obtaining, using the generative artificial intelligence model, a sample of a token based on a target distribution that excludes probabilities associated with each set of tokens in the first plurality of sets of tokens, wherein the set of tokens selected from the first plurality of sets of tokens comprises the sampled token.

33. Processing system, according to claim 25, characterized in that the generative artificial intelligence model comprises a generative artificial intelligence model trained to generate multiple tokens in response to the input prompt based on prediction prompt embeddings.

34. Processing system, according to claim 25, characterized in that the generative artificial intelligence model comprises a model including one or more non-autoregressive layers and one or more autoregressive layers.

35. Processing system, according to claim 34, characterized in that one or more autoregressive layers comprise one or more layers on top of a stack of layers representing the generative artificial intelligence model.

36. Processing system, according to claim 34, characterized by one or more autoregressive layers comprising one or more layers in a background of a stack of layers representing the generative artificial intelligence model.

37. Non-transient, computer-readable medium characterized by having executable instructions stored within it that, when executed by one or more processors, perform an operation comprising: generating, based on an input prompt and a generative artificial intelligence model, a first plurality of token sets, wherein each token set in the first plurality of token sets corresponds to a first portion of a candidate response to the input prompt; speculatively generating, using the generative artificial intelligence model, a second plurality of token sets, wherein each token set in the second plurality of token sets corresponds to a second portion of the candidate response to the input prompt based on the first plurality of token sets;While speculatively generating the second plurality of token sets, select a token set from the first plurality of token sets; and emit the selected token set from the first plurality of tokens and an associated token set from the second plurality of tokens as a response to the input prompt.

38. Product, process, system, kit, means or use, characterized by comprising one or more elements described in the descriptive report, claims, drawings, sequence listing, or summary of this application, when applicable. Petition 870250089725, dated 02 / 10 / 2025, p. 295 / 295

Images