Adaptive length speculative decoding in autoregressive generative artificial intelligence models

Abstract

Certain aspects of the present disclosure provide techniques and apparatuses for generating responses to query inputs in generative artificial intelligence models using variable draft lengths. An example method generally includes determining (e.g., measuring or accessing) one or more operational properties of a device on which an inference operation using a machine learning model is performed. A first draft set of tokens is generated using the machine learning model. A number of tokens included in the first draft set of tokens is based on the one or more operational properties of the device and a defined scheduling function of the machine learning model. The first draft set of tokens is output for validation.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 423,840, filed January 26, 2024, which is incorporated herein by reference. Background Technology

[0003] This disclosure relates to generative artificial intelligence models, and more specifically to speculative decoding in generative artificial intelligence models.

[0004] Generative AI models can be used in a variety of environments to generate responses to input prompts (also known as queries or inputs). For example, generative AI models can be used in chatbot applications that use large language models (LLMs) to generate answers or at least responses to input prompts. Other examples where generative AI models can be used include latent diffusion models, in which the model generates an image from an input text description of the content of the desired image; decision transformers, in which future actions are predicted based on a sequence of previous actions within a given environment; and so on.

[0005] Typically, generating responses to queries using generative AI models can be computationally expensive. For example, in a chatbot deployment where a large language model is used to generate responses to queries formatted as text, the response can be generated by passing through the large language model for each lexical unit (e.g., a word or part of a word) generated as part of the response. The output of each pass can be a probability distribution over a set of lexical units (e.g., words or parts of words), from which the next lexical unit (e.g., a word or part of a word) can be selected, for example, by sampling or based on maximum likelihood. Because passes through the large language model are used to generate each word (or (multiple) lexical units) in the response to the query, the computational overhead can be modeled as the product of the number of words included in the response and the computational resource overhead of performing the passes through the large language model (e.g., in terms of processing power, memory bandwidth, and / or other computational resources used), which typically increases with the number of parameters within the large language model. Summary of the Invention

[0006] Certain aspects of this disclosure provide a method for generating responses to input prompts using a generative artificial intelligence model. The method typically includes determining (e.g., measuring or otherwise accessing) one or more first operational attributes of a device on which inference operations using a machine learning model are performed. A first draft set of lexical units is generated using the machine learning model. The number of lexical units included in the first draft set is based on the measured one or more first operational attributes of the device and a scheduling function defined by the machine learning model. The first draft set of lexical units is output for verification.

[0007] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0008] The following description and related figures illustrate certain exemplary features of one or more aspects. Attached Figure Description

[0009] The accompanying drawings illustrate only certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0010] FIG. 1 This is an illustration of an operation performed according to various aspects of this disclosure to generate a response to an input query using an adaptive draft length of operating parameters of a generative artificial intelligence model and a device operating on the generative artificial intelligence model.

[0011] FIG. 2A and FIG. 2B Pseudocode is illustrated for adapting the length of a draft set of lexical units generated using a generative artificial intelligence model to the operating parameters of a device operating on a generative artificial intelligence model, according to various aspects of this disclosure.

[0012] FIG. 3 Examples of performance metrics for constant draft length and adaptive draft length speculative decoding according to various aspects of this disclosure are illustrated.

[0013] FIG. 4Example operations are illustrated for generating responses to input prompts based on adaptive draft lengths of operating parameters of a device using a generative artificial intelligence model and operating thereon, according to various aspects of this disclosure.

[0014] FIG. 5 Example processing systems configured to perform various aspects of this disclosure are described.

[0015] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0016] This disclosure provides apparatus, methods, processing systems, and computer-readable media for efficiently generating responses to input queries using generative artificial intelligence models.

[0017] Typically, generative AI models generate responses to queries input into the model. For example, a large language model (LLM) deployed within a chatbot can generate a response to a query using multiple passes through the LLM, where each successive pass is based on the query (which can be lexicalized for processing) and uses lexical units (e.g., words or parts of words) generated through previous passes through the LLM. These LLMs can typically include a large number (e.g., billions or even trillions) of weights or parameters within the model. Due to the size of these models and the operations performed on each lexical unit to predict what the next lexical unit should be in response to the query and previously generated lexical units, deploying large language models on a variety of devices with limited memory and / or processing power relative to the cloud computing instances on which the large language models typically operate. Furthermore, in some cases, the memory bandwidth involved in generating responses to queries provided as input to the model may prevent computational resources from being used for other tasks.

[0018] To improve the efficiency and throughput of large language models, speculative decoding techniques allow a smaller language model (sometimes called a draft large language model, or a draft model or approximate model) to operate alongside a larger language model (sometimes called a target large language model, or target model) (e.g., sequentially or in parallel). In such a case, the draft model can speculatively generate additional lexical units and probabilities for sampling these additional lexical units sequentially based on the current set of accepted lexical units. The target model can then generate lexical units based on those generated by the draft model. To produce the final result, the target model can perform rejection sampling on a per-lexical basis, accepting or rejecting individual lexical units generated by the draft model, such that the draft model and the target model have similar probability distributions.

[0019] In some respects, the draft model can be a pruned version of the target model, chosen to have a similar probability distribution to the target model. In other respects, the draft model can be a smaller version of the target model (e.g., trained on millions of lexical units instead of hundreds of millions or even billions of lexical units).

[0020] Certain aspects of this disclosure provide techniques and apparatus for generating responses to queries input into a generative artificial intelligence model, such as a large language model, using speculative decoding with an adaptive draft lexical length parameter. According to various aspects, the draft model can use a draft length to generate a set of lexical units as candidate responses to the query, the draft length being determined based on a defined scheduling function of the draft model and one or more operational properties of the device performing inference operations thereon (e.g., an edge device, such as a user equipment (UE), or another device on which a set of lexical units is speculatively generated and output to a target model for verification). Subsequent rounds of inference may involve adjusting the number of lexical units included in the draft's lexical set (also referred to as the lexical length). These adjustments can be performed to maximize or at least increase the rate at which lexical units are generated as responses to the input query. Simultaneously, the likelihood of reaching various operational constraints that might reduce the rate of lexical unit generation can be minimized or at least reduced (e.g., reaching a temperature threshold, frequency threshold, etc., at which processor performance degrades in an attempt to reduce the processor temperature to at or below a defined maximum operating temperature). By doing so, aspects of this disclosure can allow for variable (e.g., increased) lexical generation rates, which in turn allow for faster completion of inference tasks performed by generative AI models, reduced power consumption, and minimize or at least reduce the likelihood that the device will enter a lower-performance operating mode during inference operations using generative AI models (or other machine learning models that support variable-sized outputs).

[0021] Speculative decoding in generative artificial intelligence models

[0022] Typically, autoregressive lexical generation (e.g., in large language models) takes historical lexical units as input to generate output. That is, autoregressive lexical generation can be represented by the following expression:

[0023] in Indicates time t The generated lexical sequence has pairs of lexical units. to The conditional probability of choosing as a condition. ,and Indicates time t +1 generates a sequence of lexical units, which has the following properties: lexical units to The conditional probability of choosing as a condition. Typically, each execution of an autoregressive model generates a single lexical unit, meaning that execution is possible. N An inference to generate N A sequence of lexical units. As discussed above, speculative decoding techniques can be used to accelerate lexical generation by using a draft model that speculatively generates lexical units faster than the target model and is smaller in size, where the target model is used to validate the lexical units generated (speculously) by the draft model.

[0024] In a speculative decoding pipeline, a draft model can be speculatively autoregressively generated based on the following expression. n Each word element: , …,

[0025] in Corresponding to a point in time, Corresponding to pairs of lexical elements to The choice is conditional and in time Selected lexical units The associated conditional probability distribution, and Indicates the time of the draft model speculatively generated lexical units .

[0026] The target model uses the generated n 100 tokens, and process them in parallel according to the following expression. n 1 word element to generate n The probability distribution of each word in the lexical units:

[0027] in k Corresponding to the generatedn The lexical index of each lexical element, and Corresponding to the target model in time This refers to the lexical units generated by the draft model. The generated probability distribution.

[0028] Then, the target model can validate the lexics generated by the draft model by comparing the distributions from the draft model and the target model to determine whether a lexicographical element is accepted or rejected. For some functions and a certain threshold (Also known as acceptance rate), given a word It is possible If accepted, then the term can be rejected. Otherwise, it can be based on a certain function. First rejection position or last position n The final word element is generated at the specified location.

[0029] With an acceptance rate In this case, inference decoding can lead to a cost reduction compared to iteratively generating lexical units using a single autoregressive model. The cost savings from inference compared to iterative lexical generation can be expressed by the following expression:

[0030] in Corresponding to the number of lexical units, Corresponding to the acceptance rate used The computational cost Corresponding to the computational cost of generating a lexical set using the target model, The computational cost corresponding to generating a lexical set using a draft model, Corresponding to the computational cost of speculatively generating a lexical set using a draft model, and n This corresponds to the number of lexical units speculatively generated in a single pass through the autoregressive model. Consider where... An example. In such an example, speculative decoding can result in a 35% reduction in computational overhead compared to standalone autoregressive iterative lexical generation.

[0031] However, as discussed, per-word speculative decoding can impose a limitation on the rate of word generation, since the first word can be sampled individually by the draft model and then validated by the target model, and then the next word can be sampled by the draft model and validated by the target model. In other words, generating a response to an input prompt using per-word speculative decoding can involve executing both the draft and target models for each word generated as part of the response to the input prompt, which can consume significant computational resources (e.g., processor time, memory, memory bandwidth, etc.) to generate the response.

[0032] Example speculative decoding in generative artificial intelligence models with variable draft length

[0033] Typically, generative AI models that use speculative decoding techniques (discussed above) to generate responses to input queries can involve generative... The draft model of each draft word and in The target model is executed on each lexical unit. Lexical generation rate of speculative decoding techniques. This may typically exceed the lexical generation rate of word-based autoregressive decoding. This correspondingly increases computational costs. For example, if each word is... If the computational cost is generated, then word meta-autoregressive decoding can achieve a speed of [per second]. The total computational cost. Meanwhile, the computational cost of the target model can be [calculated per second]. Its computational cost may be significantly greater than that of word meta-autoregressive decoding.

[0034] Although speculative decoding can allow for an increased lexical generation rate However, the increased computational cost involved in speculative decoding typically utilizes additional computing resources (e.g., additional processor cycles, memory, etc.). The use of these additional computing resources often results in the processors on which the generative AI model executes generating significant amounts of heat. Because these processors are typically designed to operate at or below a defined maximum temperature, the heat generation over time can trigger various actions by the processor to control its temperature. For example, various throttling techniques can be used to limit the amount of power that the processor (and / or other components) can draw, and thus limit the frequency at which the processor (and / or other components) operates (e.g., by reducing clock speed, voltage, etc.). Such limitations on processor frequency can, in turn, reduce the rate at which tokens are generated, and therefore negatively impact the inference speed of the generative AI model.

[0035] To improve the inference speed of generative AI models (e.g., measured by the number of inferences generated within a defined time period), aspects of this disclosure allow for adaptation of the length of the lexical draft set speculatively generated by the generative AI model for validation by the target model. In some aspects, the length of the lexical draft set can vary based on various factors, including (but not limited to) a defined scheduling function designed for the processor on which the generative AI model executes, the operating parameters of the device on which the generative AI model executes, the acceptance rate of the lexical speculatively generated by the generative AI model, etc.

[0036] FIG. 1This is an illustration of an inference operation 100 performed according to various aspects of this disclosure to generate a response to an input query using an adaptive draft length of one or more operating parameters (e.g., measured parameters) of a generative artificial intelligence model and a device operating on the generative artificial intelligence model thereon.

[0037] As illustrated, the use of a generative artificial intelligence model to perform inference operations 100 can begin at box 110, where model execution is initialized. In some aspects, at box 110, input queries received from the user of the device can be fed into the generative artificial intelligence model for processing based on a baseline set of parameters. These parameters may include, for example, the default (or initial) length of the draft set of lexical units, the length of the draft set of lexical units set according to a scheduling function designed for the processor on which the generative artificial intelligence model executes. For example, the scheduling function may vary based on the processor's computing power (e.g., floating-point operations per second (FLOPS), the number of processing cores, the frequency of processor operations, instruction retirement statistics, etc.). In some aspects, the scheduling function may vary based on whether each processor core on which the generative artificial intelligence model executes is a high-performance core or a high-efficiency core (e.g., in a heterogeneous architecture, such as the big.LITTLE architecture used in ARM processors). Typically, the scheduling function may be configured to assume that performance degrades over time when additional operations are performed on the processor to account for hot generation or other properties that may cause the processor to enter a lower performance state. Larger draft word lengths can be set for processors with broader processing capabilities, such as dedicated neural processing units (NPUs), processors supporting large parallel workloads (e.g., graphics processing units (GPUs) with many processing units, each capable of executing a portion of the workload in parallel), the ability to perform a large number of FLOPS, and higher maximum frequencies. Conversely, smaller draft word lengths can be set for processors with less broad processing capabilities (e.g., processors that can support fewer parallel workloads, processors capable of performing fewer FLOPS, and lower maximum frequencies).

[0038] After the initialization model is executed, inference operation 100 proceeds to box 120, where a generative artificial intelligence model is executed to generate a draft set of lexical units.

[0039] Typically, during the initial rounds when a generative AI model generates a draft set of lexical terms for an input query, a previous set of draft lexical terms has not yet been generated in response to the input query. In some aspects, the parameters on which the draft set of lexical terms is based may include a scheduling function and the length of the draft set of lexical terms set at box 110.

[0040] In some respects, historical token acceptance rates from previous queries can be further used to determine the length of the draft set of tokens generated at box 120. For example, when determining the length of the draft set of tokens generated at box 120, token acceptance rates from previous input queries with similar intent can be considered. For similar queries with high token acceptance rates, it can be assumed that the generative AI model will generate tokens for the input query with similar accuracy, and therefore, the length of the draft set of tokens can remain constant relative to the baseline set of parameters, or even be increased. On the other hand, if the historical token acceptance rate of similar queries meets or exceeds a threshold, the length of the draft set of tokens can be increased over time. Typically, the length of the initial draft set of tokens can be based on the increase and decrease of historical token acceptance rates over time, such that the length of the initial draft achieves the target performance level.

[0041] In some respects, the amount by which the length of the draft set of lexical units increases can be based on the threshold acceptance rate or the increment between the actual acceptance rate and the threshold acceptance rate for a given length of the draft set of lexical units. For example, a smaller increase in the length of the draft set of lexical units can be defined for a smaller increment between the threshold acceptance rate and the actual acceptance rate, compared to a larger increment between the threshold acceptance rate and the actual acceptance rate. As an illustrative example, consider a threshold acceptance rate of 50%. If... n If the actual acceptance rate for the draft length of a word is 60%, then the length of the word draft set can be increased. x Each word element. At the same time, if n If the actual acceptance rate for the draft length of a word element is 70%, then the length of the word element draft set can be increased. 2 x Each word element.

[0042] However, if the generative AI model has historically generated draft sets of lexics with low acceptance rates for similar queries, it can be assumed that the generative AI model will generate a similarly low-acceptance set of lexics for the input query. Therefore, in some respects, the length of the draft set of lexics can be reduced relative to the baseline set of parameters. The amount by which the length of the draft set of lexics is reduced can be based on similar techniques discussed above regarding the length of the draft set relative to increasing the number of lexics.

[0043] The draft set of lexical units generated at box 120 can be output to the target model for validation, and at box 130, the generative model receives validation information of the draft set of lexical units from the target model. In some respects, the target model can be a larger generative AI model than the draft model (e.g., including more parameters, trained on a larger corpus, etc.) and can be used to determine whether the draft set of lexical units speculatively generated by the draft model is accurate or inaccurate. Typically, the draft model receives information identifying which lexical units are accepted by the target model, such as a bitmap based on the position of the accepted lexical units in the bitmap, the index of the first rejected lexical unit in the draft set of lexical units (because subsequent lexical units are usually also rejected), etc.

[0044] If the draft model has not yet generated terminating terms that have been validated by the target model, operation 100 may proceed to box 140. At box 140, various device parameters are measured or otherwise determined to adjust the length of the draft set of terms to be generated by the generative AI model during the next round of inference. In some examples, the temperature of the processor on which the generative AI model is being executed may be measured or otherwise determined. Additionally or alternatively, in other examples, the operating frequency of the processor on which the generative AI model is being executed may be measured or otherwise determined. Additionally or alternatively, in further examples, one or more other operating parameters that may affect the performance of the processor and / or other components, such as operating voltage, power draw (e.g., in watts), current, etc., may be measured or otherwise determined.

[0045] At box 150, operation 100 continues, adjusting the length of the draft set of lexical units to be generated by the generative AI model during the next round of inference. This adjustment can be based on: (measured) operating parameters of the device on which the generative AI model is executed, the acceptance rate of previously generated draft sets of lexical units, the historical acceptance rate of other sets of lexical units generated by the generative AI model for the input query and historical input queries, a scheduling function defined for the processor on which the generative AI model is executed, etc. In some aspects, as discussed, the (measured) operating parameters may include temperature, frequency (clock speed), etc. If at least one of the (measured) operating parameters exceeds a threshold defined for at least one of the operating parameters (e.g., maximum temperature and / or maximum frequency), the length of the draft set of lexical units can be set to a defined minimum number of lexical units. This defined minimum number of lexical units can be selected, for example, as a baseline value (e.g., 0 lexical units) or the minimum of a number of lexical units smaller than the number of lexical units included in the previously generated set of lexical units.

[0046] If at least one of the (measured) operating parameters does not exceed a threshold, the processor running on the device may not risk entering a limited performance mode (e.g., low-power, low-frequency mode), which could reduce the rate at which inference is generated using the generative AI model. Therefore, the number of lexics to be included in the next draft set of lexics generated by the generative AI model can be set at least in part based on the historical acceptance rate of lexics generated by the generative AI model. In some aspects, the acceptance rate can be the acceptance rate of the most recently generated set of lexics generated by the generative AI model. In other aspects, the acceptance rate can be a time-weighted acceptance rate, such as an acceptance rate calculated based on an exponential moving average, to allow additional data to be considered when determining the accuracy of the set of lexics generated using the generative AI model.

[0047] If the acceptance rate calculated after the current lexical generation iteration exceeds a threshold amount calculated after a previous current lexical generation, the generative AI model can be determined to be sufficiently accurate, and performance can be improved by increasing the number of lexicals generated during the next iteration of lexical generation using the generative AI model (e.g., measured from the number of lexicals generated by the generative AI model within a defined time period). However, if the acceptance rate calculated after the current lexical generation iteration drops below a threshold amount calculated after a previous current lexical generation, the generative AI model can be determined to be inaccurate, and therefore the generative AI model should generate fewer lexicals during the next lexical generation iteration.

[0048] The adjusted lexical length can be provided to the generative AI model, and operation 100 can return to box 120 to generate a new draft set of lexical units. The loop including boxes 120, 130, 140, and 150 can continue until the generative AI model includes a terminating lexical unit in the draft set of lexical units and receives an indication that the target model has accepted (verified) the terminating lexical unit. Once the terminating lexical unit is received (e.g., included in the verified set of lexical units passed from the target model to the draft model), operation 100 can proceed from box 130 to box 160, where the lexical unit speculatively generated by the generative AI model and accepted by the target model is output as a response to the input query.

[0049] FIG. 2A and FIG. 2B Pseudocode is illustrated for adapting the length of a draft set of lexical units generated using a generative artificial intelligence model to one or more (measured) operating parameters of a device operating on which a generative artificial intelligence model operates, according to various aspects of this disclosure.

[0050] FIG. 2AThe pseudocode 200A illustrated illustrates the use of (measured) processor (or other component) temperature and temperature threshold when determining the length of the draft set of lexical units during subsequent iterations of a lexical generation operation using a generative artificial intelligence model.

[0051] As illustrated, the lexical generation operation can be performed using the baseline draft length. Maximum draft length (in (representing integers) and threshold acceptance rate divergence (in This is initiated by representing real numbers. During lexical generation, the processor temperature (or other device temperature) is monitored. It can be measured or otherwise determined. If in time... The (measured) processor temperature at the location Exceeding the defined threshold This threshold can correspond to the maximum operating temperature or the maximum temperature before processor performance is throttled, and can be set according to the following expression for the time. Draft length at this location:

[0052] In other words, time t The draft length at this point can be set to a minimum value of 0, or time. The draft length at this point is less than one. Typically, because the model may not be able to generate a negative number of tokens, this expression may result in the number of tokens generated by the generative AI model being equal to 0, thus pausing the inference operation until the (measured) processor temperature (or other device temperature) is reached. Drop below the threshold temperature However, in some respects, the minimum number of tokens can be set to a positive number to allow operation even when the processor temperature (or other device temperature) exceeds a temperature threshold. It also generates a certain (relatively small) number of word units.

[0053] However, if the (measured) processor temperature (or other device temperature) Less than or equal to the threshold temperature Then time t Acceptance rate and time at the site The previous acceptance rate can be used to determine whether to change the draft length, and if so, in what direction to change it. If time... Acceptance rate at the site Compare time Acceptance rate at the site Improved divergence above the threshold This can increase the length of the draft. In some respects, in terms of time... The length of the draft at that point can be increased using the following expression:

[0054] That is, unless Otherwise in time Draft length at the location The length can be increased relative to the previous draft because the number of lexical units included in the draft set may not exceed the defined maximum. In other respects, time... The draft length at this point can be increased based on other expressions and / or other values ​​(e.g., in addition to increasing by 1).

[0055] If the acceptance rate Already relative to previous acceptance rates Reduced beyond the threshold This indicates that the generative model is wasting resources generating lexicals that are unlikely to be accepted by the target model. Therefore, to save computational resources and reduce the workload on the processor (and the corresponding heat generated by performing as many lexical generation operations as possible within a given time period), the draft length can be reduced. In some respects, in terms of time... The length of the draft at that point can be reduced using the following expression:

[0056] In other words, time Draft length at the location It can be relative to time The length of the draft was reduced. In other respects, time... The draft length can be reduced based on other expressions and / or other values ​​(e.g., in addition to reducing it by 1). In some respects, the draft length can be reduced up to the minimum number of tokens; for example, regardless of historical (time-weighted) acceptance rate, time... Draft length at the location It allows the generation of a defined minimum number of tokens (e.g., 1 token) so that a response to the input query can continue to be generated.

[0057] Finally, if the acceptance rate It has not deviated from the previous acceptance rate. Exceeding the threshold Then in time Draft length at the location Can be maintained with time Draft length at the location same.

[0058] FIG. 2BThe pseudocode 200B shown in the example uses the processor frequency (Also known as clock speed) and threshold processor frequency Measurements (or other determinations) are used to determine whether to adjust the draft length, thereby avoiding lower-performance operation scenarios for the processor. Otherwise, relative to time... Length of the draft, time The determination of the draft length can be kept consistent with FIG. 2A The same pseudocode 200A is shown.

[0059] FIG. 3 Examples of performance metrics for constant draft length and adaptive draft length speculative decoding used for comparison are illustrated according to various aspects of this disclosure.

[0060] As illustrated, graphs 312, 314, and 316 illustrate various performance and operational parameter measurements (or other determinations) for the constant draft length of the four lexical units in each iteration of the draft set generated by the generative AI model, as illustrated in graph 310. Meanwhile, graphs 322, 324, and 326 illustrate various performance and operational parameter measurements (or other determinations) for the variable draft length illustrated in graph 320. The number of lexical units included in each iteration of the draft set generated by the generative AI model can vary based on operational parameter measurements (or other determinations) and historical acceptance rate information, as discussed above.

[0061] As illustrated in Figure 312, using a constant draft length, the (measured) processor temperature may eventually exceed a threshold temperature (e.g., at the thirteenth draft lexicon iteration stage). Because the (measured) processor temperature exceeds the threshold temperature, the processor can be throttled from a high clock speed to a low clock speed, as illustrated in Figure 314. As illustrated, throttling the processor from a high clock speed to a low clock speed can result in a significant reduction in the lexicon generation rate for generative AI models: in this example, the lexicon generation rate is reduced by fifty percent, from 20 lexicons per second to 10 lexicons per second.

[0062] However, using a variable draft length, as illustrated in Figure 320, the (measured) temperature of the processor, as depicted in Temperature Figure 322, can remain below the throttling threshold temperature. Therefore, unlike the scenario illustrated in Clock Speed ​​Figure 314, where the processor is throttled while performing inference operations relative to the input query, Clock Speed ​​Figure 324 illustrates no throttling of the processor (because the processor temperature has not yet exceeded the threshold). Accordingly, the rate of lexical generation may be affected by the choice of draft length for the next set of lexical units generated by the generative AI model executing on the device. Therefore, the rate of lexical generation may be unaffected (or less affected) by the clock speed of the processor on which the generative AI model executes. Thus, as illustrated in Lexical Rate Figure 326, the lexical generation rate of a generative AI model using a variable draft length can vary over time, but may not decrease as drastically as the lexical generation rate illustrated in Figure 316. Furthermore, even as inference operations continue over time, the lexical generation rate using a generative AI model with variable draft length remains closer to the theoretical maximum lexical generation rate than that using a generative AI model with fixed draft length. Therefore, inference operations using a generative AI model with variable draft length can be completed faster than those using a generative AI model with fixed draft length.

[0063] Example for generating responses to input queries using generative artificial intelligence models and variable draft length Operation

[0064] FIG. 4 An example operation 400 is illustrated, which, according to various aspects of this disclosure, can be executed by a computing device to generate a response to an input prompt using an adaptive draft length of measured (or otherwise determined) operational parameters of a generative artificial intelligence model and a device operating on top of the generative artificial intelligence model. Operation 400 can be executed by a device on which at least a generative artificial intelligence model (e.g., a draft model, a target model, a model that can act as both a draft model and a target model in a self-speculating decoding pipeline, etc.) can be deployed, such as a smartphone, tablet computer, laptop computer, desktop computer, server, cloud computing instance hosted in a distributed computing environment, etc.

[0065] As illustrated, operation 400 begins at box 410, where one or more first operational attributes of a device (e.g., one or more of its components) on which an inference operation using a machine learning model is performed is determined (e.g., measured or accessed).

[0066] At box 420, operation 400 continues to generate a first draft set of lexical units using a machine learning model. Typically, the number of lexical units included in the first draft set is based at least in part on one or more first operation attributes of the device and a scheduling function defined by the machine learning model. In some respects, the scheduling function defined by the machine learning model may define the maximum number of lexical units that can be included in the draft set of lexical units and the rate at which the maximum number of lexical units may change over time (e.g., decreasing over time to account for various processor characteristics, such as hot generation over time).

[0067] In some respects, one or more first operational attributes include the temperature of the device (e.g., its components). The number of morphemes included in the first draft set of morphemes may be further based on a comparison of the device temperature or the device component temperature with a threshold temperature (e.g., a throttling threshold temperature).

[0068] In some respects, one or more first operating parameters include processor frequency (or clock speed). The number of lexical units included in the first draft set of lexical units may be further based on a comparison of the processor frequency with a threshold frequency (e.g., a throttling threshold frequency).

[0069] In some aspects, the number of lexical units included in the first draft set of lexical units may be a defined minimum number of lexical units when at least one of the device's operational attributes exceeds a threshold. In other aspects, the minimum number of lexical units is determined based on the number of lexical units generated during previous lexical unit generation rounds using a machine learning model.

[0070] At box 430, operation 400 continues to output a first draft set of lexical units for validation. The first draft set of lexical units can be output to a second machine learning model (e.g., a target model) for validation. This second machine learning model can be executed on the same device as the machine learning model or on a different device, and can be a larger model than the machine learning model (e.g., with more parameters).

[0071] In some aspects, operation 400 may further proceed to box 440, wherein the acceptance rate of the first draft set of lexical units is calculated based on the verification of a first draft set of lexical units (e.g., generated by a second machine learning model), the number of lexical units included in the first draft set of lexical units, and the number of lexical units in the subset of lexical units corresponding to the accepted lexical units from the first draft set of lexical units. For example, the number of lexical units in the subset of lexical units corresponding to the accepted lexical units from the first draft set of lexical units can be determined based on information received from the target machine learning model, which identifies the subset of lexical units from the first draft set of lexical units accepted by the target machine learning model. This information may include, for example, a bitmap identifying accepted and rejected lexical units, an index of the first rejected lexical unit in the first draft set of lexical units, etc.

[0072] In some respects, operation 400 can proceed to box 450, where one or more second operating parameters of the device are determined (e.g., measured or accessed).

[0073] In some aspects, operation 400 can proceed to box 460, where a machine learning model is used to generate a second draft set of lexical terms. The number of lexical terms included in the second draft set can be based on one or more second operation attributes of the device, a defined scheduling function, and an acceptance rate.

[0074] In some respects, when the acceptance rate of the first draft set of a word element exceeds the sum of the acceptance rates of the previously generated draft sets of the word element and a threshold, the number of words elements included in the second draft set of the word element is greater than the number of words elements included in the first draft set of the word element.

[0075] In some respects, when the acceptance rate of the first draft set of a word element is less than the difference between the acceptance rate of the previously generated draft set of the word element and a threshold, the number of words elements included in the second draft set of the word element is less than the number of words elements included in the first draft set of the word element.

[0076] In some respects, when the acceptance rate of the first draft set of lexical units is within a threshold range (e.g., the same range) of the acceptance rate of the previously generated draft set of lexical units, the number of lexical units included in the second draft set of lexical units is equal to the number of lexical units included in the first draft set of lexical units.

[0077] In some respects, the acceptance rate can be further calculated based on the exponential moving average of the number of tokens accepted over multiple iterations of the machine learning model and the number of draft tokens generated.

[0078] In some respects, operation 400 can proceed to box 470, where a second draft set of lexical units is output for validation. Similar to the first draft set of lexical units, the second draft set of lexical units can be output to a second machine learning model (e.g., a target model) for validation. This second machine learning model can be executed on the same device as the machine learning model or on a different device, and can be a larger model than the machine learning model (e.g., with more parameters).

[0079] In some respects, operation 400 can proceed to box 480, where a response is output based on the validation of a first draft set of lexical units and the validation of a second draft set of lexical units. Typically, the validation of the second draft set of lexical units includes the acceptance of terminating lexical units in the second draft set of lexical units.

[0080] Example for generating responses to input queries using generative artificial intelligence models and variable draft length Processing system

[0081] FIG. 5 An example processing system 500 for generating responses to queries input into a generative artificial intelligence model based on variable draft length is described, such as in this paper, for example, relative to... FIGS. 1-4 As described.

[0082] Processing system 500 includes a central processing unit (CPU) 502, which in some examples may be a multi-core CPU. Instructions executed at CPU 502 may be loaded, for example, from program memory associated with CPU 502 or from a memory partition (e.g., memory 524).

[0083] The processing system 500 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 504, a digital signal processor (DSP) 506, a neural processing unit (NPU) 508, and a connectivity component 512.

[0084] NPUs, such as the NPU 508, are typically configured to implement dedicated circuitry for the control and arithmetic logic used to execute machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. An NPU may sometimes be alternatively referred to as a neural signal processor (NSP), tensor processing unit (TPU), neural network processor (NNP), intelligent processing unit (IPU), vision processing unit (VPU), or graphics processing unit.

[0085] NPUs such as the NPU 508 are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, such NPUs may be part of a dedicated neural network accelerator.

[0086] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0087] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0088] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take a new piece of data as input and process it rapidly through an already trained model to generate model output (e.g., inference).

[0089] In some implementations, the NPU 508 is part of one or more of the CPU 502, GPU 504, and / or DSP 506. These can be located in the user equipment (UE) of a wireless communication system or on another computing device.

[0090] In some examples, connectivity component 512 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Connectivity component 512 may be further coupled to one or more antennas 514.

[0091] The processing system 500 may also include one or more sensor processing units 516 associated with the sensor in any way, one or more image signal processors (ISPs) 518 associated with the image sensor in any way, and / or a navigation processor 520, which may include satellite-based positioning system components (e.g., GPS or GLONASS) and inertial positioning system components.

[0092] The processing system 500 may also include one or more input and / or output devices 522, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, etc.

[0093] In some examples, one or more processors in the processing system 500 may be based on the ARM or RISC-V instruction set.

[0094] The processing system 500 also includes memory 524, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, memory 524 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 500.

[0095] Specifically, in this example, the memory 524 includes an operation parameter determination component 524A, a draft lexical generation component 524B, a lexical output component 524C, an acceptance rate calculation component 524D, and a machine learning model component 524E. The depicted components and other undepicted components can be configured to perform various aspects of the methods described herein.

[0096] Typically, the processing system 500 and / or its components can be configured to perform the methods described herein.

[0097] Example clauses

[0098] Specific implementation details of various aspects of this disclosure are described in the following numbered clauses.

[0099] Clause 1: A processor-implemented method comprising: determining one or more first operational attributes of a device on which an inference operation using a machine learning model is performed; using the machine learning model to generate a first draft set of lexical units, including the number of lexical units in the first draft set based on one or more operational attributes of the device as measured and a scheduling function defined by the machine learning model; and outputting the first draft set of lexical units for verification.

[0100] Clause 2: The method according to Clause 1 further includes: receiving information identifying a subset of lexical units in a first draft set that are accepted by a target machine learning model; and calculating an acceptance rate based on the number of lexical units included in the first draft set of lexical units and the number of lexical units in the subset of lexical units.

[0101] Clause 3: The method according to Clause 2 further includes: determining one or more second operational attributes of the device; using the machine learning model to generate a second draft set of lexical units, including the number of lexical units in the second draft set of lexical units based on one or more measured second operational attributes of the device, the defined scheduling function, and the acceptance rate; and outputting the second draft set of lexical units for verification.

[0102] Clause 4: According to the method of Clause 3, wherein when the acceptance rate of the first draft set of the word element exceeds the sum of the acceptance rates of the previously generated draft sets of the word element and a threshold, the number of word elements included in the second draft set of the word element is greater than the number of word elements included in the first draft set of the word element.

[0103] Clause 5: The method according to Clause 3 or 4, wherein when the acceptance rate of the first draft set of the word element is less than the difference between the acceptance rate of the previously generated draft set of the word element and a threshold, the number of words included in the second draft set of the word element is less than the number of words included in the first draft set of the word element.

[0104] Clause 6: The method according to any one of Clauses 3 to 5, wherein when the acceptance rate of the first draft set of the lexical is within a threshold range of the acceptance rate of the previously generated draft set of the lexical, the number of lexical included in the second draft set of the lexical is equal to the number of lexical included in the first draft set of the lexical.

[0105] Clause 7: The method according to any one of Clauses 2 to 6, wherein the acceptance rate is further calculated based on an exponential moving average of the number of accepted lexical units and the number of generated draft lexical units in multiple iterations of executing the machine learning model.

[0106] Clause 8: The method according to any one of Clauses 1 to 7, wherein the one or more first operational attributes include device temperature, and wherein the number of morphemes generated that are included in the first draft set of morphemes is further based on a comparison of the device temperature with a threshold temperature.

[0107] Clause 9: The method according to any one of Clauses 1 to 8, wherein the one or more first operational attributes include a processor frequency, and wherein the number of lexical units generated in the first draft set of lexical units is further based on a comparison of the processor frequency with a threshold frequency.

[0108] Clause 10: The method according to any one of Clauses 1 to 9, wherein when at least one of the one or more operational attributes of the device exceeds a threshold, the number of lexical units includes a defined minimum number of lexical units.

[0109] Clause 11: The method according to Clause 10, wherein the minimum number of lexical units is determined based on the number of lexical units generated during previous lexical unit generation rounds using the machine learning model.

[0110] Clause 12: A processing system comprising: at least one memory storing executable instructions; and one or more processors coupled to the at least one memory and configured to execute the executable instructions to cause the processing system to perform any one of Clauses 1 to 11.

[0111] Clause 13: A processing system comprising: components for performing operations according to any one of Clauses 1 to 11.

[0112] Clause 14: A non-transitory computer-readable medium having executable instructions stored thereon, which, when executed by one or more processors, perform the operations pursuant to any one of Clauses 1 to 11.

[0113] Additional notes

[0114] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined in others. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Moreover, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be embodied by one or more elements of these claims.

[0115] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0116] As used in this article, the phrase “at least one of” in a list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0117] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0118] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above may be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0119] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless otherwise specifically stated, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC § 112(f) unless the element is expressly referred to by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known later, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system, the processing system comprising: At least one memory, wherein executable instructions are stored in the at least one memory; and One or more processors, said one or more processors coupled to said at least one memory and configured to execute said executable instructions to cause the processing system to: Determine one or more first operational attributes of the device on which inference operations using a machine learning model are performed; The machine learning model is used to generate a first draft set of lexical units, including the number of lexical units in the first draft set of lexical units based on one or more first operational attributes of the device and a scheduling function defined by the machine learning model. as well as The first draft set of output lexical units is used for validation by the target machine learning model.

2. The processing system of claim 1, wherein the one or more processors are further configured to cause the processing system to: Based on the output of the first draft set of lexics used for verification by the target machine learning model, information identifying a subset of lexics accepted by the target machine learning model from the first draft set of lexics; and The acceptance rate is calculated based on the number of lexical units included in the first draft set of lexical units and the number of lexical units in the subset of lexical units.

3. The processing system of claim 2, wherein the one or more processors are further configured to cause the processing system to: Determine one or more second operational attributes of the device; The machine learning model is used to generate a second draft set of lexical units, including the number of lexical units in the second draft set of lexical units based on the one or more second operational attributes of the device, the defined scheduling function, and the acceptance rate; as well as The second draft set of output lexical units is used for validation by the target machine learning model.

4. The processing system of claim 3, wherein when the acceptance rate of the first draft set of the word element exceeds the sum of the acceptance rates of the previously generated draft sets of the word element and a threshold, the number of word elements included in the second draft set of the word element is greater than the number of word elements included in the first draft set of the word element.

5. The processing system of claim 3, wherein when the acceptance rate of the first draft set of the word element is less than the difference between the acceptance rate of the previously generated draft set of the word element and a threshold, the number of word elements included in the second draft set of the word element is less than the number of word elements included in the first draft set of the word element.

6. The processing system of claim 3, wherein when the acceptance rate of the first draft set of the lexical is within a threshold range of the acceptance rate of the previously generated draft set of the lexical, the number of lexicals included in the second draft set of the lexical is equal to the number of lexicals included in the first draft set of the lexical.

7. The processing system of claim 2, wherein the acceptance rate is further calculated based on an exponential moving average of the number of tokens accepted and the number of draft tokens generated during multiple iterations of executing the machine learning model.

8. The processing system of claim 1, wherein the one or more first operational attributes include the temperature of the device or the temperature of a component of the device, and wherein the number of morphemes generated that are included in the first draft set of morphemes is further based on a comparison of the temperature of the device or the component of the device with a threshold temperature.

9. The processing system of claim 1, wherein the one or more first operational attributes include a processor frequency, and wherein the number of lexical units generated in the first draft set of lexical units is further based on a comparison of the processor frequency with a threshold frequency.

10. The processing system of claim 1, wherein when at least one of the first operational attributes of the device exceeds a threshold, the number of lexical units includes a defined minimum number of lexical units.

11. The processing system of claim 10, wherein the minimum number of lexical units is determined based on the number of lexical units generated during previous lexical unit generation rounds using the machine learning model.

12. A processor-implemented method, the processor-implemented method comprising: Determine one or more first operational attributes of the device on which inference operations using a machine learning model are performed; The machine learning model is used to generate a first draft set of lexical units, including the number of lexical units in the first draft set of lexical units based on one or more first operational attributes of the device and a scheduling function defined by the machine learning model. as well as The first draft set of output lexical units is used for validation by the target machine learning model.

13. The method according to claim 12, further comprising: The first draft set based on the output lexical units is used for verification by the target machine learning model, and information identifying a subset of lexical units accepted by the target machine learning model from the first draft set of lexical units is received. as well as The acceptance rate is calculated based on the number of lexical units included in the first draft set of lexical units and the number of lexical units in the subset of lexical units.

14. The method according to claim 13, further comprising: Determine one or more second operational attributes of the device; The machine learning model is used to generate a second draft set of lexical units, including the number of lexical units in the second draft set based on one or more second operational attributes of the device, the defined scheduling function, and the acceptance rate; and The second draft set of output lexical units is used for validation by the target machine learning model.

15. The method of claim 14, wherein when the acceptance rate of the first draft set of the lexical exceeds the sum of the acceptance rates of the previously generated draft sets of the lexical and a threshold, the number of lexicals included in the second draft set of the lexical is greater than the number of lexicals included in the first draft set of the lexical.

16. The method of claim 14, wherein when the acceptance rate of the first draft set of the lexicon is less than the difference between the acceptance rate of the previously generated draft set of the lexicon and a threshold, the number of lexicons included in the second draft set of the lexicon is less than the number of lexicons included in the first draft set of the lexicon.

17. The method of claim 14, wherein when the acceptance rate of the first draft set of the lexicon is within a threshold range of the acceptance rate of the previously generated draft set of the lexicon, the number of lexicons included in the second draft set of the lexicon is equal to the number of lexicons included in the first draft set of the lexicon.

18. The method of claim 13, wherein the acceptance rate is further calculated based on an exponential moving average of the number of tokens accepted and the number of draft tokens generated during multiple iterations of executing the machine learning model.

19. The method of claim 12, wherein the one or more first operational attributes include the temperature of the device or the temperature of a component of the device, and wherein the number of terms generated that are included in the first draft set of terms is further based on a comparison of the temperature of the device or the component of the device with a threshold temperature.

20. The method of claim 12, wherein the one or more first operational attributes include processor frequency, and wherein the number of lexical units generated in the first draft set of lexical units is further based on a comparison of the processor frequency with a threshold frequency.

21. The method of claim 12, wherein when at least one of the first operational attributes of the device exceeds a threshold, the number of lexical units includes a defined minimum number of lexical units.

22. The method of claim 21, wherein the minimum number of lexical units is determined based on the number of lexical units generated during previous lexical unit generation rounds using the machine learning model.

Images