High efficiency decoding using large and small generative artificial intelligence models

Abstract

Certain aspects of the present disclosure provide techniques and apparatus for generating a response to an input query using a generative artificial intelligence model. The method generally includes receiving an input query for processing. An embedded representation of the received input query is generated using a first generative artificial intelligence model. The embedding representation generally includes embedding of the received input query in a first dimension. The embedded representation is projected into a projected representation of the received input query. Generally, the projected representation includes a representation in a second dimension. A response to the received input query is generated using the second generative artificial intelligence model and the projection representation, and the generated response is output.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 486,653, filed October 13, 2023, which is incorporated herein by reference. Background Technology

[0003] The aspects disclosed herein relate to generative artificial intelligence models.

[0004] Generative AI models can be used in a variety of environments to generate responses to input queries. For example, generative AI models can be used in chatbot applications where large language models (LLMs) are used to generate answers, or at least responses, to input queries. Other examples of where generative AI models can be used include stable diffusion, where a model generates an image based on an input text description of the content of a desired image; and decision transformers, where future actions are predicted based on a sequence of previous actions within a given environment.

[0005] Generally, using generative AI models to generate responses to queries can be computationally expensive. For example, in a chatbot deployment that uses a large language model to generate responses to queries formatted as text queries, the response to the query can be generated by passing through the large language model 10 times for each lexical (e.g., word or part of a word) generated as part of the response. For example, the output of each pass could be a probability distribution of a set of lexical (words), from which the next lexical (word) can be selected by sampling or based on maximum likelihood. Because each word (lexical) is generated by passing through the large language model 10 times in response to the query, the computational cost can be modeled as the product of the number of words included in the response and the computational resource cost of executing the passes through the large language model (e.g., in terms of processing power, memory bandwidth, and / or other computational resources used), which typically increases with the number of parameters within the large language model. Summary of the Invention

[0006] Certain aspects of this disclosure provide a processor implementation method for generating a response to an input query using a generative artificial intelligence model. The method typically includes receiving an input query for processing. An embedding representation of the received input query is generated using a first generative artificial intelligence model. This embedding representation typically includes an embedding of the received input query in a first dimension. The embedding representation is projected onto a projected representation of the received input query. Generally, the projected representation includes a representation in a second dimension, and the second dimension is smaller than the first dimension. A response to the received input query is generated using a second generative artificial intelligence model and the projected representation, and the generated response is output.

[0007] Other aspects include: processing systems configured to perform the foregoing methods and those methods described herein; a non-transitory computer-readable medium including instructions that, when executed by one or more processors of the processing system, cause the processing system to perform the foregoing methods and those methods described herein; a computer program product embodied on a computer-readable storage medium including code for performing the foregoing methods and those methods further described herein; and a processing system including components for performing the foregoing methods and those methods further described herein.

[0008] The following description and related figures illustrate in detail certain exemplary features of one or more aspects. Attached Figure Description

[0009] The accompanying drawings depict only certain aspects of this disclosure and are therefore not intended to limit the scope of this disclosure.

[0010] Figure 1 Example pipelines for generating responses to inputs using large-scale generative artificial intelligence models and small-scale generative artificial intelligence models are illustrated according to various aspects of this disclosure.

[0011] Figure 2 Examples of iterative generation of word tuples using large-scale generative artificial intelligence models and small-scale generative artificial intelligence models according to various aspects of this disclosure are illustrated.

[0012] Figure 3 An example pipeline for generating responses to inputs using large and small language models, according to various aspects of this disclosure, is illustrated.

[0013] Figure 4 An example pipeline for generating responses to inputs using large and small language models, according to various aspects of this disclosure, is illustrated.

[0014] Figure 5 Example operations for generating responses to input queries using large-scale generative artificial intelligence models and small-scale generative artificial intelligence models, according to various aspects of this disclosure, are illustrated.

[0015] Figure 6 Example processing systems configured to perform various aspects of this disclosure are described.

[0016] For ease of understanding, the same reference numerals are used where possible to designate common elements shared by all figures. Elements and features conceived in one aspect can be beneficially incorporated into other aspects without further elaboration. Detailed Implementation

[0017] Various aspects of this disclosure provide apparatus, methods, processing systems, and computer-readable media for efficiently generating responses to input queries using generative artificial intelligence models.

[0018] Generally, generative AI models generate responses to queries input into the model. For example, a large language model (LLM) deployed within a chatbot can generate responses to queries using multiple iterations through the large language model, each subsequent iteration based on the query and using lexical units (or words or parts of words) generated through previous iterations of the large language model. These large language models can typically include millions or even billions of weights or parameters. Due to the scale of these models and the operation performed on each lexical unit to predict the next lexical unit that should be generated in response to the query and previously generated lexical units, deploying large language models on various devices with limited memory, storage, and / or processing power relative to the cloud computing execution instances that typically run large language models on them may be impractical or even infeasible. Furthermore, the computational complexity involved in generating responses to queries provided as input to the model can involve significant energy consumption, processing time, memory utilization, and other resource utilization, potentially rendering computational resources unavailable for other tasks.

[0019] Various techniques can be used to improve the efficiency of generating responses to input queries using generative AI models. In some examples, a cascade of generative AI models can be used to process queries. A cascade of generative AI models can include multiple generative models and a scoring model, each trained on a different number of tokens. The scoring model typically predicts the probability that a particular generative model from the multiple generative models is likely to generate the correct answer to the input. These predicted probabilities, along with thresholds, can be used to determine when to accept a response generated by multiple generative models and continue generating subsequent responses to the input and any previously generated tokens.

[0020] In other examples, speculative decoding techniques allow smaller generative models (sometimes referred to as draft large language models, also known as draft models or small models) to run in parallel with larger generative models (sometimes referred to as target large language models, also known as target models or large models)). In such cases, the draft model (which can be a pruned version of the chosen target model such that the draft model and the target model have similar probability distributions, or it can be a smaller version of the target model (e.g., trained on millions of lexical units rather than hundreds of millions or even billions of lexical units)) speculatively generates additional lexical units and probabilities for sampling these additional lexical units based on a currently acceptable set of lexical units. The target model can then generate lexical units based on those generated by the draft model. To generate the final product, the target model can perform rejection sampling on a per-lexical basis to accept or reject lexical units generated by the draft model, such that the draft model and the target model have similar probability distributions.

[0021] However, these techniques still rely on using large generative models to generate lexical units. As discussed, these large generative models are capable of generating responses to input queries based on each lexical unit. However, generating lexical units using large generative models typically involves significant computational costs, far exceeding those incurred using smaller generative models. Furthermore, the additional computational costs associated with generating lexical units using large generative models may be wasted because smaller generative models may be able to generate many lexical units that are included in the response to the input query.

[0022] This disclosure provides techniques for generating responses to input queries using a large generative model (which may be referred to as a first generative model) and a small generative model (which may be referred to as a second generative model), the large generative model being larger than the small generative model. Generally, the large generative model generates an embedding representation of the input query, which can be projected onto a lower-dimensional space. The small generative model then uses the embedded representation of the projected input query to generate a response to the input query. Generally, the small generative model uses the embedded representation of the projected input query to generate a number of lexical units, up to a threshold number of lexical units. The embedded representations of the generated lexical units generated by the large generative model and the embedded representation of the input query can be used as input to the small generative model to generate subsequent lexical units. By using a large generative model to generate an embedded representation of the input query (which may be computationally cheaper than generating lexical units using a generative model), aspects of this disclosure allow a smaller generative model to be tuned to generate a response to the input query based on knowledge from a higher-dimensional domain extracted from the input query (e.g., the dimension of the domain associated with the large generative model, which may be higher than the dimension of the domain associated with the smaller generative model). Therefore, the generation of the response to the input query can be performed using fewer computational resources (e.g., processor cycles, memory utilization, etc.) compared to the amount of computational resources used to autoregressively generate the response to the input query using a large generative model.

[0023] Generate responses to input queries using both large-scale and small-scale generative AI models. Example

[0024] Figure 1 An example pipeline 100 for generating responses to inputs using large-scale generative artificial intelligence models and small-scale generative artificial intelligence models, according to various aspects of this disclosure, is illustrated.

[0025] As illustrated in the figure, pipeline 100 includes a large generative AI model 110, a projection module 130, and a small generative AI model 140 (also referred to herein as "SGM"). Generally, the large generative AI model 110 and the small generative AI model 140 can be models trained to generate responses with varying numbers of parameters (e.g., lexical units). For example, the large generative AI model 110 can be trained to generate responses based on a training data corpus comprising hundreds of billions or even trillions of lexical units, while the small generative AI model 140 can be trained to generate responses from a range based on a training data corpus comprising a significantly smaller number of lexical units. Because the number of parameters associated with the large generative AI model 110 is significantly greater than the number of parameters associated with the small generative AI model 140, the computational cost involved in generating a response to an input query (or input question) using the large generative AI model 110 may be significantly higher than the computational cost involved in generating a response to an input query using the small generative AI model 140. However, while generating a response to an input query using a large generative artificial intelligence model 110 may be a computationally expensive process, generating an embedded representation 120 of the input query using a large generative artificial intelligence model 110 may be a computationally cheap process.

[0026] Therefore, in order to efficiently generate responses to input queries, the large generative AI model 110 generates an embedding representation 120 of the input query. The embedding representation 120 of the input query can typically be a set of data (e.g., vectors) representing the input query in a high-dimensional space (e.g., a space defined by a large number of parameters to which representations of the input query can be mapped). Because the embedding representation 120 represents the input query in a high-dimensional space, it typically encodes the semantics of the input query into a representation that can be used downstream of the large generative AI model 110 in generating responses to the input query. In some aspects, the embedding representation 120 may correspond to a summary of the input query, which retains semantically important information from the input query but discards semantically unimportant information (e.g., filler words, semantically unimportant phrases, etc. in a natural language query).

[0027] Because the embedding representation 120 is generated by the large generative AI model 110 in a higher dimension than the small generative AI model 140, the embedding representation can embed information from a large number of dimensions in a compact representation, which can be used by the small generative AI model 140 to generate a response, while reducing the computational cost of using the large generative AI model 110 to generate a response to the input query (which can be a significantly more resource-intensive and computationally expensive process than generating the embedding representation of the input query).

[0028] In some respects, as illustrated in the figure, the embedding representation 120 may be input into the projection module 130. The projection module 130 may be a model trained to align a small generative AI model 140 with a large generative AI model 110, such that the output of the large generative AI model 110 is usable by the small generative AI model 140. To make the output of the large generative AI model 110 usable by the small generative AI model 140, in some respects, the projection module 130 may be a model that reduces the dimension of the embedding representation 120 of the input query to a dimension corresponding to the parameter range used to train the small generative AI model 140.

[0029] In some aspects, such as Figure 1 As illustrated, projection module 130 may be a learning layer inserted between a large generative AI model 110 and a small generative AI model 140. In this example, the large generative AI model 110 and the small generative AI model 140 may be frozen pre-trained models, while projection module 130 may be a model trained based on embeddings generated by the large generative AI model 110 and the small generative AI model 140 for query samples included in the training dataset (not in...). Figure 1 (Example shown below). In some respects, and as follows regarding... Figure 3 and Figure 4 As discussed in further detail, the projection module 130 may be a component of a small generative artificial intelligence model 140, which projects the embedded representation 120 of the input query (and, in some respects, the lexical units or their embedded representations generated by the small generative artificial intelligence model 140) into a space having dimensions corresponding to the range of parameters used to train the small generative artificial intelligence model 140.

[0030] Small generative AI model 140 typically uses an embedded representation of the projection of the input query (and, in some aspects, lexical units or their embedded representations previously generated by small generative AI model 140) to generate a response to the input query. In some aspects, the response may include one or more lexical units corresponding to words in the natural language output generated as a response to a natural language input query, which are generated autoregressively by small generative AI model 140. Generally, when generating lexical units to be included in the response to the input query, small generative AI model 140 may generate lexical units based on historical lexical units according to the following expression:

[0031] in Indicates time t The generated tokens have to The conditional probability of choosing as a condition. The lexical sequence, and Indicates time t +1 generates a set of word elements to The conditional probability of choosing as a condition. The sequence of lexical units. Typically, individual lexical units can be generated per iteration using a small generative AI model 140.

[0032] The response generated to the input query can be output by a small generative AI model 140. In some aspects, the response can be provided together with the input query to a large generative AI model 110 to begin generating subsequent lexical groups that will be included as part of the response to the input query. In some aspects, where the small generative AI model 140 has determined that no additional lexicals are included in the response generated by the small generative AI model 140, or the small generative model 140 has met some other termination condition (e.g., no lexical has a probability exceeding a threshold probability to be selected for inclusion in the generated response, or the response includes the maximum number of lexicals), the small generative AI model 140 can output the generated response and further terminate the inference operation regarding the input query.

[0033] Figure 2 Example 200 illustrates the iterative generation of word tuples using large-scale generative artificial intelligence models and small-scale generative artificial intelligence models according to various aspects of this disclosure.

[0034] In Example 200, an input query is received for processing at a large generative AI model 110. As discussed above, instead of generating a response to the received input query, the large generative AI model 110 generates an embedding representation 210A, which can be fed as input to a smaller generative AI model 140. The embedding representation 210A typically corresponds to a representation of the input query in a multi-dimensional space. The embedding representation 210A may represent a semantic summary of the input query in the multi-dimensional space, and the semantic summary may include information about the semantically relevant parts of the input query and the relationships between these semantically relevant parts. In some aspects, the embedding representation 210A may not assign weights to semantically irrelevant parts of the input query, such as stop words, articles, or other linguistic components that do not significantly affect the meaning of the input query.

[0035] Embedded representation 210A can be provided as input to a small generative artificial intelligence model 140 for generating the first word tuple 220. Although Figure 2Without illustration, before the small generative AI model 140 generates a response to an input query, the embedding representation 210A can be projected from the larger dimensional space of the large generative AI model 110 to the smaller dimensional space of the small generative AI model 140. Generally, the small generative AI model 140 can be configured to autoregressively generate up to a threshold number of inputs given to the small generative AI model 140. N The lexical units. Generally speaking, as discussed above, a small generative AI model 140 can generate initial lexical units based on the embedding representation 210A of the input query. The first word tuple 220 is generated autoregressively (as discussed, it can be projected into a lower-dimensional space associated with the small generative AI model 140). In some respects, the initial word tuples... The term can be selected based on a probability distribution generated by a small generative artificial intelligence model 140 given an input of an embedded representation 210A of the input query, where the terminology... The lexicon that corresponds to the lexicon with the highest probability score in the probability distribution, or the lexicon with a probability score within the threshold of the highest probability score.

[0036] Lexicon to It can be generated by a small generative AI model 140 based on a conditional probability distribution generated by the small generative AI model 140. The conditional probability distribution typically corresponds to a probability distribution over a lexical range, conditional on the input query (or its projected representation) and the lexical previously generated by the small generative AI model 140. That is, the second lexical... It can be based on the input query (or its embedded representation 210A) and the first term. The selection is used to generate the third word. Simultaneously, queries can be based on input and the first term. Second word element The selection is used to generate [the token]. This process can continue for each consecutively generated token until [the token is selected]. N Each lexical unit is generated by a small generative artificial intelligence model 140.

[0037] While the probability distributions associated with the large generative AI model 110 and the small generative AI model 140 may be mirror images of each other, or at least similar, at the start of the lexical generation process, the lexicals generated by the small generative AI model 140 may, over time, cause the probability distribution associated with the small generative AI model 140 to deviate from the probability distribution of the large generative AI model 110. Therefore, without control, in some cases, the small generative AI model 140 may begin to generate a response to the input query that deviates significantly from the response originally generated by the large generative AI model 110. To minimize, or at least reduce, this deviation, aspects of this disclosure may define a threshold number of lexicals that the small generative AI model 140 can generate before the large generative AI model 110 generates updated embeddings to incorporate information about the input query and the lexicals generated by the small generative AI model 140 into the input of the small generative AI model 140.

[0038] To generate a second tuple 222 (together with the first tuple 220) to serve as part of the overall response to an input query, the input query and the first tuple 220 can be fed into a large generative AI model 110 for processing into an updated embedding representation 210B. The updated embedding representation 210B can thus correspond to a semantically aware representation of the input query and the first tuple 220, and therefore can be used by a small generative AI model 140 to generate the second tuple 222. As discussed above with respect to the embedding representation 210A of the input query, the large generative AI model 110 typically generates a representation of the input query (in some respects, together with the first tuple 220), and in some respects, can generate embeddings in a space with a higher dimension than that of the small generative AI model 140. As in the examples discussed above, once a threshold number of tuples in the second tuple 222 have been generated, aspects of this disclosure can output the second tuple 222 for further use (e.g., as a modulating signal to influence the generation of another tuple).

[0039] In some cases, to allow the small generative AI model 140 to accurately generate responses to input queries (e.g., responses similar to, if not identical to, those proposed by the large generative AI model 110), the large generative AI model 110 and the small generative AI model 140 may be jointly fine-tuned. In some aspects, this joint fine-tuning of the large generative AI model 110 and the small generative AI model 140 may involve adjusting one or more parameters of one or both of the large generative AI model 110 and / or the small generative AI model 140 such that, after the small generative AI model 140 generates a threshold number of tokens, the probability distributions associated with the large generative AI model 110 and the small generative AI model 140 deviate by less than a certain threshold amount.

[0040] Figure 3 An example pipeline 300 for generating responses to input queries using large and small language models, according to various aspects of this disclosure, is illustrated. The example pipeline 300 can be used, for example, to generate the responses described above and... Figure 2 The lexical tuples 220 and 222 are shown.

[0041] As illustrated in the figure, in pipeline 300, a response to a received input query can be generated by feeding the received input query into a large generative AI model 110. As discussed above, the large generative AI model 110 generates an embedded representation 120 of the input query. The generated embedded representation 120 of the input query is then provided as input to a small generative AI model 140 (or a block within a small generative AI model) for generating a response to the received input query.

[0042] In pipeline 300, the small generative artificial intelligence model 140 includes a projection module 310, a cascade module 320, and a causal self-attention (CSA) block 330. The projection module 310 may correspond to... Figure 1 The projection module 130, illustrated and discussed in more detail above, generally includes a trained machine learning model 310 that projects an embedded representation 120 of an input query from a first dimension associated with a large generative AI model 110 to a second (smaller) dimension associated with a small generative AI model 140. By doing so, relevant features or other relevant data extracted by the large generative AI model 110 can be used by the small generative AI model 140 to generate a response, or at least a portion of a response, as discussed.

[0043] The projection of the embedding representation 120 of the input query can be input to a cascade module 320, which combines the projection of the embedding representation 120 of the input query with the embedding of the selected output lexical to establish a combined embedding. The combined embedding is then input to a causal self-attention block 330 for processing. For example, the causal self-attention block 330 can be a transformer neural network that generates output using key, query, and value data. In this example, the causal self-attention block 330 generates lexicals to be included in the response to the input query based on the embedding representation of the projection of the input query and (if any such information exists) information about previously generated lexicals that form part of the response to the input query (e.g., embedding representations). Generally, the causal self-attention block 330 may use information about previously generated lexicals to influence the selection of the next lexical to be included in the response to the input query, but may not use information about downstream lexicals to influence the selection of the next lexical.

[0044] When pipeline 300 processes the input query, causal self-attention block 330 can receive a projection of the embedding representation of the input query as input. In response, causal self-attention block 330 generates a first lexical unit based on the projection of the embedding representation 120 of the input query. Subsequently, the cascade module 320 can project the embedding representation 120 of the input query and the first word. The embedding representations are concatenated, and the projection of the concatenated input query's embedding representation 120 and the first word are provided. The embedding representation enters the causal self-attention block 330 to generate the second lexical. The cascading of the embedding representations of the lexical units generated by the causal self-attention block 330 and the projection of the embedding representation 120 of the input query can continue until (as discussed above) the small generative AI model 140 has generated a threshold number of lexical units. At this point, the combination of the input query and the lexical units generated by the small generative AI model 140 can be input into the large generative AI model 110. The updated embedding representation generated by the large generative AI model 110 can be used as input to the small generative AI model 140 to trigger the generation of another set of lexical units using the techniques discussed herein.

[0045] Figure 4 An example pipeline 400 for generating responses to input queries using large and small language models, according to various aspects of this disclosure, is illustrated. The example pipeline 400 can be used, for example, to generate the responses discussed above and... Figure 2 The lexical tuples 220 and 222 are shown.

[0046] As illustrated in the diagram, production line 400 can initially be connected to... Figure 3The illustrated pipeline 300 operates similarly, feeding the input query into a large generative artificial intelligence model 110 and ultimately generating an embedded representation 120 of the input query.

[0047] To generate a set of lexical units corresponding to a response to an input query, or at least a portion thereof, the embedding representation 120 of the input query can be input to a projection module 410 for projection from a space in a first dimension associated with a large generative AI model 110 to a (smaller) space in a second dimension associated with a small generative AI model 140. This projection can result in the generation of keys (K) and values ​​(V), which are provided as input to a cross-attention block 420 for generating lexical units to be included in the response to the received input query. Generally, the cross-attention block 420 generates the output of lexical units to be included in the response based on different sequences of embeddings input to the cross-attention block 420. Within the small generative AI model 140, the embedding representation 120 of the input query can be projected into key inputs and value inputs, and the query input (Q) into the cross-attention block 420 can be an embedding associated with the next predicted lexical unit that will be output as a portion of the response to the received input query.

[0048] After the cross-attention block 420 selects lexics included in the lexical group corresponding to the input query, the lexical embeddings can be combined with the input query's embedding representation 120 to generate an updated embedding representation. This updated embedding representation can be fed as input to the projection module 410 for projection onto the keys and values ​​in the second dimension, which are used as input to the cross-attention block 420. In doing so, the context on which the cross-attention block 420 generates lexics can be autoregressively extended, allowing the small generative AI model 140 to generate subsequent lexics in the response based on previously generated lexics in the response and the input query.

[0049] Generate responses to input queries using both large-scale and small-scale generative AI models. Example operations

[0050] Figure 5 Example operation 500 according to various aspects of this disclosure is illustrated, which can be executed by a computing device to generate a response to an input query using a generative artificial intelligence model. The computing device for performing operation 500 can be a device on which a generative artificial intelligence model can be deployed, such as a smartphone, tablet computer, laptop computer, desktop computer, server, cloud computing example hosted in a distributed computing environment, or the like.

[0051] As illustrated in the figure, operation 500 begins at box 510, receiving an input query for processing. The input query may include, for example, a question, a prompt, or some other query that triggers the generation of a response using one or more generative AI models. These generative AI models may include models that generate natural language responses to natural language queries (also known as “large language models”), models that generate visual content in response to received natural language queries, or other models capable of generating responses to input queries or prompts.

[0052] At box 520, operation 500 continues to use the first generative artificial intelligence model to generate an embedding representation of the received input query in the first dimension. As discussed, the embedding representation of the received input query typically includes a representation of the received input query that preserves semantically relevant information contained within the received input query and removes semantically irrelevant information from the received input query. The embedding representation can be, for example, a vector in a multi-dimensional space, where individual points in the vector exemplify relationships between different parts (e.g., words or groups of words) of the received input query.

[0053] At box 530, operation 500 continues to project the embedded representation of the received input query onto the projected representation of the received input query. Generally, the projected representation of the received input query includes the representation in the second dimension. In some respects, the second dimension may be smaller than the first dimension.

[0054] At box 540, operation 500 continues to use a second generative artificial intelligence model and projective representation to generate a response to the received input query. In some respects, the response can be generated by autoregressively generating a first set of lexical units including a threshold number.

[0055] In some aspects, operation 500 may further include using a first generative artificial intelligence model to generate an updated embedding representation. The updated embedding representation typically includes the received input query and the generated first set of lexical units embedded in a first dimension. The updated embedding representation may be projected onto a projected representation of the received input query and the generated first set of lexical units in a second dimension. Using a second generative artificial intelligence model and the projected updated embedding representation, a second set of lexical units may be generated. The second set of lexical units may include a threshold number of lexical units.

[0056] In some aspects, generating a response using a second generative artificial intelligence model and a projected representation of the received input query may include generating one or more first lexical units based on the projected representation of the received input query. The projected representation of the received input query and the generated one or more first lexical units may be concatenated, and the concatenation of the received input query and information related to the generated one or more first lexical units can be used to generate second lexical units. In some aspects, concatenating the projected representation and information related to the generated one or more first lexical units may include concatenating the projected representation and the embedding representation of the one or more first lexical units.

[0057] In some aspects, generating a response using a second generative artificial intelligence model and a projected representation of the received input query may include generating one or more first lexical units based on the projected representation of the received input query. A combination of an embedded representation and one or more first lexical units may be projected onto the received input query and the projected representation of one or more first lexical units. Second lexical units may be generated based on the received input query and the projected representation of one or more first lexical units.

[0058] At box 550, operation 500 continues to output the generated response.

[0059] In some respects, the first generative AI model includes a model with a greater number of parameters than the second generative AI model. For example, the number of parameters may be related to the size of the training corpora used to train the first and second generative AI models.

[0060] In some respects, the first generative AI model and the second generative AI model are trained together for the same target task (e.g., using transfer learning techniques, where knowledge learned during the training of the first generative AI model is reused to train the second generative AI model), and the first generative AI model and the second generative AI model can be trained on the same number of lexical units.

[0061] In some respects, the first generative AI model can be a large language model trained to generate embedding representations in the first dimension. Meanwhile, the second generative AI model can be a small language model trained to generate responses based on inputs in the second dimension.

[0062] Generate responses to input queries using both large-scale and small-scale generative AI models. Example processing system

[0063] Figure 6 An example processing system 600 is described, which is used to generate responses to input queries using large-scale generative artificial intelligence models and small-scale generative artificial intelligence models, such as those described in this paper. Figure 5 The example described.

[0064] Processing system 600 includes a central processing unit (CPU) 602, which in some examples may be a multi-core CPU. Instructions executed at CPU 602 may be loaded, for example, from program memory associated with CPU 602 or from a memory partition (e.g., memory 624).

[0065] The processing system 600 also includes additional processing components tailored for specific functions, such as a graphics processing unit (GPU) 604, a digital signal processor (DSP) 606, a neural processing unit (NPU) 608, and a connectivity component 612.

[0066] NPUs, such as the NPU 608, are typically configured to implement dedicated circuitry for implementing control and arithmetic logic for executing machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), vision processing units (VPUs), or graphics processing units.

[0067] NPUs (such as the NPU 608) are configured to accelerate the execution of common machine learning tasks, such as image classification, machine translation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-a-chip (SoC), while in other examples, such NPUs may be part of a dedicated neural network accelerator.

[0068] An NPU can be optimized for either training or inference, or in some cases configured to balance performance between the two. For an NPU capable of performing both training and inference, these two tasks can generally still be performed independently.

[0069] NPUs designed to accelerate training are typically configured to speed up the optimization of new models. This involves taking an existing dataset (usually labeled or sublabeled), iterating over the dataset, and subsequently tuning model parameters (such as weights and biases) to improve model performance—a highly computationally intensive operation. Generally, optimization based on incorrect predictions involves backtracking through the layers of the model and determining gradients to reduce prediction errors.

[0070] NPUs designed to accelerate inference are typically configured to operate on the full model. Therefore, such NPUs can be configured to take a new piece of data as input and process it rapidly through an already trained model to generate model output (e.g., inference).

[0071] In some implementations, the NPU 608 is part of one or more of the CPU 602, GPU 604, and / or DSP 606. These can be located in the user equipment (UE) of a wireless communication system or on another computing device.

[0072] In some examples, connectivity component 612 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Connectivity component 612 may be further coupled to one or more antennas (not shown).

[0073] In some examples, one or more processors in the processing system 600 may be based on the ARM or RISC-V instruction set.

[0074] The processing system 600 also includes a memory 624, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 624 includes computer-executable components that can be executed by one or more of the aforementioned processors of the processing system 600.

[0075] Specifically, in this example, memory 624 includes a query receiving component 624A, an embedded representation generation component 624B, a projection component 626C, a response generation component 624D, a response output component 624E, and a generative model 624F. The depicted components and other undepicted components can be configured to perform various aspects of the methods described herein.

[0076] Generally, the processing system 600 and / or its components can be configured to perform the methods described herein.

[0077] Example Terms

[0078] Specific details of various aspects of this disclosure are described in the following numbered clauses: Clause 1: A processor-implemented method comprising: receiving an input query for processing; using a first generative artificial intelligence model to generate an embedded representation of the received input query in a first dimension; projecting the embedded representation of the received input query onto a projected representation of the received input query, wherein the projected representation includes a representation in a second dimension; using a second generative artificial intelligence model and the projected representation to generate a response to the received input query; and outputting the generated response.

[0079] Clause 2: The method according to Clause 1, wherein the first generative artificial intelligence model includes a model having a greater number of parameters than the number of parameters included in the second generative artificial intelligence model.

[0080] Clause 3: The method according to Clause 1 or 2, wherein the first generative AI model and the second generative AI model include models trained together on the same target task, such that the first generative AI model and the second generative AI model are trained on the same number of lexical units.

[0081] Clause 4: The method according to any one of Clauses 1 to 3, wherein using the second generative artificial intelligence model and the projective representation to generate the response comprises autoregressively generating a first set of lexical units including a threshold number of lexical units.

[0082] Clause 5: The method according to Clause 4 further comprises: using the first generative artificial intelligence model to generate an updated embedding representation, the updated embedding representation including the received input query and the embedded representation of the generated first set of lexical units in the first dimension; projecting the updated embedding representation onto an updated embedding representation of the projection of the received input query and the generated first set of lexical units in the second dimension; and using the second generative artificial intelligence model and the projected updated embedding representation to generate a second set of lexical units including the threshold number of lexical units.

[0083] Clause 6: The method according to any one of Clauses 1 to 5, wherein generating the response using the second generative artificial intelligence model and the projective representation comprises: generating one or more first lexical units based on the projective representation; concatenating the projective representation and information associated with the generated one or more first lexical units; and generating second lexical units based on the concatenated projective representation and the information associated with the generated one or more first lexical units.

[0084] Clause 7: The method according to Clause 6, wherein concatenating the projected representation and the information associated with the generated one or more first lexical units includes concatenating the projected representation and the embedded representation of the one or more first lexical units.

[0085] Clause 8: The method according to any one of Clauses 1 to 7, wherein generating the response using the second generative artificial intelligence model and the projection representation comprises: generating one or more first lexical units based on the projection representation; projecting a combination of the embedding representation and the one or more first lexical units onto the received input query and the projection representation of the one or more first lexical units in the second dimension; and generating second lexical units based on the received input query and the projection representation of the one or more first lexical units.

[0086] Clause 9: The method according to any one of Clauses 1 to 8, wherein: the first generative artificial intelligence model includes a large language model trained to generate the embedded representation in the first dimension, and the second generative artificial intelligence model includes a small language model trained to generate a response based on the input in the second dimension.

[0087] Clause 10: The method according to any one of Clauses 1 to 9, wherein the second dimension is smaller than the first dimension.

[0088] Clause 11: A processing system comprising: a memory storing executable instructions; and one or more processors coupled to the memory and configured to execute the executable instructions to cause the processing system to perform a method according to any one of Clauses 1 to 10.

[0089] Clause 12: A processing system comprising components for performing the method according to any one of Clauses 1 to 10.

[0090] Clause 13: A computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of Clauses 1 to 10.

[0091] Additional Notes

[0092] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, the function and arrangement of the elements discussed may be changed without departing from the scope of this disclosure. Various processes or components may be omitted, substituted, or added as appropriate in various examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Additionally, the scope of this disclosure is intended to cover such apparatuses or methods practiced using other structures, functionalities, or structures and functionalities that complement or replace the various aspects of this disclosure set forth herein. It should be understood that any aspect of this disclosure disclosed herein may be embodied by one or more elements of these claims.

[0093] As used herein, the term “exemplary” means “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as preferred or superior to other aspects.

[0094] As used in this article, the phrase “at least one of the items” refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbb, cc, and ccc, or any other ordering of a, b, and c).

[0095] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include calculation, computation, processing, derivation, research, searching (e.g., looking in a table, database, or other data structure), ascertainment, and similar actions. Additionally, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determine" can include parsing, selecting, picking, building, etc.

[0096] The methods disclosed herein include one or more steps or actions for implementing the methods. The steps and / or actions of the methods may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims. Furthermore, the various operations of the methods described above can be performed by any suitable component capable of performing the corresponding function. This component may include various hardware and / or software components and / or modules, including but not limited to circuits, application-specific integrated circuits (ASICs), or processors. Generally, where the operations illustrated in the figures are present, those operations may have corresponding components with similar numbering plus functional components.

[0097] The following claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless specifically stated otherwise, reference to the singular form of an element is not intended to mean “one and only one,” but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No element of any claim should be interpreted in accordance with 35 USC § 112(f) unless the element is expressly referred to by the phrase “for a component of,” or, in the case of a method claim, by the phrase “for a step of.” All structural and functional equivalents of the elements throughout the various aspects described herein that are known to a person of ordinary skill in the art, or will be known later, are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is explicitly stated in the claims.

Claims

1. A processing system, the processing system comprising: At least one memory, wherein executable instructions are stored in the at least one memory; and One or more processors, the one or more processors being configured to execute the executable instructions, so that the processing system: Receive input queries for processing; The first generative artificial intelligence model is used to generate an embedded representation of the received input query in the first dimension; The embedded representation of the received input query is projected onto the projected representation of the received input query, wherein the projected representation includes a representation in the second dimension; The second generative artificial intelligence model and the projective representation are used to generate a response to the received input query; as well as Output the generated response.

2. The processing system according to claim 1, wherein the first generative artificial intelligence model includes a model having a greater number of parameters than the number of parameters included in the second generative artificial intelligence model.

3. The processing system according to claim 1, wherein the first generative artificial intelligence model and the second generative artificial intelligence model include models trained together on the same target task, such that the first generative artificial intelligence model and the second generative artificial intelligence model are trained on the same number of lexical units.

4. The processing system of claim 1, wherein, in order to generate the response using the second generative artificial intelligence model and the projective representation, the one or more processors are configured to cause the processing system to autoregressively generate a first set of lexical units including a threshold number of lexical units.

5. The processing system of claim 4, wherein the one or more processors are further configured to cause the processing system to: The first generative artificial intelligence model is used to generate an updated embedding representation, which includes the received input query and the embedding of the first set of lexical units in the first dimension; The updated embedding representation is projected onto the updated embedding representation of the received input query and the projection of the generated first set of lexical units in the second dimension; as well as The second generative artificial intelligence model and the updated embedding representation of the projection are used to generate a second set of lexical units that include the threshold number of lexical units.

6. The processing system of claim 1, wherein, in order to generate the response using the second generative artificial intelligence model and the projective representation, the one or more processors are configured to cause the processing system to: One or more first word elements are generated based on the projection representation; The projection representation is concatenated with information related to one or more first lexical units generated; as well as The second lexical unit is generated based on the cascaded projection representation and the information associated with the generated one or more first lexical units.

7. The processing system of claim 6, wherein, in order to concatenate the projected representation and the information associated with the generated one or more first lexical units, the one or more processors are configured to cause the processing system to concatenate the projected representation and the embedded representation of the one or more first lexical units.

8. The processing system of claim 1, wherein, in order to generate the response using the second generative artificial intelligence model and the projective representation, the one or more processors are configured to cause the processing system to: One or more first word elements are generated based on the projection representation; The combination of the embedded representation and the one or more first lexical units is projected onto the received input query and the projected representation of the one or more first lexical units in the second dimension; as well as The second lexical unit is generated based on the received input query and the projection representation of the one or more first lexical units.

9. The processing system according to claim 1, wherein: The first generative artificial intelligence model includes a large language model trained to generate the embedded representation in the first dimension, and The second generative artificial intelligence model includes a small language model trained to generate responses based on inputs in the second dimension.

10. The processing system according to claim 1, wherein the second dimension is smaller than the first dimension.

11. A processor-implemented method, the processor-implemented method comprising: Receive input queries for processing; The first generative artificial intelligence model is used to generate an embedded representation of the received input query in the first dimension; The embedded representation of the received input query is projected onto the projected representation of the received input query, wherein the projected representation includes a representation in the second dimension; The second generative artificial intelligence model and the projective representation are used to generate a response to the received input query; as well as Output the generated response.

12. The method of claim 11, wherein the first generative artificial intelligence model includes a model having a greater number of parameters than the number of parameters included in the second generative artificial intelligence model.

13. The method of claim 11, wherein the first generative AI model and the second generative AI model comprise models trained together on the same target task, such that the first generative AI model and the second generative AI model are trained on the same number of lexical units.

14. The method of claim 11, wherein using the second generative artificial intelligence model and the projective representation to generate the response comprises autoregressively generating a first set of lexical units including a threshold number.

15. The method according to claim 14, further comprising: The first generative artificial intelligence model is used to generate an updated embedding representation, which includes the received input query and the embedding of the first set of lexical units in the first dimension; The updated embedding representation is projected onto the updated embedding representation of the received input query and the projection of the generated first set of lexical units in the second dimension; as well as The second generative artificial intelligence model and the updated embedding representation of the projection are used to generate a second set of lexical units that include the threshold number of lexical units.

16. The method of claim 11, wherein generating the response using the second generative artificial intelligence model and the projective representation comprises: One or more first word elements are generated based on the projection representation; The projection representation is concatenated with information related to one or more first lexical units generated; as well as The second lexical unit is generated based on the cascaded projection representation and the information associated with the generated one or more first lexical units.

17. The method of claim 16, wherein concatenating the projection representation and the information associated with the generated one or more first lexical units comprises concatenating the projection representation and the embedding representation of the one or more first lexical units.

18. The method of claim 11, wherein generating the response using the second generative artificial intelligence model and the projective representation comprises: One or more first word elements are generated based on the projection representation; The combination of the embedded representation and the one or more first lexical units is projected onto the received input query and the projected representation of the one or more first lexical units in the second dimension; as well as The second lexical unit is generated based on the received input query and the projection representation of the one or more first lexical units.

19. The method according to claim 11, wherein: The first generative artificial intelligence model includes a large language model trained to generate the embedded representation in the first dimension, and The second generative artificial intelligence model includes a small language model trained to generate responses based on inputs in the second dimension.

20. The method of claim 11, wherein the second dimension is smaller than the first dimension.

21. A processing system, the processing system comprising: A component used to receive input queries for processing; A component used to generate an embedded representation of the received input query in the first dimension using a first generative artificial intelligence model; A component for projecting the embedded representation of the received input query into a projected representation of the received input query, wherein the projected representation includes a representation in a second dimension; A component for generating a response to a received input query using a second generative artificial intelligence model and the projective representation; and The component used to output the generated response.

22. The processing system of claim 21, wherein the first generative artificial intelligence model includes a model having a greater number of parameters than the number of parameters included in the second generative artificial intelligence model.

23. The processing system of claim 21, wherein the first generative artificial intelligence model and the second generative artificial intelligence model comprise models trained together on the same target task, such that the first generative artificial intelligence model and the second generative artificial intelligence model are trained on the same number of lexical units.

24. The processing system of claim 21, wherein the component for generating the response using the second generative artificial intelligence model and the projective representation includes a component for autoregressively generating a first set of lexical units including a threshold number.

25. The processing system according to claim 24, further comprising: A component for generating an updated embedding representation using the first generative artificial intelligence model, the updated embedding representation including the received input query and the generated first set of lexical embeddings in the first dimension; A component for projecting the updated embedding representation onto the received input query and the projection of the generated first set of lexical units in the second dimension; and A component for generating a second set of lexical units including the threshold number using the second generative artificial intelligence model and the updated embedding representation of the projection.

26. The processing system of claim 21, wherein the component for generating the response using the second generative artificial intelligence model and the projective representation comprises: A component for generating one or more first lexical units based on the projection representation; A component for concatenating the projected representation with information associated with one or more generated first lexical units; and A component for generating a second lexical unit based on the cascaded projection representation and the information associated with one or more first lexical units generated.

27. The processing system of claim 26, wherein the component for concatenating the projection representation and the information associated with the generated one or more first lexical units includes a component for concatenating the projection representation and the embedded representation of the one or more first lexical units.

28. The processing system of claim 21, wherein the component for generating the response using the second generative artificial intelligence model and the projective representation comprises: A component for generating one or more first lexical units based on the projection representation; A component for projecting the combination of the embedded representation and the one or more first lexical units onto the received input query and the projected representation of the one or more first lexical units in the second dimension; and A component for generating a second lexical unit based on the received input query and the projection representation of the one or more first lexical units.

29. The processing system according to claim 21, wherein: The first generative artificial intelligence model includes a large language model trained to generate the embedded representation in the first dimension, and The second generative artificial intelligence model includes a small language model trained to generate responses based on inputs in the second dimension.

30. A non-transitory computer-readable medium having executable instructions stored thereon, the executable instructions, when executed by one or more processors, causing the one or more processors to perform operations including: Receive input queries for processing; The first generative artificial intelligence model is used to generate an embedded representation of the received input query in the first dimension; The embedded representation of the received input query is projected onto the projected representation of the received input query, wherein the projected representation includes a representation in a second dimension, and wherein the second dimension is smaller than the first dimension; The second generative artificial intelligence model and the projective representation are used to generate a response to the received input query; as well as Output the generated response.

Images