Compute budget for generative models

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automatically determining and adapting computational resources for generative models. One of the methods includes obtaining at least two sound signals captured by at least two audio sensors at two different locations at a first time point; generating an input characterizing the at least two sound signals; and processing the input characterizing the at least two sound signals using a neural network to generate a prediction at the first time point, wherein the prediction includes: (a) a respective score for each of a plurality of sound event classes indicating a predicted class for a first sound event detected in the at least two sound signals, and (b) a predicted location of a first sound source that emitted the first sound event.

Description

COMPUTE BUDGET FOR GENERATIVE MODELSBACKGROUND

[0001] Generative models have demonstrated state of the art performance across a wide range of tasks, such as text generation (e.g., writing, summarization, translation, coding), image generation, and audio generation. Generative models use very large neural network models that are trained on vast amounts of data. For example, a large language model (LLM) can include a transformer-based neural network model with self-attention capabilities and can achieve general-purpose language understanding and generation in response to a query. Thus, generative models are being deployed in various applications, e.g., as a coding assistant, as an email writing assistant, and for generating images in a presentation.

[0002] However, generative models are often computationally intensive due to their large size and a given model may use roughly the same amount of computation for any type of task. For example, in a decoder-only transformer-based generative model, the length of the prompt and the length of the output sequence can determine the amount of computation that is performed by the generative model. Thus, the amount of computational resources that the generative model consumes when performing different types of tasks can be very large and is not flexible.SUMMARY

[0003] This specification describes systems and techniques for automatically determining and adapting the amount of computational resources that a generative model consumes when processing a user input for a desired task. In particular, the systems and techniques determine feature data (e.g., context data) of the user input and process the feature data using a machine learning (ML) model to generate a compute budget that corresponds to a setting (e.g., a number of rounds of self-critique, a level of planning, image quality, or a number of samples) of the generative model.

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of obtaining at least two sound signals captured by at least two audio sensors at two different locations at a first time point; generating an input characterizing the at least two sound signals; and processing the input characterizing the at least two sound signals using a neural network to generate a prediction at the first time point, wherein the prediction includes: (a) a respective score for each of a plurality7of sound event classes indicating a predicted class for a first sound event detected inthe at least two sound signals, and (b) a predicted location of a first sound source that emitted the first sound event. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0005] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. In particular, one embodiment includes all the following features in combination. The output includes a predicted level of the compute budget of the generative model for the user input, and different levels of the compute budget correspond to different settings of the generative model, the actions further include: mapping the predicted level of the compute budget of the generative model for the user input to a corresponding setting of the generative model: and wherein processing the user input by the generative model includes processing the user input by the generative model using the corresponding setting of the generative model. The output includes a setting of the generative model that defines the compute budget of the generative model for the user input, and processing the user input by the generative model in accordance with the compute budget defined by the output includes processing the user input by the generative model using the setting of the generative model. The compute budget defines a number of rounds of selfcritique that the generative model uses for processing the user input. The compute budget defines a level of planning that the generative model uses for processing the user input. The compute budget defines a number of samples that the generative model generates from the user input. The compute budget defines a level of quality7of the generative output that the generative model generates from the user input. The generative output is an image, and the level of quality of the generative output includes a resolution of the image. The feature data of the user input includes context data of the user input, and processing the feature data includes processing the context data of the user input using the machine learning model to generate the output that defines the compute budget of the generative model for the user input. The user input includes a request to generate an output text, and the generative model includes a large language model (LLM) trained to generate text data, the feature data arecontext data in which the output text is to be used. The user input includes a request to generate an output image, and the generative model includes a generative image model trained to generate image data, the feature data includes features of one or more images in context data in which the output image is to be used. The feature data of the user input includes a level of importance for a response to the user input. The feature data of the user input includes a level of difficulty for a task defined by the user input. The actions include updating, using metadata of the generative model, the output that defines the compute budget of the generative model for the user input; and processing the user input by the generative model based on the compute budget defined by the updated output.

[0006] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0007] The systems and methods described in this specification increase the computational efficiency of a generative model because the amount of computational resources that a generative model consumes can be adapted to features of a task, e.g., task difficult}', task importance, or the desired quality that a user expects given the context. For example, the systems and methods can perform more computation using a generative model for difficult tasks or inputs seeking high quality outputs, and can perform less computation using a generative model for easy tasks or inputs not seeking high quality' outputs.

[0008] Some existing techniques allocate the same amount of computational resources for all requests, and thus generative outputs for difficult tasks or important tasks may need to be regenerated multiple times until a satisfactory one is obtained. The system and methods can avoid the situation where outputs from the generative model need to be regenerated by determining the expected quality of the output based on the context of the input query' and adapting the amount of computational resources that a generative model consumes based on the expected quality of the output. Because inference of the generative models can be quite computationally expensive given how large the generative models are, the systems and techniques can save a significant amount of computational cost. In some implementations, by reducing or eliminating the need to regenerate outputs, the system can generate an output with better quality within a fixed amount of available computational resources. In some implementations, the systems and techniques can adjust the quality of a generative output through the compute budget mechanism disclosed herein. For example, the systems and techniques can achieve a target level of quality' of a generative output by adjusting the compute budget by a percentage that corresponds to the target level.

[0009] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG. 1 is a diagram of an example system.

[0011] FIG. 2A is a diagram of an example system to generate an output text.

[0012] FIG. 2B is a diagram of an example system to generate an output image.

[0013] FIG. 2C is a diagram of an example system to generate an output conversation.

[0014] FIG. 3 is a flow chart of an example process for using generative models in accordance with a compute budget.

[0015] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0016] FIG. 1 is a diagram of an example system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described in this specification are implemented. The one or more computers can include personal computers, mobile communication devices, servers, and other devices that can send and receive data over a network. The network (not shown), such as a local area network (“LAN”), wide area network (“WAN”), the Internet, or a combination thereof, connects the one or more computers that implements the system. The system can use a single computer or multiple computers operating in conjunction with one another, including, for example, a set of remote computers deployed as a cloud computing service.

[0017] The system 100 dynamically determines an amount of computational resources that a generative model consumes for each user input. For example, the system 100 can perform more computation using a generative model for difficult tasks or inputs seeking high quality outputs, and can perform less computation using a generative model for easy tasks or inputs not seeking high quality outputs.

[0018] A generative model is a machine learning (ML) model that generates content, including text, images, audio, or other synthetic data, based on an input. The generative model 114 can be a very large neural network model and can be trained on vast amounts ofdata. As a result, after training, performing inference using the generative model can be very computationally expensive.

[0019] During inference, the generative model 114 can generate a generative output 116, e.g., a content of a certain type, in response to a query. A query is a question or the search for a piece of information. In some implementations, the generative model 114 can generate multimodal outputs, such as an image and a corresponding text describing the image.

[0020] In some implementations, the generative model 114 can be configured to process an input sequence of tokens to generate an output sequence of tokens. The tokens can represent any appropriate type of content, e.g., text, image, video, audio, or some combination of the above. For example, the generative model can be a large language model (LLM) and can be configured to process an input sequence of tokens from a vocabulary of text tokens to generate an output sequence of tokens from vocabulary.

[0021] More generally, the generative model 114 can be any appropriate neural network that receives an input sequence made up of text tokens selected from a type of content and auto- regressively generates an output sequence made up of text tokens from the type of content. For example, the generative model 114 can be a Transformer-based language model neural network or a recurrent neural network-based language model neural network.

[0022] In some situations, the generative model 114 can be referred to as an auto-regressive neural network when the neural network used to implement the language model auto- regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence.

[0023] For example, the current input sequence when generating a token at any given position in the output sequence can include the input sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the input sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the input and the current output sequence can be separated by one or more predetermined tokens within the current input sequence.

[0024] More specifically, to generate a particular token at a particular position within an output sequence, the generative model 1 14 can process the current input sequence to generatea score distribution (e.g., a probability distribution) that assigns a respective score, e.g., a respective probability, to each token in a vocabulary’ of tokens. The language model neural network can then select, as the particular token, a token from the vocabulary using the score distribution. For example, the neural network of the language model can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0025] As a particular example, the generative model 114 can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0026] The generative model 114 can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J.Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann. H. F. Song. J. Aslanides. S. Henderson, R. Ring, S. Young. E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro. A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D.Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidingen I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. RimelL C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112. 11446, 2021; Cohn Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh- Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, PranavShyam. Girish Sastry. Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.

[0027] In some implementations, the generative model 114 can use a decoder-only architecture that includes many decoder blocks, and without using an encoder. Each decoder block can include a self-attention layer and a feed forward neural network. The transformerbased generative model is an example of a generative model that the systems and techniques herein can be applicable.

[0028] The system and techniques described herein are applicable to other types of generative models, such as diffusion models, Bayesian networks, Generali ve Adversarial Networks (GANs). and Variational Autoencoders (VAEs).

[0029] One example of the generative model 114 can be a latent diffusion model.

[0030] As another example, the generative model 1 14 can be a diffusion model that uses a text-to-image diffusion model to generate a first image, and then applies one or more superresolution diffusion models to generate a final image.

[0031] As another example, the generative model 114 can be an auto-regressive generative model that auto-regressively generates tokens representing audio, video, images, or other data.

[0032] As yet another example, the generative model 114 can be a masked token generative model that sequentially unmasks tokens that represent text, video, audio, images, or other data during generation.

[0033] A compute budget of a generative model defines an amount of computational resources that the generative model consumes when processing a user input. The system 100 can adjust the amount of computational resources that are consumed by the generative model so that the amount stays within the compute budget of the generative model in any of a variety of ways.

[0034] In some implementations, the compute budget 112 can define a number of rounds of self-critique that the generative model 114 uses for processing the user input 102. Through self-critique, the generative model analyzes and compares its initial outputs before returning a final response. For example, in one round of self-critique, an LLM can generate one or more initial outputs, can analyze the one or more initial outputs to identify errors, and then can try to modify the one or more initial outputs or regenerate new outputs based on those errors before providing a final output. More rounds of self-critique means performing more inference steps and cost more computational resources, but can result in a higher quality or more refined generative output than a generative output generated using less rounds of self-critique. For example, a LLM can perform more or fewer rounds of self-critique, resulting in higher or lower quality output text.

[0035] In some implementations, the compute budget 112 can define a level of planning that the generative model 114 uses for processing the user input 102. The generative model 114 can use hierarchical planning. In hierarchical planning, a task is decomposed in smaller tasks, which are then decomposed in even smaller tasks, and so on. After forming a hierarchical plan, the system 100 can find a path through the hierarchical plan to solve a task by determining a level of planning according to the compute budget 112. For example, the system 100 can instruct the generative model 114 to construct a more detailed plan, e.g., dividing a task into smaller tasks, which can result in using more steps, such as more detailed reasoning traces. Compared with a generative model that directly generates an output using fewer steps, the generative model 114 is more likely to generate a high quality output through more steps related to the reasoning traces of the generative algorithm. In some implementations, using a higher level of planning can increase accuracy on tasks requiring reasoning at an additional computational cost. In some implementations, using a lower level of planning can improve speed for solving tasks at the cost of lower quality, e.g.. by not solving a subtask and leaving it to the user.

[0036] In some implementations, the compute budget 112 can define a number of samples that the generative model 114 generates from the user input 102. For example, the system 100 can generate multiple samples using the generative model 114. The system 100 can increase or decrease the number of samples generated by the generative model. In some implementations, the multiple samples can be presented on a device to a user, thus providing users with more choices. In some implementations, the system can use the multiple samples for other processing, such as self-consistency evaluation, that automatically selects one of the samples as the sample to be provided as output to the user. In some implementations, the system can measure the quality (and / or diversity) of the multiple samples using one or more metrics. Based on the quality measurement, the system can select one or more samples from the multiple samples and provide the one or more samples as output to the user, e.g., for presentation on a user device to a user.

[0037] In some implementations, the compute budget 112 can define a level of quality of the generative output 116 that the generative model 114 generates from the user input 102, e.g., a resolution of an image that the generative model 114 generates from the user input 102. For example, the system 100 can generate a high-quality generative output 116. The type of quality can be determined in accordance with the type of generative model 114 used by thesystem 100. In some implementations, an image or audio generation model can be configured to generate more fine-grained tokens that capture fine-grained details of images or audios, thus resulting in a higher resolution image output or an audio output with a lower compression rate. For example, the generative model 114 can include an audio generative model such as AudioLM (https: / / google-research.github.io / seanet / audiolm / examples / ) that uses Residual Vector Quantization (RVQ) tokenization. The system 100 can generate an audio output at different levels of compression rate by adjusting a RVQ level of the audio generative model. Thus, the audio output can have finer details or more coarse details.

[0038] Some image generation models can include several upsampling stages that are configured to generate images at multiple different resolutions. The system can adapt the number of upsampling stages for running the image generation model based on the compute budget indicating a desired resolution of the output image. For example, the image generation model can include three upsampling stages that are configured to generate images at three different resolutions in three stages, e.g., 128x128, 256x256, and 512x512. The system 100 can determine that an output image with 256x256 is sufficient based on the compute budget 112. The system can process the user input using the upsampling stage of the image generation model that generates a 256x256 image, and not using the upsampling stage of the image generation model that generates a 512x512 image.

[0039] In some implementations, based on the compute budget 112, the system 100 can determine, from a plurality of generative models, a generative model 114 that is being used to process the user input 102. For example, the system 100 can determine to use a larger or smaller generative model depending on the compute budget 112.

[0040] In some implementations, the generative model 114 can be a diffusion model and the compute budget 112 can define the number of denoising steps performed by the diffusion model. For example, the diffusion model can perform a large number of denoising steps if the compute budget is high, indicating that a high quality output is desired. In some cases, the diffusion model can perform a small number of denoising steps if the compute budget is low, indicating that a high quality output is not necessary.

[0041] In some implementations, based on the compute budget 112, the system 100 can adjust an amount of tokens used for a generative model, which may impact the computation cost or quality of the generative output in a predictable way. For example, some generative models can control the number of tokens for certain features, e.g., for usage of one or more tools. In some examples, the system 100 can adjust tokens allowed during a decoding process in order to enable or disable certain functionality. Thus, the system 100 can directly controlcomputational cost of the generative model, quality of the generative output, or both, in a predictable way.

[0042] In some implementations, based on the compute budget 1 12, the system can adjust an input prompt to the generative model, e.g., after a few steps of the generation of the output, or at the start time. For example, the prompt can include a detailed description of the desired output, resulting in a higher quality output at the cost of more computation.

[0043] The system 100 receives the user input 102 to be processed by the generative model 114. The user input 102 can be an input for a task that can be performed by the generative model, such as text generation, image generation, or audio generation. In some implementations, the user input 102 can include text data, image data, video data, audio data, or a combination of some of these. The user input 102 can include a query to be processed by the generative model 1 14. For example, the user input 102 can be “draft an email to thank every one for attending the conference.” In some implementations, the user input 102 can include a prompt, which can be a natural language text that requests the generative model 114 to perform a specific task. The prompt can rephrase a query, specify a style, provide relevant context, or provide examples.

[0044] In some implementations, the system 100 can receive the user input 102 from a user interface (UI) of an application. For example, an email w riting system can receive a user input indicating a task for generating a draft email. A document writing system can receive a user input indicating a task for rewriting a paragraph in a document. An image generation system can receive a user input indicating a task for generating a new- image to be used in a presentation. A conversation system using a LLM can receive a user input on various topics.

[0045] In order to generate a compute budget 112, the system 100 can take into consideration a number of different features of the user input 102.

[0046] In particular, the system 100 can include a feature determination engine 104 that generates feature data 106 of the user input 102.

[0047] In some implementations, the feature data 106 can be context data of the user input 102. For example, the context data can be contextual signals within the application being used. For example, context data can be a number of recipients on a distribution list of a draft email, a number of collaborators in a document, the quality of existing images in the remainder of a presentation, or an amount of time spent editing a piece of text.

[0048] In some implementations, the feature data 106 can include hint data included in or inferred from the user input 102. For example, the feature data 106 can be a level of difficulty for a task defined by the user input, or a level of importance for a response to the user input,or both. The feature determination engine 104 can process the user input 102 to determine explicit or implicit cues or hints.

[0049] For example, the feature determination engine 104 can process the user input 102 to predict whether the user input 102 provides any hints that the generative output 116 is particularly important. In some examples, the feature determination engine 104 can predict task difficulty by processing historical interaction data between the system 100 and a user. For example, the feature determination engine 104 can predict task difficulty based on how often users accept an output for a similar query.

[0050] The system 100 includes a compute budget determination engine 108 that generates a compute budget 112 for generating the generative output 116 using the generative model 114.

[0051] In some implementations, the compute budget determination engine 108 can automatically determine, based on feature data 106 of the user input 102, a compute budget 112 for generating the generative output 116 using the generative model 114.

[0052] In some implementations, the compute budget determination engine 108 can process the feature data 106 of the user input 102 using a model 110 to generate an output that defines a compute budget 112 of the generative model 114 for the user input 102.

[0053] In some implementations, the model 1 10 can include one or more machine learning (ML) models, such as a classification model or a regression model. In some implementations, the model 110 can be a LLM that is trained to understand the context feature of the user input 102 and to generate a compute budget 112 for processing the user input 102 using the generative model 114.

[0054] The model 110 can be trained using supervised learning on labeled training examples. In some implementations, the system can train the ML model on user feedback data. User feedback data can include data indicating whether generative outputs were accepted. The system can maintain a list of user feedback data items, and each data item can include a query, a compute budget for the uery, and feedback data for a generative output generated in accordance with the compute budget.

[0055] For example, when a user asks to refine or regenerate a generative output for a query, the system can determine feedback data indicating that the compute budget for generating the generative output was insufficient. For example, the user may provide feedback saying, ‘'this is not great” and / or “can you regenerate?”. The system can determine feedback data indicating insufficient compute budget.

[0056] For example, when there was missing information, e.g., when a user provides feedback saying, “thanks, but can you also add X,”, the system can determine feedback dataindicating there was missing information and the compute budget was unnecessarily spent for the query. The feedback data can indicate that it may have been better to use less compute budget and output an answer using less computational resources.

[0057] In some implementations, the system can use exploration strategies to generate diverse feedback to be included in training data for training the model 110. For example, the system can randomly increase or randomly decrease the compute budget for a query and obtain feedback data indicating how a user reacts to a corresponding generative output.

[0058] In some implementations, the model 1 10 can be trained using online learning. Instead of training the model on an entire training data set, the system can train the model 110 as training data becomes available in a sequential order. Thus, the system can update the model 110 for future user inputs at each step. For example, the system can train the model 110 when user feedback data becomes available and can provide a better estimate for the compute budget for processing future user inputs using the generative model 114.

[0059] In some implementations, the output that defines the compute budget 112 can be a predicted level of the compute budget 112 of the generative model 114 for the user input 102. The model 110 can be a classification model that is trained to generate likelihood scores for a plurality of levels of the compute budget. For example, the output can be one of high, medium or low levels.

[0060] In some implementations, the output that defines the compute budget 112 can be one or more settings of the generative model 114. In some implementations, the output can be values for a set of parameters for how the inference should be performed by the generative model 114. The model 110 can be a regression model that is trained to generate a predicted value for each of one or more parameters for how the inference should be performed by the generative model 114.

[0061] The system 100 processes the user input 102 by the generative model 114 in accordance with the compute budget 112 to generate a generative output 116. The system 100 can use the compute budget 112 to parameterize the inference of the generative model 114 for the user input 102. For example, the system 100 can apply more or less rounds of selfcritiques to an output of the generative model 114. The system 100 can apply more or less planning. The system 100 can sample more or less times using the generative model 114. The system 100 can generate a higher or lower quality output.

[0062] After performing the inference of the generative model 114 in accordance with the compute budget 112. the system 100 can obtain the generative output 116. The system can present the generative output 116 on a device to a user.

[0063] FIG. 2A is a diagram of an example system 200A to generate an output text. A user is writing an email which is going to a broad distribution list. For example, the number of recipients of the email can be larger than a threshold. The system 200A receives, from a device of the user, a request 202 to generate an output text, e.g., to rephrase a paragraph in the email.

[0064] The feature determination engine 204 can determine context data 206 of the request 202. For example, the system 200A can access the “To:” field of the email and can determine that the email has a broad distribution list, e g., the number of recipients of the email is larger than a threshold.

[0065] The context data 206 can include several features of the request 202, such as the distribution list of the email, the subject line of the email, the body of the email, and the signature of the email. For example, the title of the email can include a w ord or a phrase that describes the email, such as “IMPORTANT” or “URGENT”, indicating the importance or the urgency of the email. The body of the email can indicate the style of the email, such as using formal business language or casual language. Whether the email contains a signature can indicate whether the email is a formal business email or a casual personal email.

[0066] Based on the context data 206, the compute budget determination engine 208 can determine to use more compute budget 212 to process the request 202 using an LLM 214. The compute budget determination engine 208 can process the context data 206 using a ML model 210 to generate an output that indicates a high level of compute budget 212. In some implementations, the compute budget determination engine 208 can process the context data 206 using a ML model 210 to generate a value for a setting of processing the request 202 using the LLM 214, e.g., using more rounds of self-critique.

[0067] In some implementations, the ML model 210, e.g., a LLM, can be trained to achieve language understanding of the context data. For example, a LLM can be trained to predict the importance or the desired quality of the email, by processing text data, such as the distribution list of the email, the subject line of the email, the body of the email, and the signature of the email. Furthermore, the ML model 210 can be trained to generate the compute budget based on the prediction of the importance or the desired quality of the email. For example, if the desired quality of the email is high, the ML model 210 can generate a high level of compute budget. If the desired quality of the email is normal, the ML model 210 can generate a regular level of compute budget.

[0068] The system 200A processes the request 202 by the LLM 214 in accordance with the compute budget 212 to generate the output text 216. For example, the system can generateadditional suggestions on how to improve the paragraph, a more refined version of the paragraph that has gone through multiple rounds of self-critique, or both.

[0069] FIG. 2B is a diagram of an example system 200B to generate an output image. A designer is making a presentation for their client. The presentation already includes some existing images. The system 200B receives, from a device of the user, a request 222 to generate an output image, e.g.. to generate a picture of a cat on the moon, to be used in the presentation.

[0070] The feature determination engine 224 can determine context data 226 of the request 222. In some implementations, the context data 226 can include existing images in the presentation. The quality or complexity of the existing images can indicate the importance, the complexity, or the desired quality of the output image. In some implementations, the context data 226 can include the request 222 and other text data before and after the location of the requested image. The text data can indicate the importance, complexity, or desired quality of the output image.

[0071] Based on the context data 226. the compute budget determination engine 228 can determine, using the ML model 230, the compute budget 232 to process the request 222 using an image generation model 234. In some implementations, the ML model 230 can be a computer vision ML model trained to achieve image understanding of the existing images in the presentation. For example, a convolutional neural network model can be trained to predict the importance, the complexity, or the desired quality of the output image 236. by processing the existing images. In some implementations, the ML model 230 can include a LLM trained to predict the importance, complexity, or the desired quality’ of the presentation, e.g., by processing text data, such as the request 222 and other text data before and after the location of the requested image. Furthermore, the ML model 230 can be trained to generate the compute budget based on the prediction of the importance, complexity, or the desired quality of the presentation.

[0072] For example, because the context data 226 indicates that the existing images in the presentation have a high quality, the system 200B can determine that a high-quality image would be desirable to match the rest of the presentation. The compute budget determination engine 228 can process the context data 226 using a ML model 230 to generate an output that indicates a high level of compute budget 232. In some implementations, the compute budget determination engine 228 can process the context data 226 using a ML model 230 to generate a value for a setting of processing the request 222 using the image generation model 234, e.g., a resolution of the output image 236 that is higher than a threshold.

[0073] The system 200B processes the request 222 by the image generation model 234 in accordance with the compute budget 232 to generate an output image 236. For example, the system can generate a higher resolution image which may correspond to a longer token sequence for the image generation model 234. The system 200B can insert the high-quality output image 236 into the presentation.

[0074] FIG. 2C is a diagram of an example system 200C to generate an output conversation. A user is having a dialogue conversation with a LLM 254, e.g.. in a chatbot application. The system 200C receives, from a device of the user, a conversation 242 with the chatbot. For example, the user starts a conversation with the LLM 254 about a first topic, e.g., planning for a holiday trip. As the conversation continues, the conversion can indicate the importance or the desired quality of the response to the first topic. For example, as the conversation continues, the user communicates some preferences about hotels and locations for the holiday trip. However, the user is undecided as the conversation goes and the chatbot highlights different aspects of the vacation choices. For example, the location may be crowded, or tourists complain about certain aspects of the trip.

[0075] The feature determination engine 244 can determine context data 246 of the conversation 242. The system 200C can predict the importance or the desired quality of the response to the first topic based on historical conversation. For example, the system can process historical conversation, e.g., using a LLM, and can predict a reduced level of importance of the first topic, e.g., planning for the holiday trip. For example, the system can determine that the benefit for the user is likely higher to help them decide what they want, rather than generating a perfectly planned trip.

[0076] Based on the context data 246, the compute budget determination engine 248 can determine, using the ML model 259, the compute budget 252 to process the conversation 242 using the LLM 254. In some implementations, the ML model 250 can include a LLM trained to predict the importance or the desired quality of the response to the first topic by processing the historical conversation, e.g., the historical conversation indicating reduced level of importance of generating a plan for a holiday trip. Furthermore, the ML model 250 can be trained to generate the compute budget based on the prediction of the importance or the desired quality of the response to the first topic.

[0077] For example, because the context data 246 indicates a reduced level of importance of the first topic, e.g., planning for the holiday trip, the system 200C can determine that a perfectly planned trip may not be necessary. The compute budget determination engine 248 can process the context data 246 using a ML model 250 to generate an output that indicates alow level of compute budget 252. In some implementations, the compute budget determination engine 248 can process the context data 246 using a ML model 250 to generate a value for a setting of processing the conversation 242 using the LLM 254, e.g., using less planning or sampling fewer times.

[0078] The system 200C processes the conversation 242 by the LLM 254 in accordance with the compute budget 252 to generate an output conversation 256. For example, the system can generate a less detailed plan for the holiday trip. In some implementations, the system can generate suggestions for other alternative activities for spending the holiday.

[0079] FIG. 3 is a flow chart of an example process 300 for using generative models in accordance with a compute budget. The process will be described as being performed by an appropriately programmed computer system, such as the system 100.

[0080] The system receives a user input to be processed by a generative model (302). The system determines feature data of the user input (304).

[0081] In some implementations, the user input can include a request to generate an output text, and the generative model can include a large language model (LLM) trained to generate text data. The feature data can be context data in which the output text is to be used.

[0082] In some implementations, the user input can include a request to generate an output image, and the generative model can include a generative image model trained to generate image data. The feature data can include features of one or more images in context data in which the output image is to be used.

[0083] In some implementations, the feature data of the user input can include a level of importance for a response to the user input. In some implementations, the feature data of the user input can include a level of difficulty for a task defined by the user input.

[0084] The system processes the feature data of the user input using a machine learning model to generate an output that defines a compute budget of the generative model for the user input (306). The compute budget of the generative model defines an amount of computational resources that the generative model consumes when processing the user input.

[0085] In some implementations, the compute budget can define a number of rounds of selfcritique that the generative model uses for processing the user input. In some implementations, the compute budget can define a level of planning that the generative model uses for processing the user input. In some implementations, the compute budget can define a number of samples that the generative model generates from the user input. In some implementations, the compute budget can define a level of quality of the generative output that the generative model generates from the user input. In some implementations, thegenerative output can be an image, and the level of quality of the generative output can include a resolution of the image.

[0086] In some implementations, the feature data of the user input can include context data of the user input, and processing the feature data can include processing the context data of the user input using the machine learning model to generate the output that defines the compute budget of the generative model for the user input.

[0087] The system processes the user input by the generative model in accordance with the compute budget defined by the output to generate a generative output (308).

[0088] In some implementations, the output can include a predicted level of the compute budget of the generative model for the user input, and different levels of the compute budget can correspond to different settings of the generative model. The system can map the predicted level of the compute budget of the generative model for the user input to a corresponding setting of the generative model. The system can process the user input by the generative model using the corresponding setting of the generative model.

[0089] In some implementations, the output can include a setting of the generative model that defines the compute budget of the generative model for the user input. The system can process the user input by the generative model using the setting of the generative model.

[0090] In some implementations, the system can update, using metadata of the generative model, the output that defines the compute budget of the generative model for the user input. The system can process the user input by the generative model based on the compute budget defined by the updated output. The initial output can define an estimated compute budget for generating a generative output at a target quality' given the features of the user input.However, some generative models may produce this target quality more easily than other generative models.

[0091] Metadata of the generative model can include available task libraries (e.g., a collection of prompts), training data used for training the generative model, or a combination of both. The metadata of the generative model can represent how well a task is represented in the model’s training data. For example, solving an (N+l)th math benchmark can be easier if the model’s training data or task library already has N math benchmarks, than if the same model has N summarization benchmarks. The system can perform a refinement of the output that defines the compute budget based on metadata of the generative model. The metadata of the generative model can be used to increase the compute budget, decrease the compute budget, or keep the compute budget unchanged.

[0092] In some implementations, the system can use user feedback data to refine the output that defines the compute budget for a subsequent processing using the generative model in response to the user feedback. After generating the generative output in accordance with the compute budget, the system can receive user feedback on the generative output. The system can determine a compute budget for generating the next generative output based on the user feedback.

[0093] For example, based on the user feedback, the system can determine the previous compute budget was insufficient and can determine to increase the compute budget to generate a higher quality7generative output. As another example, based on the user feedback, the system can determine the previous compute budget was more than sufficient and can determine to decrease the compute budget to generate a lower quality generative output.

[0094] This specification uses the term ‘'configured’’ in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.|00095| Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0096] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, off-the-shelf or custom-made parallel processing subsystems, e.g., a GPU or another kind of special-purpose processing subsystem. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0097] A computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0098] As used in this specification, an “engine,” or “software engine,” refers to a software implemented input / output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

[0099] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to performfunctions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g.. an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [000100] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [000101] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g.. EPROM. EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.[000102] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and pointing device, e.g., a mouse, trackball, or a presence sensitive display or other surface by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction wi th a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user bysending text messages or other forms of message to a personal device, e.g., a smartphone, running a messaging application, and receiving responsive messages from the user in return. [000103] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.[000104] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.[000105] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.[000106] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising: receiving a user input to be processed by a generative model; determining feature data of the user input; processing the feature data of the user input using a machine learning model to generate an output that defines a compute budget of the generative model for the user input, wherein the compute budget of the generative model defines an amount of computational resources that the generative model consumes when processing the user input; and processing the user input by the generative model in accordance with the compute budget defined by the output to generate a generative output.

2. The method of claim 1, wherein the output comprises a predicted level of the compute budget of the generative model for the user input, and different levels of the compute budget correspond to different settings of the generative model, the method further comprising: mapping the predicted level of the compute budget of the generative model for the user input to a corresponding setting of the generative model; and wherein processing the user input by the generative model comprises processing the user input by the generative model using the corresponding setting of the generative model.

3. The method of claim 1, wherein the output comprises a setting of the generative model that defines the compute budget of the generative model for the user input, and processing the user input by the generative model in accordance with the compute budget defined by the output comprises: processing the user input by the generative model using the setting of the generative model.

4. The method of claim 1 , wherein the compute budget defines a number of rounds of self-critique that the generative model uses for processing the user input.

5. The method of claim 1, wherein the compute budget defines a level of planning that the generative model uses for processing the user input.

6. The method of claim 1, wherein the compute budget defines a number of samples that the generative model generates from the user input.

7. The method of claim 1, wherein the compute budget defines a level of quality of the generative output that the generative model generates from the user input.

8. The method of claim 7, wherein the generative output is an image, the level of quality of the generative output comprises a resolution of the image.

9. The method of any of claims 1-8, wherein the feature data of the user input comprises context data of the user input, and processing the feature data comprises processing the context data of the user input using the machine learning model to generate the output that defines the compute budget of the generative model for the user input.

10. The method of claim 9, wherein the user input comprises a request to generate an output text, the generative model comprises a large language model (LLM) trained to generate text data, the feature data are context data in which the output text is to be used.

11. The method of claim 9, wherein the user input comprises a request to generate an output image, the generative model comprises a generative image model trained to generate image data, the feature data comprises features of one or more images in context data in which the output image is to be used.

12. The method of any of claims 1-8, wherein the feature data of the user input comprises a level of importance for a response to the user input.

13. The method of any of claims 1-8, wherein the feature data of the user input comprises a level of difficulty for a task defined by the user input.

14. The method of any of claims 1-8, comprising: updating, using metadata of the generative model, the output that defines the compute budget of the generative model for the user input; and processing the user input by the generative model based on the compute budget defined by the updated output.

15. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving a user input to be processed by a generative model; determining feature data of the user input; processing the feature data of the user input using a machine learning model to generate an output that defines a compute budget of the generative model for the user input, wherein the compute budget of the generative model defines an amount of computational resources that the generative model consumes when processing the user input; and processing the user input by the generative model in accordance with the compute budget defined by the output to generate a generative output.

16. The system of claim 15, wherein the output comprises a predicted level of the compute budget of the generative model for the user input, and different levels of the compute budget correspond to different settings of the generative model, the operations further comprise: mapping the predicted level of the compute budget of the generative model for the user input to a corresponding setting of the generative model; and wherein processing the user input by the generative model comprises processing the user input by the generative model using the corresponding setting of the generative model.

17. The system of claim 15, wherein the output comprises a setting of the generative model that defines the compute budget of the generative model for the user input, and processing the user input by the generative model in accordance with the compute budget defined by the output comprises: processing the user input by the generative model using the setting of the generative model.

18. The system of claim 15, wherein the compute budget defines a number of rounds of self-critique that the generative model uses for processing the user input.

19. The system of claim 15, wherein the compute budget defines a level of planning that the generative model uses for processing the user input.

20. One or more non-transilory storage media encoded with instructions that when executed by a computing device cause the computing device to perform operations comprising: receiving a user input to be processed by a generative model; determining feature data of the user input; processing the feature data of the user input using a machine learning model to generate an output that defines a compute budget of the generative model for the user input, wherein the compute budget of the generative model defines an amount of computational resources that the generative model consumes when processing the user input; and processing the user input by the generative model in accordance with the compute budget defined by the output to generate a generative output.

Images