Performing tasks using cached intermediate results generated by a generative neural network

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing tasks using a generative neural network, a task planning engine, a retrieval engine, and a cache. The cache stores, for each of a plurality of cached sub-tasks: (i) a key vector for the cached sub-task, (ii) a cached sub-task prompt for the cached sub-task, (iii) a plurality of cached examples of the cached sub-task, and, optionally, (iv) a context for the cached sub-task.

Description

Attorney Docket No. 56113-0758WO1PERFORMING TASKS USING CACHED INTERMEDIATE RESULTS GENERATED BY A GENERATIVE NEURAL NETWORKBACKGROUND

[0001] This specification relates to processing inputs using neural networks to generate output sequences.

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.SUMMARY

[0003] This specification describes a task execution system implemented as computer programs on one or more computers in one or more locations that uses cached examples of sub-tasks to improve the performance of a generative neural network on a wide variety of tasks.

[0004] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. By maintaining a cache that stores intermediate results of sub-tasks that have been generated by the generative neural network in the past and, when a prompt for anew task that involves similar sub-tasks is received, providing the cached intermediate results as part of an input to the generative neural network for processing to generate a new result for the new task, the task execution system can improve the performance of the generative neural network on a wide variety of tasks.

[0005] When trained at sufficient scale, generative neural networks, e.g., Transformer-based language model neural networks, exhibit the notable ability to leam new tasks after being prompted with just a few examples. The described task execution system transfers this fewshot learning ability to a computerized task execution setting by making use of intermediate results of sub-tasks stored in the cache, such that a generative neural network prompted with a sub-task input that includes similar cached intermediate results can generate a higher quality (e.g.. more accurate, more relevant, more comprehensive, or the like) intermediateAtorney Docket No. 56113-0758WO1 result for a corresponding sub-task, and can advantageously do so with improved inference latency and resource efficiency.

[0006] That is, a higher quality intermediate result can be generated by the described task execution system more quickly, with reduced computing resource consumption (e.g., fewer processor cycles), or both compared with existing systems that do not make use of cached intermediate results of similar sub-tasks obtained from a cache when generating the intermediate results. For example, when a prompt for a new task that involves similar subtasks is received, by utilizing the corresponding cached intermediate results to generate a new result for the new task, the task execution system saves time and computing resources by not having to perform multiple inference passes through the generative neural network to generate new intermediate results for these similar sub-tasks.

[0007] The task execution system can further enable the generative neural network to rapidly “learn” how to generate intermediate results for new sub-tasks in new categories or novel domains with a handful of examples, and make use of existing knowledge without any additional training or fine-tuning after the generative neural network has been trained. For example, by providing cached intermediate results corresponding to similar automated assistance tasks as part of the inputs to the generative neural network, thereby leveraging its ability to learn new tasks when conditioned on examples, the task execution system can use the generative neural network to effectively perform a particular automated assistance task (i) even though the generative neural network was only trained on training data for entirely different tasks and (ii) even if the generative neural network has previously only been used to perform similar but not exactly identical automated assistance tasks.

[0008] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 is a diagram of an example task execution system.

[0010] FIG. 2 is a flow diagram of an example process for generating a result for a task.

[0011] FIG. 3 is a flow diagram of sub-steps of one of the steps of the process of FIG. 2.

[0012] Like reference numbers and designations in the various drawings indicate like elements.Atorney Docket No. 56113-0758WO1DETAILED DESCRIPTION

[0013] FIG. 1 is a diagram of an example task execution system 100. The task execution system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations that can perform tasks then output the results 126 for the tasks.

[0014] To perform each task, the task execution system 100 obtains instruction data that characterizes the task to be performed by the system and task data that will be processed by the system in order to perform the task. Collectively, such data will be referred to in this specification as “prompt data,” or “prompt 102” for short.

[0015] The prompt 102 can include data in any of a variety of modalities, e.g., any of text, audio, images, videos, or other sensor measurements, e.g., Lidar data, EEG data, EKG data, and so on. Moreover, the prompt 102 can be obtain in any of a variety of ways.

[0016] In some cases, the instruction data can be generated automatically, e.g., by an automated assistant or some other software applications that execute on a client device. In some other cases, the task execution system 100 can receive the instruction data as a user input. The user input can include a touchscreen input, a voice input, a keyboard input, a gesture input, a mouse, trackpad, or other pointing device input, that characterizes the task to be performed. For example, the instruction data can be in the form of a request, an instruction, or a command that is in some natural language and that defines the task to be performed by the system.

[0017] In some cases, the task execution system 100 can receive the task data in association with the instruction data. For example, the task data can also be received by the system together with the instruction data in a single user input or multiple user inputs. As another example, the task data can be generated automatically, e.g., by an automated assistant or another system.

[0018] In some other cases, unlike the instruction data, the task data can include prestored data, and the task execution system 100 can obtain the task data from a storage device that is accessible by the system.

[0019] In yet other cases, the task execution system 100 can obtain the task data from another system over a data communication network, e.g., through an application programming interface (API) that is made available by the other system. For example, the other system can be a sensor system (when the task is an agent control task); or a webpage capturing system, a screen recording system, or a screenshot capturing system (when the task is an automated assistance task).Attorney Docket No. 56113-0758WO1

[0020] In general the task execution system 100 can be configured to perform any of a variety of tasks. A few examples of the tasks follow.

[0021] In some cases, the task can be a text generation task, and the task execution system 100 can be configured to perform the text generation task by generating text sequences. In these cases, the result 126 generated by the task execution system 100 can include an output sequence of text. The result 126 can be presented for display to users, e.g., to the user who submitted the instruction data, task data, or both.

[0022] As some general examples, the text generation task can be a natural language processing or understanding task, e.g., an entailment task, a paraphrase task, a textual similarity task, a sentiment task, a sentence completion task, a grammaticality task, and so on, that operates on a prompt 102 that includes an input sequence of text in some natural language to generate an output sequence of text that is similarly in some natural language.

[0023] As a particular example, the task execution system 100 can be part of a dialog system and the prompt 102 can include audio or text or image data or some combination from the most recent conversational turn submitted by a user of the dialog system during the dialog, while the result 126 includes an output sequence that is the next turn in the conversation, e.g., either text or audio that is a response to the most recent conversational turn. Optionally, the prompt can also include one or more historical conversational turns that occurred earlier in the conversation.|00024| As another particular example, the task execution system 100 can be configured to perform a machine translation task, e g., it can be part of a machine translation system, where the prompt 102 can include text in a source language while the result 126 includes an output sequence of text in a target language that is a translation of the source text into the target language.

[0025] As another particular example, the task execution system 100 can be configured to perform a computer code generation task, e.g., it can be part of a computer code generation system, where the prompt 102 can include a text description of a desired piece of code or a snippet of computer code in a programming language and the result 126 includes computer code, e.g., a snippet of code that is described by the prompt or a snippet of code that follows the prompt in a computer program.

[0026] As another particular example, the task execution system 100 can be part of a search system that facilitate searching of resources on the Internet. A resource can be any data that can be provided over the Internet. A resource can be identified by a resource address that is associated with the resource. Resources include web pages, word processing documents,Atorney Docket No. 56113-0758WO1 portable document format (PDF) documents, images, video, and news feed sources, to name a few.

[0027] In this particular example, the search system can receive search queries submitted by client devices and, in response, identify resources that are relevant to the search query in the form of search results and returns the search results to the user devices in search results pages. A search result page can include search result data generated by the search system that identifies a resource responsive to a search query, and includes a link to the resource. The search result page can additionally include a result 126 in the form of an output sequence of text that is generated by the task execution system 100 based on a prompt 102 derived from the search query.

[0028] In some cases, the task can be an image or video generation task, and the task execution system 100 can be configured to perform the image or video generation task by generating images or videos that have multiple frames (where each frame is an image) as sequences of pixels. For example, the result 126 can include a plurality of color values for pixels in an image arranged according to a specified order. For example, the task execution system 100 can generate an image or a video conditioned on the prompt 102 that includes a text description of the content of the image or the video.

[0029] In some cases, the task execution system 100 can be configured to perform an agent control task. In these cases, the prompt 102 can include (i) a text sequence in a natural language that characterizes a task to be performed by an agent in the environment and (ii) one or more observations or other data respective states of the environment, and the result 126 can be any data that indicates that the agent has successfully completed the task or that some termination criterion is satisfied, e.g., the environment enters a state that has been designated as a terminal state or a threshold amount of time has elapsed without successfully completing the task.

[0030] For example, the result 126 can include a textual indicator, e.g., in the form of an output sequence of text, an audio indicator, e.g., in the form of a verbal response, or some other visual (e.g., graphical) indicator that is a confirmation of task accomplishment. As another example, the result 126 can include an observation of the environment that it is in the designated terminal state.

[0031] The agent can be, e.g., a real-world or simulated robot, a control system for an industrial facility, or a control system that controls a different kind of agent. The observations may include sensor data captured by sensors associated with (e.g. part ol) the agent, for example visual data, LIDAR data, sonar data, agent configuration data (e.g. joint angles).Attorney Docket No. 56113-0758WO1 agent orientation data, or the like. For example, the observations may include images captured by a visual sensor of the agent or by a visual sensor located in the environment.

[0032] In some cases, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment. For example, the agent may be a robot interacting with the environment to accomplish a task, e.g., to locate an object of interest in the environment, to move an object of interest to a specified location in the environment, to physically manipulate an object of interest in the environment in a specified way, or to navigate to a specified destination in the environment; or the agent may be an autonomous or semi-autonomous land, air, or sea vehicle navigating through the environment to a specified destination in the environment.

[0033] In some cases, the environment is a simulated environment and the agent is implemented as one or more computer programs interacting with the simulated environment. For example, the environment can be a computer simulation of a real-world environment and the agent can be a simulated mechanical agent navigating through the computer simulation. For example, the simulated environment may be a motion simulation environment, e g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation.

[0034] In some cases, the environment is a suitable execution environment, e.g., a runtime environment or an operating system environment, that is implemented on one or more client devices such as smart phones, tablet computers, wearable devices, automobile systems, standalone personal assistant devices, or any other appropriate electronic device, and the agent is a virtual agent (also known as “automated assistant ” or “mobile assistant”) that may be interacted with by a user through the client devices.

[0035] In these cases, the task can be an automated assistance task. That is, the virtual agent can receive instruction data as input from the user (e.g., typed or spoken natural language input) and respond with responsive content (e.g., visual and / or audible natural language output). The virtual agent can perform a broad range of tasks within the execution environment through interactions with various local and / or third-party applications, websites, or other agents.

[0036] Examples of the automated assistance tasks include scheduling an appointment through a calendar application, making a phone call through a telephone application, sending an email through an e-mail application, sending an instant message through an instant messaging application, playing a music through a digital music player application, playing a video through a digital video player application, and so on.Atorney Docket No. 56113-0758WO1

[0037] As a particular example, scheduling an appointment through a calendar application can involve multiple sub-tasks, including, for example, a first sub-task of opening the calendar application (e.g., by way of performing an action that simulates a user input within the execution environment); a second sub-task of retrieving data, e.g., the calendar entries or to-do lists of another participant, from the calendar application (e.g., by way of making calls to application programming interfaces (APIs) of other calendar applications, systems, or services); and a third sub-task of sending data confirming the appointment (e.g., similarly by way of making calls to APIs). To perform these tasks, the task execution system 100 uses a task planning engine 105, a retrieval engine 110, and a generative neural network 120. As illustrated in FIG. 1, in some implementations, the task planning engine 105 is included as part of the task execution system 100. In some other implementations, the task planning engine 105 is implemented within a remote system that is separate from, but in data communication with, the task execution system 100.

[0038] The generative neural network 120 can have any appropriate neural network architecture that allows the neural network to map an input sequence of tokens from a vocabulary to an output sequence of tokens from the vocabulary.

[0039] The vocabulary of tokens can include any of a variety' of tokens that represent text symbols or other symbols. For example, the vocabulary of text tokens can include one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of text in a natural language and / or a computer programming language.

[0040] Additionally, or alternatively, the vocabulary of tokens can include tokens that can represent data other than text. For example, the vocabulary of tokens can include image tokens that represent a discrete set of image patch embeddings of an image that can be generated by an image encoder neural network based on processing the image patches of the image. As another example, the vocabulary of tokens can include audio tokens that represent code vectors in a codebook of a quantizer, e.g., a residual vector quantizer.

[0041] When the task is an agent control task, the input sequence of tokens can for example include (i) text tokens that characterize the agent control task to be performed by an agent in an environment (ii) image tokens that represent a discrete set of image patch embeddings of an observation image of the environment, and the output sequence of tokens can for example include tokens that define or otherwise specifies an action to be performed by the agent in response to the observation image, e.g., “open gripper” or “move arm to coordinates (x, y, z).”Attorney Docket No. 56113-0758WO1

[0042] For example, the task execution system 100 can then perform the agent control task by generating instructions for the agent that when executed will cause the agent to perform the action, by submitting a control input directly to the appropriate controls of the agent, or by using another appropriate control technique.

[0043] When the task is an automated assistance task, the input sequence of tokens can for example include (i) text tokens that characterize the automated assistance task to be performed by a virtual agent in an execution environment (ii) image tokens that represent a discrete set of image patch embeddings of a screen image of the execution environment (e.g., a screenshot image of a user interface displayed on a computing device), and the output sequence of tokens can for example include tokens that define or otherwise specifies an action to be performed by the virtual agent when the execution environment is in a state characterized by the screen image, e.g., “click the button at coordinates (x, y)” or “enter [keyword] in search box.”

[0044] For example, the task execution system 100 can perform the automated assistance task by performing the action within the execution environment, e.g.. an action that simulates a user input, e.g., that simulates a touchscreen input, a voice input, a keyboard input, a gesture input, a mouse, trackpad, or other pointing device input. As another example, the task execution system 100 can perform the automated assistance task by making calls to application programming interfaces (APIs) of other applications, systems, or services.|00045| In some implementations, the generative neural network 120 can be a Transformerbased language model neural network or a recurrent neural network-based language model. As a particular example, the generative neural network 120 can be an auto-regressive Transformer-based neural network that has, e.g., an encoder-only Transformer architecture, an encoder-decoder Transformer architecture, or a decoder-only Transformer architecture.

[0046] Examples of such architectures include those described in Colin Raffel, et al.Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, et al. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977. 2020; Tom B Brown, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020; Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv: 2204.02311; Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023; and Rohan Anil, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.Attorney Docket No. 56113-0758WO1

[0047] The task planning engine 105 is configured to process the prompt 102 to generate a plan for performing the task characterized in the prompt 102. The plan specifies one or more sub-tasks. The task planning engine 105 can generate the plan by generating one or more subtask prompts 116A-N, where each sub-task prompt corresponds to a sub-task, and has a corresponding intermediate result that should be generated by using the generative neural network 120 for the sub-task. Collectively, the one or more sub-task prompts 116A-N define the plan for performing the task characterized in the prompt 102.

[0048] The one or more sub-tasks can be arranged in any sequential or parallel order within the plan. For example, the one or more sub-tasks can include a first sub-task that must be performed prior to a second sub-task, e.g., because the second sub-task operates on an intermediate result that is generated after the first sub-task is accomplished. As another example, the one or more sub-tasks can include a first sub-task that can be performed independently from, e.g., in parallel with, a second sub-task.

[0049] The task planning engine 105 can generate the one or more sub-task prompts 116A-N from the prompt 102 in many different ways.

[0050] In some implementations, the task planning engine 105 can apply any known text classification, information extraction, or grounded semantic extraction techniques to extract any portions of the prompt 102, and then generate the one or more sub-task prompts 116A-N that correspond to the one or more sub-tasks based on the extracted portions of the prompt 102, e.g.. by concatenating a predetermined template sequence or another sequence that is generated by using the generative neural network 120 to an extracted portion of the prompt 102.

[0051] In some implementations, the task planning engine 105 can generate the one or more sub-task prompts 116A-N from the prompt 102 by interacting with the generative neural network 120 or another neural network. For example, the task planning engine 105 can generate an input sequence that includes one or more task planning examples followed by the prompt 102 and then provide the input sequence for processing by the generative neural network 120 to generate the one or more sub-task prompts that correspond to the one or more sub-tasks. For example, each task planning example can be in the form of an input-output pair, where the input describes an example task, and the output describes example sub-task prompts that should be generated by the generative neural network 120 based on processing the input.

[0052] In these implementations the task planning engine 105 can interact with the generative neural network 120 by using any of the techniques discussed in Yao, Shunyu, et al. React:Attorney Docket No. 56113-0758WO1Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022); Yao, Shunyu, et al. Keep calm and explore: Language models for action generation in text-based games. arXiv preprint arXiv: 2010.02903 (2020); and Huang, Wenlong, et al. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. International conference on machine learning. PMLR, 2022, the entire contents of which are hereby incorporated by reference.

[0053] After having used the task planning engine 105 to determine the one or more subtasks for the task characterized in the prompt 102, the task execution system 100 then proceeds to perform each of these sub-tasks by, for each of the one or more sub-tasks, generating a sub-task input 118 A-N that corresponds to the sub-task and then processing the corresponding sub-task input 118A-N using the generative neural network 120 to generate a corresponding intermediate result.

[0054] For each sub-task, the corresponding sub-task input 118A-N is an input that is generated based on (i) the sub-task prompt 116A-N that corresponds to the sub-task and that is generated by using the task planning engine 105 and (ii) data obtained by using the retrieval engine 110 from a cache 150 that is accessible by the task execution system 100.

[0055] The cache 150 maintains data that is generated as a result of historic tasks previously performed by the task execution system 100 in response to historic prompts 102 received by the task execution system 100. i.e., tasks that were performed by the system 100 before the prompt 102 is received. The data includes data at sub-task granularity that is related to various sub-tasks that have been previously performed by the task execution system 100 (referred to as “cached sub-tasks”) when performing the historic tasks.

[0056] More specifically, the cache 150 stores, for each of a plurality of cached sub-tasks: (i) a key vector for the cached sub-task and (ii) a plurality of cached examples of the cached subtask. Optionally, the cache 150 stores (iii) a cached sub-task prompt for each of the plurality of cached sub-tasks. Further optionally, the cache 150 stores (iv) a context for each of the plurality of cached sub-tasks. The cache 150 can have any appropriate data structure that, for a cached sub-task, relates the key vector, the plurality of cached examples, and, when stored, the cached sub-task prompt, the context, or both.

[0057] For each cached sub-task, the context includes metadata that relates to the cached subtask. For example, the context can include metadata that identifies a particular client device that submitted a historic prompt in response to which the cached sub-task was performed. As another example, the context can include metadata that identifies a particular execution environment, e.g., a particular runtime environment or a particular operating systemAttorney Docket No. 56113-0758WO1 environment, within which the cached sub-task was performed. As another example, the context can include metadata that identifies a particular time stamp at which the cached subtask was performed. Additionally or alternatively, the context can include metadata that identifies a particular time that it took to complete the cached sub-task.

[0058] For each cached sub-task, a cached example includes a historic intermediate result generated by using the generative neural network 120. In practice there can be at least one. and possibly many, cached examples for each cached sub-task. Because the same cached subtask may have been performed within different environments (e., different execution environments running on different client devices) or across different times, the cached examples are generally at least partially different from each other, i.e., include at least partially different intermediate results. For example, for each cached sub-task, the cache 150 can store one or more different output sequences of text, one or more different images, one or more different videos, or one or more different audios as different intermediate results for the cached sub-task.

[0059] For each cached sub-task, the cached sub-task prompt can be a historic sub-task prompt that was generated in the past by using the task planning engine 105. For each cached sub-task, the key vector for the cached sub-task is generated based on the cached sub-task prompt and, optionally, the context for the cached sub-task. In various cases, generating the key vector based on context may improve the performance of the retrieval engine 110, e.g., may facilitate retrieval of more relevant cached examples or may reduce the computing resources required for similarity computation or both.

[0060] In some implementations, the task execution system 100 generates the key vector for each cached sub-task by processing the cached sub-task prompt and, optionally, the context using an embedding neural network that includes one or more neural network layers of any appropriate type, or some other machine learning model.

[0061] Such an embedding neural network can be trained on unlabeled training data based on optimizing a self-supervised or unsupervised loss function to generate vectors in an embedding space that have a fixed dimensionality. In some implementations, the embedding neural network can be trained as part of another neural network (that e.g. has a larger architecture) on tasks that involve generating embedding space representations, e.g., text classification or semantic analysis tasks.

[0062] For each of the one or more sub-tasks determined by using the task planning engine 105 for the task characterized in the prompt 102, the task execution system 100 generates a query vector for the sub-task based on the sub-task prompt that corresponds to the sub-task.Atorney Docket No. 56113-0758WO1

[0063] In some implementations, for each sub-task, the task execution system 100 process the corresponding sub-task prompt and, optionally, a context for the sub-task that includes metadata that relates to the sub-task using an embedding neural network to generate the query vector. The embedding neural network may be, but need not, the same embedding neural network or the same machine learning model that is used to generate the key vectors for the cached sub-tasks stored in the cache 150.

[0064] For example, the context can include metadata that identifies a particular client device that submitted the prompt 102. As another example, the context can include metadata that identifies a particular execution environment, e.g., a particular runtime environment or a particular operating system environment, that is provided on the one or more computers on which the task execution system 100 is implemented. As another example, the context can include metadata that identifies a particular time stamp at which the prompt 102 is received.

[0065] In implementations where the same embedding neural network is used, the embedding space of the query vectors for the one or more sub-tasks may be a shared embedding space with the key vectors for the cached sub-tasks stored in the cache 150. That is, the query vectors and the key vectors may have a same pre-determined dimensionality, i.e., they may include a same number of numeric values, e.g., floating point values or other values.

[0066] For each query7vector generated for a corresponding one of the one or more sub-tasks, the retrieval engine 110 is configured to perform a ^-nearest neighbor (&NN) search computation to search through the cache 150 for k most similar key vectors stored in the cache 150 that are most similar to the query vector according to some similarity measure. In doing so, the task execution system 100 can identify, from among the plurality of cached subtasks, k most similar cached sub-tasks (where each cached sub-task corresponds to a key vector) to each sub-task specified in the plan that has been generated by using the task planning engine 105 for the task characterized in the prompt 102.

[0067] For some similarity7measures, e.g., Euclidean distance or Hamming distance or other distance measures, the most similar key vectors are those that are closest to the query vector (have the smallest similarity measure with the query7vector).

[0068] For some other similarity measures, e.g., inner product similarity or cosine similarity, the most similar key vectors are those that have the largest similarity measure with the query7vector.

[0069] A can generally be any positive integer, i.e., any integer greater than or equal to one, but is generally much smaller than the total number of cached sub-tasks. For example, if there are a million key vectors stored in the cache 150 (that each correspond to a different one ofAttorney Docket No. 56113-0758WO1 the plurality of cached sub-tasks), the value of k can be less than ten and, in some cases, can be less than five, or less than two.

[0070] In some cases, the cNN search computation can be an exact £NN search computation. That is, the retrieval engine 150 can compute a respective similarity' score between the query' vector and each key vector according to the similarity measure, and then select the k most similar key vectors to the query vector based on the respective similarity scores.

[0071] For example, when the similarity measure is inner product similarity or cosine similarity, key vectors that have the highest similarity' scores, or key vectors that have similarity scores greater than a given value can be selected.

[0072] Alternatively, as another example, when the similarity measure is Euclidean distance or Hamming distance or other distance measures, key vectors that have the lowest similarity scores, or key vectors that have similarity scores lower than a given value can be selected.

[0073] In some other cases, the kNN search computation can be approximate &NN search computation. For example, the retrieval engine 110 can apply can any known search space pruning, quantization, or other approximation techniques to enable faster and more computationally efficient computation of the respective similarity scores between the query vector and the key vectors. A few example quantization techniques that can be used by7the retrieval engine 150 are described in more detail in US patent US10719509 entitled “Hierarchical quantization for fast inner product search"’ and in US patent US 10255323 entitled “Quantization-based fast inner product search”.

[0074] In some implementations, the corresponding sub-task input 1 18A-N each have a fewshot prompt format, where a few, e.g., 1 to 3, 1 to 5, or 1 to 10, cached examples of the k most similar cached sub-tasks are arranged according to a predetermined syntax, followed by the actual sub-task prompt that corresponds to the sub-task.

[0075] That is, for each sub-task, the corresponding sub-task input 118A-N includes (i) the sub-task prompt 116A-N that corresponds to the sub-task and that is generated by using the task planning engine 105 and (ii) for each of the k most similar cached sub-tasks, one or more cached examples of the most similar cached sub-task that are stored in association with the most similar cached sub-task in the cache 150.

[0076] For example, each sub-task input 118A-N can be in the format of:# First example (first most similar cached sub-task).< first cached sub-task prompt > # Historic sub-task prompt for the first most similar cached sub-task.Attomey Docket No. 56113-0758WO1< first cached intermediate result > # Historic intermediate result for the first most similar cached sub-task.# Second example (second most similar cached sub-task).< second cached sub-task prompt > # Historic sub-task prompt for the second most similar cached sub-task.< second cached intermediate result > # Historic intermediate result for the second most similar cached sub-task.# Sub-task to perform.< sub-task prompt for the sub-task >

[0077] In this example, statements that follow a # are not included in the prompt (or are optional) and are only included in the example for explanation, < first cached sub-task prompt > represents the historic sub-task prompt for one of the k most similar cached subtasks which is stored in association with the most similar cached sub-task in the cache 150, < first cached intermediate result > represents the historic intermediate result for one of the k most similar cached sub-tasks which is stored (as part of a cached example) in association with the most similar cached sub-task in the cache 150, the indicates that the Second example is followed by one or more additional examples in the same format, and < sub-task prompt for the sub-task > represents the sub-task prompt 116A-N that corresponds to the subtask and that is generated by using the task planning engine 105 based on the prompt 102.

[0078] In this example, a sub-task input 118 A-N can include as few as one example (that is, there is no Second example). Moreover, in this example, the first and second most similar cached sub-tasks may be, but need not be. different ones of the plurality of cached sub-tasks. For example, K can be equal to one, and a sub-task input 118A-N can include two different cached examples that are stored in association with the same, top-1 most similar cached subtask in the cache 150. As another example, K can be equal to two, and a sub-task input 118A-N can include a cached example that is stored in association with one of the top-2 most similar cached sub-tasks in the cache 150. and a cached example that is stored in association with another one of the top-2 most similar cached sub-tasks in the cache 150.

[0079] After having generated the sub-task inputs 118A-N that correspond to the one or more sub-tasks specified in the plan generated by using the task planning engine 105, for each of the one or more sub-tasks, the task execution system 100 processes the sub-task input 118A-Attorney Docket No. 56113-0758WO1N that corresponds to the sub-task using the generative neural network 120 to generate a corresponding intermediate result for the sub-task.

[0080] In practice, generating the corresponding intermediate result for the sub-task involves generating, by the generative neural network 120, an output sequence of tokens selected from the vocabulary. As mentioned above the output sequence of tokens can represent data in any of a variety of modalities, including, for example, text, image, video, or audio.

[0081] Having obtained the corresponding intermediate result for each sub-task in this way, the task execution system 100 then generates the result 126 for the task characterized by the prompt 102 based on the corresponding intermediate results for the one or more sub-tasks. The result 126 can be generated in any of a variety' of ways, depending on the plan that specifies the one or more sub-tasks for performing the task.

[0082] For example, the result 126 can include a concatenation of a first intermediate result for a first sub-task and a second intermediate result for a second sub-task. As another example, the result 126 can include data derived from a corresponding intermediate result for a sub-task, e.g., an output sequence of text generated by using the generative neural network 120 to process data that includes the corresponding intermediate result for the sub-task.

[0083] As a particular example, when the generative neural network 120 is configured as an auto-regressive Transformer-based neural network, the generative neural network 120 can proceed to generate the remaining portion of the result 126 conditioned on the obtained intermediate results and any previous portion of the result 126 that has already been generated by the generative neural network 120.

[0084] The task execution system 100 can then provide the result 126 as an output, e.g., by providing the result 126 to be presented on a display of a client device to the user who submitted the instruction data for the task. Additionally or alternatively, after generating the result 126 (and, analogously, the intermediate results for the sub-tasks), the task execution system 100 can provide the result 126 (and, analogously, the intermediate results for the subtasks) to one or more external systems for storage or further processing.

[0085] In some implementations, for each of the one or more sub-tasks determined by using the task planning engine 105 based on the prompt 102, the task execution system 100 can add a copy of the corresponding intermediate result generated by using the generative neural network 120 as a new cached example to the cache 150. Similarly, the task execution system 100 can add a copy of the corresponding sub-task prompt generated by using the task planning engine 105 to the cache 150.Attorney Docket No. 56113-0758WO1

[0086] For example, the task execution system 100 can process the corresponding sub-task prompt using the embedding neural network to generate a new key vector, which is then compared against the key vectors stored in the cache 150. If an existing key vector in the cache 150 matches the new key vector, i.e., their similarity measure satisfies a threshold similarity measure, then the task execution system 100 can determine that the sub-task is reasonably similar to an already cached sub-task and, correspondingly, add the corresponding intermediate result as another example for the already cached sub-task (e.g., as one of subtasks 1-N in the example of FIG. 1).

[0087] Alternatively, if no existing key vector in the cache 150 matches the new key vector, i.e., none of the similarity measures computed between the plurality of cached sub-tasks and the new key vector satisfies a threshold similarity measure, then the task execution system 100 can determine that the sub-task is reasonably different from the already cached sub-tasks and, correspondingly, add the corresponding intermediate result as an example for a new cached sub-task (e.g., as sub-task N 1 in the example of FIG. 1), and store the corresponding sub-task prompt in association with the new cached sub-task.

[0088] In some implementations, for each of the one or more sub-tasks, prior to storing the corresponding sub-task prompt, the corresponding intermediate result, or both, the task execution system 100 first determines whether a quality' of the corresponding sub-task prompt or the corresponding intermediate result meets a quality threshold.|00089| For example, the quality can be determined based on aspects such as latency, size (e.g., length), accuracy of the corresponding sub-task prompt or the corresponding intermediate result, and then compared against the quality' threshold. As another example, whether a quality of the corresponding intermediate result meets the quality' threshold can be determined from a user input, e.g.. a user input that is submitted by a user on the client device after being presented with the result 126.

[0090] If the quality meets the quality7threshold, then the task execution system 100 adds the corresponding sub-task prompt, the corresponding intermediate result, or both to the cache 150. Alternatively, if the quality fails to meet the quality’ threshold, then the task execution system 100 refrains from adding the corresponding sub-task prompt and the corresponding intermediate result to the cache 150.

[0091] Through the use of the cache 150 and the retrieval engine 110, the task execution system 100 can improve the performance, efficiency, or both of the generative neural network 120 on a variety of tasks. For example, the quality (e.g., in terms of accuracy, relevancy, comprehensiveness, or another quality metric) of the results 126 generated by theAttorney Docket No. 56113-0758WO1 generative neural network 120 for the tasks that it is configured to perform can be improved. As another example, the inference latency or resource efficiency of the generative neural network 120 in generating the results 126 for the tasks can be improved. That is, a result can be generated more quickly, with reduced computing resource consumption (e.g., fewer processor cycles), or both compared with existing systems that do not make use of such a cache.

[0092] The few-shot prompting technique used by the task execution system 100 taps into the capabilities of the generative neural network 120, e.g., when it is configured as a Transformer-based language model neural network, enabling the generative neural network 120 to perform effectively on a wide range of tasks beyond their original training data, thereby enhancing its overall utility and applicability in many real-world applications.

[0093] For one, few-shot prompting allows the generative neural network 120 to quickly adapt to new sub-tasks by providing a few examples related to cached sub-task at hand. For another, few-shot prompting allows the generative neural network 120 to generalize to unseen examples or variations of the sub-task. Further, few-shot prompting facilitates incremental learning, where the generative neural network 120 can continuously improve its performance on new sub-tasks or adapt to evolving task requirements by providing additional examples as needed.

[0094] FIG. 2 is a flow diagram of an example process 200 for generating a result for a task. For convenience, the process 200 will be described as being performed by one or more systems of one or more computers located in one or more locations. For example, a task execution system, e.g., the task execution system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 200.

[0095] The system includes a generative neural network. The generative neural network can have any appropriate neural network architecture that allows the neural network to map an input sequence of tokens from a vocabulary to an output sequence of tokens from the vocabulary.

[0096] The system accesses a cache for a plurality of cached sub-tasks. The cache stores, for each of the plurality of cached sub-tasks: (i) a key vector for the cached sub-task and (ii) a plurality of cached examples for the cached sub-task, w here each cached example includes a historic intermediate result generated by using the generative neural network for the cached sub-task, and, optionally, (iii) a cached sub-task prompt for the cached sub-task, (iv) a context for the cached sub-task that includes metadata that relates to the cached sub-task, or both (iii) and (iv).Attorney Docket No. 56113-0758WO1

[0097] The system receives a prompt to perform a task using a generative neural network (step 202). As mentioned above the prompt can include instruction data that characterizes the task to be performed by the system and task data that will be processed by the system in order to perform the task.

[0098] The system obtains a pian for performing the task by completing one or more subtasks (step 204). The plan specifies the one or more sub-tasks. In some implementations, the system can do this by using a task planning engine that is implemented within the system (as mentioned above) to generate one or more sub-task prompts based on the prompt. Each subtask prompt corresponds to a sub-task. Each sub-task has a corresponding intermediate result. In some other implementations, the system can receive the plan that specifies the one or more sub-tasks from another system, e.g., a remote system that implements such a task planning engine.

[0099] In some implementations where the task planning engine generates the one or more sub-task prompts by interacting with the generative neural network, the task planning engine can similarly make use of the cache by performing a search in the cache for cached sub-tasks that relate to task decomposition. That is, each task planning example as mentioned above in FIG. 1, which is provided as part of the input sequence for processing by the generative neural network to generate a sub-task prompt, can include cached examples of one or more cached sub-tasks that relate to task decomposition stored in the cache.|000100| As a concrete example, the prompt received by the system can include instruction data that characterizes an automated assistance task to be performed by the system. For example, the prompt can be “complete a form on [URL]” where [URL] identifies a particular universal resource locator (URL) of a website at which the form can be found. [000101] In this example, by using the task planning engine, the system can generate three sub-task prompts that corresponds respectively to three sub-tasks that need to be completed in order to perform the automated assistance task. The three sub-task prompts include: a first sub-task prompt of “open a web browsing application,” a second sub-task prompt of “enter [URL] in the address bar of the web browsing application,” and a third subtask prompt of “enter [text] into the text entry field on the webpage,” where [text] defines the content that should be entered into the form.[000102] For each sub-task that is specified by the plan for performing the task, the system performs steps 206-212 to generate an intermediate result for the sub-task.[000103] The system performs a search in the cache for k most similar cached sub-tasks to the sub-task according to a similarity measure (step 206). K can generally be any positiveAttorney Docket No. 56113-0758WO1 integer, i.e., any integer greater than or equal to one. Performing the search in the cache is discussed further below with reference to FIG. 3, which is a flow diagram of sub-steps 302- 306 of step 206 of the process of FIG. 2.[000104] The system generates a query vector for the sub-task (step 302). In some implementations, the system generates the query vector for the sub-task by processing the sub-task prompt that corresponds to the sub-task and, optionally, a context for the sub-task that includes metadata that relates to the sub-task using an embedding neural network.[000105] For each of the plurality of cached sub-tasks that are stored in the cache, the system computes a similarity measure between (i) the key vector for the cached sub-task and (ii) the query vector for the sub-task (step 304). The similarity measure can be any appropriate similarity measure. Examples of similarity measures include inner product similarity’, cosine similarity, Euclidean distance, and Hamming distance, to name just a few. [000106] The system determines the k most similar cached sub-tasks based on the similarity measures (step 306). The system does this by determining k most similar key vectors to the query vector, where each of the k most similar key vectors corresponds to one of the k most similar cached sub-tasks.[000107] For some similarity’ measures, e.g., Euclidean distance or Hamming distance or other distance measures, the most similar key vectors are those that are closest to the query vector (have the smallest similarity measure with the query vector). For some other similarity measures, e.g.. inner product similarity or cosine similarity, the most similar key vectors are those that have the largest similarity measure with the query vector.[000108] In the above example of the automated assistance task, for the first sub-task corresponding to the first sub-task prompt of “open a web browsing application,” the system can for example determine one or more of: a cached sub-task to open a calendar application, a cached sub-task to open a camera application, a cached sub-task to open a messaging application, and so on, as the k most similar cached sub-tasks.[000109] The system selects, from the plurality of cached examples stored in the cache in association with each of the k most similar cached sub-tasks, one or more cached examples (step 208). In the above example of the automated assistance task, for the first sub-task, the system can for example select one or more of: a historic intermediate result of “click the logo at coordinates (x, y),” a historic intermediate result of “click the logo for the messaging application,” and so on, as the one or more cached examples.[000110] In practice, the system can select any number of cached examples for each of the k most similar cached sub-tasks in many different ways.Atorney Docket No. 56113-0758WO1[000111] For example, the system can select a fixed number of cached examples for each of the k most similar cached sub-tasks. Alternatively, the numbers of cached examples selected for the k most similar cached sub-tasks vary from one cached sub-task to another. [000112] As another example, for each of the k most similar cached sub-tasks, the system can select the one or more cached examples with some randomness, e.g., through randomly sampling.[000113] As another example, for each of the k most similar cached sub-tasks, the system can select the one or more cached examples in accordance with the chronological order in which the cached examples are stored into the cache, e.g., by selecting the oldest cached examples or the newest cached examples.[000114] As another example, for each of the k most similar cached sub-tasks, the system can select the one or more cached examples in accordance with length of the cached examples (e.g., in accordance with total number of tokens included in each cached example), e.g., by selecting the longest cached examples or the shortest cached examples.[000115] As another example, for each of the k most similar cached sub-tasks, the system can generate a sub-key vector for each of the cached examples stored in the cache in association with the cached sub-task (e.g., by using an embedding neural network to process the cached examples), and then select the one or more cached examples based on the similarity measures computed between the subkey vectors for the cached examples and the query vector for the sub-task.[000116] The system generates a sub-task input for the sub-task based on (i) the subtask prompt that corresponds to the sub-task and (ii) for each of the k most similar cached sub-tasks, the one or more selected cached examples (step 210). In some implementations, the sub-task input can have a few-shot prompt format, where the selected cached examples of the k most similar cached sub-tasks are arranged according to a predetermined syntax, followed by the actual sub-task prompt that corresponds to the sub-task.[000117] In the above example of the automated assistance task, for the first sub-task, the system can for example generate a sub-task input for the first sub-task that is in the format of:# First example (first most similar cached sub-task). open an email application # Historic sub-task prompt for the first most similar cached sub-task. click the logo at coordinates (x, y) # Historic intermediate result for the first most similar cached sub-task.Attorney Docket No. 56113-0758WO1# Second example (second most similar cached sub-task). open a messaging application # Historic sub-task prompt for the second most similar cached sub-task. click the logo for the messaging application # Historic intermediate result for the second most similar cached sub-task.# Sub-task to perform. open a web browsing application[000118] The system processes the sub-task input for the sub-task using the generative neural network to generate a corresponding intermediate result of the sub-task (step 212). In practice, generating the corresponding intermediate result for the sub-task involves generating, by the generative neural network, an output sequence of tokens selected from the vocabulary. In the above example of the automated assistance task, for the first sub-task, the system can for example generate the following output sequence of tokens as the corresponding intermediate result: “click the logo for the web browsing application at coordinates (x, y).”[000119] After having repeatedly performed steps 206-212 to generate the corresponding intermediate result for each of the one or more sub-tasks obtained in step 204, the system generates a result for the task based on the corresponding intermediate result generated for each of the one or more sub-tasks (step 214).[000120] In the above example of the automated assistance task, the system can for example execute actions within an execution environment, e.g., an action that simulates a user input, where each action is defined by a respective one of the three intermediate results generated by the generative neural network for a corresponding sub-task. The actions can be executed one after another, as the corresponding intermediate results are being generated in accordance wi th the order of the three sub-tasks specified in the plan. Accordingly, the result for the automated assistance task can for example be a confirmation of the completion of the form on the website.[000121] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to performAttorney Docket No. 56113-0758WO1 particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.[000122] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.[000123] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.[000124] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single fileAttorney Docket No. 56113-0758WO1 dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.[000125] In this specification, the term “database’' is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all. and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.[000126] Similarly, in this specification the term “engine’' is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.[000127] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [000128] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory’ or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.Attorney Docket No. 56113-0758WO1[000129] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.[000130] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.10001311 Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.[000132] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a J AX framework.[000133] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application serv er, or that includes a front end component, e.g.. a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.Attorney Docket No. 56113-0758WO1Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.[000134] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g.. for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.[000135] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.[000136] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.[000137] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require theAttorney Docket No. 56113-0758WO1 particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.[000138] What is claimed is:

Claims

Attorney Docket No. 56113-0758WO1CLAIMS1. A method performed by one or more computers, the method comprising: maintaining a cache, the cache storing, for each of a plurality of cached sub-tasks:(i) a key vector for the cached sub-task,(ii) a plurality of cached examples of the cached sub-task, each cached example comprising an intermediate result for the cached sub-task; receiving a prompt to perform a task using a generative neural network; obtaining a plan for performing the task by completing one or more sub-tasks; generating an intermediate result of each sub-task, comprising: performing a search in the cache for one or more similar cached sub-tasks to the sub-task based on using the key vector for each of the plurality of cached sub-tasks; selecting, from the plurality of cached examples stored in the cache for each of the one or more similar cached sub-tasks, one or more cached examples; generating, based on the one or more selected cached examples, an input for the sub-task; and processing the input for the sub-task using the generative neural network to generate the intermediate result for the sub-task; and generating a result for the task based on the intermediate result of each sub-task.

2. The method of claim 1, wherein the key vector for the cached sub-task is generated based on a historic input for the cached sub-task.

3. The method of any one of claims 1-2, wherein the key vector for the cached sub-task is generated based on (i) a historic prompt for the cached sub-task, (ii) a context for the cached sub-task, or both (i) and (ii).

4. The method of claim 3, wherein the cache also stores, for each of the plurality of cached sub-tasks:(i) the historic prompt for the cached sub-task, (ii) the context for the cached sub-task, or both (i) and (ii).

5. The method of any one of claims 1-4, wherein obtaining the plan for performing the task comprises:Attorney Docket No. 56113-0758WO1 performing a search in the cache for one or more cached sub-tasks that relate to task decomposition; and generating a prompt for each of the one or more sub-tasks by using the generative neural network based on processing an input sequence that comprises one or more cached examples of the one or more cached sub-tasks that relate to task decomposition.

6. The method of any one of claims 1-5, wherein performing the search in the cache for one or more similar cached sub-task to the sub-task comprises: generating a query vector for the sub-task; for each of the plurality of cached sub-tasks, computing a similarity measure between (i) the key vector for the cached sub-task and (ii) the query vector for the sub-task; and determining the one or more similar cached sub-tasks based on the similarity measures.

7. The method of any one of claims 1-6, wherein the input for the sub-task comprises: (i) a prompt for the sub-task and (ii) the one or more selected cached examples.

8. The method of any one of claims 1-7, further comprising generating a new cached example of a new cached sub-task, wherein the new cached example comprises an intermediate result for the new cached sub-task that is generated by using the generative neural network.

9. The method of claim 8, wherein generating the new cached example comprises: determining whether a quality of the intermediate result for the new cached sub-task that is generated by using the generative neural network satisfies a quality threshold.

10. The method of any one of claims 8-9, further comprising storing the new7cached example in the cache.

11. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of the methods of any of the preceding claims 1-10.Atorney Docket No. 56113-0758WO112. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective operations of any one of the methods of any of the preceding claims 1-10.

Images