Machine-generated training examples for training machine-learned models

Abstract

An example method includes obtaining a reference trace describing interactions between a machine-learned agent system and an environment, wherein the reference trace includes a reference sequence of interaction objects associated with performance of a task, each respective reference interaction object of the reference sequence of interaction objects corresponding to a respective reference environment state and a respective reference action executed on the respective reference environment state. The example method includes generating, using a machine-learned trace generation system, a forecasted trace based on the reference trace, the forecasted trace comprising a forecasted sequence of interaction objects that is predicted to continue the reference sequence of interaction objects from a branching position toward performance of the task, each respective forecasted interaction object of the forecasted sequence of interaction objects corresponding to a respective forecasted environment state and a respective forecasted action executed on the respective forecasted environment state. The example method includes training the machine-learned agent system using the forecasted trace sequence.

Description

MACHINE-GENERATED TRAINING EXAMPLES FORTRAINING MACHINE-LEARNED MODELSFIELD

[0001] The present disclosure relates generally to machine learning processes and machine-learned devices and systems. More particularly, the present disclosure relates to machine-generated training examples for training machine-learned models.BACKGROUND

[0002] A computer can receive input(s). The computer can execute instructions to process the input(s) to generate output(s) using a parameterized model. The computer can obtain feedback on its performance in generating the outputs with the model. The computer can generate feedback by evaluating its performance. The computer can receive feedback from an external source. The computer can update parameters of the model based on the feedback to improve its performance. In this manner, the computer can iteratively “learn” to generate the desired outputs. The resulting model is often referred to as a machine-learned model.SUMMARY

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0004] In an aspect, the present disclosure provides a first example method. The first example method includes obtaining a reference trace describing interactions between a machine-learned agent system and an environment, wherein the reference trace includes a reference sequence of interaction objects associated with performance of a task, each respective reference interaction object of the reference sequence of interaction objects corresponding to a respective reference environment state and a respective reference action executed on the respective reference environment state. The first example method includes generating, using a machine-learned trace generation system, a forecasted trace based on the reference trace, the forecasted trace comprising a forecasted sequence of interaction objects that is predicted to continue the reference sequence of interaction objects from a branching position toward performance of the task, each respective forecasted interaction object of theforecasted sequence of interaction objects corresponding to a respective forecasted environment state and a respective forecasted action executed on the respective forecasted environment state. The first example method includes training the machine-learned agent system using the forecasted trace sequence.

[0005] In an aspect, the present disclosure provides a second example method. The second example method includes generating a variant trace based on the baseline trace. In the second example method, generating the variant trace includes providing, to the machine- learned trace generation system, the seed representation to generate the variant trace that includes a different environment state from the reference trace, wherein the different environmental state is generated using one or more inferences obtained from the machine- learned trace generation system. In the second example method, generating the variant trace includes determining, based on a goal state of the reference trace associated with the task, a goal state of the variant trace. The second example method includes processing, using the machine-learned agent system, at least a portion of the variant trace to generate an interaction output. The second example method includes evaluating the interaction output using the goal state of the variant trace. The second example method includes training the machine-learned agent system based on the evaluation of the interaction output.

[0006] In an aspect, the present disclosure provides one or more example non- transitory, computer readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations, wherein the operations include the first example method or the second example method.

[0007] In an aspect, the present disclosure provides an example computing system. The example computing system includes one or more processors. The example computing system includes one or more non-transitory, computer readable media storing instructions that, when executed by the one or more processors, cause the example computing system to perform operations, wherein the operations include the first example method or the second example method.

[0008] Other example aspects of the present disclosure are directed to other systems, methods, apparatuses, tangible non-transitory computer-readable media, computer program products, and devices for performing functions described herein. These and other features, aspects, and advantages of various implementations will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrateimplementations of the present disclosure and, together with the description, help explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure l is a block diagram of an example system according to example implementations of aspects of the present disclosure.

[0010] Figure 2 is a block diagram of an example system using forecasted traces according to example implementations of aspects of the present disclosure.

[0011] Figure 3 is a block diagram of an example system using forecasted traces according to example implementations of aspects of the present disclosure.

[0012] Figure 4 is a block diagram of an example system using forecasted traces according to example implementations of aspects of the present disclosure.

[0013] Figure 5 is a block diagram of an example system generating traces according to example implementations of aspects of the present disclosure.

[0014] Figure 6 is a block diagram of an example labeled reference trace according to example implementations of aspects of the present disclosure.

[0015] Figure 7 is a block diagram of an example interaction object according to example implementations of aspects of the present disclosure.

[0016] Figure 8 is a block diagram of an example interaction object according to example implementations of aspects of the present disclosure.

[0017] Figure 9 is a block diagram of an example replay buffer according to example implementations of aspects of the present disclosure.

[0018] Figure 10 is a block diagram of an example system generating variant traces according to example implementations of aspects of the present disclosure.

[0019] Figure 11 is a block diagram of an example system generating variant traces according to example implementations of aspects of the present disclosure.

[0020] Figure 12 is a flow chart diagram illustrating an example method for training a machine-learned model according to example implementations of aspects of the present disclosure.

[0021] Figure 13 is a block diagram of an example processing flow for using machine-learned model(s) to process input(s) to generate output(s) according to example implementations of aspects of the present disclosure.

[0022] Figure 14 is a block diagram of an example sequence processing model according to example implementations of aspects of the present disclosure.

[0023] Figure 15 is a block diagram of an example technique for populating an example input sequence for processing by a sequence processing model according to example implementations of aspects of the present disclosure.

[0024] Figure 16 is a block diagram of an example model development platform according to example implementations of aspects of the present disclosure.

[0025] Figure 17 is a block diagram of an example training workflow for training a machine-learned model according to example implementations of aspects of the present disclosure.

[0026] Figure 18 is a block diagram of an inference system for operating one or more machine-learned model(s) to perform inference according to example implementations of aspects of the present disclosure.

[0027] Figure 19 is a block diagram of an example networked computing system according to example implementations of aspects of the present disclosure.

[0028] Figure 20 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure.

[0029] Figure 21 is a block diagram of an example computing device according to example implementations of aspects of the present disclosure.

[0030] Figure 22 is a flow chart diagram illustrating an example method for training a machine-learned model according to example implementations of aspects of the present disclosure.

[0031] Figure 23 is a flow chart diagram illustrating an example method for using a machine-learned model according to example implementations of aspects of the present disclosure.

[0032] Figure 24 is a flow chart diagram illustrating an example method for training a machine-learned model according to example implementations of aspects of the present disclosure.DETAILED DESCRIPTION

[0033] Generally, the present disclosure is directed to techniques for generating training examples using machine-learned models for training the same or different machine- learned models. For example, a machine-learned agent system can be configured to interact with other systems and other real-world environments. Training the agent system to interact with such environments can involve obtaining exemplars or demonstrations of how to perform various tasks. To increase the efficacy of these exemplars, example implementationsof the present disclosure can use parts of the exemplars as a seed state to effectively simulate numerous other interactions with the same or similar environments. In this manner, for instance, a training system can augment existing training data to multiply the effect of the training data.

[0034] For example, the augmented training data may be used to more efficiently train or retrain machine-learned models of the machine-learned agent system further improving the control of or interaction with technical systems in real-world environments such as, without limitation, for example automated control of technical tasks via actions interacting with a digital environment of mobile devices or computers; navigating one or more mobile applications in a digital environment of a mobile device for performing a technical task associated therewith; controlling a robotic system interacting with a physical environment; control and navigation of an autonomous or semi-autonomous vehicle interacting with a physical environment; and / or control of any other technical task or device within a digital and / or physical environment and the like. The augmented training data can be used to improve the robustness of the machine-learning models of the machine-learned agent system for more efficiently controlling said technical systems whilst handling a broader range of scenarios or situations previously unseen in the original training dataset.

[0035] Traditional approaches to training such agents often rely predominantly on real-world interactions, which can be costly and time-consuming. For instance, as the agent is learning, the process of trial -and-error in a real deployment context can lead to undue delays, processing errors, etc. Further, the data is generally restrained to real-time collection dependent on the environment’s responsiveness to the agent. For instance, a robot interacting with a physical environment may be constrained by the robot’s physical speed, agility, etc., such that collecting extensive real-world data can be time-consuming. The interactions may also not be easily parallelizable, as, for instance, a single robot cannot be in two places at one time, an external website or platform may not allow multiple of or a large number of concurrent sessions, etc.

[0036] Advantageously, the present disclosure provides solutions to these challenges by leveraging generative capabilities of machine-learned models to simulate realistic environment interactions, thereby reducing the dependency on real-world data collection. The simulations can be grounded in a portion of reference data (e.g., real -world or ground truth data). An example uses a machine-learned trace generation system to create forecasted traces from a reference trace or a portion thereof. The reference trace can describe the agent’s past interactions with an environment, the states of the environment subject to the action, etc. Theforecasted trace can predict future interactions with the environment, simulating how possible actions might change the environment state over one or multiple iterations (e.g., simulating multiple actions performed over multiple subsequent environment states). This forecasted trace can be used as a training example in addition to the reference trace. Because the forecasted trace may diverge somewhat from the actual actions performed and environment states recorded in the complete reference trace, a single reference trace can branch into many simulated interaction traces that diverge but remain grounded in some amount of common context (e.g., a prefix portion of a reference trace). Because they can diverge, these forecasted traces can provide an additional training effect. This, in turn, can allow the agent to train in more simulated contexts that closely mirror real-world conditions without the associated risks and costs of, for example, requiring agents to operate in real environments at training time or dealing with any subsequent adverse real-world consequence.

[0037] More particularly, example aspects of the present disclosure provide example techniques for enhancing the training of machine-learned agents through the use of a “replay buffer.” An example replay buffer can be understood as a cache or storage of past interactions that an agent has had with its environment. For instance, as an agent interacts with an environment, the agent can maintain a cache of its interactions (e.g., a fixed-size rolling buffer). At each turn (e.g., decision point), a machine-learned trace generation system can generate one or more forecasted traces that continue from the sequence stored in the replay buffer.

[0038] An example machine-learned trace generation system can receive, as input, the historical data from the replay buffer and use it to predict a sequence of interactions that might occur from a given starting point, or branching position, in the reference trace. The quality and effectiveness of these forecasted traces can be continuously improved by training the trace generation system based on how closely the forecasted interactions match the actual interactions that occur when the agent continues to act along the reference trace. This forecasting can be implemented online. For instance, while the agent operates, the machine- learned trace generation system can generate one or multiple traces to help it foresee possible outcomes of different decisions. The forecasting can be performed offline. For instance, after the agent records a complete reference trace, forecasts can be generated from sampled branching points to augment the training dataset. For example, the augmented training dataset used to improve the robustness of the machine-learning models of the machine-learned agent system for more efficiently controlling a technical system whilst handling a broader range of scenarios or situations previously unseen in the original training dataset.

[0039] The machine-learned trace generation system can use the same or different machine-learned models as the machine-learned agent system. In an example implementation, the machine-learned trace generation system and the machine-learned agent system use the same model, such that as the model is trained over a diverse set of synthesized traces (via training the agent system), it also becomes more adept at forecasting the outcomes of diverse scenarios. In an example implementation, the machine-learned trace generation system and the machine-learned agent system use different models, such that each can learn to particularly excel at each model’s respective task. The agent system model can learn to specifically excel at deci si on -making for completing tasks (possibly augmented with inputs containing forecast(s) from the trace generation system). The trace generation system model can learn to specifically excel at forecasting how an environment will change state after one or more actions are executed upon it by an agent system.

[0040] In this manner, for instance, example implementations of a machine-learned agent system according to the present disclosure can obtain multiple benefits from the forecasted traces. The agent system can train on the forecasted traces. And the agent system can receive real-time forecasts to help it explicitly account for downstream results of current decisions.

[0041] In another aspect, the present disclosure provides for improved training data augmentation by generating diverse variants of a seed reference trace. For instance, based on an initial seed representation of a reference trace, a machine-learned trace generation system can generate a plurality of new traces that are similar to the reference trace in some aspects but different from the reference trace in other aspects. This variant generation can significantly enhance the training signals within an existing dataset by not only increasing its size but also its diversity. By generating new scenarios that are variations of the original, the dataset becomes richer and more robust, providing a more comprehensive training dataset for the agent. These variations can be not just random but strategically generated to maintain some core attributes of the original data while altering others, thus ensuring that the original training signal (e.g., label, task, etc.) from the baseline reference trace is not lost.

[0042] A practical example of this could involve an agent trained to improve the human-machine interface by assisting the performance of computing device processing tasks. For instance, an example task can include interacting with applications hosted in a web environment. For instance, a practical example could involve an agent trained to improve the human-machine interface by assisting the performance of online shopping tasks or other tasks on a mobile device or computing system. Suppose the original dataset consists ofinteractions related to buying coffee on an e-commerce platform. Using the variant generation techniques described in the present disclosure, the dataset could be augmented to include interactions related to buying lunch items, where the basic structure of the task (navigating a website or mobile application, adding items to a cart, checking out, etc.) remains the same, but the specifics of the items being purchased are different. Alternatively, or additionally, features of the mobile app or website can be changed (e.g., different user interface layout, different workflow, etc.). This can help to train the agent to generalize its skills. For example, this provides a technical purpose of assisting users navigate mobile device / computing device by significantly reducing the required user navigation input when performing tasks in the digital environment of a mobile device, which further improves the human-machine interface. For example, this may be achieved by the agent generalizing its skills using the variant generation techniques enabling it to more effectively automatically controlling the input to and navigation of one or more websites and / or one or more mobile applications in the digital environment of the mobile device in the performance of tasks initiated or instructed by users.

[0043] Another example could involve an agent trained to further improve the human-machine interface by performing hands-free tasks on a mobile device such as, without limitation, for example as mapping, navigating and / or providing assistance / directions for walking, cycling, commuting and / or driving tasks. Suppose the original dataset consists of interactions related to a morning commute to travel / navigate from a first location to a second location in which travel options include bike / electric scooter, bus and / or train. Using the variant generation techniques described in the present disclosure, the dataset could be augmented to include interactions related to commuting in the evening, where the basic structure of the task (navigating one or more websites (e.g., bike / electric scooter hire / locations, train and / or bus operators) or mobile application (e.g., bike / electric scooter availability, train and / or bus ticketing / scheduling applications), identifying locations / schedules / timetables for making connections, identifying quickest and cost effective modes of travel and routes, adding any necessary bike / electric scooter reservations, train / bus ticketing items to a cart, checking out, and navigating to the respective bike / electric scooter parking locations, train stations / bus stops whilst ensuring connections are made etc.) remains the same, but the specifics of when the modes of transport, items being purchased and / or urgency of travel may be different. Alternatively, or additionally, features of the train / bus mobile apps or websites can be changed (e.g., different user interface layout, different schedules, different workflow, etc.) and / or available modes of transport can bechanged (e.g., car, taxi, plane, train and / or bus rather than bike / electric scooter). This can help to train the agent to generalize its skills.

[0044] In an example, this can be implemented using one or more multiple content generation models. For instance, a trace generation system can use an image generation model to generate rendered images of a user interface according to an instruction (e.g., based on the seed representation and altered according to an additional instruction). For example, textual content and image content describing an original mobile application interface can be provided as input along with an instruction to generate one or more interface renders “where the product is a sandwich instead of a cup of coffee.” The image generation model can receive this input, attend over the textual and image inputs, and predict likely image data outputs based on the inputs. For instance, a diffusion-based model can iteratively refine an image by executing a denoising operation conditioned on the inputs. A natively multimodal transformer-based model can regress image data tokens representing a portion of an image depicting the rendered interface. In an example, a trace generation system can generate a new rendered interface for each environmental state in a trace of states, such as by rendering a first interface state, predicting an action to perform on that state, rendering a next interface state reactive to the action, predicting another action, etc.

[0045] In another example, a trace generation system can use a code generation model to generate computer code that, when processed by a code interpreter, can be executed to render a user interface. In this manner, for instance, multiple user interface states can be simulated based on this initial generation operation, as once generated, the executable interface code can itself provide state updates, without necessarily requiring a generative machine-learned model to predict how the environment will react. In an example, a trace generation system can obtain a new rendered interface for each environmental state in a trace of states, such as by rendering a first interface state, predicting an action to perform on that state, rendering a next interface state reactive to the action, predicting another action, etc.

[0046] Although described in the context of navigating a mobile application, it is to be understood that other agent-based tasks can be performed. Example agent systems can operate to control robotic platforms, autonomous or semi -autonomous vehicles, computer operating systems, server systems, embedded systems, etc.

[0047] Example implementations of the present disclosure can provide a number of technical effects and benefits. For instance, example implementations can reduce a dependency on collecting vast amounts of real-world data, which can be resource-intensive and slow. Further, example implementations can decrease failure rates while performingonline learning in real-world environments, as more of the training can be accomplished for a given amount of reference data obtained in a controlled environment.

[0048] In an example aspect, the present disclosure provides example techniques for reducing the reliance on costly real-world interactions during the training of machine-learned agents, which can be resource-intensive and potentially failure-prone. Real-world data collection often requires substantial computational resources, as well as physical resources when dealing with robotic systems or specialized equipment. For instance, training an agent to navigate a physical environment typically involves numerous trials where the agent interacts with the environment, which can lead to wear and tear on robotic components or the consumption of power and other utilities. Additionally, interactions in digital environments, such as mobile applications or complex software systems, can incur costs related to the use of APIs, server time, client compute time, network costs, purchased services / item costs, etc. Further, real-world data collection can be slow, due to being capped at real-time data collection. Especially when agents interact with third-party systems, there may be no recourse to increase cycle time.

[0049] Advantageously, example implementations of the present disclosure can help mitigate these costs by leveraging trace generation systems to simulate realistic interactions, from which the agent can learn instead of simply relying on interactions with the real environment. By generating forecasted traces from a reference trace, the system can create a variety of potential future scenarios that remain grounded to real data. This approach allows the agent to explore these scenarios without the need for actual physical or digital interaction, thereby conserving resources. The use of a replay buffer can facilitate online implementations by managing a memory footprint (e.g., with a fixed-size buffer) and curating a relevant portion of the reference trace (e.g., only that portion recent enough to fit in the buffer).

[0050] Further, the ability to generate variations of existing interactions can help introduce additional diversity into the training data without the need for new real-world data collection. For example, if an agent has learned from interactions in a specific environment, the system can generate data for similar but slightly different environments, thus preparing the agent for a broader range of scenarios without the need to physically interact with each of those environments. This can be particularly beneficial in domains where each real-world interaction carries significant cost or delay. In summary, the techniques disclosed herein can provide a cost-effective and efficient alternative to traditional training methods that rely heavily on real-world data collection, potentially leading to quicker development cycles and more robust machine-learned agent systems.

[0051] Further, example techniques provide solutions for parallelizing training data accumulation. In traditional real-world data collection and interaction scenarios, each sequence of interactions typically requires sequential processing, where computational resources are dedicated to one interaction at a time. This can be highly resource-intensive, especially when dealing with complex environments that demand high-fidelity simulations. Further, some environments can have high latency (e.g., network-based environments), which when combined with a blocking workflow (e.g., a single-session limit) can render real-world data collection to be especially slow.

[0052] Advantageously, example implementations of the present disclosure provide for parallelizable generation of simulated traces. Multiple simulated interaction sequences can be processed simultaneously from the same or different seed states. By leveraging modem computational architectures that support parallel processing, such as multi-core processors or distributed computing systems, example implementations of the present disclosure can efficiently generate multiple forecasted traces concurrently. In an example, multiple traces can be generated by the same model instance, parallelized along a batch dimension. Multiple trace generations can leverage a shared context (e.g., obtained from a replay buffer), reducing a memory footprint or otherwise reducing a computational load when computing parallel generation streams for the respective trace generations. This approach may not only expedite the overall processing time but also optimize the utilization of available computational resources.

[0053] Furthermore, the use of parallel processing in generating variant traces from the existing dataset can also contribute to computational efficiency for training models and / or updating deployed models. By creating variations of original interactions simultaneously, the dataset can be expanded and diversified more quickly than if each training example were generated one at a time. This not only speeds up the data augmentation process but also reduces a cycle time between training cycles. This can allow for more rapid deployment of models into new environments.

[0054] A further example technical effect of example implementations of the present disclosure is the potential enhancement in computational efficiency by enabling machine- learned agents to forecast and plan for the downstream effects of their current actions, which may significantly decrease the occurrence of costly trial-and-error sessions when these agents are deployed in real-world or simulated environments. A machine-learned trace generation system can generate forecasted traces based on a reference trace. This forecasting abilityallows the agent to anticipate future scenarios and their potential outcomes based on its previous actions and the resultant states.

[0055] By employing this forecasting mechanism, the agent can effectively simulate various decision paths and their consequences before actually executing actions in a live setting. This preemptive simulation can be helpful for identifying optimal action sequences and avoiding less effective or potentially detrimental ones. For instance, in a scenario where an agent is tasked with navigating through a digital (or physical) environment, the ability to forecast the results of different navigational choices can help the agent to avoid routes that may lead to errors or inefficiencies, thereby reducing the computational resources required for error correction and re-navigation in real-time.

[0056] Moreover, the trace generation model described in the present disclosure can be particularly beneficial in complex decision-making environments where the consequences of actions are not immediately apparent. By understanding the potential long-term impacts of its decisions through simulated forecasts, the agent can make more informed choices that are likely to yield better outcomes. This strategic planning capability effectively reduces the need for engaging in extensive trial-and-error learning cycles, which are not only resourceintensive but also time-consuming.

[0057] The efficacy of this technique for long-horizon planning can be multiplied by efficient parallel generation of traces. The ability to generate multiple forecasted traces from a single branching point in the reference trace allows for a comprehensive exploration of possible futures in parallel. Maintaining a replay buffer can include maintaining, in memory, the input contexts (e.g., including cached latent model states, such as Key-Value (KV) states for a transformer block), which can facilitate rapid regeneration of new traces from the next branching point that shares the same context.

[0058] A technical effect of example implementations of the present disclosure is increased energy efficiency in performing operations using machine-learned models, thereby improving the functioning of computers implementing such models. For instance, example implementations can provide for more energy-efficient runtime execution or inference. In some scenarios, increased energy efficiency can provide for less energy to be used to perform a given task (e.g., less energy expended to maintain the model in memory, less energy expended to perform calculations within the model, etc.). In some scenarios, increased energy efficiency can provide for more task(s) to be completed for a given energy budget (e.g., a larger quantity of tasks, more complex tasks, the same task but with more accuracy or precision, etc.).

[0059] In another example aspect, example implementations can provide for more energy-efficient training operations or model updates. In some scenarios, increased energy efficiency can provide for less energy to be used to perform a given number of update iterations (e.g., less energy expended to maintain the model in memory, less energy expended to perform calculations within the model, such as computing gradients, backpropagating a loss, etc.). In some scenarios, increased energy efficiency can provide for more update iterations to be completed for a given energy budget (e.g., a larger quantity of iterations, etc.). In some scenarios, greater expressivity afforded by model architectures and training techniques of the present disclosure can provide for a given level of functionality to be obtained in fewer training iterations, thereby expending a smaller energy budget. In some scenarios, greater expressivity afforded by model architectures and training techniques of the present disclosure can provide for an extended level of functionality to be obtained in a given number of training iterations, thereby more efficiently using a given energy budget.

[0060] In this manner, for instance, the improved energy efficiency of example implementations of the present disclosure can reduce an amount of pollution or other waste associated with implementing machine-learned models and systems, thereby advancing the field of machine-learning and artificial intelligence as a whole. The amount of pollution can be reduced in toto (e.g., an absolute magnitude thereof) or on a normalized basis (e.g., energy per task, per model size, etc.). For example, an amount of CO2 released (e.g., by a power source) in association with training and execution of machine-learned models can be reduced by implementing more energy-efficient training or inference operations. An amount of heat pollution in an environment (e.g., by the processors / storage locations) can be reduced by implementing more energy-efficient training or inference operations.

[0061] Various example implementations are described herein with respect to the accompanying Figures.

[0062] Figure l is a block diagram of an example implementation of a machine- learned agent system 102 configured to interact with one or more environments 104. Machine-learned agent system 102 can output trace data to machine-learned trace generation system 106. The trace data can describe recorded states of environment 104 and actions performed on environment 104 in those states. Machine-learned trace generation system 106 can generate traces (e.g., forecasted traces, variant traces) to help improve the operation of machine-learned agent system 102. For instance, generated traces can be provided as input to machine-learned agent system 102 in an online manner for helping guide its decision-making. Generated traces can be provided as input to machine-learned agent system 102 in an offlinemanner for training in diverse scenarios. Each of machine-learned agent system 102 and machine-learned trace generation system 106 can use one or more machine-learned models 108. The model(s) used by each system can be distinct to each system or shared by the systems. Model(s) 108 can be executed on or by the respective systems or on behalf of the respective systems by a model host system.

[0063] Machine-learned agent system 102 can be or include one or more computing devices or systems. Such computing devices can be standalone or dedicated computing devices or integrated into other devices or systems (e.g., robotic platforms, mobile computing devices, multi-tenant servers, etc.).

[0064] Machine-learned agent system 102 can include various different input devices to interact with environments 104. Example input devices include network controllers, camera sensors, ranging or depth sensors, audio sensors, touch sensors (e.g., switched, capacitive, compliant, etc.), inertial measurement units, gyroscopes, etc. Other sensors can detect environmental conditions such as temperature, humidity, or motion.

[0065] Machine-learned agent system 102 can include various different output devices to interact with environment 104. Example output devices may encompass a wide range of hardware components that can facilitate interaction with both physical and digital environments. For instance, in scenarios where the agent system controls a robotic platform, the output devices might include mechanical actuators, robotic arms, or other movement- oriented mechanisms that allow the system to manipulate objects or navigate through space. In another example, in scenarios where the agent system controls an autonomous vehicle or platform, the output devices might include steering, gear changing / selection, accelerating, and / or braking mechanisms, or other driving / movement-oriented mechanisms that allow the system to control the autonomous vehicle or platform and / or navigate the autonomous vehicle or platform from at least first location to a second location. Alternatively, when the agent system is engaged with digital environments or computing platforms, the output devices could include display screens for presenting visual information, speakers for audio output, or network interfaces that enable the system to send commands or data to other digital systems. Additionally, the output devices might also involve various other components such as printers, signal emitters (e.g., radio, light, ultraviolet, etc.), or other hardware that can implement the agent’s decisions and actions in a manner that influences the state of environment 104. These devices are not limited to any specific type and may vary widely depending on the particular application and environment in which a given machine-learned agent system is deployed.

[0066] Environment(s) 104 can be or include a variety of settings in which a machine-learned agent system may interact to perform tasks. These environments may encompass both physical and digital environments. For instance, the agent system may control robotic platforms within a physical environment, where tasks could include navigation or object manipulation. Such interactions might involve the agent navigating a robotic platform through a warehouse to locate and transport items, or it could involve more precise tasks like controlling a robot assembling components on a production line.

[0067] In digital environments, an agent system can interact with or control computing platforms where data processing tasks or user interface interactions are required. This could include managing network operations within a server system, automating responses in a customer service platform, or handling file management tasks within an operating system. Additionally, the agent system may engage with embedded systems, which could involve control or other interactions within automotive computing systems or smart home devices, where tasks might range from optimizing system performance to ensuring user commands are executed correctly. The agent system can control or interact with server systems, such as to perform tasks related to maintaining server health, optimizing load balancing, or automating software deployment processes. In environments involving complex software systems, the agent could be tasked with testing software, where it might control or interact with different modules of the software to simulate user interactions or to identify potential bugs. An agent system can control or interact with mobile applications or web-based platforms. In such applications, the agent system might perform tasks such as navigating through the app, performing transactions, or personalizing user experiences based on previous interactions.

[0068] An environment 104 can be represented in one or more state data objects. A “state” of environment 104 can be a specific configuration or condition of the environment at a given point in time, which can be influenced by previous actions executed within that environment. This state may be captured and described in various forms depending on the nature of the environment and the tasks being performed by the machine-learned agent system.

[0069] For instance, if environment 104 is a digital platform, such as an e-commerce website, the state might include data objects that describe the current webpage display, items in a shopping cart, user login status, and other relevant data elements. Alternatively, if the environment is a physical one, like a robotic navigation scenario, the state might comprise spatial coordinates, sensor data describing the environment, the status of the path, the statusof physical objects within the reach of the robot or in the path of the robot, and the robot’s operational status. Alternatively, if the environment is a physical one, like an autonomous vehicle navigation scenario, the state might comprise spatial coordinates, sensor data describing the environment, the status of physical objects or vehicles in path of the autonomous vehicle, and the autonomous vehicle’s operational status.

[0070] In some implementations, the state includes a textual description of the environment. For instance, a machine-learned text generation model can process data representing the environment and generate a textual summary that describes the current state of the environment.

[0071] Each state of the environment can be associated with one or more interaction objects. Interaction objects may include a reference to a state of the environment and an action that has been or will be executed in that state.

[0072] A record of the interactions between machine-learned agent system 102 and environment 104 can be stored as a reference trace. A “trace” can be represented as a sequence of interaction objects that describe the interactions between an agent and an environment. Each interaction object within this sequence may correspond to a specific state of the environment and an action executed by the agent on that state. This sequence can be associated with the performance of a task. For instance, an end state for the trace can correspond to a completion of the task.

[0073] Machine-learned trace generation system 106 can be or include one or more computing devices or systems. Machine-learned trace generation system 106 can be implemented on one or more computing devices or systems shared with machine-learned agent system 102. Machine-learned trace generation system 106 can be implemented on one or more computing devices or systems that are not shared with machine-learned agent system 102. In an example, machine-learned trace generation system 106 is implemented locally to machine-learned agent system 102. In an example, machine-learned trace generation system 106 is implemented on a server system in a client-server relationship with a client system that implements machine-learned agent system 102.

[0074] Machine-learned trace generation system 106 can generate forecasted traces based on a trace from machine-learned agent system 102. For example, a current trace (e.g., recording a number of interaction objects up to a current time) can be passed to machine- learned trace generation system 106 for generating one or more forecasted traces. For example, machine-learned trace generation system 106 can generate a plurality of continuations for a given reference trace from machine-learned agent system 102. Thesecontinuations can diverge from one another, branching out from the reference trace at a particular point. In this manner, for instance, machine-learned trace generation system 106 can generate multiple possible outcomes from different decisions executed on the environment at a given time.

[0075] Machine-learned trace generation system 106 can use probabilistic methods to randomly sample diverging forecasted traces. For instance, machine-learned trace generation system 106 can use a machine-learned sequence generation model of machine-learned model(s) 108 to regress (e.g., autoregressively) a sequence of tokens that represent the forecasted trace. The regression can proceed by sampling next token values according to a probability distribution. The sampling can use temperature sampling, which can reshape the probability distribution to adjust a probability of sampling lower-likelihood options.

[0076] Machine-learned trace generation system 106 can algorithmically enforce diverging forecasted traces. For instance, as one or more forecasted traces are being generated by machine-learned trace generation system 106, machine-learned trace generation system 106 can evaluate a level of alignment between different forecasted traces. Based on the alignment exceeding a threshold, machine-learned trace generation system 106 can terminate generating a particular trace, regenerate a portion of a particular trace, etc. For instance, an alignment measure can include a similarity of an action performed by an agent on an environment. An alignment measure can include a similarity of an environment and a similarity of an action performed on that environment.

[0077] In an example, the alignment measure can be used during decoding to guide generated beams for a beam search. For instance, at each decoding step (or at a designated interval or responsive to a trigger), the candidates in the beam can be rescored according to alignment measure. This rescoring can lead to a ranking within the beam and can facilitate pruning from the beam based on a threshold.

[0078] Machine-learned agent system 102 can request forecasts from machine- learned trace generation system 106. Machine-learned agent system 102 can be configured to automatically receive forecasts from machine-learned trace generation system 106.

[0079] Machine-learned agent system 102 can request forecasts based on a trigger condition. For instance, forecasts can be particularly useful when a path forward is uncertain. For example, machine-learned agent system 102 may predict multiple possible actions to take on a given environment state with probabilities that are relatively close together. One or more forecasts may be used by the machine-learned agent system 102 to inform this prediction such that a preferred action becomes more clearly distinguishable from other actions (e.g.,associated with a higher probability). For instance, machine-learned agent system 102 can determine an uncertainty measure associated with an action in the reference trace. For instance, this may be based on or otherwise correspond to probabilities associated with an action (e.g., a logit value associated with the action in an output head, an aggregation over logit values associated with the action, etc.). For example, a shape of a probability distribution over an output space can be determined. A flat distribution can indicate uncertainty. A distribution with a sharp peak over a particular candidate action can indicate more certainty. Based on the uncertainty measure, machine-learned agent system 102 can request forecasts, or request a particular quantity of forecasts.

[0080] Machine-learned trace generation system 106 can generate variant traces based on a trace from machine-learned agent system 102 or other traces. For example, according to the present disclosure, a seed representation of a reference trace may be obtained, and a variant trace can be generated based on this reference trace. This process involves providing the seed representation to the machine-learned trace generation system, which then generates the variant trace that may include a different environment state from the reference trace. The different environmental state is generated using one or more inferences obtained from the machine-learned trace generation system. These variant traces may include different environment states or actions, potentially leading to different outcomes while still aiming to achieve a similar task goal as the original trace.

[0081] A seed representation can be understood as a condensed version of a reference trace. The seed representation can be a partial trace (e.g., containing a subset of the interaction objects from the reference trace). This partial trace may provide a snapshot or segment of the overall interactions that can be used for generating forecasts or simulations that continue from a specific point in the reference trace. Alternatively, the seed representation can be a summary of the trace that condenses information from the reference trace into a more compact form. This summary can specify particular actions, states, or outcomes that are retained for the purposes of training or simulation, while other attributes may be discarded to allow for exploration along those dimensions.

[0082] The seed representation can be or be based on a prompt used to regenerate the reference trace. For instance, the seed representation can be based on a set of instructions that are sufficient to induce a machine-learned trace generation model to replicate the original trace. For instance, the seed representation can guide the generation process by specifying certain parameters or conditions that the new trace should meet.

[0083] Obtaining a prompt or instruction that can cause a model to recreate the reference trace can proceed over multiple iterations. Each iteration may involve generating a candidate condensed representation, using the machine-learned trace generation system to process this representation, and evaluating the alignment of its output with the reference trace. Adjustments may be made based on this evaluation to refine the seed representation to ensure that it retains certain characteristics of the original trace while possibly omitting other details.

[0084] Once obtained, the seed representation can be used by the machine-learned trace generation system to generate variant traces. For example, the seed representation can be augmented or edited to obtain an input that generates a trace that is similar to, but not identical to, the reference trace. The seed representation can be augmented with at least a portion of the reference trace to generate a variant that branches from the reference trace after a specific branching point.

[0085] A goal state of the generated variant trace can be based on a goal state of the reference trace. A goal state of the generated variant trace can be the same as a goal state of the reference trace. A goal state of the generated variant trace can be different from a goal state of the reference trace.

[0086] For instance, a goal state for a mobile application interaction task in which machine-learned agent system 102 operates as an improved human-machine interface can be a state of the application that corresponds to, for instance, execution of a particular operation. For example, an example mobile application interaction task can be to schedule a calendar event. An associated goal state can be a state of a calendar application that indicates that the event is scheduled. A variant trace can be generated for the same task. The goal can be the same. However, other attributes can be altered. For instance, attributes of the user interface can be altered (e.g., different layout, different colors, etc.). The user experience design can also be altered. For instance, there can be additional or fewer UI interaction steps required to achieve the same goal (e.g., inject or omit confirmation modals, combine or disaggregate multiple input sequences, etc.). However, the system can recognize successful completion of the task based on scheduling of the event. In this manner, for instance, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering non-essential features.

[0087] For instance, a goal state for an object manipulation task by a robotic agent controlling a robot or robotic platform can be an environment state in which the object has been manipulated. For example, an example object manipulation task can correspond to atask template such as “move the {object} to {location}.” An associated goal state can be a state of the environment in which the object is at the location. For a reference trace, the object can be a coffee cup and the location can be a coffee table. The goal state can be a state in which the cup is on the coffee table. A variant trace can be generated for the same task - moving an object to a location. As such, the goal state can use the same template - the object being at the location - while the identity of the object and the identity of the location can be different (e.g., “lamp” to “desk”). In this manner, for instance, the training signal from the essence of a reference demonstration (e.g., moving an object) can be translated to various different contexts by altering non-essential features.

[0088] In another example, a goal state for an online customer service interaction in which machine-learned agent system 102 operates as an improved human-machine interface can be an environment state in which the customer service task has been effectively performed. An example goal state can be a state of the digital environment in which a last recorded message indicates resolution of the customer request (e.g., a response of “yes” to a query “has your concern been resolved?”). For a reference trace, the original interaction might involve a customer service agent responding to a query about a product return process using a standard text-based chat interface. A variant trace can be generated where the interaction involves a different product inquiry, such as warranty information, but still uses a text-based chat interface. In this manner, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering non-essential features, such as the type of customer query. This approach allows the machine-learned agent system to adapt to a range of customer service scenarios, enhancing its ability to handle diverse interactions without the need for extensive data collection for each specific task type, thereby improving both the efficiency and effectiveness of the training process.

[0089] In another example, a goal state for an automated package sorting task can be an environment state in which the sorting task has been accurately completed. An example goal state can be a state of the warehouse environment where packages are correctly sorted into designated bins based on their destination codes. For a reference trace, the original interaction might involve a robotic agent scanning package labels and then placing them into corresponding bins based on a predefined sorting algorithm. A variant trace can be generated where the interaction involves different package sizes or shapes, requiring adjustments in the handling or sorting strategy, yet still using the same sorting bins. In this manner, for instance, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering non-essential features, such as package dimensions. Thismethod allows the machine-learned agent system to adapt to a variety of package types, enhancing its robustness and efficiency in sorting operations without the need for extensive data collection for each specific package type, thereby improving both the efficiency and effectiveness of the training process.

[0090] In another example, a goal state for an automated online booking task in which machine-learned agent system 102 operates as an improved human-machine interface can be an environment state in which the booking has been successfully completed. An example goal state can be a state of the online booking system where a hotel reservation is confirmed, and a confirmation email is sent to the user. For a reference trace, the original interaction might involve the agent navigating a standard hotel booking website interface, selecting dates, choosing a room, and completing the booking process using typical user inputs. A variant trace can be generated where the interaction involves different types of accommodations, such as booking a bed and breakfast or a luxury resort, or adjusting for peak season rates and availability, yet still utilizing the same booking interface. In this manner, as elucidated in the present disclosure, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering non- essential features, such as the type of accommodation or date flexibility. This approach enables the machine-learned agent system to adapt to a broad spectrum of booking scenarios, thereby enhancing its utility and accuracy in handling diverse customer preferences and requirements without extensive retraining for each variation in the booking task.

[0091] In another example, a goal state for an automated email categorization task in which machine-learned agent system 102 operates as an improved human-machine interface can be an environment state in which the emails have been efficiently categorized according to their content and sender. An example goal state can be a state of the email system where incoming emails are automatically sorted into predefined folders like “Work,” “Personal,” and “Spam,” enhancing user organization and efficiency. For a reference trace, the original interaction might involve the machine learning agent system processing typical business correspondence, using keywords, sender information, and attachment types to categorize each email correctly. A variant trace can be generated where the interaction involves different types of emails, such as promotional newsletters or urgent personal messages, requiring different prioritization or categorization rules while maintaining the same foundational categorization algorithm. In this manner, as illustrated in the present disclosure, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering non-essential features, such as the type of emails or urgency levels. Thismethod enables the machine-learned agent system to adapt to a diverse array of email types and senders, effectively improving its ability to manage email overload without the need for extensive data collection for each specific email type, thereby improving both the efficiency and effectiveness of the training process.

[0092] For instance, a goal state for a mapping task in which machine-learned agent system 102 operates as an improved human-machine interface can be an environment state in which a route to reach a designated or selected location has been generated. For example, an example navigation task can correspond to a task template such as “how do I get from {location A} to {location B{ as efficiently as possible within a {time period}.” For a reference trace, the original interaction might involve machine-learned agent system 102 receiving a destination location and / or target arrival time / trip time, determining a routing strategy to the destination location based on, for example, traffic, weather, and / or terrain conditions. A variant trace can be generated where the interaction involves different target arrival times / trip times or different routing strategy to the destination location, different traffic conditions / weather conditions, and / or even a different destination location. In this manner, for instance, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering features, such as target arrival times / trip times, routing strategy, traffic conditions / weather conditions, and / or destination location.This method allows machine-learned agent system 102 to adapt to a variety of different environment conditions and / or requests, enhancing its robustness and efficiency in generating routes to the destination location without the need for extensive data collection for each specific type of traffic conditions, weather conditions, timing, and / or destination location and the like, thereby improving both the efficiency and effectiveness of the training process.

[0093] In another example, a goal state for an automated user interface testing task can be an environment state in which the user interface has been successfully validated against preset usability criteria. An example goal state can be a state of the software testing platform where all interactive elements of a mobile application function correctly and efficiently under various simulated user interactions, thus ensuring high usability and user satisfaction. For a reference trace, the original interaction might involve the machine learning agent system executing a series of predetermined actions on the application’s interface, such as clicking buttons and entering text. A variant trace can be generated where the interaction involves testing these same user interface elements but on different device types with varying screen sizes and resolutions, or under different operating system versions, requiring the system to adapt user interactions and assessment criteria to each specific testing environment.In this manner, as specified in the present disclosure, the training signal from the essence of a reference demonstration can be translated to various different contexts by altering non- essential features such as device specifications or operating conditions. This modality enables the machine-learned agent system to effectively conduct comprehensive usability testing across a broad spectrum of devices and conditions, enhancing product robustness and user experience without extensive data collection for each new test scenario.

[0094] Machine-learned model(s) 108 can be or include various types of models such as text generation models, image generation models, code generation models, and other multimodal models that process and generate diverse data types.

[0095] Text generation models can synthesize textual content conditioned on specific prompts, which can include previous textual interactions. This capability can be used to generate descriptions of environment states or actions within a trace. For example, a text generation model can output text strings containing narratives or summaries of interactions between an agent and its environments. The text can be generated based on multimodal inputs or single modality inputs. In an example, a text generation model can receive an input describing a current state of the environment (e.g., textual description, image description, other data types, etc.). A text generation model can generate a string describing an action to perform on the environment (e.g., a text generation model invoked by machine-learned agent system 102 or machine-learned trace generation system 106). A text generation model can subsequently be invoked (e.g., by machine-learned trace generation system 106) to generate a string describing how the state of the environment may change in response to performance of the action. This process can continue recursively to construct a trace.

[0096] Image generation models can create visual data from textual descriptions or modifying existing images based on new inputs. These models can be employed to visualize environment states that an agent might encounter or actions it might take. For instance, an image generation model can take a textual input describing a room layout and generate an image of that room, which can then be used to simulate how an agent would navigate within it. Subsequently the image generation model can take a multimodal input containing the image of the room and a description of an action taken in the room and generate an image of the new state of the room. In another example, an image generation model can take a textual input describing a user interface and generate an image of that user interface (e.g., optionally conditioned on images of a reference user interface). The generated image of the user interface can then be used to simulate how an agent would interact with it. For instance, machine-learned agent system 102 could ingest the image of the user interface to predict anaction to execute on the user interface. Subsequently the image generation model can take a multimodal input containing the image of the user interface and a description of an action executed on the user interface (e.g., click at coordinates x and y) and generate an image of the new state of the user interface. This process can continue recursively to construct a trace.

[0097] Code generation models can provide the ability to generate executable code based on high-level descriptions or requirements. This is particularly useful in environments where the agent’s actions or the environment’s responses are defined by software control, such as automated testing of applications or managing operations within server systems. The generated code can be executed to simulate various states of the environment without the need for probabilistic prediction of actions or environment state changes.

[0098] The use of code generation models to define environment responses can significantly enhance the speed of generating traces. Since the environment’s responses are pre-defined through executable code, the system does not need to repeatedly compute the probable outcomes of each action during the simulation. This can lead to faster generation of traces as the computational overhead associated with probabilistic methods (e.g., execution of large prediction models) is reduced. For example, a machine-learned trace generation system can use a code generation model to generate computer code that, when processed by a code interpreter, can be executed to render a user interface or other aspects of the environment. This executable code can then provide state updates automatically, without necessarily requiring a generative machine-learned model to predict how the environment will react, thus speeding up and reducing the cost of the entire process of trace generation.

[0099] Further, using generated code to deterministically define the environment’s responses can provide improved consistency and continuity across multiple turns of interaction. This can be helpful for training machine-learned agent systems to perform tasks reliably in a controlled environment before being deployed in real-world scenarios. Consistency and continuity in the environment’s responses can lead to more effective training cycles, as the agent system is not confounded by unexpected responses from an environment that would not occur in the real world.

[0100] Machine-learned models 108 can be implemented by or on behalf of machine- learned agent system 102 or machine-learned trace generation system 106. In an example, machine-learned models 108 are hosted on separate machine(s) from machine-learned agent system 102 or machine-learned trace generation system 106. In an example, machine-learned models 108 are hosted on the same machine(s) as machine-learned agent system 102 or machine-learned trace generation system 106.

[0101] Machine-learned model(s) 108 can include a model particularly adapted for machine-learned agent system 102. For example, machine-learned model (s) 108 can include a model trained for use in machine-learned agent system 102. Machine-learned model(s) 108 can include a model particularly adapted for machine-learned trace generation system 106. For example, machine-learned model(s) 108 can include a model trained for use in machine- learned trace generation system 106.

[0102] An example model is trained to generate action predictions based on an input trace. An example model is trained to generate environment states or stage changes based on an input trace. An example model is trained to generate both action predictions and environmental states or stage changes based on an input trace. Any of such models can be invoked by or on behalf of machine-learned agent system 102 or machine-learned trace generation system 106. For instance, machine-learned agent system 102 can invoke a first model trained to generate action predictions to determine what action to take next. Machine- learned trace generation system 106 can invoke a second model trained to generate environment state information to determine how a forecasted trace will react to an action. This second model can be the same as or different from the first. If the second model is different, machine-learned trace generation system 106 can invoke the first model to generate action predictions and the second model to generate environment state predictions to populate a trace. If the second model is the same, machine-learned trace generation system 106 can invoke the same model to generate multiple parts of the trace. For instance, a single model can be trained to generate both action predictions and environmental states or stage changes based on an input trace.

[0103] The generation of the trace can be implemented using one or multiple queries. For instance, each step (action or state change) can be generated responsive to a single query to generate a predicted action or a predicted state or state change. For an instruction -tuned natural language processing model, the query can include explicit textual instructions to generate a predicted action or a predicted state or state change. The query can implicitly suggest to generate a predicted action or a predicted state or state change, such as if the model itself is fine-tuned for only one of the tasks, and naturally proceeds to generate a predicted action or a predicted state or state change based on its fine-tuning regimen. Alternatively, multiple steps (action or state change) can be generated responsive to a single query. For instance, a sequence processing model can continue to generate outputs describing the actions and state changes until predicting a stop sequence element (e.g., an end-of-sequence “EOS” token).

[0104] Figure 2 is a block diagram of an example implementation in which machine- learned trace generation system operates in an online manner so that machine-learned agent system 102 can use forecasted traces to generate improved action predictions. At a current time T a reference trace 200-(7) can represent a cumulation of prior environment states and actions taken on such states. To assist machine-learned agent system 102 in predicting a subsequent action to take on the environment, machine-learned trace generation system 106 can process reference trace 200-(7) to generate a plurality of forecasted traces 202-1 to 202- / W that represent different ways that the interaction could play out. These generated traces provide multiple possible futures, each concluding with their respective end states. The end states can correspond to goal states or an arbitrarily selected cap of K timesteps in the future. In an example, each trace can have the same number of constituent states leading to the end state. In an example, traces can have different numbers of state before the end state is reached, with further forecasting terminated based on detection of a stopping condition (e.g., a model generates a stop token based on a likelihood that the trace is complete). The example end states in Figure 2 are numbered 202-(Z+ i)-l to 202-(T+K)-M.

[0105] Once the forecasted traces are generated, machine-learned agent system 102 can process reference trace 200-(7) in view of the forecasted traces to determine an action 204-(7) to take in the real environment. The decision can be based on the predicted outcomes contained within the forecasted traces, allowing the agent system to select an action that leads to a favorable end state. This method of using forecasted traces for decision-making significantly enhances the agent’s ability to perform tasks effectively by providing a foresight into the future states of the environment, thereby reducing the reliance on trial-and-error methods in unpredictable or complex scenarios.

[0106] This process can be repeated in one or multiple iterations. For instance, a prior action 204-(Z-l) from a preceding time step can be used to populate reference trace 200-(7) for the current time step. The current action 204-(7) can be used to populate a reference trace 200-(Z+l) for the next time step.

[0107] Figure 3 is a block diagram of an example implementation in which machine- learned agent system 102 may not directly consume the forecasted traces for decisionmaking. In lieu of or in addition to machine-learned agent system 102 directly consuming the forecasted traces, trace evaluation system 300 can be used to assess the forecasted traces and subsequently output an evaluation signal 302. Machine-learned agent system 102 can process the evaluation signal to help evaluate a course of action to execute.

[0108] As described above, machine-learned trace generation system can generate a plurality of forecasted traces. These traces can be processed by trace evaluation system 300, which can evaluate the traces to generate a trace evaluation signal 302. This signal can provide an assessment of the forecasted outcomes, providing feedback regarding the potential effectiveness and consequences of actions depicted in the forecasted traces.

[0109] Trace evaluation system 300 can be or include a system configured to assess possible outcomes of different actions using the forecasted traces generated by machine- learned trace generation system 106. The system can evaluate the “goodness” of various points along the trace (e.g., the end state, one or more intermediate states, etc.). The evaluation can be relative or absolute. For example, machine-learned agent system 102 can operate under one or more constraints. Trace evaluation system 300 can determine whether an input trace violates any one of those constraints at any point along the trace. For example, while a current action may not risk violation of a constraint, it may be likely that, several steps along the trace, the constraint may be violated. Explicitly reasoning over possible downstream outcomes can allow trace evaluation system 300 to check for risks of constraint violation over the remainder of the trace.

[0110] Relative evaluations can include comparisons by trace evaluation system 300 among candidate end states. For example, trace evaluation system 300 can determine which candidate action has the best likelihood of leading to an end state that corresponds to completion of a current task. The alignment between an end state and the desired goal state can be conducted using vector-based similarity measures, Boolean flags, exact matching of environment attributes, instructing a machine-learned model to return a matching answer based on inputs describing the end state(s) and the goal state, or other methods.

[0111] Trace evaluation signal 302 can be an explicit signal. Trace evaluation signal 302 can include a textual description of a preferred next action, an overview of respective outcomes of a respective set of actions, warnings regarding specific actions to avoid, etc. The signal can include statistical data or probabilistic analyses that help quantify the confidence in the signal.

[0112] Trace evaluation signal 302 can be a latent signal. For instance, trace evaluation system 300 can be composed of one or more layers of a machine-learned encoder. An output layer can directly output a latent state from the encoder to a receiving interface of machine-learned agent system 102 (e.g., a decoder portion of a machine-learned model).

[0113] Figure 4 is a block diagram of an example implementation for collecting training data for training one or more of machine-learned agent system 102 or machine-learned trace generation system 106. Machine-learned trace generation system 106 can operate offline to generate additional traces (e.g., variants of or forecasted continuations of reference trace 200) to populate a training dataset. Training system 400 can train machine- learned agent system 102 using the additional traces as demonstrations. Training system 400 can train machine-learned trace generation system 106 by evaluating the forecasted continuations against ground-truth traces (e.g., comparing a forecasted outcome against an actual outcome).

[0114] Figure 5 is a block diagram illustrating in detail an example reference trace 200-(7). Reference trace 200-(7) can include a sequence of environment states and actions, where each environment state 500-(T- ) to 500-(7) corresponds to a specific time step, and each action 502-(T- ) to 502-(7-l) is executed in response to its respective environment state. This sequence effectively represents a historical record of interactions between a system and an environment. Machine-learned trace generation system 106 uses this reference trace to synthesize forecasted traces 202-1 to 202- / W that branch from a current environment state 500-(7) (e.g., 500-(7) being a branching point) and explore possible futures based on different actions 502-(7)-l and 502(7)- / W. For instance, an action 502-(7)-l can lead to an environment state 500-(T+l)-l, on which a subsequent action 502-(T+l)-l can execute, and so on until an end state 500-(7T- )-l. Similarly, an action 502-(7)- / W can lead to an environment state 500-(T+l)-A , on which a subsequent action 502-(Z+l)-AT can execute, and so on until an end state 500-(T+K)-M.

[0115] Environment states (e.g., 500-(7)) can be represented in various different data modalities, including textual and visual data, among others. In a textual modality, for instance, the state might be described through a structured format such as JSON or XML, which could include key -value pairs describing various aspects of the environment. For example, in a digital environment like a web-based shopping system, the state might include information such as current webpage elements, user session details, shopping cart contents, and user interactions up to that point. In a visual modality, the environment state could be represented as an image or a series of images that capture the visual context of the environment. For instance, the state could include screen captures or renders of web content from the shopping system. In the context of a physical robot navigating a warehouse, the state could include real-time images from the robot’s cameras that show the current layout of the warehouse, positions of objects, and other navigational aids. The environment state can also be represented in multimodal data. For instance, screen renders or other image data can be associated with textual metadata containing, for instance, a structured data format such asJSON or XML, which could include key -value pairs describing various aspects of the environment.

[0116] Other modalities can also be employed to represent the environment state, depending on the nature of the environment and the tasks being performed. For instance, audio recordings can be used to capture interactions involving spoken commands or alerts. Similarly, other measurements or sensor data can be helpful in physical environments, providing real-time measurements of various physical parameters like temperature, humidity, or motion. In a physical environment such as the external environment around a vehicle or robot, the state could include sensor data from the vehicle’s or robot’s sensors that show the current physical state of the physical environment around the vehicle / robot including positions of obstacles, objects, terrain etc. In the context of the physical environment of an autonomous vehicle (or robot) navigating to a destination location, the state could include real-time sensor data from the vehicle’s sensors (e.g., real-time images / video from the vehicle’s cameras, vision based systems, real-time radar systems such as Light Detection and Ranging systems or other radar or pulse based systems (e.g., ultra wideband signals) etc.) that show the current layout of the external environment around the autonomous vehicle, positions of objects, obstacles, pedestrians, other vehicles, state of the terrain, and other navigational aids.

[0117] In an example, an environment can be a digital e-commerce platform where a machine-learned agent system is tasked with navigating and performing transactions as an improved human-machine interface. A state of this digital e-commerce environment can be represented using various data structures that encapsulate the relevant aspects of the website at a particular moment. For instance, an example state data structure can include elements such as the current webpage URL, HTML content, a list of item identifiers displayed on the page, prices associated with these items, user session data (such as items in the shopping cart), and user authentication status. This state data structure allows the machine-learned agent system to understand and interact with the digital environment effectively. When the machine-learned agent system performs an action, such as adding an item to the cart, the state data structure is updated to reflect this change, showing the new item in the cart and potentially updating the total cost. The present disclosure can use this updated state data structure to generate a forecasted trace, which predicts subsequent potential actions and their impacts on the environment state. This forecasted trace can include a sequence of predicted environment states and actions, such as navigating to the checkout page, applying a discount code, and finalizing the purchase.

[0118] In an example, an environment can be a smart home system where a machine- learned agent system is tasked with managing various household devices such as lighting, heating, security cameras, and multimedia systems as an improved human-machine interface. A state of this smart home environment can be represented using comprehensive state data that includes the status of each connected device, such as whether lights are on or off, the current temperature settings, the operational status of security systems, and playback status of multimedia devices. For instance, an example state data structure can include detailed elements like a dictionary of light statuses with room identifiers as keys and values indicating on or off states, a list of temperatures for different zones in the house, a Boolean value indicating whether the security system is armed, and an object describing the current media being played along with its playback progress. This state data structure enables the machine- learned agent system to assess the current conditions of the home and make informed decisions to optimize comfort and energy efficiency. For example, if the security camera detects movement outside at night and the security system is disarmed, the state data structure is updated to reflect this detection, and the machine-learned agent system can decide to turn on exterior lights and send a notification to the homeowner’s mobile device. The present disclosure can utilize this updated state data to generate a forecasted trace, predicting subsequent potential actions like rearming the security system or adjusting interior lights based on the homeowner’s response or lack thereof. This forecasted trace would include a sequence of predicted actions and their anticipated effects on the environment, providing a proactive approach to managing the smart home that anticipates the homeowner’s needs and enhances security measures based on real-time data inputs and learned behavior patterns.

[0119] In an example, an environment can be a video game where a machine-learned agent system is tasked with navigating through various levels, interacting with objects, and responding to dynamic challenges presented within the game as an improved human-machine interface. A state of this video game environment can be represented using visual state data, which includes rendered images of the game at specific moments, capturing the visual elements useful for decision-making by the agent. For instance, an example state data structure can include detailed elements such as a series of screenshots showing the current view from the agent’s perspective, metadata describing the objects visible in these images (e.g., enemies, obstacles, power-ups), and numerical data indicating an avatar’s health, score, and remaining resources. This visual and metadata-based state data structure enables the machine-learned agent system to analyze the current game scenario visually and contextually, making decisions such as dodging an attack, jumping over a barrier, or using a collected item.The present disclosure can utilize this state data to generate a forecasted trace, which predicts potential future states of the game environment based on possible actions the agent might take. For instance, if the current state data shows an enemy preparing to attack and a nearby cover, the forecasted trace might predict the outcome of the agent choosing to take cover versus assuming a defensive posture, including generated screenshots that reflect each decision’s likely consequences. These forecasted traces help guide the agent by providing a simulated lookahead into the potential outcomes of different actions, allowing for more strategic decision-making and enhancing the agent’s performance in navigating the game environment effectively.

[0120] In an example, an environment can be a mobile application used for online shopping in which machine-learned agent system 102 operates as an improved humanmachine interface. A state of this mobile application environment can be represented using visual state data which includes screenshots of the application interface at various stages of user interaction, alongside metadata detailing the visible elements such as product listings, prices, descriptions, user interface controls, and user input fields. For instance, an example state data structure can include a sequence of images showing the progression from the home screen to the checkout page, annotated with metadata that categorizes elements like “add to cart” buttons, price tags, and promotional banners, as well as dialogue boxes capturing user actions and decisions (e.g., product selections, quantity adjustments). This detailed visual and descriptive metadata allows the machine-learned agent system to visually understand the layout and functionalities of the app, much as a human user would. Utilizing this state data, the present disclosure can generate a forecasted trace predicting the future states of this mobile application based on potential navigational choices and interactions. For example, if the current state data shows that a user is viewing a product with an “add to cart” button highlighted and a promotional discount notification displayed, the forecasted trace might simulate the user’s potential actions such as adding the item to the cart, continuing to shop for more items, or proceeding to checkout. Each simulated state in the forecasted trace can include generated screenshots predicting the visual outcome of these actions, such as a new total price or an updated cart contents screen.

[0121] In an example, an environment can be a musical instrument tutoring application where a machine-learned agent system is tasked with providing real-time feedback and guidance to students learning to play a musical instrument, such as a piano. A state of this musical instrument tutoring environment can be represented using audio state data, which can include recorded segments of the student playing, along with computedmetrics indicating pitch, tempo, rhythm, etc. For instance, an example state data structure can include detailed elements such as an array of audio samples captured from the student’s recent performance, beat tracking information to assess rhythm accuracy, and dynamic range data to evaluate expression. This audio-based state data structure enables the machine-learned agent system to precisely understand the student’s performance to provide customized feedback. For example, if the state data indicates off-tempo playing or pitch inaccuracies, the agent can generate feedback messages or visual indicators on the student’s screen suggesting metronome adjustments or focusing on certain scales. The present disclosure can use this audio state data to generate a forecasted trace, predicting how the student’s skills might evolve with certain interventions, like practicing specific drills or changing the tempo. The forecasted traces can include sequences of predicted audio states following the suggested interventions, providing a simulated preview of potential improvements in the student’s playing, if the student follows the instruction (or based on an expected level of compliance with the instruction). This method enhances the learning experience by allowing the agent to understand the potential impact of different practice strategies on the student’s performance, encouraging more effective practice sessions.

[0122] Actions (e.g., 502-(7)) can be represented in various different data modalities, including textual and visual data, among others. For instance, in a textual modality, an action could be described using natural language or structured text that outlines the steps or commands involved. This textual representation could be as simple as a command like “move forward” or as complex as a detailed script describing interactions within a software application, a sequence of operations in a robotic assembly task, a sequence of control / navigation operations in an autonomous or robotic navigation task, etc. On the other hand, visual data modalities can include diagrams, flowcharts, or even more dynamic forms such as animated sequences or virtual reality environments that visually illustrate the action being taken. For example, a virtual depiction of a robot’s arm moving to a specified location or a graphical user interface showing a cursor clicking a button could serve as visual representations of actions.

[0123] For instance, in a scenario where the agent is interacting with a digital interface, an action such as “clicking a button on a webpage” can be described textually. This textual description might detail the coordinates of the click, the purpose of the button, and the expected outcome of the action. The same action can be represented visually by an image or a screenshot capturing the state of the webpage at the time of the click. This visualrepresentation can include annotations or highlights showing the exact location of the click, providing a visual context that complements the textual description.

[0124] In an example, an environment can be a digital document editing platform where users can perform various formatting and editing tasks. An action to perform on this environment might include altering the text format within a document. This action can be represented using a structured command data format that encapsulates the specific commands and parameters needed for execution. For instance, an example action data structure can include elements such as “action type” set to “format change”, “target texf ’ identified by a text selection range, “new formaf ’ detailing attributes like font type, size, color, and text alignment. This structured command can be sent from a user interface or an API call within the digital document editing platform. The present disclosure can utilize this action data structure in the machine-learned agent system to predict and execute subsequent formatting actions based on historical user interactions and preferences, enhancing user experience by suggesting formatting styles or automating repetitive tasks. Moreover, the machine-learned trace generation system can generate forecasted traces based on this action, predicting the visual and structural changes to the document.

[0125] In an example, an environment can be a smart home automation system where various devices are interconnected to enhance user comfort and efficiency. An action to perform on this environment might include adjusting the temperature setting of a smart thermostat. This action can be represented using a JSON formatted object that encapsulates the specific parameters and commands needed for execution. For instance, an example action data structure can include elements such as “action type” set to “adjust temperature”, “device id” identifying the specific thermostat within the network, “desired temperature” indicating the new temperature setting, and “time of action” specifying when the adjustment should take place. This structured command can be sent from a mobile application or voice command through a smart home hub.

[0126] In an example, an environment can be a web-based application designed for online shopping. An action to perform on this environment might include adding an item to a virtual shopping cart. This action can be represented using visual data that captures the graphical interface of the web-based application at a specific moment. For instance, an example action data structure can include elements such as “action type” set to “add to cart”, “item id” representing the unique identifier of the product, and “item image” which stores a visual representation of the item within the user interface, including itsplacement and graphical depiction on the web page. The “action coordinates” could also be included, specifying the x and y screen coordinates where the “add to cart” button is located.

[0127] In an example, an environment can be the physical environment around a robot or autonomous vehicle or platform where various sensors are interconnected with the robot / vehicle for controlling various actuators or control interfaces for steering and / or adjusting speed to assist in navigating through the physical environment. An action to perform on this physical environment might include adjusting the direction and / or speed of the autonomous vehicle or robot. This action can be represented using a JSON formatted object that encapsulates the specific parameters and commands needed for execution. For instance, an example action data structure can include elements such as “action type” set to “adjust direction”, “device id” identifying the specific directional actuator / mechanism within the vehicle or robot, “desired direction” indicating the new directional setting, and “time of action” specifying when the adjustment should take place. This structured command can be sent from the vehicle / robot agent to the control system of the autonomous vehicle / robot.

[0128] Within a given trace, different actions and environment states can be represented with the same or with different types of information or data structures.

[0129] Reference trace 200-(7) can be represented as a multimodal sequence of information. A multimodal machine-learned model (e.g., used by machine-learned agent system 102 or machine-learned trace generation system 106) can process an input sequence of textual content interleaved with image content. For instance, each interaction object of the reference trace can be represented in the input sequence by a first portion representing an embedded textual description of the corresponding environment state and a second portion representing an embedded image description of the corresponding environment state. The second portion can be contiguous with the first portion.

[0130] Figure 6 is a block diagram of an example labeled reference trace 600. Reference trace 600 can be labeled automatically or manually with reward values that respectively correspond to states in the trace. This can proceed for each time step: for instance, based on the states and actions preceding time t < T-2, reward 602-(Z-2) can be assigned (e.g., obtained from human feedback, obtained from reward model, etc.). Based on the states and actions preceding time t < T-l, reward 602-(Z-l) can be assigned (e.g., obtained from human feedback, obtained from reward model, etc.). Based on the states and actions preceding time t < T, reward 602-(7) can be assigned (e.g., obtained from human feedback, obtained from reward model, etc.).

[0131] Reference trace 600 can be associated with rewards of different granularity. Action-level rewards (e.g., rewards 602-(7), etc.) can indicate a reward associated with a specific action. Trace-level rewards can indicate an overall reward amount for the overall performance across the trace. Trace-level rewards can be independently computed (e.g., by processing an entire trace) or can be computed using an accumulation or other aggregation of action-level rewards.

[0132] Labeled reference trace 600 can be stored in a training dataset for training machine-learned agent system 102 or machine-learned trace generation system 106.

[0133] Figure 7 is a block diagram of an interaction object 700. Interaction object 700 can be a subset or portion of labeled reference trace 600. An interaction object can be a unit that captures an incremental step in a sequence of interactions. For instance, a reference trace can include a sequence of interaction objects. An interaction object can be cumulative (e.g., contain a full history of preceding interactions) or additive (e.g., only contains additional data). In an example, interaction object 700 contains an association of a particular environment state, an action executed on that environment state, the effect on the environment (reflected in a new environment state), and a reward associated with that action / new state). In this manner, for instance, interaction object 700 can be used as a training example for predicting what actions to take with respect to an initial environment state, with the predictions being evaluated based on the reward obtained. Interaction object 700 can also be used as a training example for predicting how an environment reacts to an action performed on the initial environment state, with the predictions being evaluated based on the recorded subsequent environment state.

[0134] The ellipses indicate that interaction object 700 can optionally include a cumulative record of prior states and actions. A model ingesting interaction object 700 can predict an action 502-(T-2) based on environment state 500-(T-2) and all states preceding it. A model ingesting interaction object 700 can predict an action 502-(T-2) based on environment state 500-(T-2) without reference to all states preceding it. A model ingesting interaction object 700 can predict a subsequent environment state 500-(T-l) based on action 502-(T-2), environment state 500-(T-2), and all states preceding it. A model ingesting interaction object 700 can predict a subsequent environment state 500-(T-l) based on action 502-(T-2) and environment state 500-(T-2), without reference to all states preceding it.

[0135] Figure 8 is a block diagram of a simpler additive interaction object 800. Interaction object 800-(T-l) can include a triplet of data objects: environment state 500-(T- 2), an action 502-(T-2) executed on environment state 500-(T-2), and a reward 602-(T-l).

[0136] Reference trace 600 can be composed of a sequence of additive interaction objects. Each interaction object in the sequence can be independent. Each interaction object in the sequence can point to adjacent objects to maintain an ordering. For instance, interaction object 800-(Z-l) may store a reference to action(s) or prior state(s) (e.g., action 502-(Z-2) of interaction object 800-(7— 2)) that precede the environment state that forms a part of the triplet. Interaction object 800 may store a reference to subsequent state(s) on which the reward that forms a part of the triplet is based (e.g., state 500-(7) of interaction object 800- ).

[0137] Figure 9 is a block diagram of an example replay buffer 900 at a time t = T. Replay buffer 900 can store a cache of recent environment states and actions applied thereto. For example, replay buffer 900 contains a current environment state 902-( / '), as well as at least the 77 preceding states 902-(Z-l), 902-(Z-2), . . . , 902-(7-7>). Replay buffer can contain action 904-( 7—1 ) that acted upon environment state 902-(Z-l) to precipitate the current environment state 902-( 7), action 904-( 7-2) that acted upon environment state 902-(Z-2) to precipitate environment state 902 -(7-1), and so on, at least until action 904-(7-7>) that acted upon environment state 902-(7-Z?).

[0138] The buffered interaction objects can be used by machine-learned agent system 102 to predict next actions. The buffered interaction objects can be used by machine-learned trace generation system 106 to predict trace completions (e.g., forecasted traces for T< t < K). The buffered interaction objects can be recorded into storage for accumulating a dataset of traces.

[0139] Replay buffer 900 can be a fixed-size buffer. For example, replay buffer 900 can be a rolling buffer. Replay buffer 900 can have a fixed memory footprint such that the recorded trace is truncated to fit in memory. This truncation can be performed by dropping the least recent interaction objects, transforming the least recent or least important (e.g., as judged by a machine-learned model) interaction objects into data-efficient representations (e.g., summarizing), etc.

[0140] Replay buffer 900 can contain other data associated with the interaction objects that can be cached for improved real-time processing. For instance, one or more model latents associated with a given interaction object can be cached. For instance, a machine-learned model associated with one of the machine-learned agent system 102 or the machine-learned trace generation system 106 can generated intermediate latent representations of inputs that can be re-used for later computations (e.g., KV-cache data for transformer-based models). These generated intermediate latent representations can becached in association with the entries in the replay buffer to accelerate processing of the buffered trace by machine-learned agent system 102 or machine-learned trace generation system 106. Multiple different model latents can be stored, for example if machine-learned agent system 102 uses a different model from machine-learned trace generation system 106. For instance, replay buffer 900 can include latents for model 1, 906-(7), 906-(Z-l), 906-(Z- 2), . . . , 906-(Z- ). Replay buffer 900 can include latents for model N, 908-(7), 908-(Z-l), 908-(Z-2), . . . , 908-(Z- ).

[0141] Figure 10 is a block diagram of an example system for generating variant traces using machine-learned trace generation system 106. A reference trace 1000 can contain N environment states 1002-1, 1002-2, . . . , 1002-TV, respectively acted upon by N actions 1004-1, 1004-2, . . . , 1004-7V, that culminate in a goal or end state 1006. A seed representation 1008 can be extracted from reference trace 1000 can input to machine-learned model(s) 108, which can generate trace data 1010 for machine-learned trace generation system 106. Machine-learned trace generation system 106 can use trace data 1010 to obtain generated variant traces 1012.

[0142] Seed representation 1008 can be a portion of reference trace 1000. For instance, seed representation 1008 can be an initial portion or prefix of reference trace 1000. In this manner, for instance, trace data 1010 can be a prediction by machine-learned model(s) 108 of state(s) and action(s) that follow the initial prefix.

[0143] Although illustrated as an initial portion, it is to be understood that a seed representation can be any portion of reference trace 1000, such as an intermediate portion or end portion. Trace data 1010 can be a prediction of an initial portion that precedes the seed representation. Trace data 1010 can be a prediction of an intermediate portion that is between a prefix portion and an end portion. In this manner, for instance, variant traces can be generated that diverge from a reference trace in targeted portions.

[0144] The seed representation can be all of reference trace 1000, and machine- learned model(s) 108 can be instructed to generate a variant based on the entirety of the trace. For instance, reference trace 1000 can be provided as context for what the variant is intended to diverge from. For example, reference trace 1000 can be provided to a machine-learned model 108 along with an instruction to generate a trace that is the same, except for a list of specific alterations.

[0145] Trace data 1010 can be a model output generated in response to an input containing seed representation 1008. Trace data 1010 can include the raw generation requested by machine-learned trace generation system 106.

[0146] Generated variant traces 1012 can include one or multiple traces. For instance, multiple generated traces can be generated based on reference trace 1000 or based on the same seed representation. For instance, seed representation 1008 and any corresponding instructions can be broadcast across a batch dimension of a model and multiple generations can proceed in parallel.

[0147] Figure 11 is a block diagram of an example system in which pre-processing can be performed on reference trace 1000 to obtain seed representation 1008. For example, machine-learned model(s) 108 can preprocess all or part of reference trace 1000 to generate preprocessed trace data 1102, which can be used by machine-learned trace generation system 106 to obtain seed representation 1008.

[0148] Preprocessed trace data 1102 can include a condensed representation of reference trace 1000. For instance, preprocessed trace data 1102 can be or include a summary of reference trace 1000. Preprocessed trace data 1102 can be an input (e.g., prompt) configured to reproduce reference trace 1000 if input to a trace generation model or system. In general, preprocessed trace data 1102 can represent one or more core features of reference trace 1000.

[0149] Based on preprocessed trace data 1102, machine-learned trace generation system 106 can generate seed representation 1008 that leverages the distilled essence of reference trace 1000 while introducing variations. For instance, seed representation 1008 can omit certain details to allow machine-learned model(s) 108 to explore variations along that dimension (e.g., omitting color palette information from a web interface description to allow the model to probabilistically sample color information). Seed representation 1008 can include additional details or variations which can be defined in absolute terms (e.g., “the web page is rendered on a screen in landscape orientation”) or relative terms (e.g., “the web page uses a more casual communication tone”).

[0150] Figure 12 depicts a flowchart of a method 1200 for training one or more machine-learned models according to aspects of the present disclosure. For instance, an example machine-learned model can include one or more models of machine-learned models 108. For instance, an example model can be trained to provide inferences for machine- learned agent system 102. An example model can be trained to provide inferences for machine-learned trace generation system 106.

[0151] One or more portion(s) of example method 1200 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion ofexample method 1200 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 1200 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 12 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Figure 12 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 1200 can be performed additionally, or alternatively, by other systems.

[0152] At 1202, example method 1200 can include obtaining a training instance. A set of training data can include a plurality of training instances divided between multiple datasets (e.g., a training dataset, a validation dataset, or testing dataset). A training instance can be labeled or unlabeled. Although referred to in example method 1200 as a “training” instance, it is to be understood that runtime inferences can form training instances when a model is trained using an evaluation of the model’s performance on that runtime instance (e.g., online training / learning). Example data types for the training instance and various tasks associated therewith are described throughout the present disclosure.

[0153] At 1204, example method 1200 can include processing, using one or more machine-learned models, the training instance to generate an output. The output can be directly obtained from the one or more machine-learned models or can be a downstream result of a chain of processing operations that includes an output of the one or more machine- learned models.

[0154] At 1206, example method 1200 can include receiving an evaluation signal associated with the output. The evaluation signal can be obtained using a loss function. Various determinations of loss can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal can be computed using known ground-truth labels (e.g., supervised learning), predicted or estimated labels (e.g., semi- or self-supervised learning), or without labels (e.g., unsupervised learning). The evaluation signal can be a reward (e.g., for reinforcement learning). The reward can be computed using a machine-learned reward model configured togenerate rewards based on output(s) received. The reward can be computed using feedback data describing human feedback on the output(s).

[0155] At 1208, example method 1200 can include updating the machine-learned model using the evaluation signal. For example, values for parameters of the machine-learned model(s) can be learned, in some embodiments, using various training or learning techniques, such as, for example, backwards propagation. For example, the evaluation signal can be backpropagated from the output (or another source of the evaluation signal) through the machine-learned model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the evaluation signal with respect to the parameter value(s)). For example, system(s) containing one or more machine-learned models can be trained in an end-to-end manner. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. Example method 1200 can include implementing a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0156] In some implementations, example method 1200 can be implemented for training a machine-learned model from an initialized state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0157] In some implementations, example method 1200 can be implemented for particular stages of a training procedure. For instance, in some implementations, example method 1200 can be implemented for pre-training a machine-learned model. Pre-training can include, for instance, large-scale training over potentially noisy data to achieve a broad base of performance levels across a variety of tasks / data types.

[0158] In some implementations, example method 1200 can be implemented for fine- tuning a machine-learned model. Fine-tuning can include, for instance, smaller-scale training on higher-quality (e.g., labeled, curated, etc.) data. Fine-tuning can affect all or a portion of the parameters of a machine-learned model. For example, various portions of the machine- learned model can be “frozen” for certain training stages. For example, parameters associated with an embedding space can be “frozen” during fine-tuning (e.g., to retain information learned from a broader domain(s) than present in the fine-tuning dataset(s)). In some implementations, example method 1200 uses adapter modules. Adapters can be small trainable layers that are inserted between pre-existing layers of a pre-trained model. Duringthe fine-tuning process, the original parameters of the pre-trained model are typically frozen, and only the parameters of the adapters are updated.

[0159] In some implementations, example method 1200 can be implemented to execute parameter-efficient fine-tuning methods, such as Layerwise Optimization of Residuals (LoRA). LoRA can refine pre-trained models with minimal adjustments to the original parameters. This can be achieved by introducing trainable low-rank matrices that modify the behavior of the pre-trained weights without directly altering them. In some implementations, during fine-tuning, only these auxiliary matrices are updated, which significantly reduces the number of parameters that are trained.

[0160] An example fine-tuning approach includes reinforcement learning. Reinforcement learning can be based on user feedback on model performance during use.

[0161] In some implementations, example method 1200 can be used to train a model for use by machine-learned agent system 102. This training can be a step to create agent systems that interact with a variety of environments to perform specific tasks, such as navigation, object manipulation, or user interface interactions. To illustrate this process, consider an example scenario in which machine-learned agent system 102 is designed to navigate through a digital environment, such as an online shopping platform, to perform a shopping task (e.g., finding a gift), or a physical environment, such as a real-world external environment, to perform a task (e.g., assembly task, navigation task, packing or sorting task, a movement task, etc.).

[0162] At 1202, an example training instance can include a sequence of user interactions with the online shopping platform, captured as a series of screen images and corresponding user actions (e.g., clicks, keystrokes). Each interaction object in this sequence might include a screenshot of the webpage (the environment state) and the user’s action, such as clicking on a product category. The example training instance can include all or part of a reference trace 200, 600, 1000, etc., or a generated variant or forecasted trace.

[0163] In another example, an example training instance can include a sequence of user demonstrations or control inputs for controlling a robot or autonomous vehicle in performing a task (e.g., assembly task, navigation task, packing or sorting task, a movement task), which may be captured as a series of screen images or control inputs associated with the corresponding user actions / control inputs performing the task. Each interaction object in this sequence might include a screenshot of the physical environment (the environment state) and the user’s action, such as changing direction around an object detected in the physical environment, grabbing and moving an object from one location to another location. Theexample training instance can include all or part of a reference trace 200, 600, 1000, etc., or a generated variant or forecasted trace.

[0164] At 1204, the obtained training instance can be processed using one or more machine-learned models to generate a prediction of the next user action based on the current webpage state or the physical environment state associated with the robot or vehicle. For example, if the user is on the homepage, and the page URL includes an identifier indicating that the homepage was reached by clicking through a promotional message for a sale, the predicted next action can include sorting a list of sale items on the home page by percentage discount, etc. For example, if the user is navigating the vehicle or robot through the physical environment, and the current physical environment state includes an obstacle requiring a change of direction or speed, the predicted next action can include an estimated change of direction or speed, etc.

[0165] At 1206, an evaluation signal associated with the output is received. This signal can include a reward assigned to the state resulting from the action. The reward can indicate how helpful the action was in progressing the state of the application toward completion of the task. The reward can be generated by a reward model or assigned by a human labeler. The signal can be an evaluation of accuracy or a loss. For instance, in a supervised learning scenario, the signal can assess the accuracy of the model’s prediction against actual user actions.

[0166] At 1208, the machine-learned model can be updated using the evaluation signal by updating model parameters in a direction expected to improve the feedback signal (e.g., increase a reward, decrease a loss, etc.).

[0167] Example method 1200 can be used online or offline. In an example, online learning can occur when a user is cooperatively working with an agent system to perform a task. Correction or other feedback received from the user can be used to update the agent system according to the feedback received.

[0168] In some implementations, example method 1200 can be used to train a model for use by machine-learned trace generation system 106. This training can specifically focus on enhancing the model’s ability to predict new environment states based on actions performed on preceding states. To illustrate an example training procedure, consider an example agent for use in a calendar application.

[0169] At 1202, a training example can include a reference trace obtained historical data from a calendar application. This data might include user interactions such as adding,removing, or modifying calendar events, along with the state of the calendar application before and after these actions were taken.

[0170] At 1204, at or part of the obtained training instance is processed using one or more machine-learned models. In an example, the model can be configured to output future actions and environment states. In an example, the model can be configured to output a next environment state based on an action performed on a preceding state. For example, if an action is an audio input transcribed as “add meeting at 3 PM,” and a preceding state is a calendar application in the foreground of a computing system, the model can predict the new state of the calendar application as having a meeting successfully added at the specified time.

[0171] At 1206, an evaluation signal is received. The evaluation signal can assess the accuracy of the model’s prediction against the observed outcome in the training example. If the model accurately predicts the new state of the calendar after the action, the evaluation signal would reflect a positive reinforcement. However, if the model’s prediction deviates from the actual outcome — for instance, it predicts a state of the calendar application that does not render an invite scheduled at 3 PM — the evaluation signal can involve a corrective feedback, indicating the error in the prediction. For action prediction, the signal can include a reward assigned to the state resulting from the action. The reward can indicate how helpful the action was in progressing the state of the application toward completion of the task. The reward can be generated by a reward model or assigned by a human labeler.

[0172] At 1208, the machine-learned model can be updated using the evaluation signal by updating model parameters in a direction expected to improve the feedback signal (e.g., increase a reward, decrease a loss, etc.).

[0173] Example method 1200 can be used online or offline. In an example, online learning can occur when a user is cooperatively working with an agent system to perform a task. Correction or other feedback received from the user can be used to update the agent system according to the feedback received.

[0174] Figure 13 is a block diagram of an example processing flow for using machine-learned model(s) 1 to process input(s) 2 to generate output(s) 3.

[0175] Machine-learned model(s) 1 can be or include one or multiple machine- learned models or model components. Example machine-learned models can include neural networks (e.g., deep neural networks). Example machine-learned models can include nonlinear models or linear models. Example machine-learned models can use other architectures in lieu of or in addition to neural networks. Example machine-learned models can includedecision tree based models, support vector machines, hidden Markov models, Bayesian networks, linear regression models, k-means clustering models, etc.

[0176] Machine-learned model(s) 1 can be or include, or otherwise be representative of any one or more of the machine-learned models described above with respect to the preceding figures. For example, machine-learned model(s) 1 can be or include, or otherwise be representative of any one or more of machine-learned models 108, such as a model used by machine-learned agent system 102, machine-learned trace generation system 106, trace evaluation system 300, etc. Although various features, variations, and implementations described below are described with respect to machine-learned model(s) 1, it is to be understood that such features, variations, and implementations are to be understood as described with respect to each of machine-learned models 108, such as a model used by machine-learned agent system 102, machine-learned trace generation system 106, trace evaluation system 300, etc., any other machine-learned component described herein.

[0177] Example neural networks can include feed-forward neural networks, recurrent neural networks (RNNs), including long short-term memory (LSTM) based recurrent neural networks, convolutional neural networks (CNNs), diffusion models, generative-adversarial networks, or other forms of neural networks. Example neural networks can be deep neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multiheaded self-attention models.

[0178] Machine-learned model(s) 1 can include a single or multiple instances of the same model configured to operate on data from input(s) 2. Machine-learned model(s) 1 can include multiple different models or multiple different model portions configured to operate on data from input(s) 2.

[0179] Machine-learned model(s) 1 can include an ensemble of different models that can cooperatively interact to process data from input(s) 2. For example, a model ensemble can include multiple models that have different attributes (e.g., different architectures, trained with different recipes, etc.). The ensemble can output an overall output based on the individual outputs of the constituent models. In this manner, for instance, the diverse constituent models can work together to provide system-level robustness by effectively aggregating over individual strengths and weaknesses of any given model. The respective individual outputs can be combined in a weighted combination, using a voting or routing mechanism, or a learned output layer (e.g., one or more feedforward or fully-connected layers).

[0180] Machine-learned model(s) 1 can employ a mixture-of-experts structure. See, e.g., Zhou et al., Mixture -of -Experts with Expert Choice Routing, ARXIV:2202.09368V2 (Oct. 14, 2022). For example, different portions of a model can learn (explicitly or implicitly) different expertise areas, with pathways through the model being selected by a learned routing mechanism that engages the appropriate expert for a given input (e.g., a given portion of an input, such as on a per-token basis). For example, a feedforward network can be sparsely activated for a given portion of an input based on an output of a routing mechanism that processes the portion of the input. In this manner, for instance, the group of activated weights can form an “expert” that is selected by the router. On each forward pass, only a subset of the total model weights may be engaged, thereby decreasing a quantity of operations performed for processing a given input compared to a densely activated model. In this manner, for instance, the expressive and interpretive power of a high-parameter-count model can be achieved with more compute-efficient forward passes.

[0181] Input(s) 2 can generally include or otherwise represent various types of data. Input(s) 2 can include one type or many different types of data. Output(s) 3 can be data of the same type(s) or of different types of data as compared to input(s) 2. Output(s) 3 can include one type or many different types of data.

[0182] Example data types for input(s) 2 or output(s) 3 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming languages), machine code data (e.g., binary code, assembly code, or other forms of machine-readable instructions that can be executed directly by a computer’s central processing unit), assembly code data (e.g., low-level programming languages that use symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audiovisual data, haptic data, biometric data, medical data, financial data, statistical data, geographical data, astronomical data, historical data, sensor data generally (e.g., digital or analog values, such as voltage or other absolute or relative level measurement values from a real or artificial input, such as from an audio sensor, light sensor, displacement sensor, etc.), and the like. Data can be raw or processed and can be in any format or schema.

[0183] In multimodal inputs 2 or outputs 3, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It is to be understood that any combination of data types in an input 2 or an output 3 can be present.

[0184] An example input 2 can include one or multiple data types, such as the example data types noted above. An example output 3 can include one or multiple data types, such as the example data types noted above. The data type(s) of input 2 can be the same as or different from the data type(s) of output 3. It is to be understood that the example data types noted above are provided for illustrative purposes only. Data types contemplated within the scope of the present disclosure are not limited to those examples noted above.

[0185] Figure 14 is a block diagram of an example implementation of an example machine-learned model configured to process sequences of information. For instance, an example implementation of machine-learned model(s) 1 can include machine-learned sequence processing model(s) 4. An example system can pass input(s) 2 to sequence processing model(s) 4. Sequence processing model(s) 4 can include one or more machine- learned components. Sequence processing model(s) 4 can process the data from input(s) 2 to obtain an input sequence 5. Input sequence 5 can include one or more input elements 5-1, 5- 2, . . . , 5-A , etc. obtained from input(s) 2. Sequence processing model 4 can process input sequence 5 using prediction layer(s) 6 to generate an output sequence 7. Output sequence 7 can include one or more output elements 7-1, 7-2, . . . , 7 -A, etc. generated based on input sequence 5. The system can generate output(s) 3 based on output sequence 7.

[0186] Sequence processing model(s) 4 can include one or multiple machine-learned model components configured to ingest, generate, or otherwise reason over sequences of information. For example, some example sequence processing models in the text domain are referred to as “Large Language Models,” or LLMs. See, e.g., PaLM 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2techreport.pdf (n.d.). Other example sequence processing models can operate in other domains, such as image domains, see, e.g., Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (Jun. 3, 2021), audio domains, see, e.g., Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.1 1325V1 (Jan. 26, 2023), biochemical domains, see, e.g., Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (Aug. 26, 2021), by way of example. Sequence processing model(s) 4 can process one or multiple types of data simultaneously. Sequence processing model(s) 4 can include relatively large models (e.g., more parameters, computationally expensive, etc.), relatively small models (e.g., fewer parameters, computationally lightweight, etc.), or both.

[0187] In general, sequence processing model(s) 4 can obtain input sequence 5 using data from input(s) 2. For instance, input sequence 5 can include a representation of data from input(s) 2 in a format understood by sequence processing model(s) 4. One or more machine-learned components of sequence processing model(s) 4 can ingest the data from input(s) 2, parse the data into pieces compatible with the processing architectures of sequence processing model(s) 4 (e.g., via “tokenization”), and project the pieces into an input space associated with prediction layer(s) 6 (e.g., via “embedding”).

[0188] Sequence processing model(s) 4 can ingest the data from input(s) 2 and parse the data into a sequence of elements to obtain input sequence 5. For example, a portion of input data from input(s) 2 can be broken down into pieces that collectively represent the content of the portion of the input data. The pieces can provide the elements of the sequence.

[0189] Elements 5-1, 5-2, . . . , 5-M can represent, in some cases, building blocks for capturing or expressing meaningful information in a particular data domain. For instance, the elements can describe “atomic units” across one or more domains. For example, for textual input source(s), the elements can correspond to groups of one or more words or sub-word components, such as sets of one or more characters.

[0190] For example, elements 5-1, 5-2, . . . , 5-M can represent tokens obtained using a tokenizer. For instance, a tokenizer can process a given portion of an input source and output a series of tokens (e.g., corresponding to input elements 5-1, 5-2, . . . , 5-A ) that represent the portion of the input source. Various approaches to tokenization can be used. For instance, textual input source(s) can be tokenized using a byte-pair encoding (BPE) technique. See, e.g., Kudo et al., SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, PROCEEDINGS OF THE 2018 CONFERENCE ON EMPIRICAL METHODS IN NATURAL LANGUAGE PROCESSING (System Demonstrations), pages 66-71 (October 31-November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Image-based input source(s) can be tokenized by extracting and serializing patches from an image.

[0191] In general, arbitrary data types can be serialized and processed into input sequence 5. It is to be understood that element(s) 5-1, 5-2, . . . , 5-M depicted in Figure 14 can be the tokens or can be the embedded representations thereof.

[0192] Prediction layer(s) 6 can predict one or more output elements 7-1, 7-2, . . . , 7- N based on the input elements. Prediction layer(s) 6 can include one or more machine-learned model architectures, such as one or more layers of learned parameters that manipulate and transform the input(s) to extract higher-order meaning from, and relationships between, input element(s) 5-1, 5-2, . . . , 5-M. In this manner, for instance, example prediction layer(s) 6 can predict new output element(s) in view of the context provided by input sequence 5.

[0193] Prediction layer(s) 6 can evaluate associations between portions of input sequence 5 and a particular output element. These associations can inform a prediction of the likelihood that a particular output follows the input context. For example, consider the textual snippet, “The carpenter’s toolbox was small and heavy. It was full of .” Example prediction layer(s) 6 can identify that “It” refers back to “toolbox” by determining a relationship between the respective embeddings. Example prediction layer(s) 6 can also link “It” to the attributes of the toolbox, such as “small” and “heavy.” Based on these associations, prediction layer(s) 6 can, for instance, assign a higher probability to the word “nails” than to the word “sawdust.”

[0194] A transformer is an example architecture that can be used in prediction layer(s) 4. See, e.g., Vaswani et al., Atention Is All You Need, ARXIV: 1706.03762v7 (Aug. 2, 2023). A transformer is an example of a machine-learned model architecture that uses an attention mechanism to compute associations between items within a context window. The context window can include a sequence that contains input sequence 5 and potentially one or more output element(s) 7-1, 7-2, . . . , 7-N. A transformer block can include one or more attention layer(s) and one or more post-attention layer(s) (e.g., feedforward layer(s), such as a multi-layer perceptron).

[0195] Prediction layer(s) 6 can include other machine-learned model architectures in addition to or in lieu of transformer-based architectures. For example, recurrent neural networks (RNNs) and long short-term memory (LSTM) models can also be used, as well as convolutional neural networks (CNNs). In general, prediction layer(s) 6 can leverage various kinds of artificial neural networks that can understand or generate sequences of information.

[0196] Output sequence 7 can include or otherwise represent the same or different data types as input sequence 5. For instance, input sequence 5 can represent textual data, and output sequence 7 can represent textual data. Input sequence 5 can represent image, audio, or audiovisual data, and output sequence 7 can represent textual data (e.g., describing the image, audio, or audiovisual data). It is to be understood that prediction layer(s) 6, and any other interstitial model components of sequence processing model(s) 4, can be configured to receive a variety of data types in input sequence(s) 5 and output a variety of data types in output sequence(s) 7.

[0197] Output sequence 7 can have various relationships to input sequence 5. Output sequence 7 can be a continuation of input sequence 5. Output sequence 7 can be complementary to input sequence 5. Output sequence 7 can translate, transform, augment, or otherwise modify input sequence 5. Output sequence 7 can answer, evaluate, confirm, orotherwise respond to input sequence 5. Output sequence 7 can implement (or describe instructions for implementing) an instruction provided via input sequence 5.

[0198] Output sequence 7 can be generated autoregressively. For instance, for some applications, an output of one or more prediction layer(s) 6 can be passed through one or more output layers (e.g., softmax layer) to obtain a probability distribution over an output vocabulary (e.g., a textual or symbolic vocabulary) conditioned on a set of input elements in a context window. In this manner, for instance, output sequence 7 can be autoregressively generated by sampling a likely next output element, adding that element to the context window, and re-generating the probability distribution based on the updated context window, and sampling a likely next output element, and so forth.

[0199] Output sequence 7 can also be generated non-autoregressively. For instance, multiple output elements of output sequence 7 can be predicted together without explicit sequential conditioning on each other. See, e.g., Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, ARXIV:2004.07437V3 (NOV. 16, 2020).

[0200] Output sequence 7 can include one or multiple portions or elements. In an example content generation configuration, output sequence 7 can include multiple elements corresponding to multiple portions of a generated output sequence (e.g., a textual sentence, values of a discretized waveform, computer code, etc.). In an example classification configuration, output sequence 7 can include a single element associated with a classification output. For instance, an output “vocabulary” can include a set of classes into which an input sequence is to be classified. For instance, a vision transformer block can pass latent state information to a multilayer perceptron that outputs a likely class value associated with an input image.

[0201] Figure 15 is a block diagram of an example technique for populating an example input sequence 8. Input sequence 8 can include various functional elements that form part of the model infrastructure, such as an element 8-0 obtained from a task indicator 9 that signals to any model(s) that process input sequence 8 that a particular task is being performed (e.g., to help adapt a performance of the model(s) to that particular task). Input sequence 8 can include various data elements from different data modalities. For instance, an input modality 10-1 can include one modality of data. A data-to- sequence model 11-1 can process data from input modality 10-1 to project the data into a format compatible with input sequence 8 (e.g., one or more vectors dimensioned according to the dimensions of input sequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 can include a different modality of data. A data-to-sequence model 11-2 can project data from inputmodality 10-2 into a format compatible with input sequence 8 to obtain elements 8-4, 8-5, 8- 6. Another input modality 10-3 can include yet another different modality of data. A data-to- sequence model 11-3 can project data from input modality 10-3 into a format compatible with input sequence 8 to obtain elements 8-7, 8-8, 8-9.

[0202] Input sequence 8 can be the same as or different from input sequence 5. Input sequence 8 can be a multimodal input sequence that contains elements that represent data from different modalities using a common dimensional representation. For instance, an embedding space can have P dimensions. Input sequence 8 can be configured to contain a plurality of elements that have / Jdimensions. In this manner, for instance, example implementations can facilitate information extraction and reasoning across diverse data modalities by projecting data into elements in the same embedding space for comparison, combination, or other computations therebetween.

[0203] For example, elements 8-0, . . . , 8-9 can indicate particular locations within a multidimensional embedding space. Some elements can map to a set of discrete locations in the embedding space. For instance, elements that correspond to discrete members of a predetermined vocabulary of tokens can map to discrete locations in the embedding space that are associated with those tokens. Other elements can be continuously distributed across the embedding space. For instance, some data types can be broken down into continuously defined portions (e.g., image patches) that can be described using continuously distributed locations within the embedding space.

[0204] In some implementations, the expressive power of the embedding space may not be limited to meanings associated with any particular set of tokens or other building blocks. For example, a continuous embedding space can encode a spectrum of high-order information. An individual piece of information (e.g., a token) can map to a particular point in that space: for instance, a token for the word “dog” can be projected to an embedded value that points to a particular location in the embedding space associated with canine-related information. Similarly, an image patch of an image of a dog on grass can also be projected into the embedding space. In some implementations, the projection of the image of the dog can be similar to the projection of the word “dog” while also having similarity to a projection of the word “grass,” while potentially being different from both. In some implementations, the projection of the image patch may not exactly align with any single projection of a single word. In some implementations, the projection of the image patch can align with a combination of the projections of the words “dog” and “grass.” In this manner, for instance, ahigh-order embedding space can encode information that can be independent of data modalities in which the information is expressed.

[0205] Task indicator 9 can include a model or model component configured to identify a task being performed and inject, into input sequence 8, an input value represented by element 8-0 that signals which task is being performed. For instance, the input value can be provided as a data type associated with an input modality and projected along with that input modality (e.g., the input value can be a textual task label that is embedded along with other textual data in the input; the input value can be a pixel-based representation of a task that is embedded along with other image data in the input; etc.). The input value can be provided as a data type that differs from or is at least independent from other input(s). For instance, the input value represented by element 8-0 can be a learned within a continuous embedding space.

[0206] Input modalities 10-1, 10-2, and 10-3 can be associated with various different data types (e.g., as described above with respect to input(s) 2 and output(s) 3).

[0207] Data-to-sequence models 11-1, 11-2, and 11-3 can be the same or different from each other. Data-to-sequence models 11-1, 11-2, and 11-3 can be adapted to each respective input modality 10-1, 10-2, and 10-3. For example, a textual data-to-sequence model can subdivide a portion of input text and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model can subdivide an input image and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). An arbitrary datatype data-to-sequence model can subdivide an input of that arbitrary datatype and project the subdivisions into element(s) in input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0208] Data-to-sequence models 11-1, 11-2, and 11-3 can form part of machine- learned sequence processing model(s) 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be jointly trained with or trained independently from machine-learned sequence processing model(s) 4. Data-to-sequence models 11-1, 11-2, and 11-3 can be trained end-to-end with machine-learned sequence processing model(s) 4.

[0209] Figure 16 is a block diagram of an example model development platform 12 that can facilitate creation, adaptation, and refinement of example machine-learned models (e.g., machine-learned model(s) 1, sequence processing model(s) 4, etc.). Model development platform 12 can provide a number of different toolkits that developer systems can employ in the development of new or adapted machine-learned models.

[0210] Model development platform 12 can provide one or more model libraries 13 containing building blocks for new models. Model libraries 13 can include one or more pretrained foundational models 13-1, which can provide a backbone of processing power across various tasks. Model libraries 13 can include one or more pre-trained expert models 13-2, which can be focused on performance in particular domains of expertise. Model libraries 13 can include various model primitives 13-3, which can provide low-level architectures or components (optionally pre-trained), which can be assembled in various arrangements as desired. Model primitives 13-3 can include a library of pre-trained adapters or LoRA modules that can adapt a baseline foundational model to align its outputs with a desired performance profile, augment model capabilities (e.g., to adapt to a different input modality, etc.), and the like.

[0211] Model development platform 12 can receive selections of various model components 14. Model development platform 12 can pass selected model components 14 to a workbench 15 that combines selected model components 14 into a development model 16.

[0212] Workbench 15 can facilitate further refinement and adaptation of development model 16 by leveraging a number of different toolkits integrated with model development platform 12. For example, workbench 15 can facilitate alignment of the development model 16 with a desired performance profile on various tasks using a model alignment toolkit 17.

[0213] Model alignment toolkit 17 can provide a number of tools for causing development model 16 to generate outputs aligned with desired behavioral characteristics. Alignment can include increasing an accuracy, precision, recall, etc. of model outputs. Alignment can include enforcing output styles, schema, or other preferential characteristics of model outputs. Alignment can be general or domain-specific. For instance, a pre-trained foundational model 13-1 can begin with an initial level of performance across multiple domains. Alignment of the pre-trained foundational model 13-1 can include improving a performance in a particular domain of information or tasks (e.g., even at the expense of performance in another domain of information or tasks).

[0214] Model alignment toolkit 17 can integrate one or more dataset(s) 17-1 for aligning development model 16. Curated dataset(s) 17-1 can include labeled or unlabeled training data. Dataset(s) 17-1 can be obtained from public domain datasets. Dataset(s) 17-1 can be obtained from private datasets associated with one or more developer system(s) for the alignment of bespoke machine-learned model(s) customized for private use-cases.

[0215] Pre-training pipelines 17-2 can include a machine-learned model training workflow configured to update development model 16 over large-scale, potentially noisydatasets. For example, pre-training can leverage unsupervised learning techniques (e.g., denoising, etc.) to process large numbers of training instances to update model parameters from an initialized state and achieve a desired baseline performance. Pre-training pipelines 17-2 can leverage unlabeled datasets in dataset(s) 17-1 to perform pre-training. Workbench 15 can implement a pre-training pipeline 17-2 to pre-train development model 16.

[0216] Fine-tuning pipelines 17-3 can include a machine-learned model training workflow configured to refine the model parameters of development model 16 with higher- quality data. Fine-tuning pipelines 17-3 can update development model 16 by conducting supervised training with labeled dataset(s) in dataset(s) 17-1. Fine-tuning pipelines 17-3 can update development model 16 by conducting reinforcement learning using reward signals from user feedback signals. Workbench 15 can implement a fine-tuning pipeline 17-3 to finetune development model 16.

[0217] Prompt libraries 17-4 can include sets of inputs configured to induce behavior aligned with desired performance criteria. Prompt libraries 17-4 can include few-shot prompts (e.g., inputs providing examples of desired model outputs for prepending to a desired runtime query), chain-of-thought prompts (e.g., inputs providing step-by-step reasoning within the exemplars to facilitate thorough reasoning by the model), and the like.

[0218] Example prompts can be retrieved from an available repository of prompt libraries 17-4. Example prompts can be contributed by one or more developer systems using workbench 15.

[0219] In some implementations, pre-trained or fine-tuned models can achieve satisfactory performance without exemplars in the inputs. For instance, zero-shot prompts can include inputs that lack exemplars. Zero-shot prompts can be within a domain within a training dataset or outside of the training domain(s).

[0220] Prompt libraries 17-4 can include one or more prompt engineering tools. Prompt engineering tools can provide workflows for retrieving or learning optimized prompt values. Prompt engineering tools can facilitate directly learning prompt values (e.g., input element values) based one or more training iterations. Workbench 15 can implement prompt engineering tools in development model 16.

[0221] Prompt libraries 17-4 can include pipelines for prompt generation. For example, inputs can be generated using development model 16 itself or other machine- learned models. In this manner, for instance, a first model can process information about a task and output a input for a second model to process in order to perform a step of the task.The second model can be the same as or different from the first model. Workbench 15 can implement prompt generation pipelines in development model 16.

[0222] Prompt libraries 17-4 can include pipelines for context injection. For instance, a performance of development model 16 on a particular task can improve if provided with additional context for performing the task. Prompt libraries 17-4 can include software components configured to identify desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. Workbench 15 can implement context injection pipelines in development model 16.

[0223] Although various training examples described herein with respect to model development platform 12 refer to “pre-training” and “fine-tuning,” it is to be understood that model alignment toolkit 17 can generally support a wide variety of training techniques adapted for training a wide variety of machine-learned models. Example training techniques can correspond to the example training method 1200 described above.

[0224] Model development platform 12 can include a model plugin toolkit 18. Model plugin toolkit 18 can include a variety of tools configured for augmenting the functionality of a machine-learned model by integrating the machine-learned model with other systems, devices, and software components. For instance, a machine-learned model can use tools to increase performance quality where appropriate. For instance, deterministic tasks can be offloaded to dedicated tools in lieu of probabilistically performing the task with an increased risk of error. For instance, instead of autoregressively predicting the solution to a system of equations, a machine-learned model can recognize a tool to call for obtaining the solution and pass the system of equations to the appropriate tool. The tool can be a traditional system of equations solver that can operate deterministically to resolve the system of equations. The output of the tool can be returned in response to the original query. In this manner, tool use can allow some example models to focus on the strengths of machine-learned models — e.g., understanding an intent in an unstructured request for a task — while augmenting the performance of the model by offloading certain tasks to a more focused tool for rote application of deterministic algorithms to a well-defined problem.

[0225] Model plugin toolkit 18 can include validation tools 18-1. Validation tools 18- 1 can include tools that can parse and confirm output(s) of a machine-learned model. Validation tools 18-1 can include engineered heuristics that establish certain thresholds applied to model outputs. For example, validation tools 18-1 can ground the outputs of machine-learned models to structured data sources (e.g., to mitigate “hallucinations”).

[0226] Model plugin toolkit 18 can include tooling packages 18-2 for implementing one or more tools that can include scripts or other executable code that can be executed alongside development model 16. Tooling packages 18-2 can include one or more inputs configured to cause machine-learned model(s) to implement the tools (e.g., few-shot prompts that induce a model to output tool calls in the proper syntax, etc.). Tooling packages 18-2 can include, for instance, fine-tuning training data for training a model to use a tool.

[0227] Model plugin toolkit 18 can include interfaces for calling external application programming interfaces (APIs) 18-3. For instance, in addition to or in lieu of implementing tool calls or tool code directly with development model 16, development model 16 can be aligned to output instruction that initiate API calls to send or obtain data via external systems.

[0228] Model plugin toolkit 18 can integrate with prompt libraries 17-4 to build a catalog of available tools for use with development model 16. For instance, a model can receive, in an input, a catalog of available tools, and the model can generate an output that selects a tool from the available tools and initiates a tool call for using the tool.

[0229] Model development platform 12 can include a computational optimization toolkit 19 for optimizing a computational performance of development model 16. For instance, tools for model compression 19-1 can allow development model 16 to be reduced in size while maintaining a desired level of performance. For instance, model compression 19-1 can include quantization workflows, weight pruning and sparsification techniques, etc. Tools for hardware acceleration 19-2 can facilitate the configuration of the model storage and execution formats to operate optimally on different hardware resources. For instance, hardware acceleration 19-2 can include tools for optimally sharding models for distributed processing over multiple processing units for increased bandwidth, lower unified memory requirements, etc. Tools for distillation 19-3 can provide for the training of lighter-weight models based on the knowledge encoded in development model 16. For instance, development model 16 can be a highly performant, large machine-learned model optimized using model development platform 12. To obtain a lightweight model for running in resource-constrained environments, a smaller model can be a “student model” that learns to imitate development model 16 as a “teacher model.” In this manner, for instance, the investment in learning the parameters and configurations of development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0230] Workbench 15 can implement one, multiple, or none of the toolkits implemented in model development platform 12. Workbench 15 can output an output model 20 based on development model 16. Output model 20 can be a deployment version ofdevelopment model 16. Output model 20 can be a development or training checkpoint of development model 16. Output model 20 can be a distilled, compressed, or otherwise optimized version of development model 16.

[0231] Figure 17 is a block diagram of an example training flow for training a machine-learned development model 16. One or more portion(s) of the example training flow can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of the example training flow can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of the example training flow can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. FIG. 17 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. FIG. 17 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of the example training flow can be performed additionally, or alternatively, by other systems.

[0232] Initially, development model 16 can persist in an initial state as an initialized model 21. Development model 16 can be initialized with weight values. Initial weight values can be random or based on an initialization schema. Initial weight values can be based on prior pre-training for the same or for a different model.

[0233] Initialized model 21 can undergo pre-training in a pre-training stage 22. Pretraining stage 22 can be implemented using one or more pre-training pipelines 17-2 over data from dataset(s) 17-1. Pre-training can be omitted, for example, if initialized model 21 is already pre-trained (e.g., development model 16 contains, is, or is based on a pre-trained foundational model or an expert model).

[0234] Pre-trained model 23 can then be a new version of development model 16, which can persist as development model 16 or as a new development model. Pre-trained model 23 can be the initial state if development model 16 was already pre-trained. Pre-trained model 23 can undergo fine-tuning in a fine-tuning stage 24. Fine-tuning stage 24 can be implemented using one or more fine-tuning pipelines 17-3 over data from dataset(s) 17-1.Fine-tuning can be omitted, for example, if a pre-trained model as satisfactory performance, if the model was already fine-tuned, or if other tuning approaches are preferred.

[0235] Fine-tuned model 29 can then be a new version of development model 16, which can persist as development model 16 or as a new development model. Fine-tuned model 29 can be the initial state if development model 16 was already fine-tuned. Fine-tuned model 29 can undergo refinement with user feedback 26. For instance, refinement with user feedback 26 can include reinforcement learning, optionally based on human feedback from human users of fine-tuned model 25. As reinforcement learning can be a form of fine-tuning, it is to be understood that fine-tuning stage 24 can subsume the stage for refining with user feedback 26. Refinement with user feedback 26 can produce a refined model 27. Refined model 27 can be output to downstream system(s) 28 for deployment or further development.

[0236] In some implementations, computational optimization operations can be applied before, during, or after each stage. For instance, initialized model 21 can undergo computational optimization 29-1 (e.g., using computational optimization toolkit 19) before pre-training stage 22. Pre-trained model 23 can undergo computational optimization 29-2 (e.g., using computational optimization toolkit 19) before fine-tuning stage 24. Fine-tuned model 25 can undergo computational optimization 29-3 (e.g., using computational optimization toolkit 19) before refinement with user feedback 26. Refined model 27 can undergo computational optimization 29-4 (e.g., using computational optimization toolkit 19) before output to downstream system(s) 28. Computational optimization(s) 29-1, . . . , 29-4 can all be the same, all be different, or include at least some different optimization techniques.

[0237] Figure 18 is a block diagram of an inference system for operating one or more machine-learned model(s) 1 to perform inference (e.g., for training, for deployment, etc.). A model host 31 can receive machine-learned model(s) 1. Model host 31 can host one or more model instance(s) 31-1, which can be one or multiple instances of one or multiple models. Model host 31 can host model instance(s) 31-1 using available compute resources 31-2 associated with model host 31.

[0238] Model host 31 can perform inference on behalf of one or more client(s) 32. Client(s) 32 can transmit an input request 33 to model host 31. Using input request 33, model host 31 can obtain input(s) 2 for input to machine-learned model(s) 1. Machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3. Using output(s) 3, model host 31 can return an output payload 34 for responding to input request 33 from client(s) 32. Output payload 34 can include or be based on output(s) 3.

[0239] Model host 31 can leverage various other resources and tools to augment the inference task. For instance, model host 31 can communicate with tool interfaces 35 to facilitate tool use by model instance(s) 31-1. Tool interfaces 35 can include local or remote APIs. Tool interfaces 35 can include integrated scripts or other software functionality. Model host 31 can engage online learning interface(s) 36 to facilitate ongoing improvements to machine-learned model(s) 1. For instance, online learning interface(s) 36 can be used within reinforcement learning loops to retrieve user feedback on inferences served by model host 31. Model host 31 can access runtime data source(s) 37 for augmenting input(s) 2 with additional contextual information. For instance, runtime data source(s) 37 can include a knowledge graph 37-1 that facilitates structured information retrieval for information associated with input request(s) 33 (e.g., a search engine service). Runtime data source(s) 37 can include public or private, external or local database(s) 37-2 that can store information associated with input request(s) 33 for augmenting input(s) 2. Runtime data source(s) 37 can include account data 37-3 which can be retrieved in association with a user account corresponding to a client 32 for customizing the behavior of model host 31 accordingly.

[0240] Model host 31 can be implemented by one or multiple computing devices or systems. Client(s) 2 can be implemented by one or multiple computing devices or systems, which can include computing devices or systems shared with model host 31.

[0241] For example, model host 31 can operate on a server system that provides a machine-learning service to client device(s) that operate client(s) 32 (e.g., over a local or wide-area network). Client device(s) can be end-user devices used by individuals. Client device(s) can be server systems that operate client(s) 32 to provide various functionality as a service to downstream end-user devices.

[0242] In some implementations, model host 31 can operate on a same device or system as client(s) 32. Model host 31 can be a machine-learning service that runs on-device to provide machine-learning functionality to one or multiple applications operating on a client device, which can include an application implementing client(s) 32. Model host 31 can be a part of a same application as client(s) 32. For instance, model host 31 can be a subroutine or method implemented by one part of an application, and client(s) 32 can be another subroutine or method that engages model host 31 to perform inference functions within the application. It is to be understood that model host 31 and client(s) 32 can have various different configurations.

[0243] Model instance(s) 31-1 can include one or more machine-learned models that are available for performing inference. Model instance(s) 31-1 can include weights or othermodel components that are stored on in persistent storage, temporarily cached, or loaded into high-speed memory. Model instance(s) 31-1 can include multiple instance(s) of the same model (e.g., for parallel execution of more requests on the same model). Model instance(s) 31-1 can include instance(s) of different model(s). Model instance(s) 31-1 can include cached intermediate states of active or inactive model(s) used to accelerate inference of those models. For instance, an inference session with a particular model may generate significant amounts of computational results that can be re-used for future inference runs (e.g., using a KV cache for transformer-based models). These computational results can be saved in association with that inference session so that session can be executed more efficiently when resumed.

[0244] Compute resource(s) 31-2 can include one or more processors (central processing units, graphical processing units, tensor processing units, machine-learning accelerators, etc.) connected to one or more memory devices. Compute resource(s) 31-2 can include a dynamic pool of available resources shared with other processes. Compute resource(s) 31-2 can include memory devices large enough to fit an entire model instance in a single memory instance. Compute resource(s) 31-2 can also shard model instance(s) across multiple memory devices (e.g., using data parallelization or tensor parallelization, etc.). This can be done to increase parallelization or to execute a large model using multiple memory devices which individually might not be able to fit the entire model into memory.

[0245] Input request 33 can include data for input(s) 2. Model host 31 can process input request 33 to obtain input(s) 2. Input(s) 2 can be obtained directly from input request 33 or can be retrieved using input request 33. Input request 33 can be submitted to model host 31 via an API.

[0246] Model host 31 can perform inference over batches of input requests 33 in parallel. For instance, a model instance 31-1 can be configured with an input structure that has a batch dimension. Separate input(s) 2 can be distributed across the batch dimension (e.g., rows of an array). The separate input(s) 2 can include completely different contexts. The separate input(s) 2 can be multiple inference steps of the same task. The separate input(s) 2 can be staggered in an input structure, such that any given inference cycle can be operating on different portions of the respective input(s) 2. In this manner, for instance, model host 31 can perform inference on the batch in parallel, such that output(s) 3 can also contain the batch dimension and return the inference results for the batched input(s) 2 in parallel. In this manner, for instance, batches of input request(s) 33 can be processed in parallel for higher throughput of output payload(s) 34.

[0247] Output payload 34 can include or be based on output(s) 3 from machine- learned model(s) 1. Model host 31 can process output(s) 3 to obtain output payload 34. This can include chaining multiple rounds of inference (e.g., iteratively, recursively, across the same model(s) or different model(s)) to arrive at a final output for a task to be returned in output payload 34. Output payload 34 can be transmitted to client(s) 32 via an API.

[0248] Online learning interface(s) 36 can facilitate reinforcement learning of machine-learned model(s) 1. Online learning interface(s) 36 can facilitate reinforcement learning with human feedback (RLHF). Online learning interface(s) 36 can facilitate federated learning of machine-learned model(s) 1.

[0249] Model host 31 can access a library of pre-trained adapters or LoRA modules that can adapt a baseline model to align its outputs with a desired performance profile, augment model capabilities (e.g., to adapt to a different input modality, etc.), and the like. For instance, model host 31 can receive an input request to load a customized model, and model host 31 can retrieve one or more components to adapt a baseline model to the custom profile. Model host 31 can determine that a particular functionality is needed for a particular task (e.g., based on an output of a model that preprocesses an input) and retrieve a pre-trained component accordingly.

[0250] Model host 31 can execute machine-learned model(s) 1 to perform inference for various tasks using various types of data. For example, various different input(s) 2 and output(s) 3 can be used for various different tasks. In some implementations, input(s) 2 can be or otherwise represent image data. Machine-learned model(s) 1 can process the image data to generate an output. As an example, machine-learned model(s) 1 can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an image segmentation output. As another example, machine-learned model(s) 1 can process the image data to generate an image classification output. As another example, machine-learned model(s) 1 can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, machine- learned model(s) 1 can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, machine-learned model(s) 1 can process the image data to generate an upscaled image data output. As another example, machine-learned model(s) 1 can process the image data to generate a prediction output.

[0251] In some implementations, the task is a computer vision task. In some cases, input(s) 2 includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

[0252] In some implementations, input(s) 2 can be or otherwise represent natural language data. Machine-learned model(s) 1 can process the natural language data to generate an output. As an example, machine-learned model(s) 1 can process the natural language data to generate a language encoding output. As another example, machine-learned model(s) 1 can process the natural language data to generate a latent text embedding output. As another example, machine-learned model(s) 1 can process the natural language data to generate a translation output. As another example, machine-learned model(s) 1 can process the natural language data to generate a classification output. As another example, machine-learned model(s) 1 can process the natural language data to generate a textual segmentation output. As another example, machine-learned model(s) 1 can process the natural language data to generate a semantic intent output. As another example, machine-learned model(s) 1 can process the natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, machine-learned model(s) 1 can process the natural language data to generate a prediction output (e.g., one or more predicted next portions of natural language content).

[0253] In some implementations, input(s) 2 can be or otherwise represent speech data (e.g., data describing spoken natural language, such as audio data, textual data, etc.). Machine-learned model(s) 1 can process the speech data to generate an output. As an example, machine-learned model(s) 1 can process the speech data to generate a speech recognition output. As another example, machine-learned model(s) 1 can process the speech data to generate a speech translation output. As another example, machine-learned model(s) 1 can process the speech data to generate a latent embedding output. As another example, machine-learned model(s) 1 can process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.). As another example, machine-learned model(s) 1 can process the speech data to generate a prediction output.

[0254] In some implementations, input(s) 2 can be or otherwise represent latent encoding data (e.g., a latent space representation of an input, etc.). Machine-learned model(s) 1 can process the latent encoding data to generate an output. As an example, machine- learned model(s) 1 can process the latent encoding data to generate a recognition output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a reconstruction output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a search output. As another example, machine- learned model(s) 1 can process the latent encoding data to generate a reclustering output. As another example, machine-learned model(s) 1 can process the latent encoding data to generate a prediction output.

[0255] In some implementations, input(s) 2 can be or otherwise represent statistical data. Statistical data can be, represent, or otherwise include data computed and / or calculated from some other data source. Machine-learned model(s) 1 can process the statistical data to generate an output. As an example, machine-learned model(s) 1 can process the statistical data to generate a recognition output. As another example, machine-learned model(s) 1 can process the statistical data to generate a prediction output. As another example, machine- learned model(s) 1 can process the statistical data to generate a classification output. As another example, machine-learned model(s) 1 can process the statistical data to generate a segmentation output. As another example, machine-learned model(s) 1 can process thestatistical data to generate a visualization output. As another example, machine-learned model(s) 1 can process the statistical data to generate a diagnostic output.

[0256] In some implementations, input(s) 2 can be or otherwise represent sensor data. Machine-learned model(s) 1 can process the sensor data to generate an output. As an example, machine-learned model(s) 1 can process the sensor data to generate a recognition output. As another example, machine-learned model(s) 1 can process the sensor data to generate a prediction output. As another example, machine-learned model(s) 1 can process the sensor data to generate a classification output. As another example, machine-learned model(s) 1 can process the sensor data to generate a segmentation output. As another example, machine-learned model(s) 1 can process the sensor data to generate a visualization output. As another example, machine-learned model(s) 1 can process the sensor data to generate a diagnostic output. As another example, machine-learned model(s) 1 can process the sensor data to generate a detection output.

[0257] In some implementations, machine-learned model(s) 1 can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data and the output may comprise compressed audio data. In another example, the input includes visual data (e.g. one or more images or videos), the output comprises compressed visual data, and the task is a visual data compression task. In another example, the task may comprise generating an embedding for input data (e.g. input audio or visual data). In some cases, the input includes audio data representing a spoken utterance and the task is a speech recognition task. The output may comprise a text output which is mapped to the spoken utterance. In some cases, the task comprises encrypting or decrypting input data. In some cases, the task comprises a microprocessor performance task, such as branch prediction or memory address translation.

[0258] In some implementations, the task is a generative task, and machine-learned model(s) 1 can be configured to output content generated in view of input(s) 2. For instance, input(s) 2 can be or otherwise represent data of one or more modalities that encodes context for generating additional content.

[0259] In some implementations, the task can be a text completion task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent textual data and to generate output(s) 3 that represent additional textual data that completes a textual sequence that includes input(s) 2. For instance, machine-learned model(s) 1 can be configured togenerate output(s) 3 to complete a sentence, paragraph, or portion of text that follows from a portion of text represented by input(s) 2.

[0260] In some implementations, the task can be an instruction following task. Machine-learned model(s) 1 can be configured to process input(s) 2 that represent instructions to perform a function and to generate output(s) 3 that advance a goal of satisfying the instruction function (e.g., at least a step of a multi-step procedure to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent textual data (e.g., natural language instructions for a task to be performed) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by textual instructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward accomplishing the requested functionality. For instance, an initial output can be executed by an external system or be processed by machine-learned model(s) 1 to complete an initial step of performing a function. Multiple steps can be performed, with a final output being obtained that is responsive to the initial instructions.

[0261] In some implementations, the task can be an instruction following task that is to be performed by a robot and / or autonomous vehicle (e.g., navigation task, assembly task, movement or object manipulation task, and / or any other robotic / vehicular task and the like). Machine-learned model(s) 1 can be configured to process input(s) 2 that represent instructions for the robot or autonomous vehicle to perform a function and to generate output(s) 3 that advance a goal of satisfying the instruction function (e.g., at least a step of a multi-step procedure for controlling the robot / autonomous vehicle to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent speech or textual data (e.g., natural language instructions or spoken request for a task to be performed by the robot / autonomous vehicle) and machine- learned model(s) 1 can process input(s) 2 to generate output(s) 3 that control the robot or autonomous vehicle to be responsive to the instruction function. The output(s) 3 may, without limitation, for example represent textual data and / or computer / machine data responsive to the instructions (e.g., natural language responses, programming languageresponses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by textual / spoken instructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data and / or machine data responsive to the instructions (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward controlling the robot and / or autonomous vehicle in accomplishing the requested functionality (e.g., navigation function / task, assembly function / task, movement or object manipulation function / task, and / or any other robotic / vehicular function / task and the like). For instance, an initial output can be executed by an external system or control system of the robot / vehicle or be processed by machine- learned model(s) 1 to complete an initial step of the robot / vehicle performing a function. Multiple steps can be performed for controlling the robot / vehicle, with a final output being obtained that is responsive to the initial instructions.

[0262] In some implementations, the task can be a question answering task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent a question to answer and to generate output(s) 3 that advance a goal of returning an answer to the question (e.g., at least a step of a multi-step procedure to perform the function). Output(s) 3 can represent data of the same or of a different modality as input(s) 2. For instance, input(s) 2 can represent textual data (e.g., natural language instructions for a task to be performed) and machine- learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the question (e.g., natural language responses, programming language responses, machine language responses, etc.). Input(s) 2 can represent image data (e.g., image-based instructions for a task to be performed, optionally accompanied by textual instructions) and machine-learned model(s) 1 can process input(s) 2 to generate output(s) 3 that represent textual data responsive to the question (e.g., natural language responses, programming language responses, machine language responses, etc.). One or more output(s) 3 can be iteratively or recursively generated to sequentially process and accomplish steps toward answering the question. For instance, an initial output can be executed by an external system or be processed by machine-learned model(s) 1 to complete an initial step of obtaining an answer to the question (e.g., querying a database, performing a computation, executing a script, etc.). Multiple steps can be performed, with a final output being obtained that is responsive to the question.

[0263] In some implementations, the task can be an image generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of image content. The context can include text data, image data, audio data, etc. Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent image data that depicts imagery related to the context. For instance, machine-learned model(s) 1 can be configured to generate pixel data of an image. Values for channel(s) associated with the pixels in the pixel data can be selected based on the context (e.g., based on a probability determined based on the context).

[0264] In some implementations, the task can be an audio generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of audio content. The context can include text data, image data, audio data, etc. Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent audio data related to the context. For instance, machine-learned model(s) 1 can be configured to generate waveform data in the form of an image (e.g., a spectrogram). Values for channel(s) associated with pixels of the image can be selected based on the context. Machine- learned model(s) 1 can be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. Values of the sequence can be selected based on the context (e.g., based on a probability determined based on the context).

[0265] In some implementations, the task can be a data generation task. Machine- learned model(s) 1 can be configured to process input(s) 2 that represent context regarding a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistical data, etc.). The desired data can be, for instance, synthetic data for training other machine-learned models. The context can include arbitrary data type(s). Machine-learned model(s) 1 can be configured to generate output(s) 3 that represent data that aligns with the desired data. For instance, machine-learned model(s) 1 can be configured to generate data values for populating a dataset. Values for the data object(s) can be selected based on the context (e.g., based on a probability determined based on the context).

[0266] Figure 19 is a block diagram of an example networked computing system that can perform aspects of example implementations of the present disclosure. The system can include a number of computing devices and systems that are communicatively coupled over a network 49. An example computing device 50 is described to provide an example of a computing device that can perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). An example server computing system 60 is described as an example of a server computing system that can perform any aspect of the presentdisclosure (e.g., implementing model host 31, client(s) 32, or both). Computing device 50 and server computing system(s) 60 can cooperatively interact (e.g., over network 49) to perform any aspect of the present disclosure (e.g., implementing model host 31, client(s) 32, or both). Model development platform system 70 is an example system that can host or serve model development platform(s) 12 for development of machine-learned models. Third-party system(s) 80 are example system(s) with which any of computing device 50, server computing system(s) 60, or model development platform system(s) 70 can interact in the performance of various aspects of the present disclosure (e.g., engaging third-party tools, accessing third-party databases or other resources, etc.).

[0267] Network 49 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over network 49 can be carried via any type of wired or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL). Network 49 can also be implemented via a system bus. For instance, one or more devices or systems of Figure 19 can be co-located with, contained by, or otherwise integrated into one or more other devices or systems.

[0268] Computing device 50 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 can be a client computing device. Computing device 50 can be an end-user computing device. Computing device 50 can be a computing device of a service provided that provides a service to an end user (who may use another computing device to interact with computing device 50).

[0269] Computing device 50 can include one or more processors 51 and a memory 52. Processor(s) 51 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 52 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 52 can store data 53 and instructions 54 which can be executed by processor(s) 51 to cause computing device 50 to perform operations. The operations can implement any one ormultiple features described herein. The operations can implement example methods and techniques described herein.

[0270] Computing device 50 can also include one or more input components that receive user input. For example, a user input component can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, camera, LIDAR, a physical keyboard or other buttons, or other means by which a user can provide user input.

[0271] Computing device 50 can store or include one or more machine-learned models 55. Machine-learned models 55 can include one or more machine-learned model(s) 1, such as a sequence processing model 4. Machine-learned models 55 can include one or multiple model instance(s) 31-1. Machine-learned model(s) 55 can be received from server computing system(s) 60, model development platform system 70, third party system(s) 80 (e.g., an application distribution platform), or developed locally on computing device 50. Machine-learned model(s) 55 can be loaded into memory 52 and used or otherwise implemented by processor(s) 51. Computing device 50 can implement multiple parallel instances of machine-learned model(s) 55.

[0272] Server computing system(s) 60 can include one or more processors 61 and a memory 62. Processor(s) 61 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 62 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 62 can store data 63 and instructions 64 which can be executed by processor(s) 61 to cause server computing system(s) 60 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein.

[0273] In some implementations, server computing system 60 includes or is otherwise implemented by one or multiple server computing devices. In instances in which server computing system 60 includes multiple server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0274] Server computing system 60 can store or otherwise include one or more machine-learned models 65. Machine-learned model(s) 65 can be the same as or different from machine-learned model(s) 55. Machine-learned models 65 can include one or more machine-learned model(s) 1, such as a sequence processing model 4. Machine-learned models 65 can include one or multiple model instance(s) 31-1. Machine-learned model(s) 65 can be received from computing device 50, model development platform system 70, third party system(s) 80, or developed locally on server computing system(s) 60. Machine-learned model(s) 65 can be loaded into memory 62 and used or otherwise implemented by processor(s) 61. Server computing system(s) 60 can implement multiple parallel instances of machine-learned model(s) 65.

[0275] In an example configuration, machine-learned models 65 can be included in or otherwise stored and implemented by server computing system 60 to establish a client-server relationship with computing device 50 for serving model inferences. For instance, server computing system(s) 60 can implement model host 31 on behalf of client(s) 32 on computing device 50. For instance, machine-learned models 65 can be implemented by server computing system 60 as a portion of a web service (e.g., remote machine-learned model hosting service, such as an online interface for performing machine-learned model operations over a network on server computing system(s) 60). For instance, server computing system(s) 60 can communicate with computing device 50 over a local intranet or internet connection. For instance, computing device 50 can be a workstation or endpoint in communication with server computing system(s) 60, with implementation of machine-learned models 65 being managed by server computing system(s) 60 to remotely perform inference (e.g., for runtime or training operations), with output(s) returned (e.g., cast, streamed, etc.) to computing device 50. Machine-learned models 65 can work cooperatively or interoperatively with machine- learned models 55 on computing device 50 to perform various tasks.

[0276] Model development platform system(s) 70 can include one or more processors 71 and a memory 72. Processor(s) 71 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 72 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 72 can store data 73 and instructions 74 which can be executed by processor(s) 71 to cause model development platform system(s) 70 to perform operations. The operations can implement any one or multiple features described herein. The operationscan implement example methods and techniques described herein. Example operations include the functionality described herein with respect to model development platform 12. This and other functionality can be implemented by developer tool(s) 75.

[0277] Third-party system(s) 80 can include one or more processors 81 and a memory 82. Processor(s) 81 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory 82 can include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 82 can store data 83 and instructions 84 which can be executed by processor(s) 81 to cause third-party system(s) 80 to perform operations. The operations can implement any one or multiple features described herein. The operations can implement example methods and techniques described herein. Example operations include the functionality described herein with respect to tools and other external resources called when training or performing inference with machine-learned model(s) 1, 4, 16, 20, 55, 65, etc. (e.g., third-party resource(s) 85).

[0278] Figure 19 illustrates one example arrangement of computing systems that can be used to implement the present disclosure. Other computing system configurations can be used as well. For example, in some implementations, one or both of computing system 50 or server computing system(s) 60 can implement all or a portion of the operations of model development platform system 70. For example, computing system 50 or server computing system(s) 60 can implement developer tool(s) 75 (or extensions thereof) to develop, update / train, or refine machine-learned models 1, 4, 16, 20, 55, 65, etc. using one or more techniques described herein with respect to model alignment toolkit 17. In this manner, for instance, computing system 50 or server computing system(s) 60 can develop, update / train, or refine machine-learned models based on local datasets (e.g., for model personalization / customization, as permitted by user data preference selections).

[0279] Figure 20 is a block diagram of an example computing device 98 that performs according to example embodiments of the present disclosure. Computing device 98 can be a user computing device or a server computing device (e.g., computing device 50, server computing system(s) 60, etc.). Computing device 98 can implement model host 31. For instance, computing device 98 can include a number of applications (e.g., applications 1 through N). Each application can contain its own machine learning library and machine- learned model(s). For example, each application can include a machine-learned model.Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. As illustrated in Figure 20, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0280] Figure 21 is a block diagram of an example computing device 99 that performs according to example embodiments of the present disclosure. Computing device 99 can be the same as or different from computing device 98. Computing device 99 can be a user computing device or a server computing device (e.g., computing device 50, server computing system(s) 60, etc.). Computing device 98 can implement model host 31. For instance, computing device 99 can include a number of applications (e.g., applications 1 through N). Each application can be in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0281] The central intelligence layer can include a number of machine-learned models. For example, as illustrated in Figure 21, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of computing device 99.

[0282] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for computing device 99. As illustrated in Figure 21, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0283] Figure 22 depicts a flowchart of a method 2200 for training one or more machine-learned models according to aspects of the present disclosure. For instance, anexample machine-learned model can include one or more models of machine-learned models 108. For instance, an example model can be trained to provide inferences for machine- learned agent system 102. An example model can be trained to provide inferences for machine-learned trace generation system 106.

[0284] One or more portion(s) of example method 2200 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of example method 2200 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 2200 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 22 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Figure 22 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 2200 can be performed additionally, or alternatively, by other systems.

[0285] At 2202, example method 2200 can include obtaining a reference trace describing interactions between a machine-learned agent system and an environment, wherein the reference trace includes a reference sequence of interaction objects associated with performance of a task, each respective reference interaction object of the reference sequence of interaction objects corresponding to a respective reference environment state and a respective reference action executed on the respective reference environment state.

[0286] At 2204, example method 2200 can include generating, using a machine- learned trace generation system, a forecasted trace based on the reference trace, the forecasted trace comprising a forecasted sequence of interaction objects that is predicted to continue the reference sequence of interaction objects from a branching position toward performance of the task, each respective forecasted interaction object of the forecasted sequence of interaction objects corresponding to a respective forecasted environment state and a respective forecasted action executed on the respective forecasted environment state.

[0287] At 2206, example method 2200 can include training the machine-learned agent system using the forecasted trace sequence.

[0288] Figure 23 depicts a flowchart of a method 2300 for training one or more machine-learned models according to aspects of the present disclosure. For instance, an example machine-learned model can include one or more models of machine-learned models 108. For instance, an example model can be trained to provide inferences for machine- learned agent system 102. An example model can be trained to provide inferences for machine-learned trace generation system 106.

[0289] One or more portion(s) of example method 2300 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of example method 2300 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 2300 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 23 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Figure 23 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 2300 can be performed additionally, or alternatively, by other systems.

[0290] Example method 2300 can include any one or more implementations of 2202 and 2204 as described above with respect to example method 2200.

[0291] At 2306, example method 2300 can include executing, by the machine-learned agent system, a predicted action on a current environment state, wherein the predicted action is determined based on the forecasted trace. For example, executing the predicted action may include sending data representative of the predicted action to a control system configured for controlling a technical system operating in a physical environment such as, for example, a robotic system or platform, an autonomous / semi -autonomous vehicle or platform, a manufacturing plant or power station, a home control system, and / or any other technical system operating in the physical environment. For example, executing the predicted action may include sending data representative of the predicted action to a mobile application or operating system of a computing device / mobile device for controlling the computing device / mobile device when operating in a digital environment such as, for example, a roboticsystem or platform, an autonomous / semi -autonomous vehicle or platform, a manufacturing plant or power station, a home control system, and / or any other technical system operating in the physical environment.

[0292] In some implementations, example method 2200 or example method 2300 includes determining a resulting environment state that results from execution of a predicted action on a reference environment state associated with the branching position, the resulting environment state associated with a subsequent position in the reference trace. For instance, a branching position can be a state at step t = T, with an action applied to that state forecasted to lead to different resulting states with different corresponding probabilities. For instance, a machine-learned trace generation model can be sampled to obtain various different forecasts for the resulting state.

[0293] In some implementations, example method 2200 or example method 2300 includes determining a feedback signal based on a comparison of the resulting environment state against a corresponding state in the forecasted trace that is associated with the subsequent position of the forecasted trace. For instance, an actual reference state can be obtained that corresponds to a forecasted state. For example, after forecasting a state and selecting an action based on the forecast, an actual observed state can be obtained after executing the selected action. In some implementations, example method 2200 or example method 2300 includes training the machine-learned trace generation system based on the feedback signal.

[0294] In some implementations of example method 2200 or example method 2300, the predicted action is based on the forecasted trace.

[0295] In some implementations, example method 2200 or example method 2300 includes processing, using the machine-learned trace generation system, at least a portion of the reference trace to generate the forecasted trace. In some implementations, example method 2200 or example method 2300 includes processing, using the machine-learned agent system, at least a portion of the forecasted trace to generate the predicted action.

[0296] In some implementations of example method 2200 or example method 2300, determining the resulting environment state includes executing the predicted action on the environment. In some implementations of example method 2200 or example method 2300, determining the resulting environment state includes receiving, from the environment, a state update.

[0297] In some implementations, example method 2200 or example method 2300 includes constructing the reference trace by recording actions executed and state updates received by the machine-learned agent system in a deployment environment.

[0298] In some implementations, example method 2200 or example method 2300 includes maintaining the reference trace in a buffer that retains a history of recent interaction objects, wherein the forecasted trace is generated based on the contents of the buffer.

[0299] In some implementations of example method 2200 or example method 2300, a respective forecasted trace is generated for each respective turn of a multi-turn interaction, wherein each respective turn provides a respective branching position.

[0300] In some implementations, example method 2200 or example method 2300 can include training the machine learned agent system with variant traces.

[0301] In some implementations, example method 2200 or example method 2300 can include obtaining a seed representation of the reference trace.

[0302] In some implementations, example method 2200 or example method 2300 can include generating a variant trace based on the baseline trace. In some implementations of example method 2200 or example method 2300, generating the variant trace can include providing, to the machine-learned trace generation system, the seed representation to generate the variant trace that includes a different environment state from the reference trace, wherein the different environmental state is generated using one or more inferences obtained from the machine-learned trace generation system. In some implementations of example method 2200 or example method 2300, generating the variant trace includes determining, based on a goal state of the reference trace associated with the task, a goal state of the variant trace. In some implementations, example method 2200 or example method 2300 can include processing, using the machine-learned agent system, at least a portion of the variant trace to generate an interaction output. In some implementations, example method 2200 or example method 2300 can include evaluating the interaction output using the goal state of the variant trace. In some implementations, example method 2200 or example method 2300 can include training the machine-learned agent system based on the evaluation of the interaction output.

[0303] Figure 24 depicts a flowchart of a method 2400 for training one or more machine-learned models according to aspects of the present disclosure. For instance, an example machine-learned model can include one or more models of machine-learned models 108. For instance, an example model can be trained to provide inferences for machine- learned agent system 102. An example model can be trained to provide inferences for machine-learned trace generation system 106.

[0304] One or more portion(s) of example method 2400 can be implemented by a computing system that includes one or more computing devices such as, for example, computing systems described with reference to the other figures. Each respective portion of example method 2400 can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portion(s) of example method 2400 can be implemented on the hardware components of the device(s) described herein, for example, to train one or more systems or models. Figure 24 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Figure 24 is described with reference to elements / terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method 2400 can be performed additionally, or alternatively, by other systems.

[0305] At 2402, example method 2400 can include obtaining a seed representation of a reference trace.

[0306] At 2404, example method 2400 can include generating a variant trace based on the reference trace.

[0307] At 2404-1, generating the variant trace can include providing, to a machine- learned trace generation system, the seed representation to generate the variant trace that includes a different environment state from the reference trace, wherein the different environmental state is generated using one or more inferences obtained from the machine- learned trace generation system.

[0308] At 2404-2, generating the variant trace can include determining, based on a goal state of the reference trace, a goal state of the variant trace.

[0309] At 2406, example method 2400 can include processing, using the machine- learned agent system, at least a portion of the variant trace to generate an interaction output.

[0310] At 2408, example method 2400 can include evaluating the interaction output using the goal state of the variant trace.

[0311] At 2410, example method 2400 can include training the machine-learned agent system based on the evaluation of the interaction output.

[0312] In some implementations of example method 2200, example method 2300, or example method 2400, training the machine-learned agent system includes training amachine-learned model used by the machine-learned agent system (e.g., one or more of model(s) 108).

[0313] In some implementations of example method 2200, example method 2300, or example method 2400, training the machine-learned trace generation system includes training a machine-learned model used by the machine-learned trace generation system (e.g., one or more of model(s) 108).

[0314] In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned agent system was trained based on a plurality of exemplar traces using reinforcement learning. In some implementations of example method 2200, example method 2300, or example method 2400, a training reward for a respective exemplar trace was based on an action-level reward associated with each action in the respective exemplar trace. In some implementations of example method 2200, example method 2300, or example method 2400, a training reward for a respective exemplar trace was based on a trace-level reward associated with completion of a task associated with the respective exemplar trace.

[0315] In some implementations of example method 2200, example method 2300, or example method 2400, the respective forecasted environment state includes a textual description of the environment generated by the machine-learned sequence processing model. In some implementations of example method 2200, example method 2300, or example method 2400, the respective forecasted environment state includes image data descriptive of the environment, the image data generated using regression of new image tokens conditioned on textual inputs. In some implementations of example method 2200, example method 2300, or example method 2400, the respective forecasted environment state includes image data descriptive of the environment, the image data generated using iterative de-noising of new image data conditioned on textual inputs.

[0316] In some implementations of example method 2200, example method 2300, or example method 2400, the reference trace is processed by the machine-learned trace generation system as an input sequence of textual content interleaved with image content, wherein each interaction object of the reference trace is represented in the input sequence by a first portion representing an embedded textual description of the corresponding environment state and a second portion representing an embedded image description of the corresponding environment state, the second portion being contiguous with the first portion.

[0317] In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned agent system controls a robotic platform, and thetask is a navigation task or an object manipulation task. In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned agent system controls a computing platform, and the task is a data processing task or a user interface interaction task.

[0318] In some implementations, example method 2200, example method 2300, or example method 2400 includes determining an uncertainty measure associated with an action in the reference trace. In some implementations, example method 2200, example method 2300, or example method 2400 includes determining, based on the uncertainty measure, a number of forecasted traces to generate.

[0319] In some implementations of example method 2200, example method 2300, or example method 2400, the uncertainty measure is obtained using the shape of an output distribution of the machine-learned agent system.

[0320] In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned agent system and the machine-learned trace generation system use the same machine-learned model to process inputs and generate outputs. In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned agent system and the machine-learned trace generation system use different machine-learned models to process inputs and generate outputs.

[0321] In some implementations of example method 2200, example method 2300, or example method 2400, determining the goal state includes inheriting the goal state from the reference trace based on an alignment of the variant trace with the reference trace. In some implementations of example method 2200, example method 2300, or example method 2400, determining the goal state includes generating the goal state of the variant trace using the machine-learned trace generation system.

[0322] In some implementations of example method 2200, example method 2300, or example method 2400, obtaining the seed representation includes generating a condensed representation of the reference trace, wherein the reference trace includes a sequence of interaction objects. In some implementations of example method 2200, example method 2300, or example method 2400, generating the variant trace includes processing, using the machine-learned trace generation system, the condensed representation and an initial portion of the reference trace to generate one or more interaction objects that are predicted to continue the sequence of interaction objects.

[0323] In some implementations, example method 2200, example method 2300, or example method 2400 includes In some implementations of example method 2200, example method 2300, or example method 2400, generating a condensed representation of the reference trace includes generating a plurality of candidate condensed representations over one or more iterations. In some implementations of example method 2200, example method 2300, or example method 2400, a respective iteration includes processing, using the machine- learned trace generation system, a respective candidate condensed representation to generate a respective output. In some implementations of example method 2200, example method 2300, or example method 2400, a respective iteration includes evaluating an alignment of the respective output to the reference trace. In some implementations of example method 2200, example method 2300, or example method 2400, a respective iteration includes generating a new candidate condensed representation based on the alignment. In some implementations of example method 2200, example method 2300, or example method 2400, generating a condensed representation of the reference trace includes selecting the condensed representation from the plurality of candidate condensed representations.

[0324] In some implementations, example method 2200, example method 2300, or example method 2400 includes determining an alignment between the baseline trace and the variant trace. In some implementations of example method 2200, example method 2300, or example method 2400, the alignment is determined based on a similarity comparison of an environment state of the baseline trace and an environment state of the variant trace. In some implementations of example method 2200, example method 2300, or example method 2400, the alignment is determined based on a similarity comparison of an action of the baseline trace and an action of the variant trace.

[0325] In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned trace generation system is configured to limit a divergence of the variant trace from the baseline trace.

[0326] In some implementations, example method 2200, example method 2300, or example method 2400 includes instructing the machine-learned trace generation system to generate the variant trace, wherein the instructions indicate that the variant trace is to share at least one feature of the goal state of the baseline trace.

[0327] In some implementations of example method 2200, example method 2300, or example method 2400, the baseline trace describes performance of a task in a baseline environment, and wherein the variant trace describes performance of the same task in a new environment.

[0328] In some implementations of example method 2200, example method 2300, or example method 2400, the baseline trace includes sequence of interaction objects, each object comprising a state and an action acting upon that state. In some implementations of example method 2200, example method 2300, or example method 2400, the variant trace includes sequence of interaction objects, each object comprising a state and an action acting upon that state.

[0329] In some implementations of example method 2200, example method 2300, or example method 2400, each interaction object includes an action-level reward associated with the action.

[0330] In some implementations of example method 2200, example method 2300, or example method 2400, the trace includes a trace-level reward associated with completion of a task.

[0331] In some implementations, example method 2200, example method 2300, or example method 2400 includes processing, using the machine-learned trace generation system, the baseline trace to generate a task template. In some implementations, example method 2200, example method 2300, or example method 2400 includes generating, using the machine-learned trace generation system, values to populate the task template. In some implementations, example method 2200, example method 2300, or example method 2400 includes generating, using the machine-learned trace generation system, the variant trace based on the populated task template.

[0332] In some implementations of example method 2200, example method 2300, or example method 2400, generating the variant trace includes processing one or more images describing one or more environment states of the baseline trace. In some implementations of example method 2200, example method 2300, or example method 2400, generating the variant trace includes generating, using the machine-learned trace generation system, one or more new images describing the different environment state. In some implementations of example method 2200, example method 2300, or example method 2400, processing, using the machine-learned agent system, at least the portion of the variant trace to generate the interaction output includes processing the one or more new images using the machine-learned trace generation system to predict one or more actions to execute on the different environment state.

[0333] In some implementations of example method 2200, example method 2300, or example method 2400, generating the variant trace includes processing one or more images describing one or more environment states of the baseline trace. In some implementations ofexample method 2200, example method 2300, or example method 2400, generating the variant trace includes generating, using the machine-learned trace generation system, one or more new images describing a plurality of different environment states, the plurality of different environment states comprising the different environment state and an additional different environment state in sequence with the different environment state, wherein the additional different environment state corresponds to an outcome of an action executed on the different environment state. In some implementations of example method 2200, example method 2300, or example method 2400, processing, using the machine-learned agent system, at least the portion of the variant trace to generate the interaction output includes processing the one or more new images using the machine-learned trace generation system to predict one or more actions to execute on the different environment state.

[0334] In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned trace generation system includes a multi-modal machine-learned sequence processing model configured to process textual and image inputs.

[0335] In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned trace generation system is configured to generate new images using regression of new image tokens conditioned on textual inputs. In some implementations of example method 2200, example method 2300, or example method 2400, the machine-learned trace generation system is configured to generate new images using iterative de-noising of new image data conditioned on textual inputs.

[0336] In some implementations of example method 2200, example method 2300, or example method 2400, the seed representation includes a textual description of the baseline trace. In some implementations of example method 2200, example method 2300, or example method 2400, the seed representation includes an image of an environment state of the baseline trace.

[0337] It is to be appreciated that the skilled person would understand that one or more of the above implementations, features and / or aspects of example method 2200, example method 2300, and / or example method 2400 can be, where applicable, combined together and / or modified to form a machine-learned agent system and / or machine-learned trace generation system for use in technical systems and / or computing systems and / or applications thereof as described with reference to any of Figures 1 to 24, and / or as described herein and / or as the application demands.

[0338] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken andinformation sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel. In further examples, there is provided a computer program product including computer-executable instructions which, when executed by at least one computing apparatus (or one or more processors), cause the at least one computing apparatus (or one or more processors) to perform one or more of the method(s) or process(es) described with reference to Figures 1 to 24 and / or as described herein.

[0339] While the present subject matter has been described in detail with respect to various specific example embodiments or implementations thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

[0340] Aspects of the disclosure have been described in terms of illustrative embodiments thereof. Any and all features in the following claims can be combined or rearranged in any way possible, including combinations of claims not explicitly enumerated in combination together, as the example claim dependencies listed herein should not be read as limiting the scope of possible combinations of features disclosed herein. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. Moreover, terms are described herein using lists of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for explanatory purposes only. Clauses and other sequences of items joined by a particular conjunction such as “or,” for example, can refer to “and / or,” “at least one of’, “anycombination of’ example elements listed therein, etc. Terms such as “based on” should be understood as “based at least in part on.”

[0341] The term “can” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X can perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

[0342] The term “may” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X may perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for generating demonstrations for training a machine-learned agent system to interact with environments to perform tasks, the computer- implemented method comprising: obtaining a reference trace describing interactions between a machine-learned agent system and an environment, wherein the reference trace comprises a reference sequence of interaction objects associated with performance of a task, each respective reference interaction object of the reference sequence of interaction objects corresponding to a respective reference environment state and a respective reference action executed on the respective reference environment state; generating, using a machine-learned trace generation system, a forecasted trace based on the reference trace, the forecasted trace comprising a forecasted sequence of interaction objects that is predicted to continue the reference sequence of interaction objects from a branching position toward performance of the task, each respective forecasted interaction object of the forecasted sequence of interaction objects corresponding to a respective forecasted environment state and a respective forecasted action executed on the respective forecasted environment state; and training the machine-learned agent system using the forecasted trace sequence.

2. A computer-implemented method for generating demonstrations for using a machine- learned agent system to interact with environments to perform tasks, the computer- implemented method comprising: obtaining a reference trace describing interactions between a machine-learned agent system and an environment, wherein the reference trace comprises a reference sequence of interaction objects associated with performance of a task, each respective reference interaction object of the reference sequence of interaction objects corresponding to a respective reference environment state and a respective reference action executed on the respective reference environment state; generating, using a machine-learned trace generation system, a forecasted trace based on the reference trace, the forecasted trace comprising a forecasted sequence of interaction objects that is predicted to continue the reference sequence of interaction objects from a branching position toward performance of the task, each respective forecasted interaction object of the forecasted sequence of interaction objects corresponding to a respectiveforecasted environment state and a respective forecasted action executed on the respective forecasted environment state; and executing, by the machine-learned agent system, a predicted action on a current environment state, wherein the predicted action is determined based on the forecasted trace.

3. The computer-implemented method of any of the preceding claims, wherein the machine-learned agent system was trained based on a plurality of exemplar traces using reinforcement learning, wherein a training reward for a respective exemplar trace was based on: an action-level reward associated with each action in the respective exemplar trace; or a trace-level reward associated with completion of a task associated with the respective exemplar trace.

4. The computer-implemented method of any of the preceding claims, comprising: determining a resulting environment state that results from execution of a predicted action on a reference environment state associated with the branching position, the resulting environment state associated with a subsequent position in the reference trace; determining a feedback signal based on a comparison of the resulting environment state against a corresponding state in the forecasted trace that is associated with the subsequent position of the forecasted trace; and training the machine-learned trace generation system based on the feedback signal.

5. The computer-implemented method of any of the preceding claims, wherein the predicted action is based on the forecasted trace.

6. The computer-implemented method of any of the preceding claims, comprising: processing, using the machine-learned trace generation system, at least a portion of the reference trace to generate the forecasted trace; and processing, using the machine-learned agent system, at least a portion of the forecasted trace to generate the predicted action.

7. The computer-implemented method of any of the preceding claims, wherein determining the resulting environment state comprises: executing the predicted action on the environment; andreceiving, from the environment, a state update.

8. The computer-implemented method of any of the preceding claims, comprising: constructing the reference trace by recording actions executed and state updates received by the machine-learned agent system in a deployment environment.

9. The computer-implemented method of any of the preceding claims, comprising: maintaining the reference trace in a buffer that retains a history of recent interaction objects, wherein the forecasted trace is generated based on the contents of the buffer.

10. The computer-implemented method of any of the preceding claims, wherein a respective forecasted trace is generated for each respective turn of a multi-turn interaction, wherein each respective turn provides a respective branching position.

11. The computer-implemented method of any of the preceding claims, wherein each respective forecasted environment state is generated using a machine-learned sequence processing model, wherein: the respective forecasted environment state comprises a textual description of the environment generated by the machine-learned sequence processing model; or the respective forecasted environment state comprises image data descriptive of the environment, the image data generated using: regression of new image tokens conditioned on textual inputs; or iterative de-noising of new image data conditioned on textual inputs.

12. The computer-implemented method of any of the preceding claims, wherein the reference trace is processed by the machine-learned trace generation system as an input sequence of textual content interleaved with image content, wherein each interaction object of the reference trace is represented in the input sequence by a first portion representing an embedded textual description of the corresponding environment state and a second portion representing an embedded image description of the corresponding environment state, the second portion being contiguous with the first portion.

13. The computer-implemented method of any of the preceding claims, wherein:the machine-learned agent system controls a robotic platform, and the task is a navigation task or an object manipulation task; or the machine-learned agent system controls a computing platform, and the task is a data processing task or a user interface interaction task.

14. The computer-implemented method of any of the preceding claims, comprising: determining an uncertainty measure associated with an action in the reference trace; and determining, based on the uncertainty measure, a number of forecasted traces to generate.

15. The computer-implemented method of any of the preceding claims, wherein the uncertainty measure is obtained using the shape of an output distribution of the machine- learned agent system.

16. The computer-implemented method of any of the preceding claims, wherein: the machine-learned agent system and the machine-learned trace generation system use the same machine-learned model to process inputs and generate outputs; or the machine-learned agent system and the machine-learned trace generation system use different machine-learned models to process inputs and generate outputs.

17. The computer-implemented method of any of the preceding claims, comprising: obtaining a seed representation of the reference trace; generating a variant trace based on the baseline trace, wherein generating the variant trace comprises: providing, to the machine-learned trace generation system, the seed representation to generate the variant trace that comprises a different environment state from the reference trace, wherein the different environmental state is generated using one or more inferences obtained from the machine-learned trace generation system; and determining, based on a goal state of the reference trace associated with the task, a goal state of the variant trace; processing, using the machine-learned agent system, at least a portion of the variant trace to generate an interaction output; evaluating the interaction output using the goal state of the variant trace; andtraining the machine-learned agent system based on the evaluation of the interaction output.

18. A computer-implemented method for augmenting baseline training trajectories for training a machine-learned agent system to interact with environments, the computer- implemented method comprising: obtaining a seed representation of a reference trace; generating a variant trace based on the reference trace, wherein generating the variant trace comprises: providing, to a machine-learned trace generation system, the seed representation to generate the variant trace that comprises a different environment state from the reference trace, wherein the different environmental state is generated using one or more inferences obtained from the machine-learned trace generation system; and determining, based on a goal state of the reference trace, a goal state of the variant trace; processing, using the machine-learned agent system, at least a portion of the variant trace to generate an interaction output; evaluating the interaction output using the goal state of the variant trace; and training the machine-learned agent system based on the evaluation of the interaction output.

19. The computer-implemented method of claim 17 or 18, wherein determining the goal state comprises: inheriting the goal state from the reference trace based on an alignment of the variant trace with the reference trace; or generating the goal state of the variant trace using the machine-learned trace generation system.

20. The computer-implemented method of any of claims 17 to 19, wherein obtaining the seed representation comprises: generating a condensed representation of the reference trace, wherein the reference trace comprises a sequence of interaction objects; and wherein generating the variant trace comprises:processing, using the machine-learned trace generation system, the condensed representation and an initial portion of the reference trace to generate one or more interaction objects that are predicted to continue the sequence of interaction objects;21. The computer-implemented method of any of claims 17 to 20, comprising: wherein generating a condensed representation of the reference trace comprises: generating a plurality of candidate condensed representations over one or more iterations, wherein a respective iteration comprises: processing, using the machine-learned trace generation system, a respective candidate condensed representation to generate a respective output; evaluating an alignment of the respective output to the reference trace; and generating a new candidate condensed representation based on the alignment; and selecting the condensed representation from the plurality of candidate condensed representations.

22. The computer-implemented method of any of claims 17 to 21, comprising: determining an alignment between the baseline trace and the variant trace, wherein the alignment is determined based on: a similarity comparison of an environment state of the baseline trace and an environment state of the variant trace; or a similarity comparison of an action of the baseline trace and an action of the variant trace.

23. The computer-implemented method of any of claims 17 to 22, wherein the machine- learned trace generation system is configured to limit a divergence of the variant trace from the baseline trace.

24. The computer-implemented method of any of claims 17 to 23, comprising: instructing the machine-learned trace generation system to generate the variant trace, wherein the instructions indicate that the variant trace is to share at least one feature of the goal state of the baseline trace.

25. The computer-implemented method of any of claims 17 to 24, comprising: instructing the machine-learned trace generation system to generate the variant trace, wherein the instructions indicate that the variant trace is to differ in at least one feature of the goal state of the baseline trace.

26. The computer-implemented method of any of claims 17 to 25, wherein the baseline trace describes performance of a task in a baseline environment, and wherein the variant trace describes performance of the same task in a new environment.

27. The computer-implemented method of any of claims 17 to 26: the baseline trace comprises a sequence of interaction objects, each object comprising a state and an action acting upon that state; or the variant trace comprises a sequence of interaction objects, each object comprising a state and an action acting upon that state.

28. The computer-implemented method of claim 26, wherein each interaction object comprises an action-level reward associated with the action.

29. The computer-implemented method of claim 26 or 27, wherein the trace comprises a trace-level reward associated with completion of a task.

30. The computer-implemented method of any of claims 17 to 29, comprising: processing, using the machine-learned trace generation system, the baseline trace to generate a task template; generating, using the machine-learned trace generation system, values to populate the task template; and generating, using the machine-learned trace generation system, the variant trace based on the populated task template.

31. The computer-implemented method of any of claims 17 to 30, wherein generating the variant trace comprises: processing one or more images describing one or more environment states of the baseline trace; andgenerating, using the machine-learned trace generation system, one or more new images describing the different environment state; and wherein processing, using the machine-learned agent system, at least the portion of the variant trace to generate the interaction output comprises: processing the one or more new images using the machine-learned trace generation system to predict one or more actions to execute on the different environment state.

32. The computer-implemented method of any of claims 17 to 31, wherein generating the variant trace comprises: processing one or more images describing one or more environment states of the baseline trace; and generating, using the machine-learned trace generation system, one or more new images describing a plurality of different environment states, the plurality of different environment states comprising the different environment state and an additional different environment state in sequence with the different environment state, wherein the additional different environment state corresponds to an outcome of an action executed on the different environment state; and wherein processing, using the machine-learned agent system, at least the portion of the variant trace to generate the interaction output comprises: processing the one or more new images using the machine-learned trace generation system to predict one or more actions to execute on the different environment state.

33. The computer-implemented method of any of claims 17 to 32, wherein the seed representation comprises: a textual description of the baseline trace; or an image of an environment state of the baseline trace.

34. The computer-implemented method of any of the preceding claims, wherein the machine-learned trace generation system comprises a multi-modal machine-learned sequence processing model configured to process textual and image inputs.

35. The computer-implemented method of any of the preceding claims, wherein the machine-learned trace generation system is configured to generate new images using: regression of new image tokens conditioned on textual inputs; or iterative de-noising of new image data conditioned on textual inputs.

36. The computer-implemented method of any of the preceding claims, wherein training the machine-learned agent system comprises training a machine-learned model used by the machine-learned agent system.

37. The computer-implemented method of any of the preceding claims, wherein training the machine-learned trace generation system comprises training a machine-learned model used by the machine-learned trace generation system.

38. One or more non-transitory, computer readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations, wherein the operations comprise: the computer-implemented method of any of the preceding claims.

39. A computing system, comprising: one or more processors; and one or more non-transitory, computer readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations, wherein the operations comprise: the computer-implemented method of any of claims 1 to 37.

40. A computer program product comprising computer-executable instructions which, when executed by at least one computing apparatus, cause the at least one computing apparatus to perform the computer-implemented method of any of claims 1 to 37.

Images