Training generative model to generate predicted rewards and / or use thereof in reinforcement learning

Abstract

Implementations relate to training a generative model. Some of those implementations include performing offline supervised fine-tuning (SFT) of a base generative model using an imitation learning dataset of successful task episodes. The SFT training utilizes both a behavioral cloning loss function, which compares offline predictions to ground truth data, and a reinforcement reward loss function, comparing predicted rewards to actual episode rewards. Subsequently, online reinforcement learning (RL) is performed on an instance of the base generative model. This online RL processes online data using the model instance to generate predictions and uses the SFT-trained model to generate online predicted rewards. These online predicted rewards are then used to train the instance of the base generative model, improving its performance for various robot and application control tasks.

Description

Attorney Reference No. DEEP-0023-WO-01 TRAINING GENERATIVE MODEL TO GENERATE PREDICTED REWARDSAND / OR USE THEREOF IN REINFORCEMENT LEARNINGBackground

[0001] Recent advances in generative models such as visual language models (VLMs), large language models (LLMs), etc. have shown impressive capabilities across various reasoning and generation tasks. Further, reinforcement learning (RL) has been utilized in training of such models.

[0002] However, rewards are needed in RL and in various situations generating meaningful rewards can be difficult. For example, some rewards are generated based on an engineered reward function. However, the engineered reward function may be task-specific (e.g., a particular robot task) and / or device-specific (e.g., only for a single robot type), resulting in a need for multiple reward functions for RL training for multiple tasks and / or for multiple device types. Additionally or alternatively, for some tasks (e.g., some robotic tasks) obtaining meaningful signal(s) for a reward function can be difficult or not possible. Summary

[0003] Implementations disclosed herein relate to training a generative model. Some of those implementations include performing offline supervised fine-tuning (SFT) of a base generative model. The offline SFT of the base generative model can be based on an imitation learning dataset that includes successful episodes of one or more tasks. The offline SFT results in an SFT trained generative model that has updated weights relative to the base generative model. The offline SFT can include training the base generative model utilizing both a behavioral cloning loss function and a reinforcement reward loss function. The behavioral cloning loss function facilitates a comparison between one or more corresponding offline predictions, which are reflected in corresponding output generated during the offline supervised fine-tuning, and ground truth data of a corresponding one of the successful episodes. The reinforcement reward loss function facilitates a comparison between a corresponding offline predicted reward, reflected in the corresponding output, and a reward reflected by the corresponding one of the successful episodes.Attorney Reference No. DEEP-0023-WO-01

[0004] Implementations disclosed herein additionally or alternatively include performing online reinforcement learning (RL) of an instance of the base generative model. The online RL of the instance of the base generative model can include processing an instance of online data, using the instance of the base generative model, to generate one or more online predictions to implement during the online RL. The online RL can also include processing the instance of online data, using the SFT trained generative model, to generate an online predicted reward. The online predicted reward, generated using the SFT trained generative model, can then be utilized in training the instance of the base generative model. These and other implementations offer the advantage of efficiently and robustly improving policy performance and enables the generative model to acquire new skills that generalize beyond the distribution of tasks initially seen during offline SFT.

[0005] In some examples, the instance of the base generative model for online RL can be the SFT trained generative model itself, or it can be a version of the base generative model that differs from the SFT trained generative model but has undergone at least some of the offline SFT. The online predicted reward can be a predicted quantity reflecting a predicted duration of time until a corresponding goal is achieved, such as a predicted quantity of remaining iterations until the goal is achieved. This quantity can be generated based on comparing a total quantity of observations of a successful episode to a current quantity for observation data processed and / or by comparing a final timestep of a successful episode to a current timestep of observation data processed.

[0006] The tasks of the successful episodes can include robot tasks or application control tasks. For robot tasks, the successful episodes can each include a corresponding sequence of observations and actions, and a corresponding episode goal. Generating the corresponding output during offline SFT can be based on processing, using the generative model, the corresponding episode goal and an observation from the sequence of observations and actions. Observations can include sequences of vision data instances, and episode goals can be natural language text describing a corresponding task. For application control tasks, observations can include sequences of screenshots.

[0007] Subsequent to performing the online RL, the instance of the base generative model, with updated weights from the online RL, can be provided for use by one or more devices. ForAttorney Reference No. DEEP-0023-WO-01 example, the devices can include a robot or a client / server device for controlling one or more applications. Providing the generative model for user by one or more devices can include transmitting it to the device(s) via one or more networks and / or enabling access to the generative model via an application programming interface. In some implementations, the entirety of the generative model can be updated during offline SFT training, while only a subset of the generative model is updated during online RL training. The generative model can be a pre-trained vision language model. The successful episodes in the imitation learning dataset can be performed by a first set of robots, and the online predictions implemented during online RL can be on a distinct robot of a distinct type from the first set. Additionally, the online predictions may be for a distinct task that differs from the tasks of the successful episodes.

[0008] The preceding is presented as a non-limiting overview of only some implementations disclosed herein. The appended paper and the claims provide additional details on those and other implementations. Brief Description of the Drawings

[0009] FIG.1 is a diagram illustrating a two-stage fine-tuning approach for a multimodal foundation agent.

[0010] FIG.2 is a flowchart depicting a method for training a generative model through offline supervised fine-tuning and online reinforcement learning.

[0011] FIG.3 illustrates an example robotic system according to some implementations.

[0012] FIG.4 illustrates an example computing system according to some implementations. Detailed Description

[0013] Prior to turning to the Figures, a non-limiting description of some example aspects of the disclosure is provided.

[0014] Foundation models trained on web-scale data have transformed the field of robotics. However, their application to low-level control has largely been confined to behavioral cloning. Drawing inspiration from the sample efficiency and demonstrated success of reinforcement learning (RL) fine-tuning in large language models (LLMs), a two-stage approach suitable for robotics is disclosed herein. The first stage, Supervised Fine-Tuning (SFT), involves fine-tuning a pre-trained foundation model using goal-conditioned behavioral cloning and steps-to-go prediction objectives. In the second stage, this foundation enables the derivation of a well-Attorney Reference No. DEEP-0023-WO-01 shaped reward function and a success detector. This capability eliminates the need for manual reward engineering and real-world instrumentation, thereby allowing robots to autonomously practice with minimal human supervision. Experiments conducted on both real-world and simulated robots illustrate that the combination of SFT and online self-improvement demonstrates significantly greater sample-efficiency compared to supervised learning alone. Furthermore, the combination of this approach with web-scale pre-trained foundation models facilitates the rapid acquisition of new skills, enabling robots to generalize well beyond the behaviors observed in the imitation learning datasets utilized during training. These findings underscore the transformative potential of integrating pre-trained foundation models with online fine-tuning to achieve enhanced levels of autonomy and skill acquisition in robotics and / or other domains such as control of software applications.

[0015] Recent studies have shown that foundation models can be effectively fine-tuned to function directly as low-level robot policies. These models exhibit substantial generalization and robustness capabilities, inherited from the web-scale pre-training of the foundation models from which they are derived. Such foundation agents present an opportunity for the future of robotics, where a monolithic agent can plan, reason, and subsequently execute actions within an environment. Additionally, these agents enable a closer transfer of methodologies between related fields of artificial intelligence that leverage foundation models, such as computer vision and natural language processing. Throughout this description, the term “Multimodal Foundation Agent” (MFA) can be used to refer to foundation models that operate directly within an environment.

[0016] Heretofore, the training approach for MFAs has largely been limited to behavioral cloning, which is a form of supervised learning. In contrast, observations from the literature on LLMs indicate that following initial pre-training, post-training for downstream tasks is typically segmented into two stages. The first stage is SFT, followed by a second stage of RL, where models enhance their performance on downstream tasks such as mathematics and coding, as well as aligning with human preferences (RLHF). RL-tuning of LLMs has been shown to improve downstream task performance markedly and rapidly beyond the SFT stage, establishing itself as an important stage in the training regimen of foundation models.Attorney Reference No. DEEP-0023-WO-01

[0017] There are algorithmic and engineering challenges associated with investigating RL- tuning for MFAs within the context of robotics. Implementations disclosed herein address these challenges via a two-stage framework. In a first stage, often designated herein as offline SFT, given a goal-conditioned imitation learning dataset, an MFA is fine-tuned using two objectives: behavioral cloning and predicting the number of steps-to-go to accomplish desired goals. The behavioral cloning objective can include maximizing the likelihood of a dataset action conditioned on an observation and a text sequence that represents the desired goal. The steps-to-go objective trains the MFA to predict the number of environment timesteps remaining until the robot accomplishes an intended goal. In a second stage, often designated herein designated as online RL, the model’s own steps-to-go predictions are leveraged to derive an effective reward function and a success detector. This capability allows a single human operator to monitor multiple robots as they practice downstream tasks. The reward function for online RL training can be expressed as the difference between the expected value of steps-to-go from the current observation and the expected value of steps-to-go from the next observation after taking an action, plus a terminal reward component. Such a data-driven reward design can circumvent the need for ground-truth rewards and leverages the robustness and generalization properties of the underlying foundation models.

[0018] Through extensive experiments conducted on two robot embodiments, LanguageTable and Aloha, in both real-world and simulated environments, the efficacy of the fine-tuning framework disclosed herein is demonstrated. The results indicate that online RL improves policy performance with considerable sample-efficiency and robustness. Furthermore, it is shown that distributing robot time budget between imitation data collection for offline SFT and online RL self-improvement is more efficient than allocating the entire robot time for offline SFT data collection alone. The value of web-scale pre-training of foundation models is also demonstrated. Pre-training not only results in substantial sample-efficiency but also enables robots to autonomously practice and acquire new skills, thereby generalizing far beyond the distribution of tasks observed during offline SFT.

[0019] Implementations disclosed herein highlight the transformative potential of combining pre-trained foundation models with online fine-tuning to unlock new levels of autonomy and skill acquisition in robotics and other domains.Attorney Reference No. DEEP-0023-WO-01

[0020] Implementations disclosed herein can be utilized with any one of various underlying multimodal foundation models. A multimodal foundation model can be a sequence-to- sequence based machine learning model capable of processing multiple modalities of input data, such as vision data and textual data, and generating corresponding generative data in one or more modalities such as generative vision data, generative audio data, generative textual data, and / or other forms of generative data. Some non-limiting examples of these sequence- to-sequence based machine learning models include transformer-based machine learning models (e.g., encoder-decoder transformer models, encoder-only transformer models, decoder-only transformer models, etc. that optionally employ an attention mechanism or some other form of memory), diffusion-based machine learning models, recurrent neural network-based machine learning models, generative adversarial network-based machine learning models, etc. Some particular non-limiting examples of these sequence-to-sequence based machine learning models include the Gemini family of models (e.g., Gemini 1.5 Pro, Gemini 2.5 Pro, Gemini 2.5 Flash). A multimodal foundation model can be a vision language model (VLM) and can include billion(s) of parameters, such as more than two billion parameters, more than three billion parameters, or more than five billion parameters.

[0021] One non-limiting example of a multimodal foundation model that can be used is a 3 billion parameter PaLI-3B vision-language model. A PaLI model accepts one or more images alongside text as input and produces text as output. At a high level, the architecture of a PaLI model includes two components: 1) a Vision Transformer (ViT), and 2) an encoder-decoder Transformer. Input images are processed by the ViT into a sequence of “visual tokens.” The sequence of visual tokens is concatenated with the tokenized text input and fed into the Transformer encoder, and the Transformer decoder outputs text tokens. The PaLI architecture is initialized from a Transformer encoder-decoder model (language) and a ViT (vision) that are pretrained separately in a unimodal fashion. The model is subsequently trained jointly with a variety of vision-language training objectives to obtain a multimodal foundation model.

[0022] RT-2 introduces a model family, designated RT-2, that enables VLMs to directly perform closed-loop robot control. The two VLMs considered in that work are PaLI and PaLM-E, both of which accept images alongside text as input and provide output in the form of text tokens. To enable these VLMs to operate as robot policies, continuous robot actions are discretized andAttorney Reference No. DEEP-0023-WO-01 mapped onto a linguistic token space. Given image and text inputs, the VLMs are fine-tuned via behavioral cloning (BC), which is a form of supervised learning, to predict tokenized robot actions. Implementations disclosed herein are independent of the choice of underlying model and architecture. Put another way, implementations disclosed herein work with any of a variety of underlying models and architectures. However, various examples described herein describe robot policy architectures that are equivalent to RT-2 utilizing the PaLI VLM.

[0023] A focus in this work is to design an effective and sample-efficient procedure for fine- tuning pretrained multimodal foundation models in order to obtain performant robotic MFAs. A proposed fine-tuning framework is composed of two stages: 1) offline SFT where MFAs are trained using goal-conditioned behavioral cloning as well as steps-to-go prediction objectives, and 2) online RL where MFA policies autonomously practice downstream tasks and rapidly improve themselves via self-predicted rewards.

[0024] A challenge of RL for robotics, and in particular for manipulation tasks, is the problem of reward engineering. Designing effective reward functions requires repeated trial-and-error iterations of training RL policies and patching reward definitions to arrive at intended outcomes. Furthermore, even with a perfect reward function, significant research and engineering effort must be dedicated to measuring rewards in the real-world. Thus, manual reward design is untenable as progress is made towards a future where robots are trained to accomplish increasingly broad sets of tasks. Implementations disclosed herein overcome this obstacle via learning data-driven reward functions that also inherit robustness and generalization properties from the web-scale pre-training of the foundation models used to build the MFAs.

[0025] The initial stage of the framework includes an offline SFT stage. Access is provided to a goal-conditioned imitation learning dataset that includes a collection of episodes, where an observation and an action are denoted at a timestep, and a goal is denoted for a given episode. All trajectories in the dataset are assumed to terminate in a state where the episode goal is accomplished. In the case of single-task datasets, these are treated as a goal-conditioned dataset in which all episodes share the same goal. Given a dataset D and a pre-trained multimodal foundation model, a MFA is instantiated, and the model is fine-tuned utilizing supervised learning objectives.Attorney Reference No. DEEP-0023-WO-01

[0026] A goal-conditioned behavioral cloning loss is determined by maximizing the likelihood of a dataset action conditioned on an observation and a text sequence representing the desired goal. An objective associated with a steps-to-go prediction teaches the MFA to predict a quantity of environment timesteps remaining from accomplishing an intended goal. The steps-to-go prediction can be based on a query such as “How many steps to go?”.

[0027] An algorithm for a Stage 2 Self-Improvement Loop can be implemented, having a policy model and a frozen reward computation model obtained from Stage 1 checkpoints. While a given condition is true, the current policy can be utilized to collect sufficient robot rollouts for a number of update steps with a given batch size. For each rollout, Monte Carlo returns can be computed. The Monte Carlo returns can be expressed as a function of the total reward from time t to the end of the episode under a given policy and discount factor, representing the expected cumulative reward from a given state and action. Subsequently, including observations, actions, and discounted cumulative rewards are placed into a replay buffer. The replay buffer is then shuffled, and updates are performed utilizing a reinforcement learning objective, such as REINFORCE. The replay buffer is then cleared of any remaining elements before continuing the loop.

[0028] Depending on the domain, this first stage can include additional auxiliary supervised objectives. As an example, in experiments with a LanguageTable domain, conditioned on the first and last image of an episode, the model can be configured to predict an instruction performed in that episode.

[0029] Put more formally, assume a goal-conditioned imitation learning dataset ^^ consisting ofa collection of episodes ^^ =^^= 0, where ^^^^and ^^^^denote observation and action at timestep t respectively, and ^^^^denotes the goal for episode ^^. Further assume that all trajectories in the dataset end in a state where the episode goal is accomplished. In the case of single-task datasets, they can be treated as a goal-conditioned dataset where all episodes share the same goal. Given a dataset D and pretrained multimodal foundation model, the MFA can be instantiate and fine-tune using the following supervised learning objectives, where ℒBCdenotes a goal conditioned behavioral cloning loss, and where the likelihood of a dataset action, conditioned on the observation and a text sequence Questionaction(^^^^), representingAttorney Reference No. DEEP-0023-WO-01 the desired goal, is maximized. For example, Questionaction(^^^^) = "What robot action to ^^^^?" can be utilized. The objective ℒsteps_to_goteaches the MFA to predict how many environment timesteps away the robot is from accomplishing an intended goal. For example,Questionsteps_to_go(^^^^) = "How many steps to ^^^^?" can be utilized. Supervised learningobjectives:Questionaction(^^^^))]ℒsteps_to_go(MFA) = −^^(^^^^,^^^^,^^^^)~^^[log^^MFA (length(^^) − t⃒^^^^, Questionsteps_to_go(^^^^))]

[0030] Algorithm 1 is presented below and presents the self-improvement loop according to various implementations disclosed herein. __________________________________________________________________________ Algorithm 1: Stage 2 Self-Improvement Loop __________________________________________________________________________ Input: Policy model and frozen reward computation model taken from Stage 1 checkpoints while true do Using the current policy collect enough robot rollouts for N update steps with batch size B; for each rollout do Compute Monte Carlo returns using Equation 2: ^^^^ ^∙ ^^(^^^^,^^^^, ^^^^+1,^^),Place (^^^^,^^^^,^^,^^^^) tuples in the replay buffer; endEmpty the replay buffer if there are any remaining elements; end __________________________________________________________________________

[0031] In the second stage a goal is to fine-tune the MFA with online RL, with the expectation that this process will lead to rapid and significant performance improvements on desired downstream tasks. Downstream tasks can even be significantly different from those that appeared in the dataset used for offline SFT in the first stage.

[0032] A predicted value, denoted by a quantity V(o, g), represents the expected value of “steps to go” in order to accomplish a goal “g” given an observation “o”, as predicted by theAttorney Reference No. DEEP-0023-WO-01 MFA model obtained after Stage 1. The reward function utilized for online RL training is defined as a difference between the predicted value of steps-to-go from a current observation, after an action has been taken, and the predicted value of steps-to-go from the observation prior to the action, plus a terminal reward component.

[0033] This reward function intuitively predicts how much closer a robot has moved towards accomplishing a goal after taking an action. As the reward function is derived from the predicted value, which is a function of the MFA itself, this online RL fine-tuning process is referred to as self-improvement. The choice of using the expected value in the predicted value V(o, g) is for simplicity and alignment with the notion of a value function in RL.

[0034] Put more formally let ^^(^^,^^) ≔ ^^^^MFA(steps_to_go[steps_to_go]denote the expected value of “steps to go” in order to accomplish goal g given observation o, as predicted by the MFA model obtained after Stage 1. The reward function for online RLtraining can be defined as ^^(^^^^, ^^^^, ^^^^+1, ^^) ≔ ^^(^^^^, ^^) − ^^(^^^^+1,^^)

[0035] This reward function predicts how much closer the robot got towards accomplishing goal ^^ after taking action ^^^^. As the reward function is derived from ^^(^^,^^), which is a function of the MFA itself, the online RL fine-tuning process can be referred to as “self- improvement”. The choice of using the expected value is for simplicity and alignment with the notion of a value function in RL.

[0036] For second stage fine-tuning, a frozen first stage checkpoint is employed for reward function calculations, and the second stage policy is initialized from a first stage checkpoint. The checkpoints for the reward and policy models are not necessarily identical, as the optimal validation losses for these models can occur at different points during first stage training. Within one iteration of the second stage self-improvement loop, the current policy is utilized to collect a sufficient number of robot trajectories to perform N model update steps. Subsequently, for each trajectory and per timestep, Monte Carlo returns are computed. The Monte Carlo returns represent the expected cumulative reward from a given state and action. Elements including observations, actions, and discounted cumulative rewards are then placed into a shuffled replay buffer. Subsequently, N policy updates are performed utilizing a REINFORCE loss. The replay buffer is then cleared, and the next iteration commences.Attorney Reference No. DEEP-0023-WO-01 Algorithm 1, as presented previously, outlines this second stage self-improvement procedure. In simulation experiments, it has been observed that employing a small positive multiplicative factor ‘c’ in the REINFORCE loss contributes to stable model training. For example, a value of c = 5e-2 can be utilized. Despite an objective of sample-efficient RL, on-policy RL without data reuse can be utilized due to the inherent stability of on-policy RL methods.

[0037] It can be beneficial for robot episodes to terminate upon successfully reaching an intended goal state. Otherwise, a significant portion of the collected data would include the robot residing in a successful terminal state. In settings where a ground-truth success detector is unavailable, as in real-world experiments, a success indicator derived from the frozen reward model checkpoint can be utilized. This success indicator evaluates the predicted value V(o, g) to determine if a goal has been reached within a small number of timesteps, denoted by ‘s’. Moreformally, this can be defined as success (^^,^^) ≔ ^^[^^(^^, ^^) ≤ ^^]. This formulation of successdetection has proven to be robust even in low data regimes, and significantly more reliable than explicitly including a success detection binary classification objective in Stage 1. Some implementations utilize a value of s = 3.

[0038] The second stage RL facilitates policies that more efficiently achieve intended goals while being implicitly regularized to maintain proximity to the dataset policy.

[0039] Experiments validate the self-improvement framework and address the following five questions: Q1: Does a self-improvement procedure enhance performance on downstream tasks beyond a supervised learning stage? Q2: Is a self-improvement procedure, which relies on reinforcement learning (RL), sufficiently reliable and reproducible for real-world robotics applications? Q3: Is the combination of supervised learning and self-improvement a more efficient procedure for obtaining performant policies compared to supervised learning alone?Q4: What is the contribution of web-scale pretraining of a multimodal foundation model Q5:Can pretraining knowledge embedded into the MFA be leveraged to enhance generalization abilities and facilitate second stage self-improvement on tasks that generalize beyond what was observed in an imitation dataset?

[0040] These questions are investigated utilizing the LanguageTable and Aloha robot embodiments, with experiments conducted in both simulation and real-world environments. In the experiments, the PaLI vision-language model is utilized as the base pretrained foundationAttorney Reference No. DEEP-0023-WO-01 model. In the experiments, the inputs to the PaLI MFA include two images and a text sequence, with outputs including a sequence of tokens. In the experiments, to deploy PaLI models as policies, the RT-2 policy parameterization is adopted, and tokenized actions are predicted. In the experiments, for predicting steps-to-go, the range of integers from 0 to a final timestep T is mapped onto the PaLI model’s output token space. In the first stage in the experiments, no parameters within the model are frozen, and both the Transformer and the ViT backbone are fine-tuned. In the second stage in the experiments, the ViT portion of the model is not further fine-tuned.

[0041] In the experiments, the dataset utilized for first stage offline SFT training of policies within the simulated LanguageTable domain includes 181,020 human-generated trajectories, with 78,623 distinct instructions describing the objectives of the trajectories. This dataset is subsampled to generate three new datasets, each representing 10%, 20%, and 80% of the original size. For each dataset size, the following procedure is implemented: First, first stage offline SFT of the MFA is performed. A checkpoint exhibiting the best imitation validation loss is selected as the supervised policy checkpoint, and a checkpoint showing the best steps-to-go prediction validation loss is selected for reward computation. Second stage self-improvement training is conducted with three seeds to validate the reliability of the self-improvement procedure. Although the LanguageTable dataset includes a variety of tasks, second stage fine- tuning is performed on distinct Block2Block tasks, such as "move the blue moon to the red pentagon. Second stage self-improvement training is discontinued when policy success rates appear to reach a plateau.

[0042] Across all dataset sizes (10%, 20%, 80%), the self-improvement procedure leads to significant enhancements in success rates, with a minimum 1.5x performance boost. This improvement is achieved with notable sample-efficiency in terms of the number of episodes, requiring less than 2% additional episodes collected in the second stage self-improvement. For example, by training a 10% data first stage policy with 1% additional episodes in the second stage, policies are obtained that outperform both the 20% and 80% data first stage only policy. Furthermore, the second stage process exhibits stability and reproducibility across random seeds.Attorney Reference No. DEEP-0023-WO-01

[0043] In real-world experiments, for both the 20% and 80% data settings, the second stage self-improvement procedure enhances policy success rates from approximately 60% to approximately 80%-85%, all within approximately 3% additional Block2Block episodes. To provide context, this indicates that with a total amount of experience equivalent to approximately 23% (first stage + second stage), policies are obtained that substantially exceed first stage only behavioral cloning (BC) policies (e.g., RT-2) that utilized 80% of the real-world LanguageTable dataset. Furthermore, in contrast to the 1-to-1 human-to-robot ratio during imitation learning data collection for the first stage, the second stage process requires reduced human effort due to the 1-to-many human-to-robot ratio enabled by the disclosed approach.

[0044] The framework is also validated on a second robot embodiment, the bimanual Aloha manipulation platform. Data is designed and collected for a bimanual insertion task, where a left gripper must pick up a socket, and a right gripper must pick up a peg and insert that peg into the socket. Due to the single-task nature, significantly smaller imitation datasets, more complex observations, and a 70-dimensional action space, this presents a challenging setting for further validation of the proposed process. Three imitation dataset sizes of 5K, 10K, and 15K trajectories are generated. The two-stage process is applied to 5K and 10K dataset sizes, and results for supervised learning on the 15K dataset are also reported to better contextualize the numerical findings. The only methodological differences compared to the LanguageTable domain are as follows: 1) To initialize the second stage policy checkpoint, the best validation checkpoint is not selected, as further training of the supervised policy was observed to lead to improved performance.2) Since the exact success state is difficult to observe from the robot camera observations, a small positive constant is added to the reward function when the robot achieves a successful state.

[0045] Policies trained with 5K+2.5K episodes (first stage + second stage) outperform policies trained with 10K imitation episodes (first stage only), and demonstrate a success rate comparable to those trained with 15K supervised episodes (first stage only).

[0046] As described above, the PaLI model is initialized from a pretrained ViT model (trained unimodally using vision tasks) and a pretrained language Transformer model (trained unimodally using language tasks), which are interconnected to form the PaLI architecture and subsequently co-trained on multimodal vision-language tasks. To evaluate the effect of theAttorney Reference No. DEEP-0023-WO-01 multimodal knowledge embedded within PaLI, the two-stage fine-tuning process can be executed starting from alternative variations of the PaLI model. A first variation is “Scratch”, where the PaLI architecture is utilized with randomly initialized parameters. A second variation is “Frankenstein”, where a version of the PaLI model is utilized that connects the pretrained ViT model to the pretrained language Transformer, but without the PaLI vision-language co- training.

[0047] These variations are compared on the Simulated LanguageTable domain, utilizing the 10%, 20%, and 80% dataset sizes, and performing the second stage self-improvement on the Block2Block subset of tasks. Each experiment is conducted with three random seeds. Despite extensive efforts and prolonged training runs, first stage only supervised policies derived from Scratch or Frankenstein variations demonstrated significantly inferior performance compared to PaLI first stage only policies. Consequently, the ablation study focuses on the second stage self-improvement process, where the policy is initialized from the PaLI first stage checkpoints, and the reward model employs Scratch or Frankenstein checkpoints. A clear performance hierarchy is evident, with PaLI demonstrating the best performance, followed by Frankenstein, and then the Scratch reward model. The Scratch reward models yield high variance results across random seeds and struggle to provide meaningful improvements in low-data regimes, to the extent that in the 10% data regime, second stage self-improvement could not enhance the first stage policy. While superior to Scratch, second stage self-improvement with Frankenstein reward models is also significantly less effective than using PaLI reward models. These results clearly demonstrate the value that web-scale multimodal pre-training contributes to the self- improvement procedure.

[0048] A capability enabled by the combination of the self-improvement process and the utilization of pretrained multimodal foundation models is that, during second stage self- improvement, policies can practice novel tasks not present in the imitation learning dataset.

[0049] This section investigates domain transfer between simulation and real environments. Sim2Real represents an important class of approaches for robotics with numerous successes, and can significantly reduce the amount of real-world experience required to train performant robot policies. To simplify experimentation, this section investigates the inverse problem of Real2Sim transfer on the LanguageTable domain. First stage models are trained using 80% ofAttorney Reference No. DEEP-0023-WO-01 the real-world LanguageTable dataset, and second stage self-improvement is performed in the simulated LanguageTable environment. Second models are also trained using the “Frankenstein” reward model variant to emphasize the role of foundation model pretraining in facilitating domain transfer.

[0050] With a modest number of episodes equivalent to 3% of Block2Block episodes in the simulated LanguageTable dataset, the second stage self-improvement procedure enhances policy performance from approximately 22% to approximately 59%. This performance is comparable to PaLI first stage only models trained with 80% of the simulated LanguageTable dataset. Additionally, it is demonstrated that the Frankenstein model leads to a significantly slower self-improvement procedure, underscoring the impactful role of PaLI pre-training.

[0051] The generalization capability of the second stage self-improvement approach is tested through an experiment referred to as “BananaTable”. This procedure commences with a real- world LanguageTable policy that was subjected to second stage self-improvement for Block2Block tasks and the corresponding reward model. Subsequently, further second stage self-improvement is performed for a BananaTable task, which involves substituting the LanguageTable blocks with a single prosthetic banana and requiring policies to maneuver the banana to various positions on a board. Prior to this experiment, the policy and reward function had not previously encountered a banana or a table without blocks. Thus, the generalization abilities of the underlying PaLI model are exclusively relied upon. The BananaTable scene presents visual distinctions from LanguageTable, necessitating semantic generalization. Furthermore, effectively manipulating bananas requires the acquisition of new skills compared to those utilized for LanguageTable block manipulation, thereby demanding behavioral generalization. For example, due to the geometry of a banana, an imprecise push may cause it to rotate around its own axis rather than moving in the intended direction. Within approximately 8 hours of training utilizing 2 robot stations, the policy demonstrates enhanced proficiency in accomplishing the BananaTable tasks, with a success rate improvement from approximately 63% to approximately 85%.

[0052] In some implementations disclosed herein, LanguageTable actions are represented using 4 tokens: + / −, a token represenfing a number between 0-10, + / −, and a token representing a number between 0-10. Continuous 2D actions are binned to conform to thisAttorney Reference No. DEEP-0023-WO-01 representation. In some implementations disclosed herein, timesteps until the end of an episode are represented using one token, which indicates a number between 0-50.

[0053] In some implementations disclosed herein, the Aloha action space is 5 × 14 dimensions and each dimension is represented as 1 token, resulting in the model outputting 70 tokens. Each token represents a number from 0-255. Continuous Aloha actions are discretized and binned into these 256 bins. In some implementations disclosed herein, timesteps until the end of an episode are represented using one token, which indicates a number between 0-300. In some implementations disclosed herein, current joint positions are provided as input by appending 14 tokens to the input, where each token represents a number from 0-255. Continuous Aloha joint positions are discretized and binned into these 256 bins.

[0054] The Aloha domain involves a 14 degree-of-freedom joint-space controlled robot. In contrast to the default 50Hz, the environment operates at 10Hz. A common design choice in the Aloha domain involves training policies to predict N actions into the future. For example, a value of N = 5 is utilized, resulting in an action space that is 70-dimensional (14 × 5). The Aloha environment features 4 cameras. To convert their input into two images suitable for processing by the PaLI models, two images can be stacked into a single image with a black buffer in between.

[0055] Let µ denote the policy corresponding to the imitation learning dataset ^^ (e.g. the “human policy”). Given the definition in the equation herein, it can be observed that^^(^^^^,^^) = −^^^^(^^^^,^^), wheredenotes the undiscounted value function of policy µ for the reward function -^^[^^^^satisfies ^^] (i.e.0 in goal states, -1 elsewhere). Substituting in theadditional equation herein results in ^^(^^^^, ^^^^ + ^^^^+1, ^^) = ^^ ^^(^^^^+1,^^) − ^^^^(^^^^,^^). Thus, whenperforming Stage 2 RL updates with discount factor γ, provides ^^(^^^^,^^^^, ^^^^+1, ^^) = (1 − ^^) ∙

[0056] It can be observed that ^^(^^^^, ^^^^, ^^^^+1,^^) is implicitly a shaped reward function,providing higher rewards in states where the dataset policy µ performs well. Simplifying theMonte Carlo returns provides ^^Attorney Reference No. DEEP-0023-WO-01

[0057] Such reward shaping results in a baseline subtracted from the Monte Carlo returns, leading to lower variance estimates that are useful when employing simple RL objectives, suchas REINFORCE. When γ is close to 0, we have ^^^^ = ^^^^(^^^^+1,^^) which is closely similar to asingle-step policy improvement for the −^^[^^^^satisfies^^] reward. As ^^ → 1, ^^^^ encouragespolicies to traverse trajectories along which the states have high value under the dataset policy µ (i.e. high ^^^^). Thus, performing Stage 2 RL updates with proposed reward functions disclosed herein leads to policies that more efficiently achieve intended goals while being implicitly regularized to stay close to the dataset policy µ!.

[0058] Turning now to the Figures, FIG.1 is a diagram illustrating a two-stage fine-tuning approach for a multimodal foundation agent, in accordance with implementations disclosed herein.

[0059] The first stage, labeled stage 1: supervised training (offline) 110, involves supervised fine-tuning a base generative model 112 based on demonstration data. The base generative model 112 can be a pre-trained foundation model such as a pre-trained VLM. Supervised fine- tuning of the base generative model can include utilizing a behavioral cloning objective 101 and / or utilizing a timesteps-to-go objective 102 in training based on the demonstration data.

[0060] The second stage, labeled stage 2: sample efficient online self-improvement 120, utilizes an instance 122, of the base generative model 112, where the instance 122 is one that has been at least partially trained in the stage 1: supervised training (offline) 110 stage. For example, the instance 122 can be the base generative model 112 after all stage 1 training is complete or can be from an earlier checkpoint in the stage 1 training.

[0061] In stage 2: sample efficient online self-improvement 120, online rollouts 103 are performed by robot(s) 115 which interact with the instance 122. The robot(s) 115 can include real robots operating in real environment(s) and / or simulated robots operating in simulated environment(s).

[0062] The online rollouts 103 include robot(s) 115 attempting performance of corresponding task episodes utilizing a current version of the instance 122. Data from those episodes is provided to self-improvement reward labelling & RL updates engine 124, which generatesAttorney Reference No. DEEP-0023-WO-01 rewards 104. For example, reward labelling & RL updates engine 124 can utilize a timestep-to- go objective to generate rewards 104.

[0063] The rewards 104 are provided to update engine 126 that uses the MFA rewards 104 to update the instance 122. The updated version of the instance 122 is then used for further online rollouts. This process can continue, until condition(s) are satisfied, with the update engine 126 iteratively updating the instance 122. The update engine 126 can use the rewards 104 to update all or only portion(s) of the instance 122. For example, the update engine 126 can use the rewards 104 to update only weights of parameters of a portion of the instance 122 without updating weights of a different portion of the instance 122.

[0064] FIG.2 illustrates a method 200 for training a generative model through two distinct stages: offline supervised fine-tuning (SFT) and online reinforcement learning (RL).

[0065] At block 252, the system generates an SFT trained generative model. The system generates the SFT trained generative model by performing offline SFT, of a base generative model, based on an imitation learning dataset of successful episodes of one or more tasks. For example, the imitation learning dataset can include successful demonstrations of various tasks performed by human operators or other robots. The outcome of block 252 is an SFT trained generative model, which has adjusted internal parameters (weights) based on the SFT.

[0066] Block 252 can optionally include sub-block 252A and / or sub-block 252B.

[0067] At sub-block 252A, the system uses a behavioral cloning loss function that compares corresponding offline predictions to ground truth data of a corresponding episode. This function assesses how closely the generative model’s predicted actions (offline predictions) match the actual actions taken in the successful demonstration (ground truth data).

[0068] At sub-block 252B, the system uses a reinforcement reward loss function that compares offline predicted reward to reward reflected by a corresponding episode. For example, the system can compare a reward predicted by the generative model (offline predicted reward) to a reward derived from the successful episode itself. This reward can, for instance, be based on the number of steps remaining until the task goal is achieved. For example, if the model predicts that completion of the task will take 10 more steps, but the successful demonstration shows it only took 5, this loss function would penalize the model for the discrepancy, helping it learn to predict the efficiency or proximity to success.Attorney Reference No. DEEP-0023-WO-01

[0069] At block 254, the system performs online reinforcement learning of an instance of the base generative model. Following the offline SFT, this block represents the transition to online reinforcement learning. At this point, an instance of the base generative model, which has already benefited from the offline training, is used. This instance might be the fully SFT-trained generative model itself or a version of the base generative model after some of the offline SFT has been performed. This online phase allows the model to continue learning and refining its abilities in an interactive environment.

[0070] Block 254A can include sub-block 254A, sub-block 254B, and / or sub-block 254C.

[0071] At sub-block 254A, the system processes an instance of online data, using the instance of the base generative model, to generate one or more online predictions to implement during online reinforcement learning.

[0072] At sub-block 254B, the system processes an instance of online data, using the supervised fine-tuning trained generative model, to generate an online predicted reward. Put another way, the SFT trained generative model (the one fully trained in the offline phase) is used to process the same online data from the robotic arm’s interactions. This SFT-trained model, acting as a “critic,” generates an online predicted reward, which reflects its assessment of how well the current action is progressing towards the task goal. For instance, as a robotic arm attempts to grasp a blue block, the SFT-trained model might output a high predicted reward if the arm is moving closer to the block, indicating good progress.

[0073] At sub-block 254C, the system uses the online predicted reward in training the instance of the base generative model. This iterative process of generating actions, receiving predicted rewards, and updating the model allows the robotic arm to autonomously learn and improve its performance in real-time, even for novel tasks or in environments different from those in the initial imitation dataset.

[0074] At block 256, the system provides the instance of the base generative model, with updated weights from the online reinforcement learning, for use by one or more devices. This can involve transmitting the updated model to robot(s) or other devices.

[0075] FIG.3 schematically depicts an example architecture of a robot 320. The robot 320 includes a robot control system 360, one or more operational components 340a-340n, and one or more sensors 342a-342m. The sensors 342a-342m may include, for example, vision sensors,Attorney Reference No. DEEP-0023-WO-01 light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors 342a-m are depicted as being integral with robot 320, this is not meant to be limiting. In some implementations, sensors 342a-m may be located external to robot 320, e.g., as standalone units.

[0076] Operational components 340a-340n may include, for example, one or more end effectors and / or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot 320 may have multiple degrees of freedom and each of the actuators may control the actuation of the robot 320 within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

[0077] The robot control system 360 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller(s) of the robot 320. In some implementations, the robot 320 may comprise a “brain box” that may include all or aspects of the control system 360. For example, the brain box may provide real time bursts of data to the operational components 340a-n, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 340a-n. In some implementations, the robot control system 360 may perform one or more aspects of method(s) described herein.

[0078] As described herein, in some implementations all or aspects of the control commands generated by control system 360, in controlling a robot during performance of a robotic task, can be generated based on considering different poses and / or different time stamp image(s) generated according to techniques described herein. Although control system 360 is illustrated in FIG.3 as an integral part of the robot 320, in some implementations, all or aspects of the control system 360 may be implemented in a component that is separate from, but inAttorney Reference No. DEEP-0023-WO-01 communication with, robot 320. For example, all or aspects of control system 360 may be implemented on one or more computing devices that are in wired and / or wireless communication with the robot 320, such as computing device 410.

[0079] FIG.4 is a block diagram of an example computer system 410. Computer system 410 typically includes at least one processor 414 which communicates with a number of peripheral devices via bus subsystem 412. These peripheral devices may include a storage subsystem 424, including, for example, a memory subsystem 425 and a file storage subsystem 426, user interface output devices 420, user interface input devices 422, and a network interface subsystem 416. The input and output devices allow user interaction with computer system 410. Network interface subsystem 416 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0080] User interface input devices 422 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, a use of the term “input device” is intended to include all possible types of devices and ways to input information into computer system 410 or onto a communication network.

[0081] User interface output devices 420 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, a use of the term “output device” is intended to include all possible types of devices and ways to output information from computer system 410 to the user or to another machine or computer system.

[0082] Storage subsystem 424 stores programming and data constructs that provide functionality of some or all of the modules described herein. For example, the storage subsystem 424 may include logic to perform selected aspects of method 200 and / or to implement one or more aspects of robot 300. Memory 425 used in the storage subsystem 424 can include a number of memories including a main random-access memory (RAM) 430 forAttorney Reference No. DEEP-0023-WO-01 storage of instructions and data during program execution and a read only memory (ROM) 432 in which fixed instructions are stored. A file storage subsystem 426 can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing functionality of certain implementations may be stored by file storage subsystem 426 in the storage subsystem 424, or in other machines accessible by the processor(s) 414.

[0083] Bus subsystem 412 provides a mechanism for letting the various components and subsystems of computer system 410 communicate with each other as intended. Although bus subsystem 412 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0084] Computer system 410 can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to an ever-changing nature of computers and networks, the description of computer system 410 depicted in FIG.4 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 410 are possible having more or fewer components than the computer system depicted in FIG.4.

[0085] In some implementations a method implemented by processor(s) is provided and includes performing offline supervised fine-tuning (SFT), of a base generative model, to generate an SFT trained generative model. The offline SFT is based on an imitation learning dataset of successful episodes of one or more tasks. Performing the offline SFT includes training the generative model using both a behavioral cloning loss function and a reinforcement reward loss function. The behavioral cloning loss function compares one or more corresponding offline predictions, reflected in corresponding output generated using the generative model during the offline supervised fine-tuning, to ground truth data of a corresponding one of the successful episodes. The reinforcement reward loss function compares a corresponding offline predicted reward, reflected in the corresponding output, to a reward reflected by the corresponding one of the successful episodes. The method further includes performing online reinforcement learning (RL) of an instance of the base generative model. Performing the online RL of the instance of the base generative model includesAttorney Reference No. DEEP-0023-WO-01 processing an instance of online data, using the instance of the base generative model, to generate one or more online predictions to implement during the online RL. Performing the online RL of the instance of the base generative model further includes processing the instance of online data, using the SFT trained generative model, to generate an online predicted reward. Performing the online RL of the instance of the base generative model further includes using the online predicted reward, generated using the SFT trained generative model, in training the instance of the base generative model.

[0086] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0087] In some implementations, the instance of the base generative model is the SFT trained generative model.

[0088] In some implementations, the instance of the base generative model differs from the SFT trained generative model, but is a version of the base generative model after performing at least some of the offline SFT. For example, the SFT trained generative model can be one that results from a given quantity of offline SFT training iterations – and the instance of the base generative model can be one that results from a different (e.g., lesser) given quantity of offline SFT training iterations.

[0089] In some implementations, the predicted reward is a predicted quantity that reflects a predicted duration of time until a corresponding goal is achieved. In some versions of those implementations, the predicted quantity is a predicted quantity of remaining iterations, of utilizing the instance of the base generative model, until the corresponding goal is achieved. In some of those versions or other versions, using the reinforcement reward loss function includes generating the reward, reflected by the corresponding one of the successful episodes, based on comparing a total quantity of observations of the corresponding one of the successful episodes to a current quantity for observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output. For example, if a successful episode includes a sequence of 20 observations and the 12th observation is the one being processed in generating the corresponding output, then the reward can be 8 (20-12) – which reflects a duration of iterations (and time) until the goal (achieved in the 20th observation) is reached. In some of those versions or other versions, using the reinforcement reward lossAttorney Reference No. DEEP-0023-WO-01 function includes generating the reward, reflected by the corresponding one of the successful episodes, based on comparing a final timestep of the corresponding one of the successful episodes to a current timestep of observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output. For example, if a successful episode includes a sequence of timestamped observations, then the reward can be based on subtracting a current timestamp, of the current observation processed in generating the corresponding output, from a final timestamp of the final observation of the successful episode.

[0090] In some implementations, the one or more tasks, of the successful episodes, are robot tasks. In some versions of those implementations, the successful episodes each include a corresponding sequence of observations and actions, and a corresponding episode goal and generating the corresponding output, during the offline SFT, is based on processing, using the generative model, the corresponding episode goal and an observation, of the sequence of observations and actions. In some of those versions the observations, of the sequence of observations and actions, include a sequence of vision data instances and / or the corresponding episode goals are each natural language text that describes a corresponding task performed in the successful episode.

[0091] In some implementations, the one or more tasks, of the successful episodes, include application control tasks. In some versions of those implementations, the successful episodes each include a corresponding sequence of observations and actions, and a corresponding episode goal and generating the corresponding output, during the offline SFT, is based on processing, using the generative model, the corresponding episode goal and an observation, of the sequence of observations and actions. In some of those versions the observations, of the sequence of observations and actions, include a sequence of screenshots.

[0092] In some implementations, the method further includes, subsequent to performing the online RL, providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices. In some versions of those implementations providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices comprises transmitting, via one or more networks, the instance of the base generative model, with updated weights from the online RL, to the one or moreAttorney Reference No. DEEP-0023-WO-01 devices. In some additional or alternative versions of those implementations providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices comprises enabling access, via an application programming interface, to the instance of the base generative model, with updated weights from the online RL. In some of those versions, the one or more devices include a robot and optionally, the method can further include using the instance of the base generative model, with updated weights from the online RL, in controlling the robot. In some of those versions, the one or more devices include a client device or a server device and, optionally, the method can further includes using the instance of the base generative model, with updated weights from the online RL, in controlling one or more applications.

[0093] In some implementations, in the offline SFT an entirety of the generative model is updated during the training and / or in the RL training only a subset of the generative model is updated during the training.

[0094] In some implementations, the generative model is a pre-trained vision language model.

[0095] In some implementations, the successful episodes of the imitation learning dataset are performed by a first set of one or more robots and the one or more online predictions are implemented, during the online RL, on a distinct robot that is of a distinct type from all of the one or more robots of the first set. For example, the distinct robot can include an end effector that differs from any end effector of the robots of the first set and / or can include wheel(s) and / or leg(s) that differ from any of those of the robots of the first set.

[0096] In some implementations, the one or more online predictions to implement during the online RL are for a distinct task that is distinct from all of the one or more tasks of the successful episodes. For example, the online RL can be for a drawer opening task and none of the successful episodes are for a drawer opening task. As another example, the online RL can be for a task of controlling a given application and none of the successful episodes are for controlling the given application.

[0097] In some implementations, the method further includes performing additional online reinforcement learning (RL) of an additional instance of the base generative model. Performing the additional online RL of the instance of the additional base generative model includes: processing an additional instance of additional online data, using theAttorney Reference No. DEEP-0023-WO-01 additional instance of the base generative model, to generate one or more additional online predictions to implement during the additional online RL; processing the additional instance of online data, using the SFT trained generative model, to generate an additional online predicted reward; and using the additional online predicted reward, generated using the SFT trained generative model, in training the additional instance of the base generative model. In some versions of those implementations, the online RL is for a first task and the additional online RL is for a second task that is distinct from the first task. In some of those or other versions, the one or more online predictions are implemented, during the online RL, on a first device and the one or more additional online predictions are implemented, during the additional online RL, on a second device that is of a distinct type from the first device.

[0098] In some implementations a method implemented by processor(s) is provided and includes performing offline supervised fine-tuning (SFT), of a base generative model, to generate an SFT trained generative model. The offline SFT is based on an imitation learning dataset of successful episodes of one or more tasks. Performing the offline SFT includes training the generative model using both a behavioral cloning loss function and a reinforcement reward loss function. The behavioral cloning loss function compares one or more corresponding offline predictions, reflected in corresponding output generated using the generative model during the offline supervised fine-tuning, to ground truth data of a corresponding one of the successful episodes. The reinforcement reward loss function compares a corresponding offline predicted reward, reflected in the corresponding output, to a reward reflected by the corresponding one of the successful episodes. The method further includes providing the SFT trained generative model for use in generating predicted rewards during reinforcement learning.

[0099] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0100] In some implementations, providing the SFT trained generative model for use in generating predicted rewards during reinforcement learning includes enabling access to the SFT trained generative model via an application programming interface.Attorney Reference No. DEEP-0023-WO-01

[0101] In some implementations, providing the SFT trained generative model for use in generating predicted rewards during reinforcement learning includes transmitting the SFT trained generative model over one or more networks.

[0102] In some implementations, the reward reflected by the corresponding one of the successful episodes is a quantity that indicates a duration of time until a corresponding goal, of the corresponding one of the successful episodes, is achieved. In some versions of those implementations, the method further includes generating the quantity based on comparing a total quantity of observations of the corresponding one of the successful episodes to a current quantity for observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output. In some additional or alternative versions of those implementations, the method further includes generating the quantity based on comparing a final timestep of the corresponding one of the successful episodes to a current timestep of observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output.

[0103] In some implementations, the one or more tasks, of the successful episodes, are robot tasks.

[0104] In some implementations, the one or more tasks, of the successful episodes, include application control tasks.

[0105] In some implementations a method implemented by processor(s) is provided and includes performing online reinforcement learning (RL) of an instance of a base generative model. Performing the online RL of the instance of the base generative model includes processing an instance of online data, using the instance of the base generative model, to generate one or more online predictions to implement during the online RL. Performing the online RL of the instance of the base generative model further includes processing the instance of online data, using a supervised fine-tuning (SFT) trained generative model, to generate an online predicted reward. Performing the online RL of the instance of the base generative model further includes using the online predicted reward, generated using the SFT trained generative model, in training the instance of the base generative model. The supervised fine- tuning trained generative model is generated based on performing SFT, of the base generativeAttorney Reference No. DEEP-0023-WO-01 model, based on an imitation learning dataset of successful episodes of one or more tasks and using both a behavioral cloning loss function and a reinforcement reward loss function.

[0106] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0107] In some implementations, instance of the base generative model is the SFT trained generative model.

[0108] In some implementations, the instance of the base generative model differs from the SFT trained generative model, but is a version of the base generative model after performing at least some of the SFT.

[0109] In some implementations, the predicted reward is a predicted quantity that reflects a predicted duration of time until a corresponding goal is achieved. In some of those implementations, the predicted quantity is a predicted quantity of remaining iterations, of utilizing the instance of the base generative model, until the corresponding goal is achieved.

[0110] In some implementations, the method further includes, subsequent to performing the online RL, providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices. In some of those implementations, providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices includes transmitting, via one or more networks, the instance of the base generative model, with updated weights from the online RL, to the one or more devices.

[0111] In some implementations, in the RL training only a subset of the generative model is updated during the training.

[0112] In some implementations, the successful episodes of the imitation learning dataset are performed by a first set of one or more robots and the one or more online predictions are implemented, during the online RL, on a distinct robot that is of a distinct type from all of the one or more robots of the first set.

[0113] In some implementations, the one or more online predictions to implement during the online RL are for a distinct task that is distinct from all of the one or more tasks of the successful episodes.Attorney Reference No. DEEP-0023-WO-01

[0114] In some implementations, the method further includes performing additional online reinforcement learning (RL) of an additional instance of the base generative model. Performing the additional online RL of the instance of the additional base generative model includes: processing an additional instance of additional online data, using the additional instance of the base generative model, to generate one or more additional online predictions to implement during the additional online RL; processing the additional instance of online data, using the SFT trained generative model, to generate an additional online predicted reward; and using the additional online predicted reward, generated using the SFT trained generative model, in training the additional instance of the base generative model. In some versions of those implementations, the online RL is for a first task and the additional online RL is for a second task that is distinct from the first task. In some of those or other versions, the one or more online predictions are implemented, during the online RL, on a first device and the one or more additional online predictions are implemented, during the additional online RL, on a second device that is of a distinct type from the first device.

[0115] Some implementations include a system having memory storing instructions and one or more processors (e.g., graphic processing unit(s), tensor processing unit(s), and / or central processing unit(s)) operable to execute the instructions to cause performance of one or more of the methods disclosed herein.

[0116] Some implementations include at least one transitory or non-transitory computer-readable medium including instructions that, in response to execution by one or more processors, cause the one or more processors to perform one or more of the methods disclosed herein.

[0117] Some implementations include a robot having actuators, vision component(s) (e.g., RGB camera, RGB-D camera, LIDAR, and / or other vision component(s)) memory storing instructions, and one or more processors operable to execute the instructions to cause performance of one or more of the methods disclosed herein.

[0118] Some implementations include at least one transitory or non-transitory computer-readable medium including instructions that, in response to execution by one or more processors, cause the one or more processors to perform one or more of the methods disclosed herein.Attorney Reference No. DEEP-0023-WO-01

[0119] Some implementations includes a robot having actuators, vision component(s) (e.g., RGB camera, RGB-D camera, LIDAR, and / or other vision component(s)) memory storing instructions, and one or more processors operable to execute the instructions to cause performance of one or more of the methods disclosed herein.

[0120] Some implementations includes a system having one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to perform one or more of the methods disclosed herein.

[0121] Some implementations includes a system having memory storing instructions and one or more processors operable to execute the instructions to cause performance of one or more of the methods disclosed herein.

Claims

Attorney Reference No. DEEP-0023-WO-01 Claims 1. A method implemented by one or more processors, the method comprising: performing offline supervised fine-tuning (SFT), of a base generative model, based on an imitation learning dataset of successful episodes of one or more tasks, to generate an SFT trained generative model with updated weights relative to the base generative model, performing the offline SFT comprising training the base generative model using both a behavioral cloning loss function and a reinforcement reward loss function, wherein the behavioral cloning loss function compares one or more corresponding offline predictions, reflected in corresponding output generated using the generative model during the offline supervised fine-tuning, to ground truth data of a corresponding one of the successful episodes, and wherein the reinforcement reward loss function compares a corresponding offline predicted reward, reflected in the corresponding output, to a reward reflected by the corresponding one of the successful episodes; performing online reinforcement learning (RL) of an instance of the base generative model, performing the online RL of the instance of the base generative model comprising: processing an instance of online data, using the instance of the base generative model, to generate one or more online predictions to implement during the online RL; processing the instance of online data, using the SFT trained generative model, to generate an online predicted reward; and using the online predicted reward, generated using the SFT trained generative model, in training the instance of the base generative model.

2. The method of claim 1, wherein the instance of the base generative model is the SFT trained generative model.

3. The method of claim 1, wherein the instance of the base generative model differs from the SFT trained generative model, but is a version of the base generative model after performing at least some of the offline SFT.

4. The method of any preceding claim, wherein the online predicted reward is a predicted quantity, wherein the predicted quantity reflects a predicted duration of time until a corresponding goal is achieved.Attorney Reference No. DEEP-0023-WO-01 5. The method of claim 4, wherein the predicted quantity is a predicted quantity of remaining iterations, of utilizing the instance of the base generative model, until the corresponding goal is achieved.

6. The method of claim 4 or claim 5, wherein using the reinforcement reward loss function comprises generating the reward, reflected by the corresponding one of the successful episodes, based on comparing a total quantity of observations of the corresponding one of the successful episodes to a current quantity for observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output.

7. The method of claim 4 or claim 5, wherein using the reinforcement reward loss function comprises generating the reward, reflected by the corresponding one of the successful episodes, based on comparing a final timestep of the corresponding one of the successful episodes to a current timestep of observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output.

8. The method of any preceding claim, wherein the one or more tasks, of the successful episodes, are robot tasks.

9. The method of claim 8, wherein the successful episodes each include a corresponding sequence of observations and actions, and a corresponding episode goal and wherein generating the corresponding output, during the offline SFT, is based on processing, using the generative model, the corresponding episode goal and an observation, of the sequence of observations and actions.

10. The method of claim 9, wherein the observations, of the sequence of observations and actions, include a sequence of vision data instances.

11. The method of claim 10, wherein the corresponding episode goals are each natural language text that describes a corresponding task performed in the successful episode.

12. The method of any one of claims 1 to 7, wherein the one or more tasks, of the successful episodes, include application control tasks.

13. The method of claim 12, wherein the successful episodes each include a corresponding sequence of observations and actions, and a corresponding episode goal and wherein generating the corresponding output, during the offline SFT, is based on processing, using theAttorney Reference No. DEEP-0023-WO-01 generative model, the corresponding episode goal and an observation, of the sequence of observations and actions.

14. The method of claim 13, wherein the observations, of the sequence of observations and actions, include a sequence of screenshots.

15. The method of any preceding claim, further comprising: subsequent to performing the online RL: providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices.

16. The method of claim 15, wherein providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices comprises transmitting, via one or more networks, the instance of the base generative model, with updated weights from the online RL, to the one or more devices.

17. The method of claim 15 or claim 16, wherein the one or more devices include a robot and further comprising using the instance of the base generative model, with updated weights from the online RL, in controlling the robot.

18. The method of claim 15 or claim 16, wherein the one or more devices include a client device or a server device and further comprising using the instance of the base generative model, with updated weights from the online RL, in controlling one or more applications.

19. The method of any preceding claim, wherein in the offline SFT an entirety of the generative model is updated during the training.

20. The method of any preceding claim, wherein in the RL training only a subset of the generative model is updated during the training.

21. The method of any preceding claim, wherein the generative model is a pre-trained vision language model.

22. The method of any preceding claim, wherein the successful episodes of the imitation learning dataset are performed by a first set of one or more robots and wherein the one or more online predictions are implemented, during the online RL, on a distinct robot that is of a distinct type from all of the one or more robots of the first set.Attorney Reference No. DEEP-0023-WO-01 23. The method of any preceding claim, wherein the one or more online predictions to implement during the online RL are for a distinct task that is distinct from all of the one or more tasks of the successful episodes.

24. The method of any preceding claim, further comprising: performing additional online reinforcement learning (RL) of an additional instance of the base generative model, performing the additional online RL of the instance of the additional base generative model comprising: processing an additional instance of additional online data, using the additional instance of the base generative model, to generate one or more additional online predictions to implement during the additional online RL; processing the additional instance of online data, using the SFT trained generative model, to generate an additional online predicted reward; and using the additional online predicted reward, generated using the SFT trained generative model, in training the additional instance of the base generative model.

25. The method of claim 24, wherein the online RL is for a first task and wherein the additional online RL is for a second task that is distinct from the first task.

26. The method of claim 24, wherein the one or more online predictions are implemented, during the online RL, on a first device and wherein the one or more additional online predictions are implemented, during the additional online RL, on a second device that is of a distinct type from the first device.

27. A method implemented by one or more processors, the method comprising: performing offline supervised fine-tuning (SFT), of a base generative model, based on an imitation learning dataset of successful episodes of one or more tasks, to generate an SFT trained generative model with updated weights relative to the base generative model, performing the offline SFT comprising training the generative model using both a behavioral cloning loss function and a reinforcement reward loss function, wherein the behavioral cloning loss function compares one or more corresponding offline predictions, reflected in corresponding output generated using the generative model during the offline supervised fine-tuning, to ground truth data of a corresponding one of the successful episodes, andAttorney Reference No. DEEP-0023-WO-01 wherein the reinforcement reward loss function compares a corresponding offline predicted reward, reflected in the corresponding output, to a reward reflected by the corresponding one of the successful episodes; and providing the SFT trained generative model for use in generating predicted rewards during reinforcement learning.

28. The method of claim 27, wherein providing the SFT trained generative model for use in generating predicted rewards during reinforcement learning comprises enabling access to the SFT trained generative model via an application programming interface.

29. The method of claim 27, wherein providing the SFT trained generative model for use in generating predicted rewards during reinforcement learning comprises transmitting the SFT trained generative model over one or more networks.

30. The method of any one of claims 27 to 29, wherein the reward reflected by the corresponding one of the successful episodes is a quantity that indicates a duration of time until a corresponding goal, of the corresponding one of the successful episodes, is achieved.

31. The method of claim 30, further comprising generating the quantity based on comparing a total quantity of observations of the corresponding one of the successful episodes to a current quantity for observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output.

32. The method of claim 30, further comprising generating the quantity based on comparing a final timestep of the corresponding one of the successful episodes to a current timestep of observation data, of the corresponding one of the successful episodes, processed in generating the corresponding output.

33. The method of any one of claims 27 to 32, wherein the one or more tasks, of the successful episodes, are robot tasks.

34. The method of any one of claims 27 to 32, wherein the one or more tasks, of the successful episodes, include application control tasks.

35. The method of claim 33 or 34, wherein the successful episodes each include a corresponding sequence of observations and actions, and a corresponding episode goal and wherein generating the corresponding output, during the offline SFT, is based on processing,Attorney Reference No. DEEP-0023-WO-01 using the generative model, the corresponding episode goal and an observation, of the sequence of observations and actions.

36. The method of any one of claims 27 to 35, wherein in the offline SFT an entirety of the generative model is updated during the training.

37. A method implemented by one or more processors, the method comprising: performing online reinforcement learning (RL) of an instance of a base generative model, performing the online RL of the instance of the base generative model comprising: processing an instance of online data, using the instance of the base generative model, to generate one or more online predictions to implement during the online RL; processing the instance of online data, using a supervised fine-tuning (SFT) trained generative model, to generate an online predicted reward, wherein the supervised fine-tuning trained generative model is generated based on performing SFT, of the base generative model, based on an imitation learning dataset of successful episodes of one or more tasks and using both a behavioral cloning loss function and a reinforcement reward loss function; and using the online predicted reward, generated using the SFT trained generative model, in training the instance of the base generative model.

38. The method of claim 37, wherein the instance of the base generative model is the SFT trained generative model.

39. The method of claim 37, wherein the instance of the base generative model differs from the SFT trained generative model, but is a version of the base generative model after performing at least some of the SFT.

40. The method of any one of claims 37 to 39, wherein the predicted reward is a predicted quantity, wherein the predicted quantity reflects a predicted duration of time until a corresponding goal is achieved.

41. The method of claim 40, wherein the predicted quantity is a predicted quantity of remaining iterations, of utilizing the instance of the base generative model, until the corresponding goal is achieved.

42. The method of any one of claims 37 to 41, further comprising:Attorney Reference No. DEEP-0023-WO-01 subsequent to performing the online RL: providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices.

43. The method of claim 42, wherein providing the instance of the base generative model, with updated weights from the online RL, for use by one or more devices comprises transmitting, via one or more networks, the instance of the base generative model, with updated weights from the online RL, to the one or more devices.

44. The method of claim 42 or claim 43, wherein the one or more devices include a robot and further comprising using the instance of the base generative model, with updated weights from the online RL, in controlling the robot.

45. The method of claim 42 or claim 43, wherein the one or more devices include a client device or a server device and further comprising using the instance of the base generative model, with updated weights from the online RL, in controlling one or more applications.

46. The method of any one of claims 37 to 45, wherein in the RL training only a subset of the generative model is updated during the training.

47. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 46.

48. At least one non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 46.

Images