Training a Policy Neural Network in Simulation with a Scene Synthesis Machine Learning Model

Abstract

A method, system, and apparatus for training a policy neural network for use in controlling a robot, including a computer program encoded on a computer storage medium. In particular, the policy neural network can be trained in a simulation using images generated by a scene synthesis machine learning model.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 407,129, filed September 15, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] A machine learning model receives input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the parameters of the model.

[0003] Some machine learning models are deep models that utilize multiple layers of the model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, where each layer applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0004] Generally described herein is a system implemented as a computer program on one or more computers at one or more locations that trains a policy neural network in a simulation, and that can use the policy neural network to control a robot (also known as an agent) in the real world.

[0005] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0006] Training control policies in simulation and transferring them to a real robot (sim2real) avoids many of the issues that make learning directly in real-world environments difficult. These issues include difficulty in state estimation, safety risks, and data efficiency. Furthermore, training in simulation avoids wear on the robot before actually deploying it for use in the environment.

[0007] However, creating accurate and realistic simulations is difficult and computationally expensive. In other words, generating a scene in a simulation while accurately modeling how a robot senses and interacts with the world is a challenge.

[0008] Reducing the gap between simulation and the real world—that is, increasing training realism—often involves collecting small amounts of data followed by manual tuning, using established system identification tools, or, more recently, training neural networks that are part of the system. In particular, accurately modeling the shape and visual appearance of unstructured scenes, which affect how a robot interacts with the world and how it senses its surroundings when using, for example, an RGB camera, is challenging. While the need to model RGB cameras can be partially alleviated by using depth sensors or LiDAR, which are easier to simulate and therefore result in smaller sim2real discrepancies, such compromises can limit the set of tasks a robot can learn and the range of robots to which these techniques can be applied. In general, existing approaches to photorealistic scene reconstruction and rendering limit their applicability because they do not work well for outdoor scenes and use specialized 3D scanning setups that are not widely available.

[0009] The described technology can overcome these challenges by automatically generating simulation models of visually complex scenes with highly realistic renderings and accurate geometry of RGB camera views. In particular, the described technology learns a scene synthesis model (e.g., a NeRF model) from only a single video of the real-world scene with which the robot interacts, and uses the learned model in combination with a simulator of the physics of the environment to generate a combined simulation with sufficiently high fidelity to enable transfer of visually guided control policies from simulation to reality.

[0010] Thus, the described techniques enable zero-shot or few-shot transfer of policy neural networks from simulation to the real world, even when the robot operates in visually complex scenes and relies on observations that involve images (e.g., RGB images of the environment), or when it must manipulate dynamic objects to successfully complete a real-world task.

[0011] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0012] [Figure 1] 1 illustrates an exemplary action selection system. [Figure 2] 1 is a flow diagram of an example process for training a policy neural network. [Figure 3] An example of generating a combined simulation is given below. [Figure 4] 1 is a flow diagram of an exemplary process for generating training data in a simulation. [Figure 5] An example of generating an input observation image is shown below. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference numbers and designations in the various drawings indicate like elements.

[0014] 1 illustrates an exemplary action selection system 100. Action selection system 100 is an example of a system in which the systems, components, and techniques described below are implemented as computer programs on one or more computers at one or more locations.

[0015] The action selection system 100 controls the robot 104 to interact with the environment 106 to accomplish a task by selecting an action 108 to be performed by the robot 104 at each of multiple time steps while performing an episode of the task.

[0016] The robot 104 can be any suitable type of robot, such as a robotic arm, a humanoid robot, a quadruped robot, a vehicular robot (e.g., an autonomous vehicle), or the like.

[0017] As a general example, a task may include, for example, one or more of: navigating to a specified location in an environment, identifying a particular object in an environment, manipulating a particular object in a specified manner, etc.

[0018] More generally, tasks are specified such that the reward received maximizes, for example, an episodic return upon successful completion of the task. Rewards and returns are described in more detail below.

[0019] An "episode" of a task is a sequence of interactions during which the robot attempts to perform a single instance of the task starting from some starting state of the environment. In other words, each task episode begins with the environment in an initial state, e.g., a fixed or randomly selected initial state, and ends when the robot successfully completes the task or when some termination criterion is met, e.g., the environment enters a state identified as a terminal state or the robot performs a threshold number of actions without successfully completing the task.

[0020] At each time step during any given task episode, the system 100 receives input observations 110, including images captured by a camera of the robot 104, that cause the robot 104 to perform an action from a set of actions. For example, the set of actions can include a fixed number of actions or can be a continuous action space.

[0021] Optionally, observation 110 may include other data in addition to images captured by the robot camera. For example, observation 110 may include data from other sensors of the robot (e.g., data from the robot's gyroscope, data from the robot's accelerometer, or both). Additional data that may be included in observation 110 is described in more detail below.

[0022] After the robot 104 performs the action 108, the environment 106 transitions to a new state and the system 100 receives a reward 130 from the environment 106.

[0023] Generally, the reward 130 is a scalar value that characterizes the progress of the robot 104 toward completing the task.

[0024] As a specific example, reward 130 can be a sparse binary reward that is zero if the action being performed does not result in successful completion of the task. That is, reward 130 is only non-zero, e.g., equal to 1, if the action being performed results in successful completion of the task.

[0025] As another specific example, reward 130 can be a dense reward that measures the robot's progress toward completing the task at the time of each observation received during an episode of attempting to perform the task, i.e., a non-zero reward can be received, and is received frequently, before the task is successfully completed.

[0026] While performing any given task episode, the system 100 selects actions to attempt to maximize the returns received over the course of the task episode.

[0027] That is, at each time step in the episode, the system 100 selects an action that attempts to maximize the return received for the remainder of the task episode starting at the time step.

[0028] In general, at any given time step, the return received is a combination of the rewards received in time steps following the given time step in the episode.

[0029] For example, at time step t, the return is

number

[0030] To control the robot, at each time step of the episode, the system 100 processes the observations using a policy neural network 120 to generate a policy output 122 that defines actions 108 for controlling the robot 104 in response to the observations 110.

[0031] In one example, the policy output 122 may include a respective numerical probability value for each action in a fixed set of actions. The system 102 may select an action by, for example, sampling the actions according to the probability values ​​of the action indexes or by selecting the action with the highest probability value.

[0032] In other examples, the policy output may include, for each of a fixed set of actions, a respective Q-value. The system 102 may process the Q-values ​​(e.g., using a softmax function) to generate a probability value for each action, which may be used to select an action (as described above) or to select the action with the highest Q-value.

[0033] The Q-value of an action is an estimate of the return that results from the robot taking the action in response to a current observation and then selecting future actions that the robot will take according to the current values ​​of the parameters of the policy neural network 120.

[0034] As another example, when the action space is continuous, the policy output can include parameters of a probability distribution over the continuous action space, and the system can select an action by sampling from the probability distribution or by selecting an average action. A continuous action space is a space that contains an infinite number of actions. That is, each action is represented as a vector with one or more dimensions, and for each dimension, the action vector can take any value within the range of the dimension. The only constraint is the precision of the numeric format used by system 100.

[0035] As yet another example, when the action space is continuous, the policy output can include a recursive action, i.e., a recursive vector representing an action from the continuous space, and the system 100 can select the recursive action as the action 108.

[0036] Policy neural network 120 may have any suitable architecture that enables policy neural network 120 to map inputs, including observed images, to policy outputs.

[0037] As an example, policy neural network 120 may include an "embedding" subnetwork, a "core" subnetwork, and one or more "selection" subnetworks. A subnetwork of a neural network refers to a group of one or more neural network layers within a neural network.

[0038] When the observations are images, the embedding sub-network can be a convolutional sub-network, i.e., a convolutional sub-network including one or more convolutional neural network layers configured to process the observations for a time step.

[0039] The core sub-network can be, for example, one or more long short-term memory (LSTM) neural network layers or a recurrent sub-network including (i) the output of the embedding sub-network and, optionally, (ii) data specifying any other information in the observation, such as lower-dimensional action data, previous actions, the last received reward, etc.

[0040] Each selection subnetwork can be configured to process the output of the core subnetwork to generate a corresponding output, i.e., a corresponding set of action scores or corresponding parameters of a probability distribution. For example, each selection subnetwork can be a multilayer perceptron (MLP) or other fully connected neural network. In some cases, data specifying other information in the observation can be provided as input to the selection subnetwork(s) instead of the core subnetwork.

[0041] The system 100 can then control the robot 104 by providing the actions 108 defined by the policy outputs 122 as control inputs for the robot 104 .

[0042] Generally, the environment 106 is a real-world environment, and the robot 104 interacts with the environment 106 to achieve a goal, such as locating an object of interest within the environment, moving an object of interest to a specified location within the environment, physically manipulating an object of interest within the environment in a specified manner, or navigating to a specified destination within the environment.

[0043] In these implementations, the observations 110 may include one or more of images, object position data, and sensor data, such as, for example, from image sensors, distance sensors, or position sensors, or sensor data from actuators, to capture observations as the robot interacts with its environment.

[0044] For example, the observations may include data characterizing the current state of the robot (e.g., one or more of joint positions, joint velocities, joint forces, torques or accelerations, e.g., gravity-compensated torque feedback, and global or relative poses of items held by the robot).

[0045] As other examples, the observations may also include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the robot. The observations may be defined in one, two, or three dimensions and may be absolute and / or relative observations.

[0046] The observations may also include, for example, data acquired by one of a number of sensor devices that sense the real-world environment (e.g., sensed electronic signals such as motor current or temperature signals, and / or image data or video data, for example from a camera or LIDAR sensor, e.g., data from sensors on the robot or from sensors positioned separately from the robot in the environment).

[0047] The observations may also include data characterizing the task (e.g., data specifying target states of the robot, e.g., target joint positions, velocities, forces or torques, or higher-level states such as robot coordinates or robot velocity, data specifying target states and / or positions of other objects in the environment, data specifying the position of targets in the environment, etc.).

[0048] An action can be a control input for controlling the robot (e.g., torques at the robot's joints, or more advanced control commands).

[0049] In other words, the control inputs may include, for example, position, velocity, or force / torque / acceleration data of one or more joints or other parts of the robot. The control inputs may additionally or alternatively include electronic control data such as motor control data, or more generally, data for controlling one or more electronic devices in the environment, the control of which affects the observed state of the environment.

[0050] Optionally, in any of the above embodiments, the observations at any given time step may include data from previous time steps that may be useful in characterizing the environment (e.g., actions performed at the previous time step, rewards received at the previous time step, etc.).

[0051] Before using the policy neural network 120 to control the robot 104 , the training system 190 trains the policy neural network 120 .

[0052] More specifically, system 190 trains policy neural network 120 in a simulation, i.e., system 190 trains network 120 in a computer simulation of environment 106.

[0053] For example, the system 190 can train the policy neural network 120 in a simulation and then use the trained policy neural network 120 to control the robot 104 in the environment 106 without any further training, thereby performing zero-shot transfer from simulation to real world (sim2real).

[0054] As another example, the system 190 can train the policy neural network 120 in a simulation and then further train the policy neural network 120 while controlling the robot 104 in the environment 106, thereby performing few-shot transfer from the simulation to the real world.

[0055] In particular, when training the policy neural network 120 in a simulation, the training system 190 uses a model 192 of the robot 104 and a simulator 194 that can accurately simulate the interaction of the robot 104 with the environment 106 .

[0056] The model 192 of the robot 104 is data that specifies the configuration of the robot, e.g., the robot's sensors, and the robot's physical and visual characteristics, and can be used by the simulator 194 to model the physics of the robot.

[0057] Simulator 194 can be any suitable simulator software capable of modeling the physics of the robot and any other dynamic objects in the environment. One example of such a simulator is the MuJoCo Physics Simulator, which models the dynamics of the robot and the environment and accounts for collisions between objects. In general, simulator 194 maintains a simulation state that defines the current state (e.g., position, velocity, acceleration, etc.) of any dynamic objects in the environment and maintains data specifying the physical and visual characteristics of the dynamic objects. By modeling the physics of the environment, simulator 194 can update the simulator state to reflect changes in the environment (e.g., actions taken by the robot, the motion of other objects, collisions between objects or with static objects, etc.). Simulator 194 also includes a renderer that can render images of objects taking into account the current state of the objects and the visual characteristics of the objects.

[0058] The training system 190 also uses a scene synthesis machine learning model 196 as part of its training.

[0059] Scene synthesis machine learning model 196 is a model, such as a neural network, that is configured to receive a scene input that includes a camera viewpoint and to generate as output a synthetic image of the scene from the camera viewpoint.

[0060] Generally, during training, the training system 190 can use a simulator 194 to simulate the physics of the environment (e.g., the movement of objects in the environment and the effects on the robot and the environment of actions selected by the policy neural network 120), while using a model 196 to generate a synthetic image of the environment 106 for use in generating observations that are provided as inputs to the policy neural network 120.

[0061] This training is described in more detail below with reference to FIGS.

[0062] 2 is a flow diagram of an exemplary process 200 for training a policy neural network. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system (e.g., action selection system 100 of FIG. 1 ) can be suitably programmed in accordance with this specification to perform process 200.

[0063] The system acquires multiple images of a scene in a real-world environment with which the robot will interact (step 202).

[0064] For each image, the system also obtains corresponding camera data, including the viewpoint of the camera that captured the image.

[0065] That is, the camera data includes camera pose information for each image, and more specifically defines the internal and external characteristics of the camera used to capture each image.

[0066] For example, the system can extract images and corresponding camera data from footage of a scene captured by a camera.

[0067] As one example, the system may acquire a video of a scene in a real-world environment and then extract images from the scene by selecting video frames from the video. As one example, the system may divide the video into partitions, e.g., equal partitions, and then select one or more images from each partition. For example, the system may select one or more least blurred images from each partition by selecting the least blurred image from each partition based, e.g., on the variance of the Laplacian frames.

[0068] The system can then extract the camera data for the selected image.

[0069] As an example, the system can extract camera data from the metadata of images available to the system.

[0070] As another example, the system may extract camera data by applying structure-from-motion (SfM) techniques to the images. The COLMAP package is an example of an SfM package that may be used by the system to process images in a video to extract camera data.

[0071] The camera used to capture the image of the scene can generally be any suitable camera device and need not be the same camera or have the same characteristics as the camera used by the robot to capture the observation image. Thus, the system can leverage footage taken by a common camera, for example, a camera on a common mobile device, to extract image and camera data.

[0072] Next, the system uses the multiple images and corresponding camera data to train a scene synthesis machine learning model (step 204).

[0073] As described above, a scene synthesis machine learning model is a model, such as a neural network, that is configured to receive a scene input that includes a camera viewpoint and to generate as output a synthetic image of the scene from the camera viewpoint.

[0074] In general, the scene synthesis model can be any suitable model that, after training, is capable of generating synthetic images of a scene in a real-world environment from any viewpoint.

[0075] As an example, the scene synthesis model can be a neural radiance field (NeRF) model.

[0076] The NeRF model represents radiance in terms of neural fields that reproduce the geometric structure and appearance of a scene, and can reconstruct a set of input images using backpropagation. In particular, the NeRF model can predict the radiance and occupancy in space, i.e., the underlying spatial geometry, as part of rendering an image of the scene from a given viewpoint.

[0077] In particular, a NeRF model takes as input a camera pose and generates as output a synthetic image of a scene that looks as if the image had been taken by a camera having the input camera pose. In some cases, a NeRF model also receives as input camera internal properties and generates as output a synthetic image that has the input camera pose and looks as if the image had been taken by a camera having the input camera internal properties.

[0078] The system can train any of a variety of NeRF models using any of a variety of NeRF variants. Examples of such models and loss functions for training them include J.T. Barron, B. Mildenhall, D. Verbin, P.P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” CoRR, vol. abs / 2111.12077, 2021. T.M. ¨uller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans.,Graph., vol. 41, pp. 102:1–102:15, July 2022. D. Verbin, P. Hedman, B. Mildenhall, T.E. Zickler, J.T. Barron, and P.P. Srinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance fields,” CoRR, vol. abs / 2111.12077, 2021. fields,” CoRR, vol.abs / 2112.03907, 2021. J.T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P.P. Srinivasan, “Mip-nerf: A multiscale representation for antialiasing neural radiance fields,” CoRR, vol.abs / 2103.13415, 2021.

[0079] As a specific example, the system may utilize one or more of the following transformations to improve reconstruction quality and reconstructed geometry, and to reduce rendering time:

[0080] As an example, to avoid artifacts during rendering at low resolution, the system can sample the mean of the volume on a normal distribution.

[0081] As another example, the system may use a spatial squashing formulation to support large capture regions as well as separate "proposal" networks, and a "distortion" loss to encourage compact representations.

[0082] As another example, the system can optimize separate specular and diffuse colors to improve the reconstructed geometry.

[0083] As another example, to reduce latency, the system can implement a multi-scale spatial hash grid approach, which allows, for example, a V100 GPU to render one frame in 6ms.

[0084] As another example, the system may use any suitable architecture for the multilayer perceptrons (MLPs) that make up the NeRF model. For example, the system may use an architecture that adds layer normalization before the last MLP layer, and may use, for example, swish activation rather than ReLU activation as in the original NeRF model.

[0085] As another example, a system can adapt NeRF to sample radiance volumes over a distribution. To achieve this, the system can blur the training samples with a Gaussian blur with random variance σblur∈[σmin,σmax] and provide Σ = Σsample*(1 + (σblur-σmin)) as an additional input to the final MLP of the NeRF model. This extension allows the network to interpolate samples in scale space, significantly improving reconstruction at lower resolutions. For example, using this extension can result in an average PSNR of ~31.5 vs. ~35.4 for the example image set retained.

[0086] Thus, the system trains a model that is capable of generating synthetic images of scenes in a real-world environment.

[0087] Next, the system generates training data for training the policy neural network using at least the synthetic images generated by the scene synthesis machine learning model (step 206).

[0088] That is, while collecting data during training data generation, the system generates observations of a scene in a simulation of an environment interacted by a model of a robot from synthetic images generated by a scene synthesis machine learning model. The system can then control the model of a robot in the simulation using outputs generated based on the observations. That is, the system uses the trained scene synthesis model to generate images of the state of the simulation of the environment, and these images are then provided as inputs to a policy neural network.

[0089] FIG. 3 shows an example 300 of generating a combined simulation 310 using a simulator and a scene synthesis machine learning model for use in generating training data for training a policy neural network.

[0090] As shown in Figure 3, the system receives an input video 302 of a scene in a real-world environment. In the example of Figure 3, the video is generated using a camera on a mobile device. However, more generally, the video can be generated using any suitable camera device capable of capturing video of a scene from multiple viewpoints.

[0091] The system applies COLMAP or a different SfM package to extract a set of images with corresponding camera data, including camera poses 304, from video 302. The system then trains a scene synthesis machine learning model, e.g., a NeRF model 306, that generates new synthetic images of the scene from any viewpoint / camera pose.

[0092] In general, as described above, a scene synthesis machine learning model receives as input a new viewpoint and camera internal characteristics of a camera, and produces as output a synthetic image of a scene captured from the new viewpoint and by the camera with the input camera internal characteristics.

[0093] When rendering a given image of the scene in the simulation (308), the system uses the obtained camera internal characteristics, such as focal length, distortion parameters, or both, generated as a result of calibrating the robot's camera. Thus, the simulated rendered image appears as if it were taken by the robot's camera in the real-world environment. In other words, the system models the visuals of the environment using the rendered image generated using the robot's camera internal characteristics.

[0094] The NeRF model 306 learns functions that predict the radiance and occupancy in space, i.e., the geometry of the underlying scene.

[0095] As part of generating the combined simulation 310, the system uses the trained scene synthesis model to generate a mesh of the scene. The system can then provide the mesh to the simulator for use in modeling collisions when updating the state of the simulation as part of the combined simulation 310.

[0096] In particular, the system can generate an initial mesh in a first reference frame from the trained synthesis model, and then generate mesh 309 by mapping the vertices of the initial mesh from the first reference frame of the scene synthesis model to the simulator's world reference frame.

[0097] More specifically, the system voxelizes the predicted occupancy generated by the trained scene synthesis model and uses this predicted occupancy to compute an initial mesh, for example, via a marching cubes algorithm. As described in more detail below, the camera pose obtained from COLMAP, and therefore the vertices of the collision mesh, are expressed in an arbitrary frame of reference (including any scale). As a result, the system estimates a rigid transformation and scale between this frame of reference and the simulator's world frame. For example, the system can compute an estimate by solving a least-squares optimization that constrains the normal vector at the dominant floor of the mesh to be aligned with the z-axis in the simulator. The system then calculates the relative scale between NeRF and the world by rotating the initial mesh about the z-axis to the desired alignment with the simulator's world frame and comparing the size of objects in the initial mesh with the size of objects in the real world to generate mesh 309.

[0098] The system may also replace the vertices of the mesh floor (which may have artifacts due to the lack of texture) with flat surfaces. Optionally, for faster collision calculations, the system trims the mesh 309 to the extent required for the simulation.

[0099] The system can then use the mesh 309 for collisions in a combined simulation 310 .

[0100] The system can then combine the generated mesh with a model of the robot and any other dynamic objects in a physics simulator to generate a combined simulation 310. That is, while running an episode of a task in a simulation to generate training data, the system generates a composite scene by modeling the state of the model of the robot and any other dynamic objects using the physics simulator, while (i) modeling static aspects of the scene using images synthesized using a scene synthesis neural network, and (ii) modeling collisions using the mesh 309.

[0101] This is explained in more detail below with reference to FIGS.

[0102] 4 is a flow diagram of an exemplary process 400 for generating training data for training a policy neural network. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system (e.g., action selection system 100 of FIG. 1 ) can be suitably programmed in accordance with this specification to perform process 400.

[0103] In particular, as part of generating the training data, the system uses the policy neural network at each of a number of time steps to control a model of a robot in a simulation of an environment, for example, to attempt to perform an episode of a task within the simulation.

[0104] At each time step, the system obtains an input camera viewpoint from the simulator based on the position of the robot camera at the time step within the simulated state of the real-world environment at the time step (step 402).

[0105] That is, as described above, the simulator maintains a simulation state that is updated over time. At any given time step, the simulator state identifies the current state of the robot, including, for example, the current camera viewpoint of the robot's camera. The system can use this current camera viewpoint as the input camera viewpoint for the time step.

[0106] In some implementations, the simulator operates in a different frame of reference than the scene synthesis model, e.g., trained with inputs specifying a camera viewpoint in a different frame of reference than the one used by the simulator. For example, the scene synthesis model can be configured to receive a camera viewpoint in a first frame of reference, e.g., any frame of reference generated by SfM used by the system, and estimate camera data for images in the training data while the simulator operates in the world frame of reference.

[0107] In these embodiments, as part of obtaining an input camera viewpoint from the simulator based on the position of the robot camera at a time step in the simulation of the real-world environment, the system receives an initial camera viewpoint in the world frame of reference from the simulator and generates the input camera viewpoint by mapping the initial camera viewpoint from the world frame of reference to the first frame of reference, e.g., by applying a rigid transformation and scale to the initial camera viewpoint, and generates the camera viewpoint in the first frame of reference as described above.

[0108] The system uses the scene synthesis model to generate a synthetic image of the scene from the input camera viewpoint (step 404).

[0109] That is, the system processes inputs specifying camera viewpoints using a scene synthesis model to generate, as output, a synthetic image of the scene from the input camera viewpoint. As noted above, in some cases, the input to the scene synthesis model also includes data specifying the internal characteristics of the camera capturing the image. In these cases, the system provides, as part of the input, data specifying the internal characteristics of the robot's camera in order to maximize alignment between images processed during simulation and images processed in the real world after training.

[0110] In other words, if the camera that captured the multiple images used to train the scene synthesis model is different from the robotic camera, and the camera data used to train the scene synthesis model includes camera parameters that specify internal characteristics of the camera that captured the multiple images, and the scene input further includes input camera parameters that specify internal characteristics of the input camera that the synthesized images generated by the scene synthesis machine learning should match, then the system generates each of the observations by providing a scene input that includes input camera parameters that specify the internal characteristics of the robotic camera instead of the internal characteristics of the camera that captured the multiple images.

[0111] The system generates input images of time steps from at least a composite image of the scene (step 406).

[0112] Generally, the synthetic image of the scene does not include the robot or any dynamic objects currently in the scene.

[0113] Therefore, to account for this, the system obtains from the simulator a rendering of each of one or more dynamic objects in the environment at the time step, and generates an input image for the time step by combining the respective renderings with a composite image of the scene.

[0114] That is, the simulator renders the dynamic objects of the scene (including the robot) based on the respective states of these objects and the respective visual properties of the objects as maintained by the simulator.

[0115] Generating the input image is described in more detail below with reference to FIG.

[0116] The system processes the observations, including the input image, using the policy neural network to generate a policy output (step 408), and uses the policy output to select an action (step 410), e.g., by selecting an action as described above, or by applying a search policy to the policy output to select an action.

[0117] The system provides the selected actions to the simulator for use in controlling the model of the robot to update the state of the simulation (step 412). That is, the system provides the selected actions to the simulator, and the simulator uses the selected actions to simulate the physics of the environment to update the state of the simulation, e.g., to update the state of the robot or any other dynamic objects in the environment.

[0118] The system can then generate, for each of the time steps, a respective training example that includes the observation (including the input image) at the time step and the selected action at the time step.

[0119] In general, the system also receives a respective reward for each time step from the simulator and then includes each reward in the training examples for that time step.

[0120] In some implementations, the system may regularize the received rewards before using them for training, e.g., to improve the transfer of the learned policy neural network from the simulation to the real world. As an example, the system may use the following reward components as regularization: 1. a constant penalty that encourages the robot to turn slowly whenever the robot's yaw angular velocity is greater than π rad s-1; 2. L2 regularization of joint angles toward a default standing posture; 3. When the robot is humanoid or quadruped, a penalty that encourages the robot's forward foot velocity to average 0.3 ms -1 These rewards encourage the policy neural network to learn gaits that transfer better, encouraging better exploration for faster learning.

[0121] The system can train the policy neural network through reinforcement learning using any suitable reinforcement learning technique, for example, an off-policy reinforcement learning technique using an actor-critic framework. Examples of such techniques include policy gradient techniques, Q-learning techniques, policy improvement techniques, etc. As a specific example, the reinforcement learning can be a DMPO or MPO technique.

[0122] In some implementations, the system can use an asymmetric actor-critic setup for training in a simulation, where a separate neural network that is not evaluated on the robot, i.e., the critic that is not used after training, receives privileged information. As a specific example, the critic can share the same network structure as the actor, but with the image encoder replaced with the ground truth state of the simulation (robot / object pose and velocity).

[0123] For example, the system can store the generated training examples in a replay memory, and then sample batches of training examples from the replay memory and use reinforcement learning techniques to train the policy neural network on the sampled batches of training examples.

[0124] In some implementations, the system may utilize data augmentation during training to improve the likelihood that the policy neural network will successfully transfer to the real world. For example, the NeRF model may significantly reduce the sim2real gap with realistic scene renderings, while the system may apply image augmentation to more reliably modulate image intensity characteristics, such as brightness or gain. For example, the system may perform one or more of randomizing brightness, randomizing saturation, randomizing hue, randomizing contrast, or applying a random transformation to images provided as inputs to the neural network during training.

[0125] Additionally, in some implementations, the system may use domain randomization during training to improve the likelihood of a successful transfer. Some examples of such randomization follow. As one example, the system may apply random pushes to the robot during training. As another example, the system may apply a constant delay per episode, uniformly sampled from a specified range, e.g., 10 ms to 50 ms, and optionally jitter, to all simulated sensor data to reflect varying robot latencies. As another example, at the beginning of each episode, the system may attach a random mass to a random location on the robot's torso and randomize the location of the IMU on the torso. As another example, in a task involving a ball or other dynamic object, the system may further randomize the mass and radius of the dynamic object (e.g., the ball) at the beginning of each episode.

[0126] By repeatedly running process 400 to collect training data and repeatedly training on training examples sampled from the replay memory, the system trains the policy neural network to effectively control a model of a robot in a simulation.

[0127] After training, the system can use the policy neural network to control a robot in a real-world environment.

[0128] FIG. 5 shows an example 500 of generating input images during training.

[0129] 5, the simulator maintains a physics simulation state 502. At any given time, the system uses the state 502 to generate a static scene render 504 using a scene composition model, while the simulator is used to generate a dynamic object render 506 that shows the current view of dynamic objects in the environment.

[0130] The system then generates a combined render 508 from the static scene render 504 and the dynamic object render 506. For example, the system may overlay the dynamic object rendering onto the static scene render 504, or combine the two renders in a different way.

[0131] The simulator also uses the static scene mesh 510 (generated using the scene composition model), the dynamic object meshes 512, and non-geometric properties (e.g., friction) 514 to generate inputs to a collision engine 516 and a physics engine 518 that update the simulation state 502, for example, based on the motion of the dynamic objects and the actions selected for the robot by the system.

[0132] As noted above, the system can train the policy neural network to perform any of a variety of tasks, some examples of which follow.

[0133] As an example, the task can be a navigation and obstacle avoidance task, e.g., a point-to-point visual navigation task in which the robot must reach one or more targets (specified as (x,y) coordinates in the NeRF frame of reference) while avoiding various obstacles in the environment (e.g., objects such as large plants, chairs, and walls).

[0134] During training, the system can automatically calculate the free areas of the scene using the NeRF mesh, and during simulation, the system can randomly initialize the robot to a location and orientation within these free areas and select targets in different parts of space where the robot needs to arrive.

[0135] As an example, for this task, the reward for training could include one or more of the regularization terms above, plus two task-specific terms: 1. a small bonus for reaching the goal location; and 2. a walking reward similar to that used as regularization, but instead encouraging movement in the direction of the goal at a specific speed, e.g., 0.3 ms-1. The episode ends whenever any body part of the robot other than its feet touches the scene mesh. The episode is successful if the robot reaches within 25 cm of the target without falling or colliding with any obstacles.

[0136] Another example of a task is a ball-pushing task, or more generally, an object-movement task, where the robot must move a specified object to a specified location in the environment. One example of such a task is a task where the robot must move a basketball to a corner of the workspace. The system can model the basketball as a simple orange ball. During simulation training, each episode starts with a randomly placed ball and robot. In a portion of every episode, e.g., half, the system initializes the ball directly in front of the robot to speed up learning.

[0137] As rewards, the system can use one or more of the regularization terms above and two task-specific terms: 1. A reward for minimizing the distance between the ball and the goal area, and 2. A reward for minimizing the distance between the robot and the ball if the ball is not moving towards the goal.

[0138] Many other tasks are possible, dictated by the reward received.

[0139] This specification uses the term "configured" in the context of systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the operating system to perform the operation or action. A computer program or programs configured to perform a particular operation or action means that the program or programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0140] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, such as the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions can be encoded into an artificially generated transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus.

[0141] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for data processing, including, by way of example, a programmable processor, a computer, or multiple processors or multiple computers. An apparatus can be or further include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus optionally includes, in addition to hardware, code that creates an execution environment for a computer program (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof).

[0142] A computer program, which may be called or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or to be executed on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0143] As used herein, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and run on the same computer or computers.

[0144] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry (e.g., FPGAs or ASICs), or a combination of special purpose logic circuitry and one or more programmed computers.

[0145] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The basic components of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes, or is operatively coupled to, one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks. However, a computer need not include such devices. Furthermore, a computer can be incorporated in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or portable storage device, universal serial bus (USB) flash drive, to name just a few.

[0146] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all types of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0147] To interact with a user, embodiments of the subject matter described herein can be implemented in a computer that has a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to interact with a user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer may interact with a user by sending documents to a device used by the user and receiving documents from a device used by the user (e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser). A computer may also interact with a user by sending text messages or other types of messages to a personal device (e.g., a smartphone running a messaging application) and then receiving a reply message from the user.

[0148] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units for handling, for example, the general and numerically intensive parts of the machine learning training or machine learning production, i.e., inference, workload.

[0149] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework or the Jax framework.

[0150] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components such as, for example, a data server, or that includes middleware components such as, for example, an application server, or that includes front-end components such as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network (LAN), a wide area network (WAN), e.g., the Internet.

[0151] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a user device, e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., the results of user interactions, can be received from the device by the server.

[0152] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0153] Similarly, while operations are depicted in the figures and described in the claims in a particular order, it should not be understood that desirable results require that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0154] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still produce desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0155] Aspects of the present disclosure may be as described in the following clauses. Article 1. A method implemented by one or more computers, comprising: acquiring a plurality of images of a scene in a real-world environment with which a robot interacts and, for each image, corresponding camera data including the viewpoint of the camera that captured the image; training a scene synthesis machine learning model using the plurality of images and corresponding camera data, the scene synthesis machine learning model configured to receive a scene input including a camera viewpoint and to generate as output a synthetic image of the scene from the camera viewpoint; and generating training data for training a policy neural network for use in controlling a robot in a real-world environment to perform one or more tasks using at least the synthetic images generated by the scene synthesis machine learning model, the policy neural network being configured to receive policy inputs including observations characterizing a current state of the environment and to generate as outputs policy outputs defining actions to be performed by the robot in response to the observations, the observations including images of the environment captured by a robotic camera of the robot, and generating the training data includes: A method comprising generating, from a synthetic image generated by a scene synthesis machine learning model, an observation of a scene in a simulation of an environment interacted with by a model of a robot. Article 2. 10. The method of claim 1, further comprising training a policy neural network with the training data. Article 3. 3. The method of claim 2, further comprising, after training, using the policy neural network to control an agent in a real-world environment. Article 4. Taking multiple images is Acquiring a video of a scene in a real-world environment and capturing it as a plurality of images. and selecting a plurality of video frames from the video. Article 5. Using Structure from Motion (SfM), for each of the multiple images 5. The method of clause 4, further comprising determining camera data. Article 6. Generating training data for training a policy neural network involves: Controlling a model of a robot in a simulation of an environment using a policy neural network at each of a plurality of time steps, wherein at each time step: obtaining, from the simulator, an input camera viewpoint based on the position of the robot camera at the time step within a simulated state of the real-world environment at the time step; generating a synthetic image of the scene from an input camera viewpoint using the scene synthesis model; generating time-stepped input images from at least a composite image of the scene; processing observations including an input image using a policy neural network to generate a policy output; Using the policy output to select an action; and controlling, including providing selected actions to the simulator for use in controlling the model of the robot to update a state of the simulation; generating a respective training example for each of the time steps comprising an observation for the time step and a selected action for the time step. Article 7. Generating input images of at least a time step from a synthetic image of the scene involves: obtaining, from the simulator, a rendering of each of one or more dynamic objects in the environment at the time step; 7. The method of claim 6, comprising generating input images of the time steps by combining the composite image of the scene and the respective renderings. Article 8. The scene synthesis model is configured to receive a camera viewpoint in a first reference frame, and the simulator operates in a world reference frame, and obtaining from the simulator an input camera viewpoint based on a position of the robot camera at a time step within the simulation of the real-world environment comprises: receiving an initial camera viewpoint in a world reference frame from a simulator; and generating an input camera viewpoint by mapping an initial camera viewpoint from a world frame of reference to the first frame of reference. Article 9. 9. The method of any one of clauses 6 to 8, further comprising receiving, at each time step, from the simulator, a respective reward for each of the one or more tasks, wherein the training examples include the respective rewards. Article 10. generating a mesh of the scene using the trained scene synthesis model; and and providing the mesh to the simulator for use in modeling the collision when updating the state of the simulation. Article 11. Subject to clause 8, generating a mesh may: generating an initial mesh in a first reference frame; and generating a mesh by mapping vertices of the initial mesh from the first reference frame to a world reference frame of the simulator. Article 12. 10. The method of any preceding clause, wherein the observations further include data from a robot gyroscope, a robot accelerometer, or both. Article 13. Subject to clause 2, training the policy neural network 10. The method of any of the preceding clauses, comprising training a policy neural network via reinforcement learning with domain randomization. Article 14. 10. The method of any of the preceding clauses, wherein the scene synthesis model is a neural radiance field (NeRF) model. Article 15. The camera that captured the plurality of images is different from the robot camera, the camera data further includes camera parameters that specify internal characteristics of the camera that captured the plurality of images, and the scene input further includes input camera parameters that specify internal characteristics of the input camera that the synthetic image generated by the scene synthesis machine learning model should match, and generating, from the synthetic image generated by the scene synthesis machine learning model, an observation of a scene in a simulation of an environment interacting with a model of the robot: 10. The method of claim 1, further comprising: generating each of the observations by providing a scene input including input camera parameters that specify internal characteristics of a robotic camera instead of internal characteristics of a camera that captured the plurality of images. Article 16. A system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective method set forth in any one of clauses 1 to 15. Article 17. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations recited in any one of clauses 1-15.

Claims

1. 1. A method implemented by one or more computers, comprising: acquiring a plurality of images of a scene in a real-world environment with which a robot interacts and, for each image, corresponding camera data including the viewpoint of a camera that captured the image; training a scene synthesis machine learning model using the plurality of images and the corresponding camera data, the scene synthesis machine learning model configured to receive a scene input including a camera viewpoint and to generate as output a synthetic image of the scene from the camera viewpoint; generating training data for training a policy neural network for use in controlling the robot in the real-world environment to perform one or more tasks using at least synthetic images generated by the scene synthesis machine learning model, the policy neural network being configured to receive policy inputs including observations characterizing a current state of the environment and to generate as outputs policy outputs defining actions to be taken by the robot in response to the observations, the observations including images of the environment captured by a robotic camera of the robot; and generating the training data using at least synthetic images generated by the scene synthesis machine learning model; generating, from the synthetic images generated by the scene synthesis machine learning model, observations of a scene in the simulation of the environment interacting with the model of the robot; A method comprising:

2. The method of claim 1 , further comprising training the policy neural network with the training data.

3. The method of claim 2 , further comprising, after the training, using the policy neural network to control the robot in the real-world environment.

4. acquiring the plurality of images capturing an image of the scene in the real-world environment; selecting a plurality of video frames from the video as the plurality of images; The method according to any one of claims 1 to 3, comprising:

5. The method of claim 4 , further comprising using structure-from-motion (SfM) to determine the camera data for each of the plurality of images.

6. Generating the training data for training the policy neural network includes: controlling the model of the robot in the simulation of the environment using the policy neural network at each of a plurality of time steps, wherein at each time step: obtaining, from a simulator, an input camera viewpoint based on a position of the robot camera at the time step within the simulated state of the real-world environment at the time step; generating a synthetic image of the scene from the input camera viewpoint using the scene synthesis model; generating an input image for said time step from at least said composite image of said scene; processing observations comprising the input image using the policy neural network to generate a policy output; using the policy output to select an action; and controlling, including providing the selected actions to the simulator for use in controlling the model of the robot to update the state of the simulation; generating a respective training example for each of the time steps that includes the observation for the time step and the selected action for the time step; The method according to any one of claims 1 to 5, comprising:

7. Generating an input image for the time step from the composite image of at least the scene comprises: obtaining from the simulator a rendering of each of one or more dynamic objects in the environment at the time step; generating the input image for the time step by combining the composite image and the respective renderings of the scene; The method of claim 6, comprising:

8. the scene synthesis model is configured to receive a camera viewpoint in a first reference frame, the simulator operates in a world reference frame, and obtaining an input camera viewpoint from the simulator based on a position of the robot camera at the time step within the simulation of the real-world environment includes: receiving an initial camera viewpoint in the world reference frame from the simulator; generating the input camera viewpoint by mapping the initial camera viewpoint from the world frame of reference to the first frame of reference; The method of claim 6 or 7, comprising:

9. 9. The method of claim 6, further comprising receiving, at each time step, from the simulator a respective reward for each of the one or more tasks, wherein the training examples include the respective rewards.

10. generating a mesh of the scene using the trained scene synthesis model; and providing the mesh to the simulator for use in modeling a collision when updating the state of the simulation; The method of any one of claims 1 to 9, further comprising:

11. generating the mesh generating an initial mesh in the first reference frame; generating the mesh by mapping vertices of the initial mesh from the first reference frame to the world reference frame of the simulator; The method of claim 10, when relying on claim 8, comprising:

12. The method of any one of claims 1 to 11, wherein the observations further include data from a gyroscope of the robot, an accelerometer of the robot, or both.

13. Training the policy neural network comprises: Training the policy neural network via reinforcement learning with domain randomization The method according to any one of claims 1 to 12, when claim 2 is recited, comprising:

14. The method of any one of claims 1 to 13, wherein the scene synthesis model is a neural radiance field (NeRF) model.

15. the camera that captured the plurality of images is different from the robot camera, the camera data further includes camera parameters that specify internal characteristics of the camera that captured the plurality of images, and the scene input further includes input camera parameters that specify internal characteristics of an input camera to which the synthetic image generated by the scene synthesis machine learning model should match; and generating, from the synthetic image generated by the scene synthesis machine learning model, an observation of a scene in a simulation of the environment interacting with a model of the robot, 15. The method of any one of claims 1 to 14, comprising generating each of the observations by providing a scene input including input camera parameters that specify internal characteristics of the robotic camera in place of internal characteristics of the camera that captured the plurality of images.

16. one or more computers; A system comprising one or more storage devices communicatively coupled to the one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the respective methods described in any one of claims 1 to 15.

17. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 15.

Images