Training policy neural networks in simulations using scene-synthesized machine learning models

Abstract

A method, system, and apparatus for training a policy neural network for use in controlling a robot, including a computer program encoded on a computer storage medium. In particular, the policy neural network can be trained in a simulation using images generated by a scene synthesis machine learning model.

Description

Technical Field

[0001] Cross-Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Application No. 63 / 407,129, filed on September 15, 2022, the entire contents of which are incorporated herein by reference.

Background Art

[0002] A machine learning model receives input and generates an output, for example a predictive output, based on the received input. Some machine learning models are parametric models, which generate outputs based on received inputs and values of model parameters.

[0003] Some machine learning models are deep models that utilize multiple layers of the model to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, where each layer applies a non-linear transformation to the received input to generate an output.

Summary of the Invention

[0004] Described herein are systems, generally implemented as computer programs on one or more computers at one or more locations, that train a policy neural network in simulation and are capable of controlling a real-world robot (also known as an agent) using the policy neural network.

[0005] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0006] By training control policies in simulation and then transferring them to a real robot (sim2real), many problems that make direct learning in a real-world environment difficult are avoided. Examples of these problems include the difficulty of state estimation, safety risks, and data efficiency. Furthermore, training in simulation avoids wear and tear on the robot before it is actually deployed for use in an environment.

[0007] However, creating accurate and realistic simulations is difficult and computationally expensive. In other words, generating a simulation scene while accurately modeling how a robot perceives and interacts with the world is a challenging task.

[0008] Reducing the gap between simulation and the real world, i.e., increasing the realism of training, often involves collecting small amounts of data followed by manual tuning, using established system identification tools, or, more recently, training a neural network of part of the system. In particular, accurately modeling the shape and visual appearance of unstructured scenes, which influence how a robot interacts with the world and, for example, how it perceives its surroundings when using an RGB camera, is challenging. While the need to model an RGB camera can be partially mitigated by using a depth sensor or LiDAR, which is easier to simulate and therefore results in smaller sim2real differences, such compromises may limit the set of tasks a robot can learn and restrict the range of robots to which these techniques can be applied. Generally, existing approaches to photorealistic scene reconstruction and rendering do not work well with outdoor scenes and limit their applicability by using specialized 3D scanning setups that are not widely available.

[0009] The described technology can overcome these challenges by automatically generating simulation models of visually complex scenes using highly realistic rendering and accurate shapes of RGB camera views. Specifically, the described technology learns a scene synthesis model (e.g., a NeRF model) from only a single image of a real-world scene in which the robot interacts, and uses the learned model in combination with a simulator of the physical phenomena of the environment to generate a combined simulation with sufficient fidelity to transfer visual guidance control policies from the simulation to reality.

[0010] Therefore, the described technology enables zero-shot or few-shot transfers of policy neural networks from simulation to the real world, whether the robot operates in visually complex scenes and relies on observations including images (e.g., RGB images of the environment), or when it needs to manipulate dynamic objects to successfully complete real-world tasks.

[0011] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0012] [Figure 1] An exemplary action selection system is shown. [Figure 2] This is an exemplary process flowchart for training a policy neural network. [Figure 3] This example shows how to generate a combined simulation. [Figure 4] This is an illustrative flowchart of the process for generating training data in a simulation. [Figure 5] This example shows how to generate an input observation image. [Modes for carrying out the invention]

[0013] Similar reference numbers and symbols in various drawings refer to the same elements.

[0014] Figure 1 shows an exemplary action selection system 100. The action selection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, on which the systems, components, and techniques described below are implemented.

[0015] The action selection system 100 controls the robot 104 as it interacts with the environment 106 to accomplish the task by selecting actions 108 to be performed by the robot 104 at each of several time steps while the task episode is being executed.

[0016] Robot 104 can be any suitable type of robot, such as a robotic arm, a humanoid robot, a quadruped robot, or a vehicle robot (e.g., an autonomous vehicle).

[0017] Typical examples of tasks may include one or more of the following: navigating to a specified location within an environment, identifying a specific object within an environment, or manipulating a specific object in a specified manner.

[0018] More generally, tasks are designed so that the reward received maximizes the episode return, for example, when the task is successfully completed. Rewards and returns are explained in more detail below.

[0019] An "episode" of a task is a sequence of interactions during which a robot attempts to perform a single instance of a task starting from a certain starting state of the environment. In other words, each task episode begins with the environment in an initial state, e.g., a fixed initial state or a randomly selected initial state, and ends when the robot successfully completes the task, or when some termination criterion is met, such as the environment entering a state specified as a terminal state, or the robot performing a threshold number of actions without successfully completing the task.

[0020] At each time step during any given task episode, the system 100 receives an input observation 110 that includes an image captured by a camera of the robot 104 and causes the robot 104 to perform an action from a set of actions. For example, the set of actions can include a fixed number of actions, or can be a continuous action space.

[0021] Optionally, the observation 110 can also include other data in addition to the image captured by the robot camera. For example, the observation 110 can include data from other sensors of the robot (e.g., data from the robot's gyroscope, data from the robot's accelerometer, or both). Additional data that can be included in the observation 110 is described in more detail below.

[0022] After the robot 104 performs the action 108, the environment 106 transitions to a new state, and the system 100 receives a reward 130 from the environment 106.

[0023] In general, the reward 130 is a scalar value that characterizes the progress of the robot 104 toward completing the task.

[0024] As a specific example, if the task is not successfully completed as a result of the action being performed, the reward 130 may be a sparse binary reward that is zero. That is, the reward 130 is non-zero, e.g., equal to 1, only if the task is successfully completed as a result of the performed action.

[0025] As another specific example, the reward 130 may be a dense reward that measures the robot's progress toward task completion at the time of each individual observation received during an episode attempting to perform the task. That is, non-zero rewards can be received and are frequently received before the task is successfully completed.

[0026] While executing any given task episode, the system 100 selects actions to attempt to maximize the return received over the course of the task episode.

[0027] That is, at each time step in the episode, the system 100 selects an action that attempts to maximize the return received for the remainder of the task episode starting from the time step.

[0028] In general, at any given time step, the received return is a combination of rewards received at time steps after the given time step within the episode.

[0029] For example, at time step t, the return is [Formula] can be satisfied. In the formula, i ranges over either all time steps after t in the episode, or a fixed number of time steps after t in the episode, γ is a discount factor greater than zero and less than or equal to 1, and r i is the reward at time step i.

[0030] To control the robot, at each time step of the episode, the system 100 uses a policy neural network 120 to process observations and generate a policy output 122 that defines an action 108 to control the robot 104 in response to the observation 110.

[0031] For example, policy output 122 may include a numerical probability value for each action in a fixed set of actions. System 102 can select actions, for example, by sampling actions according to the probability values ​​of the action index, or by selecting the action with the highest probability value.

[0032] In another example, the policy output may include a Q value for each of a fixed set of actions. System 102 can process the Q values ​​(for example, using a softmax function) to generate a probability value for each action, and use that probability value to select an action (as described above), or to select the action with the highest Q value.

[0033] The Q value of an action is an estimate of the return resulting from the robot performing an action in response to the current observation, and then selecting a future action to perform according to the current values ​​of the parameters of the policy neural network 120.

[0034] As another example, when the action space is continuous, the policy output may include parameters of a probability distribution across the continuous action space, and the system can select an action by sampling from the probability distribution or by selecting the mean action. A continuous action space is a space containing an uncountable number of actions. That is, each action can be represented as a vector with one or more dimensions, and for each dimension, the action vector can take any value within the range of the dimension. The only constraint is the precision of the numerical format used by system 100.

[0035] As yet another example, when the action space is continuous, the policy output may include a regression action, i.e., a regression vector representing an action from the continuous space, and system 100 may select the regression action as action 108.

[0036] The policy neural network 120 can have any suitable architecture that allows the policy neural network 120 to map inputs, including observed images, to policy outputs.

[0037] For example, a policy neural network 120 may include an "embedded" subnetwork, a "core" subnetwork, and one or more "selected" subnetworks. A subnetwork in a neural network refers to a group of one or more neural network layers within a neural network.

[0038] When the observation is an image, the embedding subnet can be a convolutional subnet, i.e., a convolutional subnet containing one or more convolutional neural network layers configured to process the observation over time steps.

[0039] The core subnetwork may be, for example, one or more long short-term memory (LSTM) neural network layers, or a recurrent subnetwork that includes (i) the output of the embedding subnetwork and optionally (ii) data specifying any other information in the observation, such as lower-dimensional action data, previous action, last received reward, etc.

[0040] Each selection subnetwork can be configured to process the output of the core subnetwork to generate a corresponding output, i.e., a corresponding set of action scores or corresponding parameters of a probability distribution. For example, each selection subnetwork can be a multilayer perceptron (MLP) or other fully connected neural network. In some cases, data specifying other information in the observation can be provided as input to the selection subnetwork instead of the core subnetwork.

[0041] Next, the system 100 can control the robot 104 by providing the action 108 defined by the policy output 122 as the control input for the robot 104.

[0042] Generally, environment 106 is a real-world environment, and robot 104 interacts with environment 106 to achieve goals such as positioning an object of interest within the environment, moving an object of interest to a specified location within the environment, physically manipulating an object of interest within the environment in a specified manner, or navigating to a specified destination within the environment.

[0043] In these embodiments, the observation 110 may include one or more of images, object position data, and sensor data, such as image sensors, distance sensors, or position sensors, or sensor data from actuators, in order to capture the observation when the robot interacts with the environment.

[0044] For example, the observations may include data characterizing the robot's current state (e.g., joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and one or more of the global or relative orientation of the item held by the robot).

[0045] Other examples include observations that similarly include one or more of the position, linear velocity or angular velocity, force, torque or acceleration, and global or relative attitude of one or more parts of the robot. Observations may be defined in one, two, or three dimensions, and may be absolute and / or relative observations.

[0046] Observations may also include data acquired by one of several sensor devices that sense the real-world environment (e.g., sensed electronic signals such as motor current or temperature signals, and / or image data or video data from, for example, a camera or LiDAR sensor, for example, data from a robot's sensors or data from sensors positioned separately from the robot in the environment).

[0047] The observations may also include data that characterizes the task (e.g., data specifying the robot's target state, such as target joint position, velocity, force or torque, or higher-level states such as the robot's coordinates or velocity, data specifying the target state or position of other objects in the environment or both, data specifying the position of the target in the environment, etc.).

[0048] An action can be a control input for controlling the robot (for example, the torque of the robot's joints, or a more advanced control command).

[0049] In other words, control inputs may include, for example, data on the position, velocity, or force / torque / acceleration of one or more joints or other parts of a robot. Control inputs may also include, additionally or alternatively, electronic control data such as motor control data, or more generally, data for controlling one or more electronic devices in the environment whose control affects the observed state of the environment.

[0050] Optionally, in any of the embodiments described above, observations at any given time step may include data from previous time steps that may be useful in characterizing the environment (e.g., actions performed in the previous time step, rewards received in the previous time step, etc.).

[0051] Before using the policy neural network 120 to control the robot 104, the training system 190 trains the policy neural network 120.

[0052] More specifically, system 190 trains the policy neural network 120 in a simulation. That is, system 190 trains the network 120 in a computer simulation of environment 106.

[0053] For example, system 190 can train a policy neural network 120 in a simulation and then use the trained policy neural network 120 to control a robot 104 in environment 106 without any further training, thus performing a zero-shot transfer from simulation to the real world (sim2real).

[0054] As another example, system 190 can perform a few-shot transfer from simulation to the real world, by training a policy neural network 120 in a simulation and then further training the policy neural network 120 while controlling a robot 104 in the environment 106.

[0055] In particular, when training the policy neural network 120 in a simulation, the training system 190 uses a model 192 of the robot 104 and a simulator 194 that can accurately simulate the interaction between the robot 104 and the environment 106.

[0056] Model 192 of robot 104 is data that specifies the robot's configuration, such as the robot's sensors, and the robot's physical and visual characteristics, and is used by simulator 194 to model the robot's physical phenomena.

[0057] Simulator 194 can be any suitable simulator software capable of modeling the physical phenomena of a robot and any other dynamic objects in an environment. An example of such a simulator is the MuJoCo physics simulator, which models the dynamics of a robot and its environment and considers collisions between objects. Generally, Simulator 194 maintains a simulation state that defines the current state (e.g., position, velocity, acceleration, etc.) of any dynamic object in its environment, and maintains data specifying the physical and visual properties of the dynamic object. By modeling the physical phenomena of the environment, Simulator 194 can update the simulator state to reflect changes in the environment (e.g., actions performed by the robot, motion of other objects, collisions between objects or with static objects, etc.). Simulator 194 also includes a renderer that can render images of objects, taking into account their current state and visual properties.

[0058] Furthermore, the training system 190 uses a scene synthesis machine learning model 196 as part of its training.

[0059] The scene synthesis machine learning model 196 is a model configured to receive a scene input, such as a neural network, which includes the camera viewpoint, and to generate a composite image of the scene from the camera viewpoint as its output.

[0060] Generally, during training, the training system 190 can use the model 196 to generate a composite image of the environment 106 for use in generating observations provided as input to the policy neural network 120, while using the simulator 194 to simulate the physical phenomena of the environment (e.g., the motion of objects in the environment, and the effects of actions selected by the policy neural network 120 on the robot and the environment).

[0061] This training will be explained in more detail below, with reference to Figures 2-4.

[0062] Figure 2 is a flowchart of an exemplary process 200 for training a policy neural network. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system (e.g., action selection system 100 in Figure 1) is appropriately programmed according to this specification and is capable of performing process 200.

[0063] The system acquires multiple images of a scene in the real-world environment in which the robot interacts (step 202).

[0064] The system also acquires corresponding camera data for each image, including the viewpoint of the camera that captured the image.

[0065] In other words, camera data includes camera pose information for each image, and more specifically, defines the internal and external characteristics of the camera used to capture each image.

[0066] For example, the system can extract these images and their corresponding camera data from video footage of a scene captured by a camera.

[0067] For example, the system can extract images from a scene by acquiring video footage of a scene in a real-world environment and then selecting video frames from the footage. For example, the system can divide the video into partitions, for example, equal partitions, and then select one or more images from each partition. For example, the system can select one or more of the least blurred images from each partition by selecting the least blurred image from each partition, for example, based on the frame distribution of the Laplacian.

[0068] Next, the system can extract camera data from the selected image.

[0069] For example, the system can extract camera data from the metadata of images available to the system.

[0070] As another example, the system can extract camera data by applying Structure from Motion (SfM) techniques to the image. One example of an SfM package that the system may use to process images within a video and extract camera data is the COLMAP package.

[0071] The camera used to capture images of the scene can generally be any suitable camera device and does not need to be the same camera or have the same characteristics as the camera used by the robot to capture the observation images. Therefore, the system can utilize video captured by a general-purpose camera, such as the camera of a typical mobile device, to extract images and camera data.

[0072] Next, the system trains a scene synthesis machine learning model using multiple images and corresponding camera data (step 204).

[0073] As described above, a scene synthesis machine learning model is a model configured to receive a scene input, such as a neural network, which includes the camera's viewpoint, and to generate a synthesized image of the scene from the camera's viewpoint as its output.

[0074] In general, a scene synthesis model can be any suitable model that, after training, can generate a composite image of a scene in a real-world environment from any viewpoint.

[0075] For example, a scene synthesis model can be a neural radiance field (NeRF) model.

[0076] The NeRF model represents radiance in a neural field that reproduces the geometric structure and appearance of a scene, and can reconstruct a set of input images using backpropagation. In particular, the NeRF model can predict radiance and occupancy in space, i.e., the underlying spatial geometry, as part of rendering an image of the scene from a given viewpoint.

[0077] In particular, the NeRF model takes a camera pose as input and generates a composite image as output that appears as if the image was taken by a camera with the input camera pose. In some cases, the NeRF model also takes camera internal characteristics as input and generates a composite image as output that appears as if the image was taken by a camera with the input camera pose and the input camera internal characteristics.

[0078] The system can train any of the various NeRF models that use any of the various NeRF variants. Examples of such models and loss functions for training these models include: JT Barron, B. Mildenhall, D. Verbin, PPSrinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” CoRR, vol.abs / 2111.12077, 2021.; T. Uller, A. Evans, C. Schieed, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans., Graph., vol.41, pp.102:1-102:15, July 2022; and D. Verbin, P. Hedman, B. Mildenhall, TEZickler, JT Barron, and PPSrinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance Examples include "fields," CoRR, vol.abs / 2112.03907, 2021, and "Mip-nerf: A multiscale representation for antialiasing neural radiance fields," CoRR, vol.abs / 2103.13415, 2021, by JT Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and PPSrinivasan.

[0079] As a specific example, the system may utilize one or more of the following variations to improve reconstruction quality and the reconstructed geometry, and to reduce rendering time.

[0080] For example, to avoid artifacts during rendering at low resolutions, the system can sample the mean of a volume on a normal distribution.

[0081] As another example, the system can use a spatial squashing formulation to support a large capture region and a separate "proposed" network, as well as a "distortion" loss to facilitate a compact representation.

[0082] As another example, to improve the reconstructed geometry, the system can optimize separate specular and diffuse colors.

[0083] As another example, to reduce latency, the system can implement a multiscale spatial hash grid approach, which would allow, for example, a V100GPU to render one frame every 6ms.

[0084] As another example, the system can use any suitable architecture for the multilayer perceptrons (MLPs) that make up the NeRF model. For example, the system can use an architecture that adds layer normalization before the last MLP layer and use swish activation instead of ReLU activation, for example, as in the original NeRF model.

[0085] As another example, the system can be adapted to sample radiance volumes on a distribution. To achieve this, the system can blur training samples with a Gaussian blur using random variance σblur∈[σmin,σmax] and provide Σ=Σsample*(1+(σblur-σmin)) as an additional input to the final MLP of the NeRF model. This extension allows the network to interpolate samples in scale space, significantly improving reconstruction at lower resolutions. For example, using this extension, the mean PSNR can be ~31.5 vs ~35.4 in an example of a retained image set.

[0086] Therefore, the system trains a model that can generate a composite image of a scene in a real-world environment.

[0087] Next, the system generates training data for training a policy neural network, using at least the synthetic images generated by the scene synthesis machine learning model (step 206).

[0088] In other words, while collecting data during training data generation, the system generates observations of the scene in a simulation of the environment interacted with by the robot model from the synthetic images generated by the scene synthesis machine learning model. The system can then use the output generated based on the observations to control the robot model in the simulation. That is, the system uses the trained scene synthesis model to generate images of the simulated state of the environment, and these images are then provided as input to a policy neural network.

[0089] Figure 3 shows an example 300 of generating a combined simulation 310 using a simulator and a scene synthesis machine learning model for generating training data to train a policy neural network.

[0090] As shown in Figure 3, the system receives input video 302 of a scene in a real-world environment. In the example in Figure 3, the video is generated using the camera of a mobile device. More generally, however, the video can be generated using any suitable camera device capable of capturing video of the scene from multiple viewpoints.

[0091] The system applies COLMAP or a different SfM package to extract a set of images from video 302 that have corresponding camera data, including camera pose 304. The system then trains a scene synthesis machine learning model, such as the NeRF model 306, which generates a new composite image of the scene from an arbitrary viewpoint / camera pose.

[0092] Generally, as described above, a scene synthesis machine learning model receives a new camera viewpoint and camera internal characteristics as input, and generates a composite image of the scene captured from the new viewpoint and by a camera with the input camera internal characteristics as output.

[0093] When rendering a given image of the scene in the simulation (308), the system uses acquired camera intrinsic properties, such as focal length, distortion parameters, or both, which are generated as a result of calibrating the robot's camera. Thus, the image rendered in the simulation looks as if it were taken by the robot's camera in the real-world environment. In other words, the system uses the rendered image generated using the robot's camera intrinsic properties to model the visual representation of the environment.

[0094] The NeRF model 306 learns a function to predict the radiance and occupancy in space, i.e., the geometry of the underlying scene.

[0095] As part of generating the combined simulation 310, the system generates a mesh of the scene using a trained scene synthesis model. The system can then provide the mesh to the simulator for use in modeling collisions when updating the simulation state as part of the combined simulation 310.

[0096] In particular, the system can generate mesh 309 by first generating an initial mesh from the trained composite model to a first reference frame, and then mapping the vertices of the initial mesh from the first reference frame of the scene composite model to the simulator's world reference frame.

[0097] More specifically, the system voxels the predicted occupancy generated by the trained scene synthesis model and uses this predicted occupancy to compute the initial mesh, for example, via the Marching Cubes algorithm. As will be described in more detail below, the camera pose obtained from COLMAP, and therefore the vertices of the collision mesh, are represented in an arbitrary reference frame (including arbitrary scales). As a result, the system estimates the stiffness transformation and scale between this reference frame and the simulator's world frame. For example, the system can compute the estimate by solving a least-squares optimization that constrains the normal vectors on the dominant floor plane of the mesh aligned to the z-axis in the simulator. Next, the system generates mesh 309 by generating the mesh 309 by computing the relative scale between NeRF and the world by rotating the initial mesh around the z-axis to the desired alignment with the simulator's world frame and comparing the size of the object in the initial mesh with the size of the object in the real world.

[0098] The system can also replace the vertices of the mesh floor (which may have artifacts due to the lack of textures) with flat surfaces. Optionally, to speed up collision calculations, the system trims Mesh309 to the extent required for the simulation.

[0099] Next, the system can use mesh 309 for collisions within the combined simulation 310.

[0100] Next, the system can combine the generated mesh with the robot model and any other dynamic objects in the physics simulator to generate a combined simulation 310. That is, while running episodes of the task in the simulation to generate training data, the system generates a combined scene by using the physics simulator to model the state of the robot model and any other dynamic objects, while (i) modeling the static aspects of the scene using the images synthesized using a scene synthesis neural network, and (ii) modeling collisions using mesh 309.

[0101] This will be explained in more detail below with reference to Figures 4 and 5.

[0102] Figure 4 is a flowchart of an exemplary process 400 for generating training data for training a policy neural network. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, an action selection system (e.g., action selection system 100 in Figure 1) is appropriately programmed according to this specification and can perform process 400.

[0103] In particular, as part of generating training data, the system uses a policy neural network at each of multiple time steps to control a model of a robot in a simulated environment, for example, to attempt to perform episodes of a task within the simulation.

[0104] At each time step, the system obtains an input camera viewpoint from the simulator based on the position of the robot camera at the time step, within the simulated state of the real-world environment at the time step (step 402).

[0105] In other words, as described above, the simulator maintains a simulation state that is updated over time. At any given time step, the simulator state identifies the current state of the robot, including, for example, the current camera view of the robot's camera. The system can use this current camera view as the input camera view for the time step.

[0106] In some embodiments, the simulator operates on a different reference frame than the scene synthesis model, for example, trained on an input that specifies the camera viewpoint on a different reference frame than the one used by the simulator. For example, the scene synthesis model can be configured to receive the camera viewpoint on a first reference frame, for example, an arbitrary reference frame generated by SfM used by the system, and to estimate the camera data of images in the training data while the simulator operates on a world reference frame.

[0107] In these embodiments, as part of obtaining an input camera viewpoint from the simulator based on the position of the robot camera at a time step in a simulation of a real-world environment, the system receives an initial camera viewpoint from the simulator in a world reference frame, generates an input camera viewpoint by mapping the initial camera viewpoint from the world reference frame to a first reference frame, and generates a camera viewpoint in the first reference frame as described above, for example by applying stiffness transformations and scaling to the initial camera viewpoint.

[0108] The system uses a scene compositing model to generate a composite image of the scene from the input camera viewpoint (step 404).

[0109] In other words, the system processes an input specifying a camera viewpoint using a scene synthesis model and generates a composite image of the scene from the input camera viewpoint as output. As described above, in some cases, the input to the scene synthesis model also includes data specifying the internal characteristics of the camera that captures the image. In these cases, the system provides, as part of the input, data specifying the internal characteristics of the robot's camera in order to maximize the alignment between the image processed during the simulation and the image processed in the real world after training.

[0110] In other words, if the camera capturing multiple images used to train the scene synthesis model is different from a robotic camera, and the camera data used to train the scene synthesis model includes camera parameters that specify the internal properties of the camera capturing multiple images, and the scene input further includes input camera parameters that specify the internal properties of the input camera to which the synthesized image produced by scene synthesis machine learning should match, then the system generates each observation by providing a scene input that includes input camera parameters that specify the internal properties of a robotic camera instead of the internal properties of a camera capturing multiple images.

[0111] The system generates input images for time steps from at least the composite image of the scene (step 406).

[0112] Generally, a composite image of a scene does not include any robots or other moving objects present in the scene at that time.

[0113] Therefore, to take this into account, the system obtains renderings of one or more dynamic objects in the environment at each time step from the simulator and generates an input image for each time step by combining the composite image of the scene with each rendering.

[0114] In other words, the simulator renders the dynamic objects in the scene (including robots) based on the respective states of these objects and the respective visual characteristics of each object, as maintained by the simulator.

[0115] The process of generating the input image will be explained in more detail below, with reference to Figure 5.

[0116] The system processes observations, including the input image, using a policy neural network to generate a policy output (step 408), and uses the policy output to select an action, for example, by selecting an action as described above, or by applying a search policy to the policy output to select an action (step 410).

[0117] The system updates the simulation state by providing the simulator with selected actions to be used when controlling the robot model (step 412). That is, the system provides the simulator with selected actions, and the simulator simulates the physical phenomena of the environment using the selected actions to update the simulation state, for example, to update the state of the robot or any other dynamic object in the environment.

[0118] Next, the system can generate a training example for each time step, which includes observations (including the input image) at that time step and the selected action at that time step.

[0119] Generally, the system also receives a reward from the simulator for each time step, and then includes each reward in the training examples for that time step.

[0120] In some embodiments, the system can regularize the rewards received before using the training rewards, for example, to improve the transfer of the learned policy neural network from simulation to the real world. As an example, the system can use the following reward components as regularization: 1. A steady penalty that prompts the robot to rotate slowly whenever the robot's yaw angular velocity is greater than π rad s-1; 2. L2 regularization of joint angles toward the default standing posture; 3. When the robot is humanoid or quadruped, when the average forward leg velocity of the robot is 0.3 ms. -1 Walking rewards encourage the model to learn walking patterns that are better transmitted to the policy neural network, leading to better exploration for faster learning.

[0121] The system can train a policy neural network through reinforcement learning using any suitable reinforcement learning technique, such as an off-policy reinforcement learning technique using an actor-critic framework. Examples of such techniques include policy gradient techniques, Q-learning techniques, and policy improvement techniques. Specifically, the reinforcement learning can be a DMPO or MPO technique.

[0122] In some embodiments, the system can use an asymmetric actor-critic setup for training in a simulation, and a separate neural network not evaluated by the robot, i.e., a critic not used after training, receives privileged information. As a specific example, the critic may share the same network structure as the actor but have an image encoder replaced with the ground truth state of the simulation (robot / object pose and velocity).

[0123] For example, the system can store the generated training examples in playback memory. The system can then sample batches of training examples from playback memory and use reinforcement learning techniques to train a policy neural network on the sampled batches of training examples.

[0124] In some embodiments, the system can utilize data augmentation during training to improve the likelihood that the policy neural network will successfully translate into the real world. For example, while NeRF models can significantly reduce the sim2real gap with rendering realistic scenes, the system can apply image augmentation to more reliably modulate image intensity characteristics such as luminance or gain. For instance, during training of images provided as input to the neural network, the system can perform one or more of the following: randomize brightness, randomize saturation, randomize hue, randomize contrast, or apply random transformations to the image.

[0125] Furthermore, in some embodiments, the system can use domain randomization during training to improve the likelihood of successful transfers. Here are some examples of such randomization: As an example, the system can apply random pushes to the robot during training. As another example, the system can apply a constant delay per episode, uniformly sampled from a specified range, for example, 10ms to 50ms, and optionally jitter, to all simulated sensor data to reflect the robot's various latencies. As yet another example, at the beginning of each episode, the system can randomize the location of the IMU on the robot's torso by assigning random masses to random locations on the torso. As yet another example, in tasks involving a ball or other dynamic object, the system can further randomize the mass and radius of the dynamic object (e.g., a ball) at the beginning of each episode.

[0126] By repeatedly running process 400 to collect training data and repeatedly training on training examples sampled from replay memory, the system trains a policy neural network to effectively control a robot model in the simulation.

[0127] After training, the system can use a policy neural network to control a robot in a real-world environment.

[0128] Figure 5 shows an example of generating input images during training.

[0129] As shown in Figure 5, the simulator maintains a physical simulation state 502. At any given time, the system uses state 502 to generate a static scene render 504 using a scene synthesis model, while using the simulator to generate a dynamic object render 506 showing the current view of dynamic objects in the environment.

[0130] Next, the system generates a render 508 that combines the static scene render 504 and the dynamic object render 506. For example, the system can overlay the rendering of the dynamic object onto the static scene render 504, or combine the two renders in a different way.

[0131] The simulator also uses a static scene mesh 510 (generated using a scene synthesis model), a dynamic object mesh 512, and non-geometric properties (e.g., friction) 514 to generate inputs to the collision engine 516 and physics engine 518 that update the simulation state 502 based on, for example, the motion of the dynamic object and the action selected for the robot by the system.

[0132] As described above, the system can train a policy neural network to perform any of a variety of tasks. Here are some examples of such tasks:

[0133] For example, the task could be a navigation and obstacle avoidance task. For instance, the task could be a point-to-point visual navigation task in which the robot must arrive at one or more targets (specified as (x,y) coordinates in the NeRF reference frame) while avoiding various obstacles in the environment (e.g., objects such as large plants, chairs, and walls).

[0134] During training, the system can automatically calculate the free areas of the scene using the NeRF mesh, and during simulation, the system can randomly initialize the robot to a location and orientation within these free areas and select targets for different parts of space where the robot needs to arrive.

[0135] For example, in this task, the reward for training may include one or more of the regularization periods described above, along with two task-specific periods: 1. A small bonus for reaching the target location; 2. A walking reward similar to the one used for regularization, but instead a reward that encourages moving in the direction of the target at a specific speed, e.g., 0.3ms-1. The episode ends if any part of the robot's body other than its feet touches the mesh in the scene. The episode is successful if the robot reaches within 25cm of the target without falling or colliding with any obstacles.

[0136] Other examples of tasks include ball-pushing tasks, or more generally, object-moving tasks where a robot needs to move a specified object to a specified location in its environment. One example of such a task is where a robot needs to move a basketball to a corner of its workspace. The system can model the basketball as a simple orange ball. During training the simulation, each episode begins with the robot and a randomly placed ball. In some of all episodes, for example, half of them, the system initializes the ball directly in front of the robot to speed up learning.

[0137] As a reward, the system can use one or more of the above regularization periods and two task-specific periods: 1. A reward for minimizing the distance between the ball and the target area; 2. A reward for minimizing the distance between the robot and the ball if the ball is not moving towards the target.

[0138] Many other tasks are possible, depending on the reward received.

[0139] This specification uses the term “configured” in relation to systems and computer program components. A system of one or more computers being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the operating system to perform that operation or action. A system of one or more computer programs being configured to perform a particular operation or action means that the program contains instructions that cause the data processing device to perform that operation or action when executed by the device.

[0140] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, or computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a tangible, non-temporary storage medium, which is executed by or controls the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively, or in addition, program instructions can be encoded in artificially generated transmitted signals, such as machine-generated electrical, optical, or electromagnetic signals generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0141] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for data processing, including, for example, programmable processors, computers, or multiple processors or multiple computers. A device may be, or further include, a special-purpose logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). Optionally, in addition to hardware, a device may include code that generates an execution environment for computer programs (e.g., code that constitutes processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof).

[0142] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file containing one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file containing one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or it may be deployed to run on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0143] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0144] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits (e.g., FPGAs or ASICs) or by a combination of special-purpose logic circuits and one or more programmed computers.

[0145] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or integrated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to receive data from them, transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be integrated into other devices, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, a Universal Serial Bus (USB) flash drive, etc.

[0146] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROMs and DVD-ROM disks, and all forms of non-volatile memory, media, and memory devices.

[0147] To interact with a user, embodiments of the subject matter described herein can be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal screen) monitor, and a keyboard and pointing device, such as a mouse or trackball, on which the user can input to the computer. Other types of devices can also be used to interact with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending a document to a device used by the user and receiving a document from the device used by the user (for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser). The computer may also interact with the user by sending a text message or other form of message to a personal device (for example, a smartphone running a messaging application) and then receiving a response message from the user.

[0148] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for processing machine learning training or machine learning production, i.e., inference, and the general and numerical computation portions of the workload.

[0149] Machine learning models can be implemented and deployed using machine learning frameworks such as the TensorFlow framework or the Jax framework.

[0150] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component such as a data server, or a middleware component such as an application server, or a frontend component such as a client computer having a graphical user interface, a web browser, or an application through which a user can interact with the implementation of the subject matter described herein, or in any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communications of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0151] A computing system can include clients and servers. Clients and servers are generally geographically distant from each other and typically communicate through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interaction, can be received by the server from the device.

[0152] This specification includes details of many specific embodiments, but these should not be interpreted as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a subcombination or a variation of a subcombination.

[0153] Similarly, while the drawings depict operations and the claims describe them in a specific order, it should not be understood that, in order to obtain the desired results, such operations must be performed in a specific or sequential order, or that all indicated operations must be performed. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0154] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield the desired results. As an example, in the process shown in the accompanying diagram, it is not necessary to follow the specific order or sequence shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0155] The nature of this disclosure may be as described in the following clauses. Article 1. A method performed by one or more computers, This involves acquiring multiple images of a scene in a real-world environment in which the robot interacts, and for each image, corresponding camera data including the viewpoint of the camera that captured the image. The process involves training a scene synthesis machine learning model using multiple images and corresponding camera data, wherein the scene synthesis machine learning model is configured to receive a scene input including the camera viewpoint and to generate a composite image of the scene from the camera viewpoint as its output. The method involves generating training data for training a policy neural network for use in controlling a robot in a real-world environment to perform one or more tasks, using at least a synthetic image generated by a scene synthesis machine learning model, wherein the policy neural network is configured to receive a policy input which includes observations characterizing the current state of the environment, and to produce a policy output which defines an action to be performed by the robot in response to the observations, and the observations include an image of the environment captured by the robot's robotic camera, and generating training data is the process. A method comprising generating, from a scene synthesis machine learning model, an observation of a scene in a simulation of an environment interacted with by a robot model. Article 2. The method according to Clause 1, further comprising training a policy neural network with training data. Article 3. The method according to Clause 2, further comprising controlling the agent in a real-world environment using a policy neural network after training. Article 4. Acquiring multiple images is To acquire video footage of a scene in a real-world environment, and to obtain it as multiple images, A method of any of the preceding clauses, including selecting multiple video frames from a video. Article 5. Using Structure From Motion (SfM), for each of the multiple images The method according to Clause 4, further including determining camera data. Article 6. Generating training data for training a policy neural network is Controlling a robot model in an environment simulation using a policy neural network at each of multiple time steps, From the simulator, the input camera viewpoint is obtained based on the position of the robot camera at each time step, within the simulated state of the real-world environment at each time step. Using a scene compositing model, generate a composite image of the scene from the input camera viewpoint. At a minimum, generate input images for time steps from a composite image of the scene. To generate policy output, a policy neural network is used to process observations, including the input image. Selecting an action using policy output, and Controlling the robot model includes providing the simulator with selected actions to use when controlling it, in order to update the simulation state. A method of any of the preceding clauses, comprising generating a training example for each time step, which includes the observation of a time step and a selected action for that time step. Article 7. At the very least, generating time-step input images from a composite image of the scene is possible. From the simulator, obtain renderings of one or more dynamic objects in the environment at each time step, The method according to Clause 6, which includes generating input images for time steps by combining a composite image of a scene with its respective renderings. Article 8. The scene synthesis model is configured to receive the camera viewpoint in a first reference frame, and the simulator operates on a world reference frame. From the simulator, the input camera viewpoint is obtained based on the position of the robot camera at time steps within the simulation of the real-world environment. The simulator receives the initial camera view in a world standard frame, The method of Clause 6 or Clause 7, which includes generating an input camera viewpoint by mapping an initial camera viewpoint from a world standard frame to a first standard frame. Article 9. The training example further includes receiving the respective reward from the simulator for each of one or more tasks at each time step, and the training example includes the method according to any one of clauses 6 to 8, including the respective reward. Article 10. Using a trained scene synthesis model, generate a mesh for the scene, The method of any of the preceding clauses, further comprising providing a mesh to the simulator for use in modeling collisions when updating the simulation state. Article 11. If subject to Clause 8, generating a mesh is: The process involves generating an initial mesh in the first reference frame, The method according to clause 10, comprising generating a mesh by mapping the vertices of an initial mesh from a first reference frame to a simulator's world reference frame. Article 12. Observations are as described in any of the preceding clauses, further including data from the robot's gyroscope, robot's accelerometer, or both. Article 13. If subject to Clause 2, training a policy neural network is: A method of any of the preceding clauses, including training a policy neural network through reinforcement learning with domain randomization. Article 14. The scene synthesis model is a neural radiance field (NeRF) model, as described in any of the preceding clauses. Article 15. Unlike a robot camera, a camera that captures multiple images includes camera parameters that specify the internal characteristics of the camera that captured multiple images, and the scene input includes input camera parameters that specify the internal characteristics of the input camera that the composite image generated by scene synthesis machine learning should match, and generating observations of the scene in a simulation of the environment in which the robot model interacts from the composite image generated by the scene synthesis machine learning model is possible. The method of any of the preceding clauses, comprising generating each observation by providing a scene input that includes an input camera parameter specifying the internal characteristics of a robotic camera instead of the internal characteristics of a camera that has captured multiple images. Article 16. A system comprising one or more computers and one or more storage devices communicatively coupled to one or more computers, wherein one or more storage devices, when executed by one or more computers, store instructions causing one or more computers to perform the actions of any one of the methods described in any one of the clauses 1 to 15. Article 17. One or more non-temporary computer storage media that, when executed by one or more computers, store instructions causing one or more computers to perform the actions of any one of the methods described in any one of the clauses 1 to 15.

Claims

1. A method performed by one or more computers, The process involves acquiring multiple images of a scene in a real-world environment in which the robot interacts, and for each image, corresponding camera data including the viewpoint of the camera that captured the image. Training a scene synthesis machine learning model using the aforementioned multiple images and the corresponding camera data, wherein the scene synthesis machine learning model is configured to receive a scene input including a camera viewpoint and to generate a composite image of the scene from the camera viewpoint as an output. The process involves generating training data for training a policy neural network for use in controlling the robot in the real-world environment to perform one or more tasks, using at least the synthetic images generated by the scene synthesis machine learning model, wherein the policy neural network is configured to receive a policy input including observations characterizing the current state of the environment, and to produce as an output policy output defining actions to be performed by the robot in response to the observations, wherein the observations include images of the environment captured by the robot's robotic camera, and generating the training data is the process described above. This includes generating an observation of the scene in a simulation of the environment in which the robot model interacts, from a composite image generated by the scene synthesis machine learning model, Methods that include...

2. The method according to claim 1, further comprising training the policy neural network with the training data.

3. The method according to claim 2, further comprising using the policy neural network to control the robot in the real-world environment after the training.

4. Acquiring the aforementioned multiple images means To acquire video footage of the scene in the aforementioned real-world environment, Selecting multiple video frames from the video as the aforementioned multiple images, The method according to claim 1, including the method described in claim 1.

5. The method according to claim 4, further comprising determining the camera data for each of the plurality of images using Structure from Motion (SfM).

6. Generating the training data for training the policy neural network is Controlling the model of the robot in the simulation of the environment using the policy neural network in each of a plurality of time steps, From the simulator, an input camera viewpoint is obtained based on the position of the robot camera at the time step, within the simulated state of the real-world environment at the time step. Using the aforementioned scene synthesis machine learning model, generate a composite image of the scene from the input camera viewpoint. At least generating the input image for the time step from the composite image of the scene, To generate a policy output, the policy neural network is used to process the observations, including the input image. Select an action using the aforementioned policy output, and Controlling, which includes providing the simulator with the selected actions to be used when controlling the model of the robot in order to update the state of the simulation, To generate training examples for each of the time steps, including the observations of the time step and the selected actions of the time step, The method according to claim 1, including the method described in claim 1.

7. At least generating the input image for the time step from the composite image of the scene is From the simulator, obtain renderings of one or more dynamic objects in the environment at the time step, The input image for the time step is generated by combining the composite image of the scene and the respective renderings. The method according to claim 6, including the method described in claim 6.

8. The scene synthesis machine learning model is configured to receive a camera viewpoint in a first reference frame, the simulator operates in a world reference frame, and the simulator obtains an input camera viewpoint from the simulator based on the position of the robot camera at the time step in the simulation of the real world environment. The simulator receives the initial camera viewpoint in the world standard frame, The input camera viewpoint is generated by mapping the initial camera viewpoint from the world standard frame to the first standard frame. The method according to claim 6, including the method described in claim 6.

9. The method according to claim 6, further comprising receiving a reward from the simulator for each of the one or more tasks at each time step, wherein the training example includes the respective rewards.

10. Using the aforementioned trained scene synthesis machine learning model, generate a mesh of the scene, To provide the simulator with the mesh for use in modeling collisions when updating the state of the simulation. The method according to claim 8, further comprising:

11. Generating the aforementioned mesh The process involves generating an initial mesh using the first reference frame described above, The mesh is generated by mapping the vertices of the initial mesh from the first reference frame to the world reference frame of the simulator. The method according to claim 10, including the method described in claim 10.

12. The method according to claim 1, wherein the observation further includes data from the robot's gyroscope, the robot's accelerometer, or both.

13. Training the aforementioned policy neural network is Training the policy neural network through reinforcement learning using domain randomization. The method according to claim 2, including the method described in claim 2.

14. The method according to claim 1, wherein the scene synthesis machine learning model is a neural radiance field (NeRF) model.

15. Unlike the robot camera, the camera that captures the multiple images further includes camera parameters that specify the internal characteristics of the camera that captures the multiple images, the scene input further includes input camera parameters that specify the internal characteristics of the input camera that the composite image generated by the scene synthesis machine learning model should match, and generating an observation of the scene in a simulation of the environment interacting with the robot model from the composite image generated by the scene synthesis machine learning model is: The method according to claim 1, comprising generating each of the observations by providing a scene input that includes an input camera parameter specifying the internal characteristics of the robot camera instead of the internal characteristics of the camera that captured the plurality of images.

16. One or more computers, A system comprising one or more storage devices communicably coupled to one or more computers, wherein, when executed by one or more computers, the one or more storage devices store instructions causing one or more computers to perform an operation according to any one of claims 1 to 15.

17. One or more non-temporary computer storage media, which, when executed by one or more computers, store instructions causing one or more computers to perform an operation according to any one of claims 1 to 15.

Images