Machine learning method and apparatus for predicting the movement of an object in a robot's environment based on images capturing the object and based on parameters related to the robot's future movement in the environment

Abstract

To provide deep machine learning methods and apparatus allowing prediction of motion(s) (if any) that will occur to object(s) in an environment of a robot in response to a particular movement of the robot in the environment.SOLUTION: The invention is directed to training a deep neural network model to predict at least one transformation (if any) of an image of a robot's environment, which should occur as a result of implementing at least a portion of a particular movement of the robot in the environment. The training of the deep neural network model allows prediction of the transformation based on input that includes the image and robot movement parameters that define the portion of the particular movement.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] Many robots are programmed to utilize one or more end effectors to manipulate one or more objects. For example, a robot may utilize an end effector to apply forces to an object and cause motion of the object. For example, a robot may utilize a grasping end effector or other end effectors to displace an object without necessarily grasping the object. Also, for example, a robot may utilize a grasping end effector, such as a "collision" gripper or a "penetration" gripper (e.g., physically penetrating an object using a pin, needle, etc.), to pick up an object from a first location, move the object to a second location, and place the object at the second location. Summary of the Invention

[0002] Some embodiments herein relate generally to deep machine learning methods and apparatuses for predicting motion(s), if any, that will occur for object(s) in a robot's environment in response to a particular motion of the robot within the environment. Some embodiments are directed to training a deep neural network model to predict at least one transformation, if any, of an image of the robot's environment that would occur as a result of performing at least a portion of a particular motion of the robot within the environment. The trained deep neural network model can predict the transformation based on inputs including (1) an image and (2) robot motion parameters that define a portion of the particular motion. The predicted transformation can be utilized to transform the image to generate a predicted image of the robot's environment, the predicted image predicting the robot's environment when the portion of the particular motion occurs. In other words, the predicted image indicates a prediction of the robot's environment after the portion of the particular motion occurs, and can be utilized, for example, to predict motion(s) of object(s) in the environment that would occur as a result of the particular motion.

[0003] The predicted motion(s) may be utilized for various purposes, such as, for example, to determine whether to provide control commands to the robot's actuator(s) to achieve a particular motion. For example, the predicted motion(s) may be compared to a desired motion(s), and if the predicted motion(s) conform to the desired motion(s), the control commands may be implemented. In this manner, the outcome of a particular motion may be effectively "visualized" prior to the implementation of the particular motion, and the particular motion may be implemented if the outcome is desirable. As described herein, in various embodiments, the deep neural network model predicts images at each of multiple future time steps based on candidate motion parameters for these future time steps, thereby enabling effective visualization multiple time steps into the future.

[0004] In some implementations, a method is provided that includes generating candidate robot motion parameters that define at least a portion of a candidate motion executable within an environment of the robot by one or more components of the robot. The method further includes identifying a current image captured by a vision sensor associated with the robot. The current image captures at least a portion of the environment of the robot. The method further includes applying the current image and the candidate robot motion parameters as inputs to a neural network to be trained, and generating at least one predicted transformation of the current image based on application of the current image and the candidate robot motion parameters to the neural network to be trained. The method further includes transforming the current image based on the at least one predicted transformation to generate the at least one predicted image. The predicted image predicts the portion of the environment of the robot when at least a portion of the candidate motion is executed within the environment by the components of the robot.

[0005] The above and other implementations can optionally include one or more of the following features.

[0006] In some implementations, the method further includes determining to perform a candidate movement based on the predicted image, and providing one or more control commands to one or more actuators of the robot to perform the candidate movement.

[0007] In some implementations, the method further includes determining to perform an alternative movement instead of the candidate movement based on the predicted image, and providing one or more control commands to one or more actuators of the robot to perform the alternative movement.

[0008] In some implementations, the method further includes generating at least one synthesis mask based on application of the current image and the candidate robot motion parameters to the neural network to be trained. Transforming the current image is further based on the at least one synthesis mask. By way of example, the at least one predicted transformation can include a plurality of predicted transformations, the at least one synthesis mask can include a plurality of synthesis masks, and transforming the current image based on the at least one predicted transformation to generate a predicted image can include generating a plurality of predicted images based on the plurality of predicted transformations, and synthesizing the predicted image based on the plurality of synthesis masks to generate a predicted image.

[0009] In some implementations, the method further includes generating second candidate robot motion parameters that define at least a portion of a second candidate motion executable in the environment by one or more of the components in place of the candidate motion. In these implementations, the method may further include applying the current image and the second candidate robot motion parameters as inputs to the trained neural network, generating at least one second predicted transformation of the current image based on the application of the current image and the second candidate robot motion parameters to the trained neural network, and transforming one or more of the pixels of the current image based on the second predicted transformation to generate the at least one second predicted image. The second predicted image predicts a portion of the environment of the robot when at least a portion of the second candidate motion is executed in the environment by the components of the robot. In some versions of these implementations, the method further includes selecting either the candidate motion or the second candidate motion based on the predicted image and the second predicted image, and providing one or more control commands to one or more actuators of the robot to execute the selected one of the candidate motion and the second candidate motion.

[0010] In some embodiments, the method further includes generating continuing candidate robot motion parameters defining another portion of the candidate motion subsequent to the portion of the candidate motion, applying the predicted image and the continuing candidate robot motion parameters to the trained neural network, generating at least one continuing predicted transformation of the predicted image based on the application of the predicted image and the continuing candidate robot motion parameters to the trained neural network, and transforming the predicted image based on the continuing predicted transformation to generate the continuing predicted image. In some of these embodiments, the method further includes determining to perform the candidate motion based on the predicted image and the continuing predicted images, and providing one or more control commands to one or more actuators of the robot to perform the candidate motion.

[0011] In some embodiments, the neural network to be trained includes multiple stacked convolutional long short-term memory layers.

[0012] In some implementations, at least the predicted transformation of pixels of the current image includes parameters of one or more spatial transformers. In some of these implementations, transforming the current image includes applying the one or more spatial transformers to the current image using the parameters.

[0013] In some implementations, the predicted transformation of at least one of the pixels of the current image includes one or more normalized distributions, each corresponding to one or more of the pixels. In some of these implementations, transforming the current image includes applying the normalized distributions to the current image using a convolution operation. In some versions of these implementations, each of the normalized distributions corresponds to a corresponding one of the pixels.

[0014] In some embodiments, applying the current image and the candidate robot motion parameters as inputs to the neural network to be trained includes applying the current image as inputs to an initial layer of the neural network to be trained, and applying the candidate robot motion parameters to an additional layer of the neural network to be trained, where the additional layer is downstream of the initial layer.

[0015] In some embodiments, the candidate robot parameters include an initial robot state and an action that indicates a subsequent robot state. In some of these embodiments, the initial robot state is a current pose of the end effector and the action is a commanded pose of the end effector.

[0016] In some implementations, a method is provided that includes identifying a current image captured by a visual sensor associated with a robot. The method further includes identifying a current state of the robot and identifying candidate actions for transitioning the robot from the current state to a candidate state. The method further includes applying the current image, the current state, and the candidate actions as inputs to a neural network to be trained. The method further includes generating at least one predicted image based on application of the current image, the current state, and the candidate actions to the neural network to be trained. The method further includes determining to perform the candidate action based on the predicted image. The method further includes providing one or more control commands to one or more actuators of the robot to perform the candidate action in response to determining to perform the candidate action.

[0017] In some embodiments, a method is provided that includes identifying a plurality of training examples generated based on sensor output from a sensor associated with one or more robots during an attempt to move a plurality of objects by the robot. Each of the training examples includes a group of sequential images from a corresponding attempt to move the objects. Each of the images captures one or more corresponding objects in an environment at a corresponding instance of time as well as, for each sequential image, a state of the robot at the corresponding instance of time and an action to be applied to transition the state of the robot at the corresponding instance of time to a new state corresponding to a next sequential image of the sequential images. The method further includes training a neural network based on the training examples.

[0018] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by a processor (e.g., a central processing unit (CPU) or a graphics processing unit (GPU)) to perform a method, such as one or more of the methods described above and / or elsewhere herein. Still other embodiments may include a system of one or more computers and / or one or more robots including one or more processors operable to execute the stored instructions to perform a method, such as one or more of the methods described above and / or elsewhere herein. Still other embodiments may include a computer program including machine-readable instructions that, when executed by a computing device, cause the execution of one or more of the methods described above and / or elsewhere herein.

[0019] It should be understood that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 illustrates an example of an environment in which object movement attempts may be performed by a robot and data associated with the object movement attempts may be utilized to generate training examples and / or the training examples may be utilized to train a neural network. [Diagram 2] FIG. 2 illustrates one of the robots of FIG. 1 and an example of the movement of the robot's grasping end effector along a path. [Diagram 3] 4 is a flow diagram illustrating an example method for performing an object movement attempt and storing data associated with the object movement attempt. [Figure 4] 1 is a flow diagram illustrating an example method for generating training examples based on data related to robotic object movement attempts. [Diagram 5] 1 is a flow diagram illustrating an example of a method for training a neural network based on training examples. [Figure 6A] FIG. 1 illustrates an example neural network architecture; shows example inputs that may be provided to the neural network; shows example outputs of the neural network; and illustrates how the example outputs may be used to generate a predicted image. [Figure 6B] FIG. 1 illustrates an example neural network architecture; shows example inputs that may be provided to the neural network; shows example outputs of the neural network; and illustrates how the example outputs may be used to generate a predicted image. [Figure 7] 1 is a flow diagram illustrating an example method of utilizing a neural network that is trained to generate predicted image(s) that predict a robot's environment after robot movement has occurred and / or perform one or more actions based on the predicted images. [Figure 8] FIG. 1 illustrates a schematic diagram of an example of a robot architecture. [Figure 9] FIG. 1 is a schematic diagram illustrating an example of a computer system architecture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] One challenge of a robot or other agent learning to interact with the world is predicting how its actions will affect objects in its environment. Many methods for learning the dynamics of physical interactions require manually labeled object information. However, to scale real-world interaction learning to various scenes and objects, acquiring manually labeled object information becomes increasingly impractical. To learn about the motion of physical objects without the need for object labels, embodiments described herein use an action-conditioned motion prediction model that explicitly models pixel motion by predicting a distribution over pixel motion from previous frames. For example, embodiments can advect pixels from previous image frames and synthesize them on the new frame, rather than constructing new images from scratch. Because the model explicitly predicts motion, it is at least partially invariant to the appearance of the object, allowing the model to generalize to previously unseen objects. To explore video prediction for real-world interaction agents, a large (e.g., 50,000 or more) dataset of robot interactions with pushing motions can be utilized. Using such a dataset, a motion prediction model can be trained that allows accurate prediction of the motion of an object(s) in an image conditioned on the future actions of the robot, allowing the use of the motion prediction model to "visually reason" about different futures based on different courses of action.

[0022] Object detection, tracking, and motion prediction are fundamental problems in computer vision, and predicting the effects of physical interactions is a challenge for agents operating in the world, such as robots, autonomous cars, and drones. Some existing techniques for learning to predict the effects of physical interactions rely on large, manually labeled datasets. However, if unlabeled raw video data (consisting of sequential image frames) from interacting agents is used to learn about physical interactions, the interacting agents can autonomously gather virtually unlimited experience through their own exploration. Learning representations that can predict future videos without labels has applications in action recognition and prediction, leading to the learning of predictive models that, when conditioned on the agent's actions, can be used by the agent for planning and decision-making.

[0023] However, learning to predict physical phenomena poses numerous challenges, as real-world physical interactions tend to be complex and stochastic, and learning from raw video requires dealing with the high dimensionality of image pixels and the partial observability of object motion from the video. Previous video prediction methods have typically considered short-term predictions, small image patches, or synthetic images. Such previous methods follow the paradigm of reconstructing future frames from the model's internal state. In some implementations described herein, the motion prediction model is not required to store the appearance of objects and backgrounds. Such appearance information may instead be obtained from previous image frames (e.g., previous image frames captured by the robot's visual sensor or previous reconstructed image frames), allowing the motion prediction model to focus on motion prediction. Each of the prediction models described herein can merge appearance information from a previous frame(s) with the motion predicted by the model. As a result, each of the models may be better (relative to conventional techniques) at predicting future video sequences for multiple steps, even when they contain objects not seen during training.

[0024] To combine appearance and predicted motion, the pixel motion relative to the previous image can be generated as an output for the prediction model described herein. Applying the motion prediction model to the image frame forms the next image frame. Various motion prediction models may be utilized, three of which are described in detail herein. The first model, sometimes referred to herein as the dynamic neural advection (DNA) model, outputs a distribution over the locations in the previous image frame for each pixel in the new frame. A predicted pixel value is then calculated as the expectation value under this distribution. A variation on the DNA model, sometimes referred to herein as the convolutional dynamic neural advection (CDNA) model, outputs parameters of multiple regularized convolution kernels to be applied to the previous image frame to calculate the new pixel value. The final approach, sometimes referred to herein as the spatial transformer predictor (STP) model, outputs parameters of multiple affine transformations to be applied to the previous image frame. In the case of the DNA and STP models, each predicted transformation is intended to process a separate object. To combine the predictions into a single image, the model also predicts a synthesis mask for each of the transformations. DNA and CDNA may be simpler and / or easier to implement than STP, and the object-centric CDNA and STP models may also provide interpretable internal representations.

[0025] Various embodiments described herein present a technique for long-term prediction in real-world images by predicting pixel motion. When conditioned on actions taken by an agent (e.g., a robot), the model can learn to infer different futures from different actions (before these different actions are performed). To learn about physical interactions from videos, large datasets with complex object interactions can be used. As an example, a dataset of 50,000 robot pushing movements consisting of 1.4 million image frames with corresponding actions at each time step can be used.

[0026] To learn about object motion while remaining invariant to appearance, a class of motion prediction models can be utilized that directly use appearance information from previous frames to construct pixel motion predictions. The models calculate the next frame by first predicting object motion in pixel space and then merging these predictions via masking. Some potential motion prediction models are described below, including how to effectively merge the predicted motions of multiple objects into a single next image prediction. The motion prediction models described herein that predict the motion(s) of object(s) without attempting to reconstruct the appearance of the object(s) can be partially invariant to appearance and can be effectively generalized to previously unseen objects. Three examples of motion prediction models are now briefly described in turn.

[0027] Dynamic Neural Advection (DNA) Motion Prediction Model In the DNA motion prediction model, a distribution over positions is predicted in the previous frame for each pixel in the new frame. The predicted pixel value is calculated as the expectation under this distribution. Pixel motion is constrained to a local region under the regularizing assumption that pixels do not move long distances. This allows the dimensionality of the prediction to be kept low.

[0028] Formally, predictive motion transformation

number

number

number

number

number

[0029] Convolutional Dynamic Neural Advection (CDNA) Motion Prediction Model Under the assumption that the same mechanism can be used to predict the motion of different objects in different regions of an image, the CDNA motion prediction model can present a more object-centric approach to predicting motion. Instead of predicting a different distribution for each pixel, the model predicts multiple discrete distributions, each of which is applied to the entire image via a convolution (with constrained weights) that computes the expectation of the motion distributions of all pixels. Pixels on the same rigid object move together and can therefore share the same transformation. More formally, the motion distributions of the previous image are

number

number

number

number

number

number

number

number

[0030] Spatial Transformer Predictor (STP) Motion Estimation Model The STP motion estimation model creates multiple sets of parameters for a 2D affine image transformation and applies the transformation using a bilinear sampling kernel. More formally, the set of affine parameters

number

number

number

number

number

number

number

[0031] The DNA, CDNA, and STP motion prediction models can each be constructed and trained to focus on learning physical phenomena rather than object appearances, so that such models can generalize better to unseen objects compared to models that directly reconstruct pixels or predict differences from previous frames.

[0032] The CDNA and STP motion estimation models produce multiple object motion estimates that can be used to generate multiple transformed images (one transformed image per object motion estimate). The multiple transformed images need to be combined into a single image. To do that, the CDNA and STP models can also predict a set of masks to be applied to the transformed images. These masks indicate how much each transformed image affects each pixel. A softmax over the channels of the masks can ensure that they sum to 1. More formally, the synthesis of the predicted images

number

number

[0033] For each motion estimation model including DNA, a "background mask" can be included where the model is allowed to copy pixels directly from a previous image (e.g., an image frame captured in an initial iteration or the immediately preceding predicted image in a subsequent iteration). Additionally, to fill previously excluded regions that may not be well represented by nearby pixels, the model is allowed to generate pixels from the image and include these in the final masking step. Besides improving performance, this also makes the background mask interpretable.

[0034] To generate the motion estimates discussed above, stacked convolutional LSTMs can be used within the motion prediction model. Recursion through convolutions works for multi-step video prediction because it exploits spatial invariances of image representations since the laws of physics are largely consistent across space. As a result, models using convolutional recursion require significantly fewer parameters and / or can use these parameters more efficiently.

[0035] In an interaction setting, the agent's candidate actions and internal state (e.g., robot gripper pose) also affect the next image, and both can be integrated into the model by tiling the concatenated internal state and candidate action(s) vectors across the spatial extent of the lowest dimensional activation map. Note, however, that the agent's internal state (e.g., current robot gripper pose) is input to the network only at the first time step and must be predicted from actions at future time steps in some implementations. For example, the robot gripper pose can be predicted at future time steps based on the change in the current robot gripper pose taking into account the candidate action(s). In other words, the robot gripper pose at future time steps can be determined based on the assumption that the candidate action(s) of the previous time step(s) have been performed. The neural network can be trained using the I2 reconstruction loss. Alternative losses could complement this method.

[0036] One application of action-conditioned video prediction is vision-based robot control tagging. The next step is to use the learned model for decision-making at the task. Unsupervised learning from video can allow an agent to learn about the world on its own, without human intervention, which can be beneficial for scaling up interaction learning. To investigate action-conditioned video prediction for robotic tasks, a dataset on real-world physical object interactions can be used. For example, a dataset can be generated using 10 robot arms pushing hundreds of objects in a bin (resulting in a total of 50,000 interaction sequences across 1 million video frames). In addition to including images (e.g., RGB or RGBD images), each frame can also be annotated with the gripper pose at the frame's time step (which may be referred to herein as the "internal state" or "robot state") and an action at that time step (which may correspond to the gripper pose at a subsequent (e.g., next) time step) or a motion vector or command to reach the gripper pose at a subsequent time step.

[0037] In some implementations, at run time, an initial image(s) as well as an initial robot state and candidate action(s) may be provided as inputs to a neural network model to be trained. The initial robot state may be, for example, the current gripper pose, and the candidate action(s) may be candidate actions that each move the current gripper pose to a corresponding new gripper pose. The model may then be published with each subsequent time step that passes in the predicted image from the previous time step, the candidate actions for that time step, and the updated robot state. As explained above, the updated robot state for a given iteration may be the state that would have been assumed had the candidate action(s) of the previous time step(s) been applied (i.e., the initial robot state modified by the candidate action(s) of the previous time step(s)). In some implementations, the neural network is trained for 8 future time steps for all recursive models and may be used for up to 18 future time steps. Other amounts of future time steps may be used for training and / or use.

[0038] Some implementations of the techniques described herein may additionally or alternatively be utilized to generate models that predict future videos without considering robot states and / or actions. For example, a dataset consisting of videos of human actors performing various actions in a room may be used to train the model. These videos may optionally be subsampled (e.g., to 10 frames per second) so that there is significant movement within a reasonable time frame of the video. Since the model is no longer conditioned on actions, X (e.g., 10) video frames may be fed and the network trained to make the next X (e.g., 10) frames.

[0039] Prediction of future object motion in the context of physical interaction may be utilized within intelligent interaction systems. Action-conditioned prediction of future video frames of the kind described herein may enable an interaction agent, such as a robot, to infer different futures based on available candidate actions. Such mechanisms may be used to plan actions to achieve certain goals, anticipate possible future problems (e.g., in the context of autonomous vehicles, obstacle avoidance), and recognize interesting new phenomena in the context of exploration. As one particular example, a goal state of an object may be defined. For example, a human may utilize user interface input to define a goal state of an object. For example, a human may manipulate an object via an interface that displays an image captured by a robot, where the image includes the object and the manipulation involves changing the pose of the object. Also, for example, an internal system of the robot can define a goal state for the object in accomplishing a task directed at the object (e.g., moving the object to a new pose). Various candidate actions can be considered utilizing the motion prediction model to generate a number of predicted images. A predicted image depicting the object closest to its goal state can be determined, and a candidate action applied to generate the predicted image can be selected and applied by the robot to move the object to its goal state. In this manner, the motion prediction model can be utilized to predict the impact of various candidate actions on the robot's environment, and the candidate action that resulted in a predicted image that best follows the desired environmental impact can be selected. This allows for an initial consideration of the impact of various robot actions without actually performing these actions, and a subsequent selection of one or more of these robot actions to actually perform based on the considered impact.

[0040] Some embodiments of the techniques described herein are directed to training a neural network, such as a neural network including stacked long short-term memory (LSTM) layers, to enable the use of the trained neural network to predict transformations that will occur to an image of the robot's environment in response to specific movements of the robot within the environment. In some embodiments, the trained neural network is adapted to recognize an image generated by a visual sensor (I t ) and accepts candidate robot motion parameters (p), such as parameters defining the current robot state and / or one or more candidate actions to be performed to transition the current robot state to a different robot state. t ) to the neural network to be trained. t ) and candidate robot motion parameters (p t ) is applied to the candidate robot motion parameters (p t ) in response to the implementation of the image (I t ) via a neural network. In some of these embodiments, the predicted transformation may be used to generate at least one predicted transformation for an image (I t ) to the predicted image (PI t ) is used to convert the predicted image (PI t) and / or one or more additional predicted images generated based on additional predicted transformations of the neural network may be utilized to determine whether the candidate robot motion parameters should be implemented. For example, the predicted image(s) may be analyzed to determine predicted motion(s) of an object(s) in an environment that will occur based on the candidate robot motion parameters, and the candidate robot motion parameters may be implemented if the predicted motion(s) are desirable. Additional details of these and other implementations of this technology are provided below.

[0041] Various embodiments of training and utilizing a motion prediction neural network are described with reference to Figures 1-7. Figure 1 illustrates an example of an environment in which object movement attempts may be performed by a robot (e.g., robots 180A, 180B, and / or other robots) and data associated with the object movement attempts may be utilized to generate training examples and / or the training examples may be utilized to train a motion prediction neural network.

[0042] Examples of robots 180A and 180B are shown in FIG. 1. Robots 180A and 180B are "robot arms" that have multiple degrees of freedom to allow traversal of grasping end effectors 182A and 182B along any of multiple potential paths through the motion of the robot to position grasping end effectors 182A and 182B in desired poses. For example, referring to FIG. 2, an example of robot 180A is shown traversing its end effector along path 201. FIG. 2 includes phantom and non-phantom images of robot 180A showing two different poses of a set of poses assumed by robot 180A and its end effector as it traverses along path 201. Referring again to FIG. 1, robots 180A and 180B further control two opposing "jaws" of their corresponding grasping end effectors 182A and 182B, respectively, to actuate the jaws between at least an open position and a closed position (and / or optionally multiple "partially closed" positions).

[0043] Examples of visual sensors 184A and 184B are also shown in FIG. 1. In FIG. 1, visual sensor 184A is mounted in a fixed orientation relative to the base of robot 180A or other stationary reference point. Visual sensor 184B is also mounted in a fixed orientation relative to the base of robot 180B or other stationary reference point. As shown in FIG. 1, the orientation of visual sensor 184A relative to robot 180A is different from the orientation of visual sensor 184B relative to robot 180B. In some implementations, such different orientations may be beneficial to enable the generation of different training examples that may be utilized to train neural networks that are robust to and / or independent of camera calibration. Visual sensors 184A and 184B are sensors capable of generating images related to the shape, color, depth, and / or other characteristics of objects within the line of sight of the sensors. Visual sensors 184A and 184B may be, for example, monographic cameras, stereographic cameras, and / or 3D laser scanners. A 3D laser scanner includes one or more lasers that emit light and one or more sensors that collect data related to the reflection of the emitted light. The 3D laser scanner can be, for example, a time-of-flight 3D laser scanner or a triangulation-based 3D laser scanner, and can include a position sensitive detector (PSD) or other optical position sensor.

[0044] The visual sensor 184A has a view of at least a portion of the workspace of the robot 180A, such as a portion of the workspace that includes examples of objects 191A. The surfaces on which the objects 191A rest are not shown in FIG. 1, but these objects can rest on desks, bins, and / or other surfaces. The objects 191A include a spatula, a stapler, and a pencil. In other implementations, more objects, fewer objects, additional objects, and / or alternative objects can be provided during all or some of the object movement attempts of the robot 180A described herein. The visual sensor 184B has a view of at least a portion of the workspace of the robot 180B, such as a portion of the workspace that includes examples of objects 191B. The surfaces on which the objects 191B rest are not shown in FIG. 1, but these objects can rest on desks, bins, and / or other surfaces. The objects 191B include a pencil, a stapler, and a pair of glasses. In other embodiments, more objects, fewer objects, additional objects, and / or alternative objects may be provided during all or part of an object movement attempt of robot 180B as described herein.

[0045] 1, additional and / or alternative robots may be utilized, including additional robot arms similar to robots 180A and 180B, robots having other robot arm morphologies, robots having humanoid morphologies, robots having animal morphologies, robots that move via one or more wheels (e.g., robots that balance themselves), underwater robots, unmanned aerial vehicles ("UAVs"), and others. Also, while a particular grasping end effector is illustrated in FIG. 1, additional and / or alternative end effectors may be utilized. For example, end effectors that are not capable of grasping may be utilized. Additionally, while a particular mounting of visual sensors 184A and 184B is illustrated in FIG. 1, additional and / or alternative mountings may be utilized. For example, in some embodiments, the visual sensors may be mounted directly to the robot, such as on a non-actuable component of the robot or on an actuable component of the robot (e.g., on an end effector or on a component near the end effector). Also, for example, in some embodiments, the visual sensors may be mounted on a non-stationary structure separate from its associated robot and / or may be non-stationarily mounted on a structure separate from its associated robot.

[0046] Robots 180A, 180B, and / or other robots may be utilized to perform a large number of object movement attempts, and data associated with the object movement attempts may be utilized by training example generation system 110 to generate training examples. In some implementations, the object movement attempts include random and / or pseudo-random movement of the robot's end effector attempting to move one or more objects in the robot's environment, such as an attempt by robot 180A to move one or more of objects 191A.

[0047] In some implementations, all or aspects of training example generation system 110 may be implemented on robot 180A and / or robot 180B (e.g., via one or more processors of robots 180A and 180B). For example, robots 180A and 180B may each include an instance of training example generation system 110. In some implementations, all or aspects of training example generation system 110 may be implemented on one or more computer systems separate from but in network communication with robots 180A and 180B.

[0048] Each object movement attempt by robot 180A, 180B, and / or other robots consists of T separate time steps or instances. Each time step includes a current image (I T ) are stored, and the current robot parameters (p T ) are also stored and the robot selects a motion for the next time step. In some implementations, the current robot parameters for a time step may include, for example, the current state of the robot (e.g., the current end effector pose at that time step) and an action (e.g., an action to be performed to perform the motion for the next time step). In some implementations, the action for a time step may be determined based on the state of the robot at a subsequent time step compared to the current state of the robot. For example, the action may be a motion vector (e.g., a vector from the current state to the final state) to move from the current state of the robot to a final state of the robot at the end of the object motion attempt. In some implementations, the robot may be commanded via impedance control of its end effector during one or more object motion attempts. In some implementations, one or more (e.g., all) object motion attempts may last from about 3-5 seconds. In some implementations, robot 180A, 180B, and / or other robots may be programmed to move out of the field of view of the visual sensor during an object movement attempt, and an image may be captured by the visual sensor when the robot is out of field of view, and the image may be associated with the immediate object movement attempt.

[0049] In some implementations, the object movement attempts can include random pushing attempts and / or "sweep to center" attempts. Random pushing attempts can optionally involve random (e.g., truly random) pushing of the end effector within a constrained area, such as an area near the object 191A / 191B, an area based on a bin that contains the object 191A / 191B, etc. The random sweep attempt may be a random (random and / or pseudorandom) motion. The center sweep attempt may start from a random location near or on the outer boundary of the constrained area within which the object's attempted movement is restricted, and the end effector may meander toward the center of the constrained area. For example, the constrained area may generally follow a bin, and the random sweep attempt may start from a random location near the perimeter of the bin and randomly meander toward the center of the bin. The random sweep attempt may be beneficial in preventing objects from piling up at the edges of the bin.

[0050] The motion attempt of each object is (I T ,p T ) that is, each frame of the training example is represented by at least the image (I t ) and the robot motion parameters (p T ) Training examples of multiple robots' multiple object movement attempts are stored in a training example database 117 by the training example generation system 110.

[0051] Data generated by and / or derived from the data generated by sensors associated with the robot may be stored in one or more non-transitory computer-readable media local to the robot and / or remote from the robot. In some implementations, an image captured by a vision sensor may include multiple channels, such as a red channel, a blue channel, a green channel, and / or a depth channel. Each channel of the image defines a value for each of multiple pixels of the image, such as a value from 0 to 255 for each of the pixels of the image. In some implementations, an image observed at an initial time step (I T) may be concatenated with additional images in the training examples. The additional images may be additional images preceding the corresponding object motion attempt, where the additional images do not include the grasping end effector and / or other components of the robot, or where the additional images include the end effector and / or other robot components in a different pose (e.g., an end effector and / or other robot components that do not overlap with the pose of the current image). For example, the additional images may be captured after all preceding object motion attempts, but before end effector motion for the object motion attempts begins, when the grasping end effector is moved out of the field of view of the vision sensor.

[0052] The training engine 120 trains the neural network 125 based on the training examples in the training example database 117. In some implementations, training the CNN 125 includes iteratively updating the neural network 125 based on application of the training examples to the neural network 125. The trained CNN 125 is trained to predict transformations that will occur to an image of the robot's environment in response to specific movements of the robot within the environment.

[0053] 3 is a flow diagram illustrating an example of a method 300 for performing an object movement attempt and storing data associated with the object movement attempt. For convenience, the operations of this flow diagram are described with reference to a system that performs the operations. The system may include one or more components of a robot, such as a processor and / or robot control system of the robot 180A, 180B, 840, and / or other robots. Additionally, while the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0054] In block 352, the system begins attempting to move the object. In block 354, the system stores an image of the environment where the end effector is not present in the image. For example, the system may store an image of the environment where the end effector is not present outside the field of view of the visual sensor (i.e., does not block the view of the environment). The grasping end effector may be moved and images may be captured of instances when the grasping end effector is out of the field of view. The images may then be stored and associated with the attempted movement of the object. In some implementations, block 354 may be omitted.

[0055] In block 356, the system determines and implements the motion. For example, the system may generate one or more movement commands that actuate one or more of the actuators that control the pose of the end effector, thereby changing the pose of the end effector.

[0056] In some implementations and / or iterations of block 356, the motion commands may be random within a given space, such as a workspace reachable by the end effector, a constrained space within which the end effector is confined for attempting to move an object, and / or a space defined by position and / or torque limits of actuators controlling the pose of the end effector. For example, the motion commands generated by the system in block 356 to perform a movement may be random within a given space. As used herein, random may include truly random or pseudo-random.

[0057] In some implementations, at the first iteration of block 356 per object movement attempt, the end effector may be "out of position" based on moving out of field of view at block 354. In some of these implementations, prior to the first iteration of block 356, the end effector may be moved randomly or otherwise "back into position." For example, the end effector may be moved back to a set "start position" and / or moved to a randomly selected location within a given space.

[0058] In block 358, the system stores (1) an image capturing the robot's environment at a current instance (time stamp) of an object movement attempt, and (2) the robot parameters at the current instance. For example, the system may store a current image generated by a vision sensor associated with the robot and associate the image with the current instance (e.g., a time stamp). Also, for example, the system may determine robot parameters based on data from one or more joint position sensors at the robot's joints and / or torque sensors at the robot, and the system may store these parameters. The system may determine and store the robot parameters in task space, joint space, and / or another space.

[0059] In block 360, the system determines whether the current instance is the last instance of an object movement attempt. In some implementations, the system may increment an instance counter in blocks 352, 354, 356, or 358 and / or increment a time counter when time passes and determine whether the current instance is the last instance based on a comparison of the value of the counter to a threshold. For example, the counter may be a time counter and the threshold may be 3 seconds, 4 seconds, 5 seconds, and / or other values. In some implementations, the threshold may change between one or more iterations of method 300.

[0060] If, at block 360, the system determines that the current instance is not the last instance of the object movement attempt, the system returns to block 356, where the system determines and performs additional movements, and then proceeds to block 358, where the system stores the images and robot parameters for the current instance (of the additional movements). In many implementations, blocks 356, 358, 360, and / or other blocks are executed with a relatively high frequency, which allows a relatively large amount of data to be stored for each object movement attempt.

[0061] If, at block 360, the system determines that the current instance is the last instance of the object movement attempt, the system proceeds to block 366, where the system resets counters (e.g., instance counters and / or time counters) and proceeds back to block 352 to begin another object movement attempt.

[0062] In some implementations, the method 300 of FIG. 3 may be performed on each of a plurality of robots, optionally operating in parallel during one or more (e.g., all) of each iteration of the method 300. This may allow a greater number of object movement attempts to be accomplished in a given period of time than if only one robot were operating the method 300. Additionally, in implementations in which one or more of the plurality of robots includes an associated visual sensor that has a pose relative to its own robot that is different from the pose of one or more visual sensors relative to other robots, training examples generated based on the object movement attempts from the plurality of robots may provide robustness to visual sensor poses in neural networks that are trained based on these training examples. Additionally, in embodiments in which the end effectors and / or other hardware components of multiple robots change and / or wear differently, and / or different robots interact with different objects (e.g., objects of different sizes, objects of different weights, objects of different shapes, objects of different translucency, objects of different materials) and / or within different environments (e.g., different surfaces, different lighting, different environmental obstacles), training examples generated based on object movement attempts from the multiple robots can provide robustness against various robot configurations and / or environmental configurations.

[0063] In some implementations, the objects reachable by a given robot and for which object motion attempts are made may differ during different iterations of method 300. For example, a human operator and / or another robot may add and / or remove objects from the robot's workspace between one or more object motion attempts of the robot. This may increase the diversity of the training data. In some implementations, environmental factors such as lighting, surfaces, obstacles, etc. may additionally and / or instead vary during different iterations of method 300, which may also increase the diversity of the training data.

[0064] FIG. 4 is a flow diagram illustrating an example of a method 400 for generating training examples based on data related to a robot's attempts to move an object. For convenience, the operations of this flow diagram are described with reference to a system that performs the operations. The system may include one or more components of the robot and / or another computer system, such as a processor and / or robot control system of the robot 180A, 180B, 840, and / or a processor of the training example generation system 110 and / or other systems that may optionally be implemented separately from the robot. Additionally, while the operations of the method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0065] At block 452, the system begins training example generation. At block 454, the system selects an object motion attempt. For example, the system may access a database that includes data related to a plurality of stored object motion attempts and select one of the stored object motion attempts. The selected object motion attempt may be, for example, an object motion attempt generated based on the method 300 of FIG.

[0066] At block 456, the system selects a group of sequential frames of the object's attempted movement. For example, the system may select a first frame of the object's attempted movement as a first instance of the group, the next in time frame as a second instance of the group, the next in time frame as a third instance of the group, etc. As another example, the system may select a fifth frame of the object's attempted movement as a first instance of the group, the sixth frame of the object as a second instance of the group, etc.

[0067] At block 458, the system assigns sequential frames as training examples. As described, each sequential frame may include an image for a corresponding time step, a robot state for the corresponding time step, and an action for the corresponding time step. In some implementations, at block 456 or block 458, the system may optionally process the images of the frames. For example, the system may optionally resize the images to fit a defined size of the input layer of the neural network, remove one or more channels from the images, and / or normalize the values ​​of the depth channel (in implementations where the images include a depth channel).

[0068] At block 460, the system determines whether additional training examples should be generated. If so, the system proceeds back to block 454 to select an object motion attempt (e.g., a different object motion attempt). If not, training example generation ends at block 468.

[0069] In some implementations, determining whether to generate additional training examples may include determining whether there are any remaining unprocessed object movement attempts. In some implementations, determining whether to generate additional training examples may additionally and / or alternatively include determining whether a threshold number of training examples have already been generated and / or other criteria have been satisfied.

[0070] Another iteration of method 400 may be performed again. For example, method 400 may be performed again in response to at least a threshold number of additional object movement attempts being performed.

[0071] Although methods 300 and 400 are shown in separate figures herein for clarity, it should be understood that one or more blocks of method 400 may be performed by the same component that performs one or more blocks of method 300. For example, one or more (e.g., all) of the blocks of methods 300 and 400 may be performed by a processor of a robot. It should also be understood that one or more blocks of method 400 may be performed in combination with, preceding, or following one or more blocks of method 300.

[0072] FIG. 5 is a flow diagram illustrating an example method 500 for training a neural network based on training examples. For convenience, the operations of this flow diagram are described with reference to a system that performs the operations. The system may include one or more components of a computer system, such as a processor (e.g., a GPU) of the training engine 120 and / or other computer system that operates with respect to the neural network (e.g., with respect to the neural network 125). Additionally, while the operations of the method 500 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0073] In block 552, the system begins training. In block 554, the system selects training examples. For example, the system can select training examples generated according to method 400 of FIG.

[0074] At block 556, the system applies the image of the first frame of the selected training example to an initial layer of the neural network. For example, the system may apply the image of the first frame to an initial convolutional layer of the neural network. As described herein, in some implementations, the training example may optionally include additional images that at least partially omit the end effector and / or other robotic components. In some of these implementations, the system concatenates the image of the first frame with the additional images and applies the concatenated image to the initial layer. In some other of these implementations, the image of the first frame has already been concatenated with the additional images in the training example.

[0075] At block 558, the system applies the robot motion parameters for the first frame to additional layers of a neural network. For example, the system may apply the robot motion parameters for the first frame to additional layers of a neural network (e.g., a convolutional LSTM layer) downstream from the initial layer to which the image is applied at block 556. As described herein, the robot motion parameters for the first frame may include a robot state at a time step of the initial frame and an action to be performed at the time step of the initial frame.

[0076] At block 560, the system generates a predicted image on the neural network based on the applied image and the applied robot motion parameters. The generation of the predicted image depends on the structure of the neural network and may include one or more of the techniques described herein for DNA, CDNA, and STP variants of the neural network.

[0077] At block 562, the system performs backpropagation and / or other training techniques on the neural network based on a comparison of the predicted image to an image of the second frame of the training examples. In some implementations, the system determines an I2 reconstruction loss based on the comparison and updates the neural network based on the I2 reconstruction loss.

[0078] In block 564, the system applies the most recently predicted image as an input to an initial layer of the neural network. In the first iteration of block 564, this is the predicted image of block 560. In some implementations, in block 564, the system can instead apply an image of a frame of the training examples that corresponds to the most recently predicted image as an input to an additional layer of the neural network. In the first iteration of block 564, this would be an image of the second frame of the training examples.

[0079] In block 566, the system applies the robot motion parameters for the next frame to an additional layer of the neural network. In the first iteration of block 564, this would be the robot motion parameters for the second frame.

[0080] At block 568, the system generates a predicted image based on application of the image and the applied robot motion parameters to the neural network. The generation of the predicted image depends on the structure of the neural network and may be performed using techniques described herein for DNA, cDNA, and STP variants of the neural network. may include one or more of:

[0081] At block 570, the system performs backpropagation and / or other training techniques on the neural network based on a comparison of the predicted image to the next frame image of the training examples. In the first iteration of block 570, the next frame image should be the third frame image. In some implementations, the system determines an I2 reconstruction loss based on the comparison and updates the neural network based on the I2 reconstruction loss.

[0082] In block 572, the system determines whether there are additional frames to consider in the training examples. For example, if the training is for eight iterations, the system may determine that there are additional frames to consider if eight frames have not yet been considered. If the system determines in block 572 that there are additional frames to consider, the system proceeds back to block 564 and performs another iteration of blocks 564, 566, 568, 570, and 572.

[0083] If the system determines at block 572 that there are no additional frames to consider, the system proceeds to block 574. At block 574, the system determines whether there are additional training examples. If the system determines there are additional training examples, the system returns to block 554 and selects another training example. In some implementations, determining whether there are additional training examples may include determining whether there are remaining training examples that have not been utilized in training the neural network. In some implementations, determining whether there are additional training examples may additionally and / or instead include determining whether a threshold number of training examples have been utilized and / or other criteria have been satisfied.

[0084] If the system determines that there are no additional training examples and / or any other criteria have been met, the system proceeds to block 576 .

[0085] At block 576, training of the neural network may be completed. The trained neural network may be provided for use by one or more robots in determining one or more movements to perform within an environment. For example, a robot may utilize the trained neural network in performing method 700 of FIG.

[0086] 6A and 6B show an example architecture of a neural network 600, example inputs that may be provided to the neural network, example outputs of the neural network 600, and how the example outputs may be used to generate a predicted image. The neural network 600 of FIG. 6A and 6B is an example of a neural network that may be trained according to the method 500 of FIG. 5. The neural network 600 of FIG. 6A and 6B is also an example of a neural network that may be trained and then utilized within an implementation of the method 700 of FIG. 7.

[0087] The example neural network 600 in FIG. 6A and FIG. 6B is an example of a CDNA motion prediction model. The neural network 600 may be the neural network 125 in FIG. 1 and is one of the three proposed motion prediction models described herein. In the neural network 600 in FIG. 6A and FIG. 6B, a convolutional layer 661, convolutional LSTM layers 672-677, and convolutional layer 662 are configured to generate an image 601 (e.g., an image captured by a camera in the first iteration, an image predicted immediately before in the subsequent iterations) and a convolutional LSTM layer 672-677. The output of the ten normalized CDNA transform kernels 682 is generated as the output for the lowest dimensional layer of the network (convolutional LSTM layer 675), and the output of a synthesis mask 684 (e.g., an 11-channel mask) is generated as the output for the last layer (convolutional layer 62). As described herein, the synthesis mask 684 may include one channel for a static background and ten additional channels (each corresponding to a transformed image generated based on a corresponding one of the ten CDNA transform kernels 682). The CDNA kernels 682 are applied to transform 693 the image 601 (e.g., the image captured by the camera in the first iteration, and the immediately preceding predicted image in the subsequent iteration) into ten different transformed images 683. The ten different transformed images are synthesized according to the synthesis mask 684 to generate a predicted image 685 in a masked synthesis 694. Transformation 693 of image 601 into ten different transformed images 683 can include convolving image 601 with CDNA kernels 682 (e.g., convolving image 601 with a first kernel of CDNA kernels 682 to generate a first one of transformed images 683, convolving image 601 with a second kernel of CDNA kernels to generate a second one of transformed images 683, etc.). The composite mask 684 sums to 1 at each pixel due to the softmax over the applied channels.

[0088] Various skip connections are shown in Figures 6A and 6B. Specifically, a skip connection from the output of convolutional LSTM layer 673 to the input of convolutional LSTM layer 677 and a skip connection from the output of convolutional LSTM layer 671 to the input of convolutional layer 662. Examples of dimensions of various layers and outputs are also shown. For example, image 601 may be 64 pixels by 64 pixels and may include three channels (indicated by "64 x 64 x 4"). Also, for example, convolutional LSTM layer 675 may have an 8 x 8 dimension (indicated by "8 x 8") and may apply a 5 x 5 convolution (indicated by "5 x 5").

[0089] A description of an additional example of neural network 600 will now be provided. Neural network 600 includes a core trunk consisting of one stride 2 5×5 convolutional layer 661 followed by convolutional LSTM layers 671-677. Each of the convolutional LSTM layers 671-677 has weights arranged in a 5×5 convolution, with the output of the preceding LSTM directly fed into the next LSTM. The convolutional LSTM layers 673 and 675 are preceded by stride 2 downsampling to reduce resolution, and the convolutional LSTM layers 675, 676, and 677 are preceded by upsampling. The end of the LSTM stack is followed by an upsampling stage and a final convolutional layer 662, which synthesizes various transformed images 683 and outputs a full resolution synthesis mask 684 for synthesis against a static background. Masked compositing 694 can use the compositing mask 684 and the transformed image 683 to generate a predicted image 685. Note that in the case of the STP motion prediction model, a mask is also generated for compositing the various transformed images (generated based on the transformation matrix), and in the case of both STP and DNA, a compositing mask is generated for compositing against a static background.

[0090] To preserve high resolution information, there are skip connections from the output of convolutional LSTM layer 671 to the input of convolutional layer 662 and from the output of convolutional LSTM layer 673 to the input of convolutional LSTM layer 677. These skip connections concatenate the skip layer activations with the activations of the preceding layer before sending the activations to the subsequent layer. For example, the input to convolutional LSTM layer 677 consists of the concatenation of the output from convolutional LSTM layer 673 and the output from convolutional LSTM layer 676. The model also includes as inputs robot motion parameters of robot state 602 (e.g., gripper pose) and robot action 603 (e.g., gripper movement commands). For example, The robot state 602 can be five-dimensional, with vector (x,y,z) and angle (pitch,yaw) values. The robot action can also be five-dimensional, with vector and angle values ​​for a subsequent (e.g., next) time step when the action is applied, and / or a motion vector when the action is applied. For example, the robot action can be the vector (x,y,z) and angle (pitch,yaw) of a commanded gripper pose. The commanded gripper pose can be the commanded gripper pose for the next time step or the commanded gripper pose for a future time step (e.g., the target pose of the gripper at the last time step). The vectors of robot states 602 and robot actions 603 are first tiled into an 8x8 response map 681 with 10 channels, which are then concatenated in 693 with respect to channels to the input of a convolutional LSTM layer 675, which is concatenated with the output from convolutional LSTM layer 674. The robot state 602 of the current iteration may be the current robot state, and the robot state 602 of a subsequent iteration may be linearly predicted from the preceding current robot states and preceding robot actions, although additional or alternative robot state prediction models may be used.

[0091] In the case of action-conditioned robot manipulation tasks, all three models (DNA, CDNA, and STP) include as input the current state of the robot (e.g., gripper pose) and the action (e.g., gripper movement command). The three models differ in the form of the transformation applied to the previous image. The object-centric CDNA and STP models output the transformation parameters after the convolutional LSTM layer 675. In both cases, the output of the convolutional LSTM layer 675 is flattened and linearly transformed, either directly to the filter parameters in the case of CDNA, or via one 100-unit hidden layer in the case of STP. In the case of CDNA, there can be 10 CDNA filters, which can be 5×5 in size and normalized to sum to 1 via spatial softmax, so that each filter represents a distribution over the positions in the previous image from which the new pixel value is obtained. The STP parameters can correspond to 10 3×2 affine transformation matrices. The transformations are applied to the previous image to create 10 separate transformed images. The CDNA transformation corresponds to a convolution (although the kernel is the output of the network), while the STP transformation is an affine transformation. The DNA model differs from the other two in that the transformation parameters are output at the same place as the last convolution layer 662, i.e., the mask. This is because the DNA model outputs a transformation map that is as large as the entire image. For each image pixel, the DNA model outputs a 5x5 convolution kernel that can be applied to the previous image to obtain a new pixel value, just like the CDNA model. However, this kernel is spatially varying, so this model is not equivalent to CDNA. This transformation produces only one transformed image. After the transformation, the transformed image(s) and the previous image are composited together based on the mask. The previous image is included as a static "background" image, and as described herein, the mask for the background image tends to pick out static parts of the scene. The final image is formed by multiplying each transformed image and the background image by its mask value and adding all of the masked images together.

[0092] Although a particular convolutional neural network is shown in Figures 6A and 6B, variations are possible, e.g., more or fewer LSTM layers may be provided, one or more layers may be sized differently than those provided as examples, etc. Also, for example, alternative predicted transformations and / or synthesis masks may be generated for the model.

[0093] The neural networks of FIGS. 6A and 6B, or other neural networks, after being trained according to the techniques described herein, may be utilized for a variety of purposes. With reference to FIG. 7, an example flow diagram of a method 700 is shown that utilizes a trained neural network to generate predicted image(s) that predict the robot's environment after a robot movement has occurred and / or performs one or more actions based on the predicted image(s). For convenience, the operations of the flow diagram are described with reference to a system that performs the operations. The system may include one or more components of the robot, such as the processor (e.g., GPU and / or CPU) and / or robot control system of the robot 180A, 180B, 840, and / or other robots. In performing one or more blocks of the method 700, the system may operate via a trained neural network that may be stored, for example, locally on the robot and / or remotely from the robot. Additionally, while the operations of the method 700 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0094] In block 752, the system generates candidate robot motion parameters for at least some of the candidate motions of the robot. The candidate robot motion parameters may include, for example, a current robot state (e.g., the current robot state or the initial robot state of the candidate robot motion parameters) and one or more actions to be performed (e.g., a commanded robot state). The candidate robot motion parameters define at least some of the candidate motions executable (but not yet executed) by one or more components of the robot in the robot's environment. For example, the motions may include motion of an end effector of the robot from a first pose to a second pose, and the candidate robot motion parameters may define various parameters associated with the motion from the first pose to the second pose, or parameters associated with the motion from the first pose to an intermediate pose between the first pose and the second pose. For example, the candidate robot motion parameters may define only a portion of the candidate motion (e.g., only a percentage of the first portion of the candidate motion or only the first X time steps of the candidate motion).

[0095] The motion parameters may include, for example, joint space motion vectors (e.g., joint angle motions) for achieving the portion of the candidate motion, transformations of the end effector pose with respect to the portion of the candidate motion, joint space torque vectors for achieving the portion of the candidate motion, and / or other parameters. Note that the particular motion parameters and / or the shape of the motion parameters will depend on the input parameters of the trained neural network that will be utilized in future blocks.

[0096] In some implementations, the candidate movements are generated by another system, and the candidate robot motion parameters of the candidate movements are applied to the neural network in method 700 to determine how the candidate movements would affect an object(s) in the environment if the candidate movements were implemented. In some of these implementations, the candidate movements may be implemented based on how they affect the object(s), refined based on how they affect the object(s), or not implemented based on how they affect the object(s). In some implementations, the system generates multiple distinct candidate movements and applies the candidate robot motion parameters of each of the candidate movements to the neural network in multiple iterations of method 700 to determine how each affects an object(s) in the environment. In some of these implementations, one of the movements may be selected for implementation based on how they affect the object(s).

[0097] In block 754, the system captures one or more environmental objects in the robot's environment. In some embodiments, such as an initial iteration of method 700 for a portion of a candidate movement, the image is the current image. In some embodiments, the system also identifies additional images that at least partially omit the end effector and / or other robotic components, such as additional images of environmental objects captured by the visual sensor when the end effector was at least partially outside the field of view of the visual sensor. In some embodiments, the system concatenates the image and the additional images to generate a concatenated image. In some embodiments, the system optionally performs processing of the image and / or the concatenated image (e.g., to fit the size to an input of a neural network).

[0098] In block 756, the system applies the image and the candidate robot motion parameters (e.g., the candidate robot states and the candidate action(s)) to the neural network to be trained. For example, the system may apply the image to an initial layer of the neural network to be trained. The system may also apply the current robot state and the candidate robot motion parameters to additional layers of the neural network to be trained downstream of the initial layer.

[0099] In block 758, the system generates a predicted transformation of the image of block 754 on the trained neural network. The predicted transformation is generated based on applying the image and the candidate robot motion parameters (along with the current robot state) to the trained neural network in block 756 and determining the predicted transformation based on the learned weights of the trained neural network.

[0100] At block 760, the system transforms the image based on the predicted transformation of block 758 to generate a predicted image. For example, if the predicted transformation of block 758 includes parameters of one or more spatial transformers, the system may utilize those parameters to apply one or more spatial transformers to the current image. Also, for example, if the predicted transformation of block 758 includes one or more normalized distributions, each corresponding to one or more of the pixels, the transformation of the current image may include applying the normalized distributions to the current image using a convolution operation. In some of these implementations, each of the normalized distributions corresponds to a corresponding one of the pixels. A synthesis mask may also be utilized as described herein, where parameters of the synthesis mask are generated based on application of the image (and optionally additional images) and candidate robot motion parameters to the trained neural network at block 756. For example, the synthesis mask may be utilized to generate a single predicted image from multiple transformed images generated using a CDNA model and / or an STP model as described herein. Additionally, the synthesis mask(s) may include a background mask that may be applied to the current image to copy pixels from the current image when generating the predicted image.

[0101] The predicted image predicts the portion of the environment that would be captured by the image if at least a portion of the candidate movement indicated by the parameters of block 752 is executed in the environment by the components of the robot. In other words, if at least a portion of the candidate movement causes a movement of one or more objects in the environment, the predicted image may represent the object(s) after the movement.

[0102] In some implementations, the system determines whether there are additional candidate robot motion parameters to consider for the candidate motion. For example, the system determines whether there are additional parts of the candidate motion to consider, and if so, the system adds the block The process proceeds back to block 752 to generate candidate robot motion parameters for the portion, then proceeds to block 754 to identify an image, then proceeds to blocks 756-758 to generate another predicted image based on the additional candidate robot motion parameters. In some implementations, the image identified in the additional iteration of block 754 is the predicted image generated in the immediately preceding iteration of block 760. This process may be repeated, each time utilizing the predicted image from the immediately preceding iteration as the image identified in the iteration of block 754 and applied in that iteration of block 756, thereby allowing a predicted image to be generated for each of multiple future time steps. In this manner, the predicted image may be conditioned on the current image and utilized to determine the motion of the object(s) over multiple time steps based on the candidate robot motion parameters.

[0103] At block 762, the system performs one or more actions based on the predicted image and optionally the “current image.” For example, the system may determine that a movement caused by the candidate movement is desirable based on a comparison of the predicted image to the current image. Based on the determination that the movement is desirable, the system may execute the candidate movement, e.g., by providing one or more control commands to one or more actuators of the robot to realize the candidate movement. Also, for example, if multiple predicted images are determined for a candidate movement, the system may determine that a movement caused by the candidate movement is desirable based on the predicted image and / or the current image. Based on the determination that the movement is desirable, the system may execute the candidate movement, e.g., by providing one or more control commands to one or more actuators of the robot to realize the candidate movement.

[0104] In some implementations, one or more iterations of blocks 752-760 are performed for one or more additional candidate movements, and at block 764, the system may select one of the candidate movements based on the predicted images for these iterations. For example, one or more iterations of blocks 752-760 may be performed for a first candidate movement and independently for a second candidate movement. The system may select one of the first candidate movement and the second candidate movement based on the predicted images of the first candidate movement and the predicted images of the second candidate movement. The system may execute the selected candidate movement by providing one or more corresponding control commands to one or more actuators of the robot.

[0105] 8 illustrates generally an example of the architecture of a robot 840. The robot 840 includes a robot control system 860, one or more motion components 840a-840n, and one or more sensors 842a-842m. The sensors 842a-842m may include, for example, visual sensors, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and the like. Although the sensors 842a-m are illustrated as being integral to the robot 820, this is not intended to be limiting. In some implementations, the sensors 842a-m may be located external to the robot 820, for example as stand-alone units.

[0106] Motion components 840a-840n may include, for example, one or more end effectors and / or one or more servo motors or other actuators for implementing motion of one or more components of the robot. For example, robot 820 may have multiple degrees of freedom, and each of the actuators may be capable of controlling actuation of robot 820 in one or more of the degrees of freedom in response to a control command. As used herein, the term actuator may refer to an actuator and may include a servo motor or actuators that actuate the actuator to translate a received control command into one or more actuators that drive the actuator. encompasses a mechanical or electrical device (e.g., a motor) that creates movement, in addition to any driver(s) that convert the control command into a number of signals. Thus, providing a control command to an actuator can include providing the control command to a driver that converts the control command into a signal appropriate to drive an electrical or mechanical device to create the desired movement.

[0107] The robot control system 860 may be implemented in one or more processors, such as a CPU, GPU, and / or other controller(s) of the robot 820. In some implementations, the robot 820 may include a "brain box" that may include all or aspects of the control system 860. For example, the brain box may provide real-time bursts of data to the motion components 840a-n, each of which may include a set of one or more control commands that dictate, among other things, parameters of the movement (if any) of one or more of the motion components 840a-n. In some implementations, the robot control system 860 may perform one or more aspects of the methods 300, 400, 500, and / or 700 described herein.

[0108] As described herein, in some implementations, all or aspects of the control commands generated by the control system 860 in moving one or more components of the robot may be based on a predicted image(s) generated utilizing a predicted transformation(s) determined via a trained neural network. For example, a visual sensor of sensors 842a-m may capture a current image, and the robot control system 860 may generate candidate robot motion parameters. The robot control system 860 may provide the trained neural network with the current image and the candidate robot motion parameters, generate a predicted transformation based on the application, generate a predicted image based on the predicted transformation, and generate one or more end effector control commands utilizing the predicted image to control the motion of the robot. Although the control system 860 is illustrated in FIG. 8 as an integral part of the robot 820, in some implementations, all or aspects of the control system 860 may be implemented in a component separate from but in communication with the robot 820. For example, all or aspects of the control system 860 may be implemented on one or more computing devices, such as computing device 910, in wired and / or wireless communication with the robot 820.

[0109] 9 is a block diagram of an example computing device 910 that may be optionally utilized to perform one or more aspects of the techniques described herein. The computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924, including, for example, a memory subsystem 925 and a file storage subsystem 926, a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices enable user interaction with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0110] The user interface input devices 922 may be a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, etc. Input devices may include audio input devices, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and forms for inputting information into the computing device 910 or a communications network.

[0111] The user interface output devices 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel display such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and forms for outputting information from the computing device 910 to a user or to another machine or computing device.

[0112] The storage subsystem 924 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 924 may include logic to perform selected aspects of the methods of FIGS.

[0113] These software modules are generally executed by the processor 914 alone or in combination with other processors. The memory 925 used within the storage subsystem 924 may include a number of memories including a primary random access memory (RAM) 930 for storage of instructions and data during program execution, and a read only memory (ROM) 932 in which fixed instructions are stored. The file storage subsystem 926 may provide persistent storage of program files and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of certain embodiments may be stored within the storage subsystem 924 or within other machines accessible by the processor 914 via the file storage subsystem 926.

[0114] The bus subsystem 912 provides a mechanism that allows the various components and subsystems of the computing device 910 to communicate with each other as intended. Although the bus subsystem 912 is illustrated generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0115] The computing device 910 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 910 shown in FIG. 9 is intended only to be a specific example to illustrate some implementations. Numerous other configurations of the computing device 910 are possible, having more or fewer components than the computing device shown in FIG.

Claims

1. 1. A method executed by one or more processors, comprising: generating candidate robot motion parameters, the candidate robot motion parameters defining at least a portion of a first candidate motion of a plurality of candidate motions executable by one or more components of the robot in an environment of the robot; identifying a current image captured by a visual sensor associated with the robot, the current image capturing at least a portion of the environment of the robot; applying the current image and the candidate robot motion parameters as inputs to a trained neural network; generating at least one predicted transformation of the current image, the predicted transformation being generated based on the application of the current image and the candidate robot motion parameters to the trained neural network; transforming the current image based on the at least one predicted transformation to generate at least one predicted image, the predicted image representing the portion of the environment of the robot when a first candidate motion is executed in the environment by components of the robot; determining, based on the predicted image, that the first candidate motion is not suitable for achieving a predetermined goal; in response to determining that the first candidate motion is not suitable for achieving a predetermined goal, determining to perform a second candidate motion instead of the first candidate motion, the second candidate motion being a candidate motion other than the first candidate motion among the plurality of candidate motions; providing one or more control commands to one or more actuators of the robot to execute the second candidate motion; A method comprising:

2. generating at least one composite mask based on the application of the current image and the candidate robot motion parameters to the trained neural network; The method of claim 1 , wherein transforming the current image is further based on the at least one synthesis mask.

3. the at least one predicted transformation comprises a plurality of predicted transformations; the at least one composite mask includes a plurality of composite masks; transforming the current image based on the at least one predicted transformation to generate the predicted image, generating a plurality of predicted images based on the plurality of predicted transformations; synthesizing the plurality of predicted images based on the plurality of synthesis masks to generate the predicted image.

3. The method of claim 2, comprising:

4. generating second candidate robot motion parameters, the second candidate robot motion parameters defining at least a portion of the second candidate motion; applying the current image and the second candidate robot motion parameters as inputs to the trained neural network; generating at least one second predicted transformation of the current image, the second predicted transformation being generated based on the application of the current image and the second candidate robot motion parameters to the trained neural network; transforming one or more pixels of the current image based on the second predicted transformation to generate at least one second predicted image, the second predicted image predicting the part of the robot in the environment if at least the part of the second candidate motion is executed in the environment by the component of the robot; The method of claim 1, further comprising:

5. selecting the second candidate motion based on the second predicted image.

5. The method of claim 4, further comprising:

6. generating subsequent candidate robot motion parameters, the subsequent candidate robot motion parameters defining another portion of the second candidate motion that follows the portion of the second candidate motion; applying the predicted image and the continuing candidate robot motion parameters to the trained neural network; generating at least one ongoing predicted transformation of the predicted image, the ongoing predicted transformation being generated based on the application of the predicted image and the ongoing candidate robot motion parameters to the trained neural network; transforming the predicted image based on the continuing predicted transformation to generate a continuing predicted image; 6. The method of claim 5, further comprising:

7. selecting the second candidate motion further based on the subsequent predicted images.

7. The method of claim 6, further comprising:

8. 2. The method of claim 1, wherein the trained neural network comprises multiple stacked convolutional long short-term memory layers.

9. The method of claim 1 , wherein the at least one predicted transformation of pixels of the current image comprises parameters of one or more spatial transformers.

10. The method of claim 9 , wherein transforming the current image comprises applying the one or more spatial transformers to the current image using the parameters.

11. 2. The method of claim 1, wherein the at least one predicted transformation of a pixel of the current image comprises one or more normalized distributions, each of the one or more normalized distributions corresponding to one or more of the pixels.

12. The method of claim 11 , wherein transforming the current image comprises applying the one or more normalized distributions to the current image using a convolution operation.

13. The method of claim 11 , wherein each of the one or more normalized distributions corresponds to a corresponding one of the pixels.

14. applying the current image and the candidate robot motion parameters as inputs to the trained neural network; applying the current image as an input to an initial layer of the trained neural network; applying the candidate robot motion parameters to an additional layer of the trained neural network, the additional layer being downstream of the initial layer; 2. The method of claim 1, comprising:

15. 1. A system comprising: A visual sensor for observing the environment; a trained neural network recorded on one or more non-transitory computer readable media; and At least one processor and wherein the at least one processor: identifying a current image captured by the visual sensor associated with a robot; Identifying a current state of the robot; identifying candidate actions for transitioning the robot from the current state to a candidate state; applying the current image, the current state, and the candidate actions as inputs to the trained neural network; generating at least one predicted image based on the application of the current image, the current state, and the candidate actions to the trained neural network; determining, based on the predicted image, that the candidate action is not suitable for achieving a particular goal; in response to determining that the candidate action is not suitable for achieving a particular goal, determining to execute a second candidate action in place of the candidate action; providing one or more control commands to one or more actuators of the robot to execute the second candidate action; A system configured to:

16. 1. A method executed by one or more processors, comprising: determining a goal state of an object in an environment of the robot based on user interface input; generating candidate robot motion parameters, the candidate robot motion parameters defining at least a portion of candidate motions executable by the robot in the environment; identifying a current image captured by a visual sensor associated with the robot, the current image capturing at least a portion of the environment including the object in a current state; generating at least one predicted image, the at least one predicted image representing a predicted state of the object when the candidate motion is executed in the environment by the robot, the generating of the predicted image comprising: applying the current image and the candidate robot motion parameters as inputs to a trained neural network, whereby the neural network generates the at least one predicted image; determining to execute the candidate movement based on a comparison of the goal state of the object to the predicted state of the object; providing one or more control commands to one or more actuators of the robot to execute the candidate movements; A method comprising:

17. 17. The method of claim 16, wherein the user interface input is entered via an interface that displays the current image or other image capturing at least the portion of the environment and the objects in the current state.

18. The step of generating at least one predicted image comprises: generating at least one predicted transformation of the current image, the predicted transformation being generated based on the application of the current image and the candidate robot motion parameters to the trained neural network; transforming the current image based on the at least one predicted transformation to generate at least one predicted image; 17. The method of claim 16, further comprising:

19. The method of claim 18 , wherein the at least one predicted transformation of pixels of the current image comprises parameters of one or more spatial transformers.

20. 20. A computer readable medium having a computer program comprising machine readable instructions for performing the method of any one of claims 1 to 14 and claims 16 to 19 when the computer program is executed by a computing device.

Images