Robotic manipulation using domain-invariant 3D representations predicted from 2.5D visual data

Abstract

Embodiments are directed to training a point cloud prediction model, wherein the point cloud prediction model can be used to process a single view two and a half (2.5D) observation of an object to generate a domain invariant three-dimensional (3D) representation of the object. Embodiments additionally or alternatively involve utilizing the domain invariant 3D representation to train a robotic manipulation policy model that uses the domain invariant 3D representation of a simulated object to be manipulated as at least a portion of the input to the robotic manipulation policy model during training. Embodiments additionally or alternatively involve utilizing the trained robotic manipulation policy model in controlling a robot based on output generated by processing the generated domain invariant 3D representation with the robotic manipulation policy model.

Description

Technical Field

[0001] Embodiments disclosed herein relate to training a point cloud prediction model, and in particular, training a single view point cloud prediction model that can be used to process an object. Figure 2 A point cloud prediction model (a machine learning model, such as a neural network model) of one and a half-dimensional (2.5D) observations to generate a domain-invariant three-dimensional (3D) representation of an object (e.g., a 3D point cloud of the object). Background Art

[0002] Various machine learning-based methods for robot control have been proposed. Some of these methods train machine learning models (e.g., deep neural network models) that can be used to generate one or more predictions used in controlling the robot, and use training data based only on data from real-world physical robots to train the machine learning models. However, these and / or other methods may have one or more disadvantages. For example, generating training data based on data from real-world physical robots requires extensive use of one or more physical robots when generating data for training data. This may be time consuming (e.g., actually navigating a large number of paths requires a lot of time), may consume a lot of resources (e.g., the power required to operate the robot), may cause wear and tear on the robot being used, and / or may require a lot of manual intervention.

[0003] In view of these and / or other considerations, it has been proposed to use robot simulators to generate simulated robot data, which can be utilized in generating simulated training data that can be utilized in training machine learning models. However, there is often a meaningful "reality gap" between a real robot and real environment and a simulated robot and / or simulated environment simulated by a robot simulator. This may result in the generation of simulated training data that does not accurately reflect what would happen in a real environment. This may affect the performance of a machine learning model trained on such simulated training data, and / or it may be necessary to also utilize a large amount of real-world training data in training to help alleviate the reality gap. Summary of the invention

[0004] Embodiments disclosed herein relate to training a single view that can be used to process an object. Figure 2A point cloud prediction model (such as a machine learning model such as a neural network model) of one and a half-dimensional (2.5D) observations to generate a domain-invariant three-dimensional (3D) representation of an object (e.g., a 3D point cloud of the object). Various embodiments are also directed to utilizing the domain-invariant 3D representation to train (e.g., at least partially in simulation) a robotic manipulation policy model (e.g., an evaluation network or other policy model) that uses the domain-invariant 3D representation of a simulated object to be manipulated as at least a portion of the input to the robotic manipulation policy model during training. Various embodiments are additionally or alternatively directed to utilizing the trained robotic manipulation policy model in controlling a robot based on output generated by processing the generated domain-invariant 3D representation with the robotic manipulation policy model.

[0005] A domain-invariant 3D representation is generated based on processing 2.5D observations captured by a camera of the robot using a trained shape prediction network (e.g., a point cloud prediction network or other 3D shape prediction network). The 2.5D observations can be images including one or more color channels (e.g., red, green, and blue channels) and a depth channel. In other words, each pixel of the image can have a depth channel as well as one or more color channels. The camera can be, for example, an RGB-D camera including one or more sensors that capture visual data, wherein the visual data collectively (and optionally after processing) defines an image having a plurality of pixels, and collectively defines a depth channel as well as one or more additional channels (e.g., red, green, and blue channels) for each of the pixels. Various types of RGB-D cameras can be utilized, including passive RGB-D cameras and active RGB-D cameras (e.g., cameras including speckle projectors or utilizing light sources and time of flight). As described herein, each 2.5D image utilized in generating the domain-invariant 3D representation can be a single-view Figure 2 .5D images.

[0006] Various efficiencies can be achieved by training shape prediction models and / or robotic manipulation strategies using simulated data. For example, ground truth data used in training one or two models can be effectively obtained from simulation. In addition, using domain-invariant 3D representations as input to the robotic manipulation strategy model can enable the network to be trained primarily (or entirely) based on simulated training data, while alleviating the reality gap when the robotic manipulation strategy model is used on a real-world robot. For example, a domain-invariant 3D representation of a 3D point cloud as an object describes the 3D shape of an object that can have a minimal reality gap / no reality gap when simulated. Such a 3D point cloud is invariant to texture or environmental changes, which may have a significant reality gap when simulated. In addition, the domain-invariant 3D representation can be compact (in terms of data), while being semantically interpretable and directly applicable to object manipulation. This can enable such representations to be effectively processed using a robotic manipulation strategy model while achieving high accuracy and / or robustness.

[0007] In addition, the domain-invariant 3D representation can be efficiently transformed between frames, such as the frame of a camera capturing a 2.5D observation (used to generate a 3D representation) to the frame of a robot end-effector. As described herein, a robotic manipulation policy can optionally be trained to process the transformed domain-invariant 3D representation (and optionally only the transformed representation) in generating a probability (or other value) of a corresponding robot end-effector pose (used in generating the transformed domain-invariant 3D representation) that results in a successful manipulation (e.g., grasping). Since the domain-invariant 3D representation is compact (in terms of data), such processing can be computationally efficient and / or the model(s) representing the policy (e.g., neural network model(s)) can be compact (in terms of data). For example, processing the domain-invariant 3D representation can be more efficient than processing the full RGB image(s). In addition, for example, processing the transformed domain-invariant 3D representation without processing the end-effector pose used to generate the transformed domain-invariant 3D representation can be more efficient than processing both the domain-invariant 3D representation and the end-effector pose. Furthermore, the texture and environment invariance of the domain-invariant 3D representation enables it to be effectively applied to model robotic manipulation policies for various textured objects and / or various environments.

[0008] The domain-invariant 3D representation of the object can be a complete 3D point cloud of the object. The shape prediction model can be trained using a large amount of simulation training data and a small amount (or no) of real-world training data. For example, 50,000 or more (e.g., 60,000, 70,000) events in the simulation can be used, while less than one thousand (e.g., less than 600, less than 500) events in the real world can be used. The 2.5D observation of the object processed to generate the domain-invariant 3D representation can be an RGB-D (where D is depth) image. In various embodiments, additional object mask channels (in addition to the RGB-D channels) can also be used as input to the shape prediction model to, for example, enable processing of multiple objects in the 2.5D observation of the object. The mask can be generated based on processing the 2.5D observation of the object (e.g., at least its 2D portion) using an object detection network such as a Mask-RCNN network. For example, assume that the target object is an apple and the 2.5D image includes apples and bananas. The object detection network may be used to determine pixels of the 2.5D image that correspond to apples, and an additional mask generated as additional channels where those pixels in the channel have values ​​indicating that the target object is present in the corresponding pixels.

[0009] In some embodiments, the robot manipulation strategy model is an evaluation prediction network, wherein the evaluation prediction network can be used to generate a manipulation result prediction of a candidate end-effector posture based on a domain-invariant 3D representation and a candidate end-effector posture. For example, where the manipulation is grasping, the evaluation grasp prediction network can be used to predict the probability that the candidate grasping posture will succeed based on the candidate grasping posture (of the end effector) and the domain-invariant 3D representation of the object. For example, the domain-invariant 3D representation as a 3D point cloud can be transformed into a frame relative to the candidate grasping posture. The evaluation grasp prediction network can then be used to process the transformed 3D point cloud to generate the probability that the candidate grasping posture will succeed. The evaluation grasp prediction network can be used to consider various candidate grasping postures, and the candidate grasping posture with the highest probability is selected for attempting to grasp. In other embodiments, the robot manipulation strategy model can be used for other robot tasks, such as, for example, pushing an object, pulling an object, etc. In some embodiments, the robot manipulation strategy can be used to process a non-transformed domain-invariant 3D representation and a candidate end-effector posture to generate a manipulation result prediction. In other words, in those embodiments, both the non-transformed domain-invariant 3D representation and the candidate end-effector pose can be applied as input instead of the transformed domain-invariant 3D representation. In addition, in some embodiments, the robot manipulation strategy can include an action prediction network instead of or in addition to the evaluation prediction network. For example, the action prediction network can be used to process the non-transformed domain-invariant 3D representation to generate an output indicating an action prediction for the robot task. For example, the output can indicate (directly or indirectly) the end-effector pose.

[0010] As described above, in various embodiments, the domain invariant 3D representation may be optionally transformed into a frame of the end-effector pose being considered. For example, an initial domain invariant 3D representation may be generated by processing a 2.5D image (and optionally, a mask) using a trained shape prediction model. The initial domain invariant 3D representation may then be transformed into a frame of a candidate end-effector pose. The transformed domain invariant 3D representation may then be processed using a robot manipulation policy model to, for example, generate a prediction of manipulation success using a candidate end-effector pose. Therefore, in those various embodiments, the transformed domain invariant 3D representation may be processed using a robot manipulation policy without directly processing the candidate end-effector pose using the robot manipulation policy. More specifically, the candidate end-effector pose is reflected by the transformed domain invariant 3D representation of the frame transformed into the candidate end-effector pose. An embodiment using a transformed domain invariant 3D representation instead of a non-transformed initial domain invariant 3D representation and a separate representation of the candidate end-effector pose may be more efficiently trained and / or may be more robust and / or accurate during inference.

[0011] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of those embodiments and other embodiments are described in more detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example environment is shown in which implementations disclosed herein may be implemented.

[0013] Figure 2 An example of using a trained point cloud prediction network to generate a predicted 3D point cloud based on RGB-D images captured by a robot is shown. Figure 2 An example of using the predicted 3D point cloud in controlling a robot is also shown.

[0014] Figure 3A and Figure 3B A flowchart showing an example method for training a point cloud prediction network using simulated data and real-world data.

[0015] Figure 4 A flowchart showing an example method for training an evaluation network using predicted 3D point clouds.

[0016] Figure 5 A flow chart showing an example method of using a trained evaluation network and predicting a 3D point cloud in controlling a robot.

[0017] Figure 6 An example architecture of a robot is schematically depicted.

[0018] Figure 7 An example architecture of a computer system is schematically depicted. DETAILED DESCRIPTION

[0019] Some embodiments disclosed herein relate to training a single view that can be used to process an object. Figure 2 A point cloud prediction network (such as a machine learning model such as a neural network model) based on a one-half-dimensional (2.5D) observation to generate a domain-invariant three-dimensional (3D) representation of an object (e.g., a 3D point cloud of the object). In some of those embodiments, self-supervision is utilized in training the point cloud prediction network. Some embodiments may additionally or alternatively involve utilizing the domain-invariant 3D representation to train a robotic manipulation policy model (e.g., an evaluation network) using the domain-invariant 3D representation of an object to be manipulated as at least a portion of the input to the robotic manipulation policy model during training. Various embodiments may additionally or alternatively involve utilizing a trained robotic manipulation policy model in controlling a robot based on an output generated by processing the generated domain-invariant 3D representation using the trained robotic manipulation policy model.

[0020] The 3D point cloud of an object generated by processing 2.5D observations of the object using a point cloud prediction network can be domain-invariant, semantically interpretable, and directly applicable to object manipulation (e.g., grabbing, pushing, placing, and / or (multiple) other manipulations). Compared with other 3D representations (such as voxel grids and triangle meshes), the generated 3D point cloud can be lightweight and flexible. This can result in the use of fewer resources (e.g., memory resources) when the 3D point cloud is processed using a trained robot manipulation strategy in a control robot. In addition, the generated 3D point cloud can describe the complete 3D shape of the object, where the complete 3D shape is invariant to surface texture or environmental conditions. When the generated 3D point cloud is used in a training evaluation network, and the generated 3D point cloud and the supervisory signal used in the training policy network are at least partially based on simulated data, this can alleviate the reality gap. As used herein, the "reality gap" is the difference between a real robot and / or a real environment and a simulated robot and / or a simulated environment simulated by a robot simulator. Furthermore, the generated 3D point cloud can be directly used to localize objects in the scene, thus simplifying the task(s) when training the critic network or other policy networks.

[0021] Before turning to the figures, some specific non-limiting examples of training a point cloud prediction network and training a policy network as a grasp evaluation network are provided. As a specific example of training a point cloud prediction network, assume that the RGB-D observation set in is a separate simulated or real-world observation that captures the target object. The goal can be to learn a domain-invariant point cloud representation that reflects the 3D geometry of the target object. N RGB-D images can be obtained by using a mobile manipulator that moves around and takes snapshots of various workspaces from different angles (which is achieved in the simulated world by changing the vantage point of the simulated observation). The depth values ​​from the RGB-D observations form a 2.5D representation of the object (e.g., the visible parts affected by noise), and therefore do not provide the full 3D geometry. Furthermore, there is a realistic gap between the depth values ​​in simulation and the real world, thus making a policy trained only in simulation very ineffective in the real world.

[0022] Continuing with this particular example, self-supervised labeling of observations may arise. While target point clouds for supervised learning of deep networks can be easily obtained in simulation, this task becomes very expensive (e.g., in terms of computer resources) and time consuming for real-world data. Furthermore, the presence of noise and unmodeled nonlinear properties in real-world depth sensors makes learning more difficult, especially in the context of transfer learning. To address this challenge, embodiments disclosed herein use view-based supervision with a differential reprojection operator to generate self-supervised labels.

[0023] As an example, the object Point Cloud can be represented as a set of K points where x k , z k are respectively about the kth point p along the xyz axis k Without loss of generality, we can assume that the point cloud coordinates are defined in the camera frame. Furthermore, we can assume that the ground truth point cloud annotations are not directly available in the real world data, so multi-view projections can be used as supervisory signals (e.g., for both real world and simulated data). For example, the camera intrinsic matrix E can be used to extract the point cloud from the camera frame. Get the 2D projection in image space (For example, from (x k ,y k ,z k ) homogeneous coordinates of the projection (u k ,v k ,1)):

[0024]

[0025]

[0026] For localization, the corresponding tight bounding box can be obtained from the 2D projection: where u mid ,v mid , h denotes the bounding box center and size, respectively. By moving the mobile manipulator around the workspace, N RGB-D images / snapshots can be collected from various scenes in simulation and the real world. For real-world datasets, Mask-RCNN or other object detection networks can be used to detect the object bounding box B at each frame. n and its associated mask. For the simulated dataset, the bounding box and its associated mask can be obtained directly. Note that there may be multiple objects in many snapshots. The data associated with the mth object in the nth frame can be obtained by (.) m,n The number of objects in the nth frame can be represented by C n The mask of each object can be used to extract its associated depth value from the depth channel at each observation, and the camera intrinsic matrix E n Used to obtain the depth value This allows to determine that for all 1≤n≤N and of and B m,n .

[0027] The point cloud prediction network can be used to generate predicted point clouds which in turn can be used to determine using equations (1a) and (1b) (as described above) and The loss function used to train the point cloud prediction network can be defined as:

[0028]

[0029] where λ B ,λ M ,λ θ is the weighting coefficient, is the Huber loss between the estimated bounding box and the labeled bounding box, and is the projected point cloud prediction loss.

[0030] In some embodiments, the point cloud prediction network may include several encoder-decoder modules and fully connected layers to predict the point cloud. In some embodiments, the input channels of the point cloud prediction network may include (multiple) color channels (e.g., R, G, and B), depth channels, and object mask channels. In addition, the input channels may optionally be dynamic cropping of the base image (e.g., based on object detection), which is focused on the target object. In addition, in order to take into account dynamic cropping, additional inputs may optionally be provided downstream of the initial input as side inputs. For example, additional inputs may be provided immediately after the initial encoder to provide the point cloud prediction network with adaptive camera intrinsics generated from cropping. The additional inputs may be one or more camera intrinsics that define one or more intrinsic parameters of the camera that take into account cropping of the image.

[0031] We now provide a specific example of utilizing the predicted 3D point cloud in a policy network trained as a grasping policy network. The grasping policy network can be composed of Represents the evaluation network and can be used to predict point clouds based on and the transformation from the robot base to the camera frame to predict a sample grasp s∈R of the target object 4 A sample grasp s = (p, ψ) may be a candidate end-effector pose and may consist of a 3D gripper position p and a gripper yaw rotation ψ relative to the robot base.

[0032] The predicted point cloud can be First transform to the proposed grasping frame, thereby generating a transformed predicted point cloud. In some embodiments, the To perform the transformation, can be directly calculated based on the sample grasping posture s. The order of the points in the image is scrambled to allow the evaluation network to adapt to changes in the order of the point cloud. In some embodiments, the evaluation network can include multiple fully connected layers, each followed by a ReLU activation function with BatchNorm. The last layer of the evaluation network can optionally be linear and reduce the output size to one. It can be followed by a sigmoid activation function to provide a grasp success (e.g., a probability measure from 0 to 1).

[0033] In various embodiments, a majority, a vast majority (e.g., 90% or more), or all of the training data used to train the grasping strategy can be generated in simulation. As an example, a heuristic grasping strategy for data collection in simulation can include: (1) calculating the volume center of the target object based on the predicted point cloud; (2) Set the translation part of the grab posture to Add some random noise ε∈R 3 ,Right now and (3) randomly draw a yaw angle from a uniform distribution in the range [-π / 2,π / 2]. Grasping success can then be evaluated by moving the simulated end-effector to the pre-grasp pose s*, where s* is a pose offset from s by some height difference constant (e.g., above), i.e., s * -s=(0,0,δh,0). This pre-grasp pose s* enables the simulated robot end effector to be aligned relative to the simulated object before attempting a simulated grasp. The simulated robot is then moved to pose s and commanded to attempt a grasp (e.g., closing its parallel claw gripper when the end effector is a parallel claw gripper). The simulated robot is then commanded to lift the object (e.g., by moving back to s*). The grasp success is then evaluated by checking whether the simulated object has moved above its original pose. This evaluation can be easily done in simulation because there is access to the ground truth object pose in simulation. Training data can be collected by running the simulated robots in parallel and stored for training the off-policy grasping network.

[0034] Now turning to the attached figure, Figure 1 An example environment is shown in which implementations disclosed herein may be implemented. Figure 1A point cloud training system 140 is included, which is implemented by one or more computer systems. The point cloud training system 140 interfaces with one or more simulators 120, one or more robots (e.g., robot 190), and / or one or more human-held cameras in obtaining RGB-D images of various environments with various objects. For example, the robot 190 can partially or completely move around the table 250 while directing its vision component 199 (e.g., RGB-D camera) toward the table 250 to generate RGB-D images from the vision component 199. Other camera trajectories can also be captured with different environments, different objects and / or object arrangements, and optionally using different vision components. The RGB-D images can be generated additionally or alternatively using a simulator 120 with simulated environments and objects. In addition, RGB-D images from a human-held vision component can be utilized additionally or alternatively.

[0035] The point cloud training system 140 generates self-supervised training data using RGB-D images, and trains the point cloud prediction network 170 based on such training data. In various embodiments, the point cloud training system 140 may perform Figure 3A and Figure 3B One or more (e.g., all) aspects of method 300.

[0036] Point cloud training system 140 Figure 1 14 is shown as including an object mask engine 142, a ground truth depth engine 144, a predicted 3D point cloud engine 146, a 3D point cloud projection engine 148, and a loss engine 149. In other embodiments, fewer or more engines may be provided, and / or one or more aspects of one or more engines may be combined. An embodiment of the illustrated engine is now described with reference to an example of generating training data and training based on such an example. However, please note that the training of the point cloud prediction network will be based on thousands of instances of training data, and batch training techniques may optionally be utilized.

[0037] When generating an instance of self-supervised training data based on an RGB-D image (e.g., a rendered image from a simulation, or a real-world image), the object mask engine 142 generates an object mask of an object captured in the RGB-D image. For example, in the case where the RGB-D image is a real-world image, the object mask engine 142 can use the object detection network 172 to detect (multiple) object bounding boxes of (multiple) objects in the RGB-D image and associated masks for each of (multiple) objects. For example, the object detection network 172 can be a mask-RCNN network or other training network. Similarly, for example, in the case where the RGB-D image is a simulated image, (multiple) bounding boxes and (multiple) associated masks can be obtained directly from the simulated data. In the case where multiple objects are captured in the RGB-D image, one of the objects can be selected for use in the instance of generating self-supervised training data. Optionally, other objects in the multiple objects in the RGB-D image can each be used in a corresponding additional instance of generating self-supervised training data. In other words, multiple training instances can be generated based on an RGB-D image including multiple objects, each of which is used for a corresponding single object in the object.

[0038] In an instance where self-supervised training data is generated based on an RGB-D image, the ground truth depth engine 144 can generate a ground truth depth image based on an object mask and a depth channel of the RGB-D image. For example, the ground truth depth engine 144 can generate a depth image to include depth values ​​corresponding to those pixels from the depth channel for those pixels in an object mask (generated by the object mask engine 142) representing an object. Other pixels of the ground truth depth image can be zero or other null values. Optionally, the ground truth depth image can be limited to those pixels included in the bounding box determined by the object mask engine 142 and / or can include (e.g., in an additional channel) a bounding box determined by the object mask engine 142.

[0039] An example of self-supervised training data may include a training example input of an RGB-D image (or at least a crop of those pixels included in the generated bounding box), an object mask, and optionally camera intrinsics that take into account the crop of the RGB-D image. An example of self-supervised training data may include a training example output of a ground truth depth image.

[0040] The predicted 3D point cloud engine 146 can use the point cloud prediction network 170 to process the training instance input to generate a predicted 3D point cloud of the object. For example, the predicted 3D point cloud engine 146 can apply the RGB-D image (or at least a crop) and the object mask to the (multiple) initial layers of the point cloud prediction network 170, and apply the camera intrinsic parameters as side inputs downstream of the (multiple) initial layers (e.g., following the (multiple) initial encoding layers). The predicted 3D point cloud engine 146 can use the point cloud prediction network 170 to generate a predicted 3D point cloud based on the applied input and using the current weights of the point cloud prediction network 170.

[0041] The 3D point cloud projection engine 148 generates a projection of the predicted 3D point cloud generated by the predicted 3D point cloud engine 146. The generated projection can be a predicted depth image of the simulated object and can be generated based on the predicted 3D point cloud. For example, the 3D point cloud projection engine 148 can use camera intrinsic parameters to generate a projection to obtain a 2D projection in image space from the point cloud. For example, equations (1a) and (1b) (as described above) can be used. The 3D point cloud projection engine 148 can also optionally generate a bounding box that can be obtained from the 2D projection.

[0042] The loss engine 149 generates a loss based at least in part on a comparison of a projection of the predicted 3D point cloud (generated by engine 148) and a ground truth depth image (generated by engine 144). The loss engine 149 may further generate a loss based on a comparison of a ground truth bounding box optionally generated by engine 144 and a predicted bounding box optionally generated by engine 148. As an example, the loss engine 149 may generate a loss based on equation (2) (as described above). Note that in various embodiments, the loss engine 149 may generate a batch loss based on multiple instances of training data in a batch. The loss engine 149 then updates one or more weights of the point cloud prediction network 170 based on the generated loss. For example, the loss engine 149 may back-propagate the loss to update the weights of the point cloud prediction network 170.

[0043] Once the point cloud prediction network 170 is trained, a real-world robot (e.g., robot 190) can utilize it in generating predicted 3D point clouds based on RGB-D images captured by the robot. In addition, the predicted 3D point clouds can be utilized by the robot in controlling one or more of its actuators. In some embodiments, the trained point cloud prediction network 170 is utilized in a training evaluation network or other robot policy network to process the predicted 3D point clouds (generated using the trained point cloud prediction network 170) to generate corresponding outputs that are utilized in controlling the robot. Now regarding Figure 1 The assessment training system 150 describes one non-limiting example of such training, which is implemented by one or more computer systems.

[0044] The evaluation training system 150 interfaces with the trained point cloud prediction network 170, one or more simulators 120, and optionally one or more robots (e.g., robot 190) in the training evaluation network 174. In various embodiments, the evaluation training system 150 may perform Figure 4 One or more (e.g., all) aspects of method 400.

[0045] Evaluation training system 150 Figure 1 154, a predicted 3D point cloud transformation engine 156, and a loss engine 159. In other embodiments, fewer or more engines may be provided, and / or one or more aspects of one or more engines may be combined. The embodiments of the illustrated engines are now described with reference to examples of generating training data and training based on such examples. However, it should be noted that the training of the evaluation network 174 network will be based on thousands of instances of training data, and batch training techniques may optionally be utilized.

[0046] In an instance of generating training data, a simulator 120 simulating the environment, object(s), and robot may be used to render a simulated RGB-D image. The RGB-D image may capture at least one of the object(s) in the simulated environment. The object mask engine 152 may generate a mask of the target object using techniques such as those described for the object mask engine 142 (of the point cloud training system 140). When the RGB-D image is a rendered simulated image, the mask may be generated using data from the simulator 120 or, optionally, using an object detection network 172. When it is a real-world image, the object mask engine 152 may optionally utilize an object detection network 172 in generating the object mask. The object mask engine 152 may also optionally generate a bounding box for the object.

[0047] In an example of generating training data, the predicted 3D point cloud engine 154 can process the RGB-D image (or at least a crop based on the bounding box generated by the object mask engine 152) and the object mask to the (multiple) initial layers of the point cloud prediction network 170, and can optionally apply the camera intrinsic parameters (of the camera that captured the RGB-D image) as a side input downstream of the (multiple) initial layers (e.g., following the (multiple) initial encoding layers). The predicted 3D point cloud engine 146 can use the point cloud prediction network 170 to generate a predicted 3D point cloud based on the applied inputs and using the trained weights of the point cloud prediction network 170.

[0048] The predicted 3D point cloud transformation engine 156 can generate a transformation of the predicted 3D point cloud using the end-effector pose of the manipulation attempt to be used for the training instance. In other words, the predicted 3D point cloud transformation engine 156 can transform the 3D point cloud into an end-effector frame relative to the end-effector pose to be used for the manipulation attempt. For example, the 3D point cloud transformation engine 156 can generate the transformation using equation (3) (as described above) and the end-effector pose.

[0049] The evaluation engine 158 can use the evaluation network to process the transformation of the predicted 3D point cloud to generate a prediction of a successful manipulation. The manipulation can then be attempted, for example, by a simulated robot of the simulator 120 using an end-effector pose. Data from the simulator 120 can then be used to determine whether the manipulation was successful, as well as a ground truth metric of the successful manipulation determined based on the determination (e.g., "1" if successful, "0" if unsuccessful). The loss engine 159 can then generate a loss based on comparing the prediction of the successful manipulation with the ground truth metric of the successful manipulation. The loss engine 159 then updates one or more weights of the evaluation network 174 based on the generated loss. For example, the loss engine 159 can back-propagate the loss to update the weights of the evaluation network 174.

[0050] Example robot 190 in Figure 1 1 is shown as an example of a robot that can utilize the point cloud prediction network 170 and / or the evaluation network 174 in performing various robotic tasks when trained. Robot 190 is also an example of a robot that can be used to generate data for training the point cloud prediction network 170 and / or the evaluation network 174.

[0051] The robot 190 includes a robot arm 194 having a gripping end effector 196, wherein the gripping end effector 196 takes the form of a gripper having two opposing actuatable members. The robot 190 also includes a base 193 having wheels 197a and 197b provided on opposite sides thereof for movement of the robot 190. The base 193 may include, for example, one or more motors for driving corresponding wheels 197a and 197b to achieve a desired direction, speed and / or acceleration of movement of the robot 190.

[0052] The robot 190 also includes a vision component 199. The vision component 199 generates images related to the shape, color, depth, and / or other characteristics of the object(s) within the line of sight of its sensor(s). The vision component 199 can be, for example, a passive or active stereo camera that generates the RGB-D images described herein. Figure 1A particular robot 190 is shown in the drawings, but additional and / or alternative robots may be utilized, including additional robotic arms similar to robot 190, robots in the form of robotic arms, robots in humanoid form, robots in animal form, other robots that move via one or more wheels, unmanned aerial vehicles (UAVs), etc. Figure 1 Specific end effectors are shown in FIG. 1 , but additional and / or alternative end effectors may be utilized to interact with an object.

[0053] The simulator(s) 120 are implemented by one or more computer systems and can be used to simulate various environments including corresponding environmental objects and corresponding placement locations. Various simulators can be utilized, such as physics engines that simulate collision detection, soft and rigid body dynamics, etc. A non-limiting example of such a simulator is the BULLET physics engine.

[0054] Referring now to 2, there is shown a method of using a trained point cloud prediction network 170 and based on a point cloud prediction algorithm generated by a robot (e.g., Figure 1 An example of generating a predicted 3D point cloud 201 using an RGB-D image 201 captured by a robot 190). Figure 2 Further shown is an example of using the predicted 3D point cloud 201 in controlling a robot. Figure 2 Can involve execution Figure 5 One or more (e.g., all) of the blocks of method 500.

[0055] exist Figure 2 , an RGB-D image 201 is processed by a mask engine 142 using an object detection network 172 to generate a mask of an object 202 captured by the RGB-D image 201. The mask of the object 202 is applied as an initial input to the point cloud prediction network 170 along with the color channels 203 and depth channels 204 of the RGB-D image (e.g., at least those portions corresponding to the bounding box of the object, optionally determined by the object mask engine 142). The illustrated example of the point cloud prediction network includes two pairs of encoder-decoders (170A1 and 170A2; and 170B1 and 170B2). It may also optionally include additional or alternative layers not shown. Camera intrinsics 205 may also optionally be applied as a side input 205 as shown (e.g., after encoder 170A1). A predicted 3D point cloud 206 is generated based on processing the input using the point cloud prediction network 170.

[0056] The predicted 3D point cloud transformation engine 148 generates a transformation 208 of the 3D point cloud that is based on a transformation of the predicted 3D point cloud 206 considering the candidate end-effector pose 207. The evaluation engine 148 can generate a prediction of a successful manipulation based on processing the transformation 208 of the 3D point cloud using the evaluation network 174. If the candidate end-effector pose 207 used to generate the transformation 208 of the 3D point cloud is utilized in the attempted manipulation, the prediction of a successful manipulation indicates a measure of a successful manipulation. Based on processing the transformation of the corresponding 3D point cloud, the evaluation engine 148 can generate a prediction of a successful manipulation for each of a plurality of candidate end-effector poses. For example, the evaluation engine 148 can utilize a cross-entropy method (CEM) or other technique in sampling the plurality of candidate end-effector poses 207, and can select one with the best prediction of a successful manipulation as the selected end-effector pose 209. The selected end-effector pose 209 may then be used to control the robot's actuators to cause the robot's end-effector to move to the selected end-effector pose 209 and then attempt manipulation of the object.

[0057] Reference now Figure 3A and Figure 3B , describes an example method 300 for training a point cloud prediction network using simulated data and real data. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of one or more computing devices (such as point cloud training system 140). In addition, although the operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0058] At block 352 , the system renders a simulated image that includes color channel(s) and depth channel(s) and that captures simulated object(s) of a simulated environment of the simulator.

[0059] At block 354, the system generates an object mask of the simulated object in the simulated object(s) captured by the simulated image. The system may also optionally generate a bounding box of the simulated object in the simulated image.

[0060] At block 356 , the system generates a ground truth depth image of the simulated object based on the object mask and the depth channel.

[0061] At block 358, the system generates a predicted 3D point cloud of the simulated object based on processing the object mask and (multiple) channels (e.g., including (multiple) color channels and depth channels) of at least a portion of the simulated image using the point cloud prediction network. For example, at least a portion of the simulated image that captures the simulated object can be processed. For example, a cropped portion can be processed, where the cropping corresponds to a bounding box of the simulated object. In some embodiments, in generating the predicted 3D point cloud, the system also uses the point cloud prediction network to process camera intrinsics.

[0062] At block 360 , the system generates a projection of the predicted 3D point cloud of block 358 .

[0063] At block 362, the system generates a loss based on a comparison of the projection of the predicted 3D point cloud (block 360) and the ground truth depth image of the simulated object (block 356). As described herein, in various embodiments, the loss can be generated based on a batch of training examples (e.g., based on a comparison of multiple projections and ground truth depth images).

[0064] At box 364, the system updates the weight(s) of the point cloud prediction network based on the generated loss.

[0065] At block 366, the system determines whether to perform further training of the point cloud prediction network. In some embodiments, the system may determine whether to perform further training based on whether there are any unprocessed training instances and / or whether other training criteria have not been met. Other training criteria may include a threshold number of training epochs, a threshold duration of training time, and / or other criteria / criteria.

[0066] If the system determines at an iteration of block 366 that the point cloud prediction network does not require further training, the system may consider the point cloud prediction network to have completed training and may optionally continue to block 368 .

[0067] At block 368, the system may train the robotic manipulation policy network using the point cloud predicted using the trained point cloud prediction network. For example, the system may perform Figure 4 Method 400 of training an evaluation network using point clouds predicted using a trained point cloud prediction network.

[0068] If the system determines at the iteration of block 366 that further training is needed, the system may return to block 352 or continue to Figure 3B Box 372. Figure 3B The point cloud prediction network is trained based on real-world data. Figure 3A In some embodiments, the system may train the point cloud prediction network based only on the simulated data (without performing Figure 3BIn some embodiments, the system may train the point cloud prediction network primarily based on simulated data, but may also train on some simulated data. In those embodiments, the system may continue to block 372 in response to at least some of the yes determinations at block 366. For example, the system may continue to block 372 at least intermittently.

[0069] Reference now Figure 3B At block 372, the system captures a real image that includes color channel(s) and depth channel(s) and captures real object(s) of the real environment. The system may also optionally generate a bounding box for the simulated object.

[0070] At block 374, the system uses the object detection network to generate an object mask of the real object in the real object(s). The system may also optionally generate a bounding box of the real object in the real image.

[0071] At box 376, the system generates a ground truth depth image of the real object based on the object mask and the depth channel.

[0072] At box 378, the system generates a predicted 3D point cloud of the real object based on processing the object mask and (multiple) channels (e.g., including (multiple) color channels and / or depth channels) of at least a portion of the real image using the point cloud prediction network. For example, at least a portion of the real image that captures the real object can be processed. For example, a cropped portion can be processed, where the cropping corresponds to a bounding box of the real object. In some embodiments, in generating the predicted 3D point cloud, the system also uses the point cloud prediction network to process camera intrinsic parameters.

[0073] At block 380 , the system generates a projection of the predicted 3D point cloud of block 378 .

[0074] At box 382, ​​the system generates a loss based on a comparison of the projection of the predicted 3D point cloud (380) and the ground truth depth image (376) of the real object.

[0075] At block 384, the system updates the weights of the point cloud prediction network based on the generated loss, and the system returns to Figure 3A Block 366 of determines whether the point cloud prediction network requires any further training. As described herein, in various embodiments, a loss can be generated based on a batch of training examples (e.g., based on comparison of multiple projections and ground truth depth images).

[0076] Reference now Figure 4, an example method 400 for training an evaluation network using a predicted 3D point cloud is described. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include various components of one or more computing devices (such as the evaluation training system 150). In addition, although the operations of the method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0077] At block 452, the system identifies an image that includes color channel(s) and depth channel(s) and captures object(s) of an environment. The image may be a real image and the environment may be a real environment, or the image may be a rendered simulated image and the environment may be a simulated environment.

[0078] At block 454, the system generates an object mask for the object in the object(s). The system may also optionally generate a bounding box for the object in the image.

[0079] At block 456, the system generates a predicted 3D point cloud for the object based on processing the object mask and (multiple) channels (e.g., including (multiple) color channels and / or depth channels) of at least a portion of the image using the trained point cloud prediction network. For example, at least a portion of the image capturing the object can be processed. For example, a cropped portion can be processed, where the cropping corresponds to a bounding box of the object. In some embodiments, in generating the predicted 3D point cloud, the system also processes camera intrinsics using the point cloud prediction network.

[0080] At block 458, the system identifies an end-effector gesture used in a manipulation attempt of the object. The manipulation attempt may be a real-world manipulation attempt when the image is a real image, or a simulated manipulation attempt when the image is a simulated image.

[0081] At block 460 , the system generates a transformation of the predicted 3D point cloud based on the end-effector pose.

[0082] At block 462 , the system generates a prediction of a successful manipulation of the object based on processing the transformation of the predicted 3D point cloud using the critique network.

[0083] At block 464, the system generates a loss based on a comparison of the prediction of a successful maneuver and a ground truth metric of a successful maneuver. The ground truth metric of a successful maneuver is based on whether the maneuver attempt was successful. For example, following a maneuver attempt, the success of the simulated maneuver attempt may be determined based on data from a simulator.

[0084] At block 466, the system updates the weights of the evaluation network based on the generated loss. As described herein, in various embodiments, the loss can be generated based on a batch of training examples (e.g., based on a comparison of a plurality of predictions of successful maneuvers and a ground truth metric of successful maneuvers).

[0085] At block 468, the system determines whether to perform further training of the evaluation network. In some embodiments, the system may determine whether to perform further training based on whether there are any unprocessed training instances and / or whether other training criteria have not been met. Other training criteria may include a threshold number of training epochs, a threshold duration of training time, and / or other criteria / standards.

[0086] If the system determines at the iteration of block 468 that further training of the evaluation network should be performed, the system may return to block 454 .

[0087] If the system determines at the iteration of block 468 that the evaluation network does not require further training, the system may continue to block 470 .

[0088] At block 470, the system uses the evaluation network in robot control. Figure 5 In the method 500, the system can use the evaluation network in controlling a robot.

[0089] Reference now Figure 5 , describes an example method 500 for using a trained evaluation network and predicting a 3D point cloud in controlling a robot. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. The system may include a robot (e.g., Figure 1 Robot 190, Figure 6 The robot 600 and / or (multiple) other robots) and / or one or more computing devices in communication with the robot (e.g., Figure 7 710). In addition, although the operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0090] At block 552, the system identifies an image that includes color channel(s) and depth channel(s) and captures object(s) of the environment. The image may be captured by, for example, an RGB-D camera of the robot.

[0091] At block 554, the system generates an object mask for the object in the object(s). The system may also optionally generate a bounding box for the object in the image.

[0092] At block 556, the system generates a predicted 3D point cloud for the object based on processing the object mask and (multiple) channels (e.g., including (multiple) color channels and / or depth channels) of at least a portion of the image using the point cloud prediction network. For example, at least a portion of an image capturing the object can be processed. For example, a cropped portion can be processed, where the cropping corresponds to a bounding box of the object. In some embodiments, in generating the predicted 3D point cloud, the system also processes camera intrinsics using the point cloud prediction network.

[0093] At block 558 , the system identifies candidate end-effector poses.

[0094] At block 560 , the system generates a transformation of the predicted 3D point cloud based on the candidate end-effector poses.

[0095] At block 562 , the system generates a prediction of a successful manipulation of the object based on processing the transformation of the predicted 3D point cloud using the critique network.

[0096] At block 564, the system determines whether more (multiple) candidate end-effector poses should be considered. For example, the system can utilize CEM or other techniques to consider multiple end-effector poses. For example, in a first iteration of CEM, N candidate end-effector poses can be randomly sampled and considered, and then, in a next iteration of CEM, additional candidate end-effector poses can be sampled from the vicinity of the candidate end-effector pose of the first iteration that has the best prediction for a successful grasp. Additional iterations of CEM can optionally be performed.

[0097] If the system determines at iteration 564 that there are more candidate end-effector pose(s), the system may return to block 558 to identify additional candidate end-effector poses.

[0098] If the system determines at the iteration of block 564 that there are no more candidate end-effector pose(s), the system may continue to block 566 .

[0099] At block 566, the system selects a given candidate end-effector pose based on the prediction of successful manipulation(s), and controls the robot's actuator(s) to achieve the given candidate end-effector pose (i.e., move the end-effector to the pose) and then attempts a manipulation of the object. When the evaluation network is trained for grasp attempts, the manipulation can be, for example, a grasp attempt.

[0100] Figure 6 An example architecture of a robot 600 is schematically depicted. Figure 1 The robot 190 can implement Figure 6The robot 600 includes a robot control system 660, one or more operating components 604a-604n, and one or more sensors 608a-608m. The sensors 608a-608m may include, for example, visual sensors (e.g., (multiple) cameras, 3D scanners), light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and the like. Although the sensors 608a-608m are depicted as being integrated with the robot 600, this is not meant to be limiting. In some embodiments, the sensors 608a-608m may be located external to the robot 600, for example as a stand-alone unit.

[0101] The operating components 604a-604n may include, for example, one or more end effectors (e.g., gripping end effectors) and / or one or more servo motors or other actuators to accomplish movement of one or more components of the robot. For example, the robot 600 may have multiple degrees of freedom, and each of the actuators may control actuation of the robot 600 in one or more of the degrees of freedom in response to a control command. As used herein, the term actuator includes a mechanical device or electrical device (e.g., a motor) that produces motion, in addition to any (multiple) drivers that may be associated with the actuator and convert a received control command into one or more signals for driving the actuator. Thus, providing a control command to an actuator may include providing a control command to a driver, wherein the driver converts the control command into an appropriate signal for driving the electrical device or mechanical device to produce the desired motion.

[0102] The control system 602 may be implemented in one or more processors (such as a CPU, GPU) and / or other controller(s) of the robot 600. In some implementations, the robot 600 may include a "brain box" that may include all or aspects of the control system 602. For example, the brain box may provide real-time bursts of data to the operating components 604a-n, wherein each of the real-time bursts includes a set of one or more control commands, wherein the one or more control commands indicate, among other things, parameters of the motion (if any) of each of the one or more operating components 604a-n.

[0103] Although the control system 602 Figure 6600, but in some embodiments, all or aspects of the control system 602 may be implemented in a component separate from the robot 600 but in communication therewith. For example, all or aspects of the control system 602 may be implemented on one or more computing devices (such as computing device 710) that communicate with the robot 600 in wired and / or wireless communication. In some embodiments, all or aspects of the control commands generated by the control system in performing a robot task may be action-based, the action being selected based on utilization of a trained robot manipulation policy model as described herein. For example, in selecting an action, a shape prediction network may be utilized to transform 2.5D visual data from a camera of the robot into a domain-invariant 3D representation, and the domain-invariant 3D representation may be processed using a trained robot manipulation policy model.

[0104] Figure 7 7 is a block diagram of an example computing device 710 that may optionally be used to perform one or more aspects of the techniques described herein. The computing device 710 typically includes at least one processor 714 that communicates with a number of peripheral devices via a bus subsystem 712. These peripheral devices may include a storage subsystem 724 that includes, for example, a memory subsystem 725 and a file storage subsystem 726, a user interface output device 720, a user interface input device 722, and a network interface subsystem 716. The input and output devices allow for user interaction with the computing device 710. The network interface subsystem 716 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0105] User interface input devices 722 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into computing device 710 or over a communication network.

[0106] User interface output device 720 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visual image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from computing device 710 to a user or to another machine or computing device.

[0107] Storage subsystem 724 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, storage subsystem 724 may include logic to perform selected aspects of the methods described herein.

[0108] These software modules are generally executed by the processor 714 alone or in combination with other processors. The memory 725 used in the storage subsystem 724 may include multiple memories, including a main random access memory (RAM) 730 for storing instructions and data during program execution and a read-only memory (ROM) 732 for storing fixed instructions. The file storage subsystem 726 may provide permanent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media box. Modules implementing the functions of certain embodiments may be stored by the file storage subsystem 726 in the storage subsystem 724, or in other machines accessible to the processor (s) 714.

[0109] The bus subsystem 712 provides a mechanism for the various components and subsystems of the computing device 710 to communicate with each other as intended. Although the bus subsystem 712 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0110] The computing device 710 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, for the purpose of illustrating some embodiments, Figure 7 The description of computing device 710 depicted in FIG. 7 is intended only as a specific example. Many other configurations of computing device 710 may have more Figure 7 The computing devices depicted may have greater or fewer components.

[0111] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing the functions described herein and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or multiple specific applications to which the teachings are applied. Using only routine experiments, those skilled in the art will recognize or be able to determine many equivalents of the specific embodiments described herein. Therefore, it should be understood that the foregoing embodiments are presented only by way of example, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

[0112] In some embodiments, a method implemented by one or more processors of a robot is provided, the method comprising identifying a 2.5D image of an object captured by a camera of the robot and capturing the robot's environment. The method also includes processing at least the 2.5D image using a shape prediction network to generate a domain-invariant 3D representation of the object. The method also includes using a robot manipulation policy network to determine an action to be applied to the robot to process the domain-invariant 3D representation of the object or a transformation of the domain-invariant 3D representation. The method also includes providing a control command to one or more actuators of the robot so that the action is performed by the robot.

[0113] These and other implementations of the technology may include one or more of the following features.

[0114] In some implementations, the domain-invariant 3D representation of the object is a 3D point cloud of the object.

[0115] In some embodiments, the method further comprises processing the 2.5D image to generate a mask indicating which pixels of the 2.5D image comprise the object. These embodiments further comprise processing the mask and the 2.5D image in generating a domain invariant 3D representation of the object using the shape prediction network.

[0116] In some embodiments, the method includes processing a transformation of a domain-invariant 3D representation of an object in determining an action using a robot manipulation policy network. In some versions of those embodiments, the transformation is generated by transforming the domain-invariant 3D representation into a frame of the robot end-effector using a robot end-effector pose. In some additional or alternative versions, generating a metric (e.g., probability) that the robot end-effector pose will result in successful manipulation of the object using processing of the robot manipulation policy network. In some of those additional or alternative versions, determining an action includes determining an attempted manipulation with the robot end-effector pose based on a metric that satisfies a threshold. The manipulation of the object can be, for example, a grasping of the object.

[0117] In some embodiments, a method implemented by one or more processors of a robot is provided, the method comprising identifying an image captured by a camera of the robot. The image captures an object to be manipulated by the robot, and the image comprises a plurality of channels, wherein the plurality of channels comprises one or more color channels and a depth channel. The method further comprises generating an object mask of the object to be manipulated by the robot. Generating the object mask may comprise processing one or more of the channels of the image using an object detection network. The method further comprises generating a three-dimensional (3D) point cloud of the object. Generating the 3D point cloud of the object may comprise processing, using a point cloud prediction network: all channels of at least a portion of the image, and the generated object mask of the object. The method further comprises using the generated 3D point cloud in controlling one or more actuators of the robot.

[0118] These and other implementations of the technology may include one or more of the following features.

[0119] In some embodiments, an object detection network is trained to process an image to generate a bounding box and a mask of an object in the image. In some of those embodiments, the method further includes generating a bounding box of an object to be manipulated by the robot. Generating the bounding box may include processing one or more of the channels of the image using the object detection network. In some versions of those embodiments, at least a portion of the image processed using the point cloud prediction network is a crop of the image selected based on the bounding box. For example, the method may also include selecting a crop of the image based on pixels of the image included in the crop being surrounded by the bounding box. In some versions where the at least a portion is a crop, generating a 3D point cloud of the object may also include processing one or more camera intrinsic parameters using the point cloud prediction network, the one or more camera intrinsic parameters defining one or more intrinsic parameters of a camera that considers the crop of the image. In some of those versions, in processing the one or more camera intrinsic parameters using the point cloud prediction network, the one or more camera intrinsic parameters are applied as side inputs to the point cloud prediction network downstream of the initial inputs of at least a portion of the image and the generated object mask being applied as initial inputs. For example, all channels of at least a portion of an image and a generated object mask of an object may be initially processed using an initial encoder of a point cloud prediction network, and camera intrinsics may be applied as side inputs after the initial encoder and before processing using an initial decoder of the point cloud prediction network.

[0120] In some embodiments, using the generated 3D point cloud in controlling one or more actuators of the robot includes generating a prediction of successful manipulation of the object based on the 3D point cloud of the object or a transformation of the 3D point cloud, and controlling the one or more actuators of the robot based on the prediction of successful manipulation. In some versions of those embodiments, generating the prediction of successful manipulation of the object includes generating the prediction of successful manipulation by processing the generated 3D point cloud of the object or the transformation of the 3D point cloud using a critique network. In some of those versions, generating the prediction of successful manipulation includes: identifying a candidate end-effector pose of an end-effector of the robot; generating a transformation of the 3D point cloud by transforming the 3D point cloud into an end-effector frame relative to the end-effector pose; and generating the prediction of successful manipulation by processing the transformation of the 3D point cloud using the critique network. In addition, in some of those versions, controlling the one or more actuators of the robot based on the prediction of successful manipulation may include: selecting a candidate end-effector pose based on the prediction of successful manipulation satisfying at least one criterion; and in response to selecting the candidate end-effector pose: controlling the one or more actuators of the robot so that the end-effector moves to the candidate end-effector pose. Optionally, the manipulation is a grasp, and controlling one or more actuators of the robot based on the prediction of a successful manipulation further comprises: causing the end effector to attempt to grasp the object after the end effector is in the candidate end effector pose. Additionally, in some versions, the method further comprises: identifying additional candidate end effector poses for the end effector; generating additional transformations of the 3D point cloud by transforming the 3D point cloud into additional end effector frames relative to the additional end effector poses; and generating additional predictions of successful manipulation by processing the additional transformations of the 3D point cloud using the evaluation network. At least one criterion utilized in selecting the candidate end effector pose based on the prediction of a successful manipulation may include the prediction of a successful manipulation being more indicative of success than the additional predictions of a successful manipulation.

[0121] In some implementations, the point cloud prediction network includes a plurality of encoder-decoder modules, and at least one fully connected layer.

[0122] In some embodiments, a method for training a point cloud prediction network is provided, the method being implemented by one or more processors. The method includes rendering a simulated image of a simulated environment of a simulator. The simulated image captures at least one simulated object of the simulated environment, and the simulated image includes a plurality of channels, wherein the plurality of channels include one or more color channels and a depth channel. The method also includes: generating an object mask of the simulated object; and generating a ground truth depth image of the object based on the object mask and the depth channel of the simulated image. The method also includes generating a predicted three-dimensional (3D) point cloud of the simulated object. Generating a predicted 3D point cloud of the simulated object includes processing using a point cloud prediction network: all channels of at least a portion of the image, and the object mask of the generated simulated object. The method also includes generating a projection of the predicted 3D point cloud. The projection is a predicted depth image of the simulated object based on the predicted 3D point cloud. The method also includes generating a loss based at least in part on a comparison of the following: a projection of the predicted 3D point cloud, and a ground truth depth image of the simulated object. The method also includes updating one or more weights of the point cloud prediction network based at least in part on the generated loss.

[0123] These and other implementations of the technology may include one or more of the following features.

[0124] In various embodiments, using a comparison of projections and ground truth depth images in determining a loss provides one or more advantages. For example, training on such a loss in simulation can alleviate the reality gap of a point cloud prediction network relative to, for example, alternatively utilizing a comparison of the predicted 3D point cloud itself with a ground truth 3D point cloud (obtained directly from simulation). Similarly, for example, such a loss can also be effectively utilized in training a point cloud prediction network based on real-world data. For example, real-world ground truth depth images can be efficiently generated. On the other hand, obtaining ground truth 3D point cloud data in the real world can be time consuming and / or computationally expensive.

[0125] In some implementations, generating a projection of the 3D point cloud includes generating a projection of the predicted 3D point cloud using intrinsic parameters of a simulated camera used to render the simulated image.

[0126] In some embodiments, the method further comprises determining a bounding box of the simulated object. In some of those embodiments, at least a portion of the image processed using the point cloud prediction network is a crop of the image selected based on the bounding box. For example, a crop of the image can be selected based on pixels of the image included in the crop being surrounded by a bounding box.

[0127] In some embodiments, the method further includes capturing a real image of the real environment. The real image captures at least one real object, and the real image includes multiple channels, wherein the multiple channels include one or more color channels and a depth channel. In those embodiments, the method further includes generating an additional object mask of the real object. Generating the additional object mask includes processing one or more channels of the real image using an object detection network. In those embodiments, the method further includes: generating an additional ground truth depth image of the real object based on the additional object mask and the depth channel of the real image; and generating an additional predicted three-dimensional (3D) point cloud of the real object. Generating an additional predicted 3D point cloud of the real object includes processing using a point cloud prediction network: all channels of at least a portion of the real image, and the generated object mask of the real object. In those embodiments, the method further includes: generating additional projections of the additional predicted 3D point cloud; generating additional losses based at least in part on comparisons of: projections of the additional predicted 3D point cloud, and additional ground truth depth images of the real object; and updating one or more weights of the point cloud prediction network based at least in part on the generated updated losses.

[0128] In some embodiments, the method further includes determining that the training of the point cloud prediction network satisfies one or more criteria. The one or more criteria may include, for example, a threshold amount of training epochs, a threshold duration of training, convergence of training, an evaluation indicating at least a threshold level of accuracy and / or robustness, and / or one or more of other criteria. The method may also include: in response to determining that the training of the point cloud prediction network satisfies one or more criteria: training an evaluation network using additional point clouds predicted using the trained point cloud prediction network.

[0129] Various embodiments disclosed herein may include a non-transitory computer-readable storage medium storing instructions that can be executed by one or more processors (e.g., (multiple) central processing units (CPUs), (multiple) graphics processing units (GPUs), and / or (multiple) tensor processing units (TPUs)) to perform a method (such as one or more of the methods described herein). Another embodiment may include a system of one or more computers and / or one or more robots including one or more processors that are operable to execute stored instructions to perform a method (such as one or more of the methods described herein).

Claims

1. A method implemented by one or more processors of a robot, the method comprising: identifying an image captured by a camera of a robot, the image capturing an object to be manipulated by the robot, and the image comprising a plurality of channels, wherein the plurality of channels comprises one or more color channels and a depth channel; Generating an object mask of an object to be manipulated by the robot, wherein generating the object mask comprises: Processing one or more of the channels of the image using an object detection network; Generating a three-dimensional (3D) point cloud of the object, wherein generating the 3D point cloud of the object comprises: Use the point cloud prediction network to process: all channels of at least a portion of the image, and the generated object mask of the object; Generating a prediction of successful manipulation of the object, wherein generating the prediction of successful manipulation of the object comprises: Generate a prediction of successful manipulation by processing the transformation of the 3D point cloud using the robot manipulation policy model, wherein generating the prediction of successful manipulation includes: sampling a candidate end-effector pose of the end-effector of the robot from a plurality of candidate end-effector poses; generating a transformation of the 3D point cloud by transforming the 3D point cloud into an end-effector frame relative to the candidate end-effector pose, and generating a prediction of successful manipulation by processing the transformation of the 3D point cloud using a robot manipulation policy model; and One or more actuators of the robot are controlled based on the prediction of a successful maneuver.

2. The method according to claim 1, wherein: The object detection network is trained to process an image to generate bounding boxes and masks of objects in the image, and further includes: Generating a bounding box of an object to be manipulated by the robot, wherein generating the bounding box comprises: One or more of the channels of the image are processed using the object detection network.

3. The method according to claim 2, wherein: At least a portion of the image processed using the point cloud prediction network is a crop of the image selected based on the bounding box. 4 . The method of claim 3 , further comprising selecting a crop of the image based on pixels of the image included in the crop being enclosed by the bounding box.

5. The method according to claim 3, wherein: Generating a 3D point cloud of the object further comprises: Use the point cloud prediction network to process: One or more camera intrinsic parameters, the one or more camera intrinsic parameters defining one or more intrinsic parameters of the camera that take into account cropping of the image.

6. The method according to claim 5, wherein: In processing the one or more camera intrinsic parameters using the point cloud prediction network, the one or more camera intrinsic parameters are applied as side inputs to the point cloud prediction network downstream of all channels of at least a portion of the image and the generated object mask of the object being applied as initial inputs.

7. The method according to claim 6, wherein: All channels of at least a portion of the image and the generated object mask of the object are initially processed using an initial encoder of the point cloud prediction network, and wherein the camera intrinsics are applied as side inputs after the initial encoder and before processing using an initial decoder of the point cloud prediction network.

8. The method according to claim 1, wherein: Controlling one or more actuators of the robot based on a prediction of a successful maneuver comprises: selecting the candidate end-effector pose based on a prediction of a successful manipulation satisfying at least one criterion; and In response to selecting the candidate end-effector pose: One or more actuators of the robot are controlled to cause the end effector to move to the candidate end effector pose.

9. The method according to claim 8, wherein: The manipulation is grasping, and wherein controlling one or more actuators of the robot based on a prediction of a successful manipulation further comprises: After the end-effector is in the candidate end-effector pose, causing the end-effector to attempt to grasp the object.

10. The method according to claim 8, further comprising: sampling an additional candidate end-effector pose for the end-effector from the plurality of candidate end-effector poses; generating an additional transformation of the 3D point cloud by transforming the 3D point cloud into an additional end-effector frame relative to the additional candidate end-effector pose; as well as generating additional predictions of successful manipulation by processing additional transformations of the 3D point cloud using the critique network; Wherein the at least one criterion utilized in selecting the candidate end-effector pose based on the prediction of a successful manipulation comprises the prediction of a successful manipulation being more indicative of success than an additional prediction of a successful manipulation.

11. The method according to any one of claims 1 to 7, wherein: The point cloud prediction network includes multiple encoder-decoder modules and at least one fully connected layer.

12. The method according to claim 1, wherein: controlling one or more actuators of the robot based on the prediction of successful manipulation and based on an additional prediction of successful manipulation of the object, and further comprising: Generating an additional prediction of successful manipulation of the object, wherein generating the additional prediction of successful manipulation of the object comprises: Generating an additional prediction of a successful manipulation by processing an additional transformation of the 3D point cloud using the robot manipulation policy model, wherein generating the additional prediction of a successful manipulation comprises: sampling an additional candidate end-effector pose from among a plurality of candidate end-effector poses of an end-effector of the robot; generating an additional transformation of the 3D point cloud by transforming the 3D point cloud into an additional end-effector frame relative to an additional candidate end-effector pose, and Additional predictions of successful manipulation are generated by processing additional transformations of the 3D point cloud using the robot manipulation policy model.

13. A method for training a point cloud prediction network, the method being implemented by one or more processors and comprising: rendering a simulated image of a simulated environment of the simulator, the simulated image capturing at least one simulated object of the simulated environment, and the simulated image comprising a plurality of channels, wherein the plurality of channels comprises one or more color channels and a depth channel; generating an object mask of the simulated object; generating a ground truth depth image of the object based on the object mask and the depth channel of the simulated image; Generate a predicted three-dimensional (3D) point cloud of the simulated object, wherein generating the predicted 3D point cloud of the simulated object comprises: Use the point cloud prediction network to process: all channels of at least a portion of the image, and the generated object mask of the simulated object; generating a projection of the predicted 3D point cloud, the projection being a predicted depth image of the simulated object based on the predicted 3D point cloud; The loss is generated based at least in part on a comparison of: the projection of the predicted 3D point cloud, and A ground truth depth image of the simulated object; and One or more weights of the point cloud prediction network are updated based at least in part on the generated loss.

14. The method according to claim 13, wherein: Generating a projection of the 3D point cloud includes generating a projection of the predicted 3D point cloud using intrinsic parameters of a simulated camera used to render the simulated image.

15. The method according to claim 13 or 14, further comprising: determining a bounding box of the simulated object; Wherein, at least a portion of the image processed using the point cloud prediction network is a crop of the image selected based on the bounding box.

16. The method of claim 15, further comprising selecting a crop of the image based on pixels of the image included in the crop being enclosed by the bounding box.

17. The method according to any one of claims 13 to 14, further comprising: capturing a real image of a real environment, the real image capturing at least one real object, and the real image comprising a plurality of channels, wherein the plurality of channels comprises one or more color channels and a depth channel; Generate an additional object mask of the real object, wherein generating the additional object mask comprises: Using an object detection network to process one or more of the channels of the real image; generating an additional ground truth depth image of the real object based on the additional object mask and a depth channel of the real image; Generating an additional predicted three-dimensional (3D) point cloud of the real object, wherein generating the additional predicted 3D point cloud of the real object comprises: Use the point cloud prediction network to process: all channels of at least a portion of the real image, and the generated object mask of the real object; generating an additional projection of the additional predicted 3D point cloud, the additional projection being an additional predicted depth image of the real object based on the additional predicted 3D point cloud; The additional loss is generated based at least in part on a comparison of: the projection of the additional predicted 3D point cloud, and Additional ground truth depth images of the real objects; and One or more weights of the point cloud prediction network are updated based at least in part on the generated updated loss.

18. The method according to any one of claims 13 to 14, further comprising: Determining that training of the point cloud prediction network satisfies one or more criteria; as well as In response to determining that the training of the point cloud prediction network satisfies the one or more criteria: The evaluation network is trained using additional point clouds predicted using the trained point cloud prediction network.

19. A system for a robot, comprising: one or more actuators operably coupled to the robot; one or more processors; as well as A memory comprising computer readable instructions which, when executed by the one or more processors, cause the system to perform a method according to any one of claims 1 to 12.

20. A computer program product comprising computer readable instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-18.

Images