Object-to-robot pose estimation from a single RGB image

Abstract

To provide methods and systems of performing object-to-object pose estimation from a single image.SOLUTION: A method comprises: identifying an image of a first object and a target object, the image captured by a camera external to the first object and the target object; processing the image, using a first neural network, to estimate a first pose of the target object with respect to the camera; processing the image, using a second neural network, to estimate a second pose of the first object with respect to the camera; and calculating a third pose of the first object with respect to the target object, using the first pose and the second pose.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This application is a continuation-in-part of U.S. Patent Application No. 16 / 405,662, entitled "DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL," filed May 7, 2019 (Reference: 741851 / 18-RE-0161-US02), and claims priority to U.S. Provisional Patent Application No. 62 / 672,767, entitled "DETECTION AND POSE ESTIMATION OF HOUSEHOLD OBJECTS FOR HUMAN-ROBOT INTERACTION," filed May 17, 2018 (Reference: 18-RE-0161US01), which are incorporated by reference in their entireties herein.

[0002] This application is a continuation-in-part of U.S. patent application Ser. No. 16 / 657,220, entitled “POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS,” filed Oct. 18, 2019 (Reference: 1R2674.006901 / 19-SE-0341US01), which is incorporated by reference in its entirety herein.

[0003] The present disclosure relates to systems and methods for pose estimation. [Background technology]

[0004] Pose estimation generally refers to a computer vision technique that determines the Euclidean position and orientation of an object, usually relative to a particular camera. Pose estimation has many applications, but is particularly useful in the context of robotic manipulation systems. Until now, robotic manipulation systems have required a camera mounted on the robot itself (i.e., a handheld camera) to capture images of the object, and / or a camera external to the robot to capture images of the object. In both cases, the camera must be calibrated to the robot in order to then estimate the pose of the captured object relative to the robot.

[0005] Calibration of a local camera only needs to be performed once to determine the camera's position relative to the robot, but due to the rigid mounting of the camera on the robot, the camera has a limited field of view, which prevents it from seeing the surrounding context and being easily adjustable when an object moves out of the camera's field of view. An external camera can have a wider field of view and can optionally assist the local camera, but it needs to be calibrated every time it is moved even slightly. However, calibration is typically a tedious, delicate, and error-prone offline process. It should be noted that while these issues are described specifically with respect to robotic manipulation systems, the same limitations apply to any pose estimation system operable to estimate the pose of one object relative to another object (i.e., which may or may not be moving).

[0006] There is a need to address these and / or other problems associated with the prior art. Summary of the Invention [Means for solving the problem]

[0007] Disclosed is a method, computer readable medium, and system for object-to-object pose estimation from images. In use, images of a first object and a target object are identified and the images are captured by a camera external to the first object and the target object. Additionally, the images are processed using a first neural network to estimate a first pose of the target object relative to the camera. Further, the images are processed using a second neural network to estimate a second pose of the first object relative to the camera. Furthermore, a third pose of the first object relative to the target object is calculated using the first pose and the second pose. [Brief description of the drawings]

[0008] [Figure 1]1 is a diagram of a method for object-to-object pose estimation from images, according to one embodiment. [Diagram 2] 1 is a diagram of a system for object-to-object pose estimation from images, according to one embodiment. [Diagram 3] FIG. 3 is a block diagram associated with the first neural network of FIG. 2, according to one embodiment. [Figure 4] FIG. 3 is a block diagram associated with the second neural network of FIG. 2, according to one embodiment. [Diagram 5] FIG. 1 is a diagram of a robotic grasping system controlled using a robot-to-object pose estimation system, according to one embodiment. [Figure 6A] FIG. 1 illustrates inference and / or training logic, according to at least one embodiment. [Figure 6B] FIG. 1 illustrates inference and / or training logic, according to at least one embodiment. [Figure 7] FIG. 1 illustrates training and deployment of a neural network, according to at least one embodiment. [Figure 8] FIG. 1 illustrates an example data center system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] 1 illustrates a method 100 for object-to-object pose estimation from images, according to one embodiment. Method 100 may be performed by any computing system, which may include one or more computing devices, one or more computer processors, non-transitory memory, circuitry, etc. In one embodiment, the non-transitory memory may store instructions executable by one or more computing devices and / or one or more computer processors to perform method 100. In another embodiment, a circuit may be configured to perform method 100.

[0010] As shown in operation 102, images of a first object and a target object are identified, and the images are captured by cameras external to the first object and the target object. In the context of this description, the target object and the first object are physically separate objects (e.g., in some proximity to each other). In one embodiment, the first object may be a robotic gripping system having a robotic arm for gripping the object. In furtherance of this embodiment, the target object may be a known object that is gripped by the robotic gripping system. In another embodiment, the first object may be another autonomous object, such as an autonomous vehicle, that interacts with (or makes decisions related to) the target object, such as another car. However, of course, the first object and the target object may be any objects for which a pose between the two is desirable (e.g., for computer vision applications).

[0011] As described above, a camera external to the first object and the second object captures images of the first object and the target object. For the purposes of this description, the camera is external to the first object and the target object by being positioned independent of the first object and the target object. For example, the camera may not be mounted on either the first object or the target object. In one embodiment, the camera may be mounted on a tripod or other rigid surface to capture images of the first object and the target object.

[0012] The image of the target object may be a red-green-blue (RGB) image in one embodiment, or a grayscale image in another embodiment. The image may or may not include depth. In another embodiment, the image may be a single image. In a further embodiment, the image may be a combination of images obtained from various types of sensors, such as a six-channel image that may include red, green, blue, infrared, ultraviolet, and radar. Furthermore, the camera may be a monocular RGB camera. However, the camera can of course capture any number of channels of any wavelength of light (visible or invisible to humans), and even non-light wavelengths. For example, the camera may utilize infrared, ultraviolet, microwave, radar, sonar, or other technologies to capture the image.

[0013] Additionally, as shown in operation 104, the image is processed using a first neural network to estimate a first pose of the target object relative to the camera. The first neural network may be trained to output two-dimensional (2D) image locations (x, y coordinates) of key points of the target object. These 2D image locations may then be used to estimate the pose of the target object relative to the camera along with the 3D coordinates of the target object model. In one embodiment, a PnP algorithm may be used to calculate the pose of the target object relative to the camera. In another embodiment, the first pose of the target object relative to the camera may include a three-dimensional (3D) rotation and translation of the target object relative to the camera.

[0014] By way of example, the first neural network may be one disclosed in U.S. Patent Application No. 16 / 405,662, entitled “DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL,” filed May 7, 2019 (Reference: 741851 / 18-RE-0161-US02), which is incorporated by reference herein in its entirety. Further details relating to an embodiment of the first neural network are described below with reference to FIG.

[0015] Further, as shown in operation 106, the image is processed using a second neural network to estimate a second pose of the first object relative to the camera. The second neural network may be trained to output two-dimensional (2D) image locations (x, y coordinates) of key points of the first object. These 2D image locations may then be used to estimate the pose of the first object relative to the camera along with the 3D coordinates of the first object model. In one embodiment, a perspective-n-point (PnP) algorithm may be used to calculate the pose of the first object relative to the camera. In another embodiment, the second pose of the first object relative to the camera may include a 3D rotation and translation of the first object relative to the camera.

[0016] As an option, the second neural network can perform online calibration of the camera, which allows the camera to be moved during runtime, including during operation of a robotic grasping system or other autonomous system.

[0017] By way of example, the second neural network may be one disclosed in U.S. Patent Application No. 16 / 657,220, entitled "POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS," filed on October 18, 2019 (Reference: 1R2674.006901 / 19-SE-0341US01), which is incorporated by reference herein in its entirety. Further details relating to an embodiment of the second neural network are described below with reference to FIG.

[0018] Further still, as shown in operation 108, a third pose of the first object relative to the target object is calculated using the first pose and the second pose. Thus, a pose estimate from the first object to the target object may be calculated. In one embodiment, the third pose may be the pose of the target object relative to the coordinate frame of the first object. In another embodiment, the third pose may be calculated by multiplying the inverse of an output of one of the neural networks by an output of another one of the neural networks, such as the inverse first pose multiplied by the second pose, or the inverse of the second pose multiplied by the first pose, etc.

[0019] In the context where the first object is a robotic gripping system, the robotic gripping system may further be caused to grip a target object (i.e., a known object) using a third pose. For example, the third pose may be provided to the robotic gripping system to enable the robotic gripping system to locate and grip the target object. Of course, in other embodiments, the third pose may be used for other purposes, including, for example, controlling an autonomous vehicle or having an autonomous vehicle make decisions based on the relative location of the target object.

[0020] As an option, the first pose and / or the second pose can be refined prior to calculating the third pose. For example, the first pose can be refined by iteratively matching the image with a synthetic projection of the model in the first pose and then adjusting parameters of the first pose based on the results of the iterative matching. Similarly, the second pose can be refined by iteratively matching the image with a synthetic projection of the model in the second pose and then adjusting parameters of the second pose based on the results of the iterative matching.

[0021] For this purpose, a pose between a first object and a target object may be estimated using two neural networks. In particular, one neural network may estimate a first pose of the first object relative to the camera, and a second neural network may estimate a second pose of the first object relative to the camera. The outputs of the neural networks (i.e., the first pose and the second pose) may then be used as a basis for determining a pose between the first object and the target object.

[0022] Further explanatory information will now be described in relation to various optional architectures and features that can implement the aforementioned framework as desired by the user. It should be specifically noted that the following information is described for illustrative purposes and should not be construed as limiting in any manner. Any of the following features may be optionally incorporated with or without the exclusion of other features described.

[0023] Figure 2 illustrates a system 200 for object-to-object pose estimation from images, according to one embodiment. For example, system 200 may implement method 100 of Figure 1. System 200, in one embodiment, may be located in the cloud and thus remote with respect to the object. In another embodiment, system 200 may be located on a computing device local to the object.

[0024] As shown, system 200 includes a first module 201 including a first neural network 202, a second module 203 including a second neural network 204, and a processor 206. Images are input to first neural network 202 and second neural network 204. The images are of a first object and a target object and are captured by a camera external to both the first object and the target object. The camera may be external to system 200. However, the images may be provided to system 200 by the camera, via a network, shared memory, or in any other manner.

[0025] The first neural network 202 receives an image as input and processes the image to estimate a first pose of the target object relative to the camera (i.e., the pose from the target object to the camera). The first neural network 202 may be a deep neural network that estimates the 6DoF pose (e.g., 3D rotation and translation) of the known object relative to the camera. This network 202 may consist of a multi-stage convolutional network that converts the input RGB image into a set of belief maps, one for each key point. In one embodiment, n=9 key points are used to represent the vertices of a bounding cuboid along with the centroids. In addition to the belief maps, the network 202 may output n-1 affinity maps, one for each key point that is not a centroid. Each map is a 2D field of unit vectors pointing to the nearest object centroid. The maps can be used to individuate the objects in a post-processing step, allowing the system to handle multiple instances of each object. The pose can be determined by a first module 201 that applies the PnP algorithm to key points detected as peaks in the belief map.

[0026] In one embodiment, the input to network 202 may consist of a 533x400 image processed by a VGG-based feature extractor, resulting in a 50x50x512 feature block. These features may be processed by a series of six stages (each containing seven convolutional layers) that output and refine the belief map described above.

[0027] The second neural network 204 receives an image as input and processes the image to estimate a second pose of the first object relative to the camera (i.e., the pose from the first object to the camera). In one embodiment, the second neural network 204 may be a deep neural network that estimates a 6DoF pose of the robot relative to the camera. This network 204 may consist of an encoder-decoder that converts the input RGB image into a set of belief maps, one for each key point. Since there is only one robot in a scene, an affinity field may not be necessary.

[0028] In one embodiment, key points placed on the robot joints may be defined to achieve pose stability when the arm is mostly outside the camera's field of view, which occurs when the camera is viewing the scene from a close distance. In one embodiment, the input to the network 204 is a 400x400 image, downsampled and center cropped from the original 640x480. The layers of the network 204 may be as follows: the encoder follows the same layer structure as VGG, while the decoder is built from 4 upsampling layers followed by 2 convolution layers to give a belief map of key points. The second module 203 applies the PnP algorithm to the key points detected as peaks in the belief map.

[0029] As an option, the first pose and / or the second pose can be refined to reduce errors in the estimation due to limited training data and / or network capacity. This refinement can be performed by adjusting the pose parameters by iteratively matching the input images with synthetic projections of the model with the current pose.

[0030] The first neural network 202 and the second neural network 204 may or may not operate in parallel if desired, since they do not require each other's outputs. The outputs of each of the first neural network 202 and the second neural network 204 are provided to a processor 206. The processor 206 uses the first pose and the second pose to calculate a third pose of the first object relative to the target object. The third pose may be calculated using Equation 1, described according to one embodiment.

number

[0031] Where:

number

number

number

[0032] In an exemplary embodiment where the first object is a robotic grasping system and the target object is a known object, the processor 206 can now cause the robotic grasping system to grasp the target object using the third pose. The processor 206 can communicate with the robotic grasping system over a network, such as the Internet (e.g., if the system 200 is located remotely relative to the object) or Ethernet (e.g., if the system 200 is located locally relative to the object).

[0033] Because the first neural network 202 only estimates the pose of the target object relative to the camera, only the first neural network 202 (and not necessarily the second neural network 204) may be trained on the target object. In this manner, the target object may be "known" to the first neural network 202, or in other words, a "known object." Of course, the first neural network 202 may also be trained on other categories or types of target objects.

[0034] Similarly, only the second neural network 204 (and not necessarily the first neural network 202) may be trained on the first object, since the second neural network 204 only estimates the pose of the first object relative to the camera. In other words, the first object may be "known" to the second neural network 204. However, it should be noted that the second neural network 204 may also be trained on other objects, such as other robots, or other autonomous objects.

[0035] To this end, in contrast to end-to-end learning, the modular approach presented in the present system 200 allows the system to be reused without the need to retrain all networks. For example, to apply the system 200 to a new robot, only the second network 204 needs to be retrained. Similarly, to apply the system 200 to a new object, only the first network 202 needs to be trained on this object, independent of the robot and other objects. Moreover, this modular approach allows for easier testing and refinement of the individual components to ensure accuracy and reliability.

[0036] Figure 3 is a block diagram associated with the first neural network 202 of Figure 2, according to one embodiment. Of course, the block diagram is described as one possible embodiment associated with the first neural network 202 of Figure 2. This embodiment is described in more detail in U.S. Patent Application No. 16 / 405,662, entitled "DETECTING AND ESTIMATING THE POSE OF AN OBJECT USING A NEURAL NETWORK MODEL," filed May 7, 2019 (Reference: 741851 / 18-RE-0161-US02), which is incorporated herein by reference in its entirety.

[0037] In the illustrated embodiment, the pose estimation system 300 includes a key point module 310, a set of multi-stage modules 305, and a pose unit 320. Although the pose estimation system 300 is described in the context of a processing unit, one or more of the key point module 310, the set of multi-stage modules 305, and the pose unit 320 may be implemented by programs, custom circuitry, or a combination of custom circuitry and programs. For example, the key point module 310 may be implemented by a GPU, a central processing unit (CPU), or any processor capable of processing images to generate key point data.

[0038] The pose estimation system 300 receives an image captured by a single camera. The image may include one or more objects for detection. In one embodiment, in images without any depth data, the image includes color data per pixel. The pose estimation system 300 first detects key points associated with the object, and then estimates 2D projections of vertices that define a bounding volume surrounding the object associated with the key points. The key points may include the center of gravity of the object, and the vertices of the bounding volume surrounding the object. The key points are not explicitly visible in the image, but are instead inferred by the pose estimation system 300. In other words, the object of interest is visible in the image, except for portions of the object that may be occluded, and the key points associated with the object of interest are not explicitly visible in the image. The 2D locations of the key points are estimated by the pose estimation system 300 using only the image data. The pose unit 320 recovers the 3D pose of the object using the estimated 2D locations, the camera intrinsic parameters, and the object dimensions.

[0039] The keypoint module 310 receives an image containing an object and outputs image features. In one embodiment, the keypoint module 310 includes a multi-layer convolutional neural network (i.e., the first network 202). In one embodiment, the keypoint module 310 includes the first 10 layers of a Visual Geometry Group (VGG-19) neural network pre-trained using the ImageNet training database, followed by two 3x3 convolution layers to reduce the feature dimensionality from 512 to 256, and then from 256 to 128. The keypoint module 310 outputs three channel features, one per channel (e.g., RGB).

[0040] The image features are input to a set of multi-stage modules 305. In one embodiment, the set of multi-stage modules 305 includes a first multi-stage module 305 configured to detect the center of gravity of the object, and an additional multi-stage module 305 configured in parallel to detect vertices of a bounding volume surrounding the object. In one embodiment, the set of multi-stage modules 305 includes a single multi-stage module 305 that is used to process the image features in multiple passes to sequentially detect the center of gravity and vertices of the bounding volume. In one embodiment, the multi-stage module 305 is configured to detect the vertices without detecting the center of gravity.

[0041] Each multi-stage module 305 includes T stages of belief map units 315. In one embodiment, the number of stages is equal to six (e.g., T=6). Belief map unit 315-1 is the first stage, belief map unit 315-2 is the second stage, etc. Image features extracted by the key point module 310 are passed to each of the belief map units 315 in the multi-stage module 305. In one embodiment, the key point module 310 and the multi-stage module 305 include a feed-forward neural network (i.e., the first neural network 202) that receives an RGB image of size wxhx3 as input and branches to generate two different outputs, e.g., belief maps. In one embodiment, w=640 and h=480. The stages of the Belief Map Unit 315 operate sequentially, with each stage (Belief Map Unit 315) taking into account not only image features but also the output of the previous stage.

[0042] The stages of the belief map unit 315 in each multi-stage module 305 generate belief maps for the estimation of a single 2D location associated with an object in an image. A first belief map contains probability values ​​for the object's centroid, and additional belief maps contain probability values ​​for the vertices of a bounding volume surrounding the object.

[0043] In one embodiment, the 2D locations of the detected vertices are the 2D coordinates of the 3D bounding vertices that each enclose the object and are projected into image space for the scene. Representing each object by a 3D bounding box defines an abstract representation of each object that is sufficient for pose estimation, yet is independent of the details of the object's shape. If the bounding volumes are 3D bounding boxes, nine multi-stage modules 305 can be used to generate belief maps for the centroids and eight vertices in parallel. The pose unit 325 estimates the 2D coordinates of the 3D bounding box vertices projected into image space, and then infers object locations and poses in 3D space from the perspective-n-point (PnP) using either traditional computer vision algorithms or another neural network. PnP estimates the pose of an object using a set of n locations in 3D space and a projection of the n locations in image space. In one embodiment, the pose estimation system 300 estimates the 3D pose of a known object in clutter from a single RGB image in real time.

[0044] In one embodiment, each stage of the belief map unit 315 is a stage of a Convolutional Neural Network (CNN). When each stage is a CNN, each stage exploits an increasingly larger effective receptive field as data passes through the neural network. This property enables the stages of the belief map unit 315 to resolve ambiguities by incorporating an increasingly larger amount of context in later stages.

[0045] In one embodiment, the stages of belief map units 315 receive the 128 dimensional features extracted by key point module 310. In one embodiment, belief map unit 315-1 includes three 3x3x128 layers and one 1x1x512 layer. In one embodiment, belief map unit 315-2 is a 1x1x9 layer. In one embodiment, belief map units 315-3 through 315-T are identical to the first stage except that each receives a 153 dimensional input (128+16+9=153) and includes five 7x7x128 layers and one 1x1x128 layer before the 1x1x128 or 1x1x16 layers. In one embodiment, each of belief map units 315 w / 8 and h / 8 , with rectified linear unit (ReLU) activation functions interleaved globally.

[0046] 4 is a block diagram associated with the use of the second neural network 204 of FIG. 2, according to one embodiment. Of course, the block diagram is described as one possible embodiment associated with the second neural network 204 of FIG. 2. This embodiment of the second neural network 204 is described in more detail in U.S. Patent Application No. 16 / 657,220, entitled “POSE DETERMINATION USING ONE OR MORE NEURAL NETWORKS,” filed on October 18, 2019 (Reference: 1R2674.006901 / 19-SE-0341US01), which is incorporated by reference herein in its entirety.

[0047] As shown, a captured image 402 of a robot is provided as an input to the trained neural network 204. It should be noted that while a robot is described in the context of this example, the block diagrams described herein are equally applicable to other objects (e.g., other autonomous systems). As an option, some pre-processing or enhancement of the image may be performed, such as to adjust resolution, color depth, or contrast, prior to processing. In at least one example, the network 204 may be trained specifically for the type of robot 204, since different robots may have different shapes, sizes, configurations, kinematics, and characteristics.

[0048] In use, the neural network 204 can analyze the input images 402 and output a set of belief maps 406 as a set of inferences. Alternatively, other dimension-determining inferences can be generated to locate feature points. For example, the neural network 204 can infer one belief map 406 for each robot feature identified.

[0049] In at least one embodiment, a model of the robot used for training can identify specific features to be tracked. These features can be learned through a training process. In another embodiment, features can be located on various moving parts or components of the robot such that a pose of the robot can be determined from these features. In a further embodiment, the features can be selected such that each pose of the robot corresponds to one and only one setting of the features, and each setting of the features corresponds to one and only one robot pose. This uniqueness can allow the camera-to-robot pose to be determined based on the unique orientation of the features as represented in the captured image data.

[0050] For network 204, an autoencoder network can detect key points. In at least one embodiment, neural network 204 receives as input an RGB image of size wxhx3 and outputs n belief maps 406 of the form wxhxn. In at least one embodiment, RGBD or stereoscopic images can be received as input as well. Optionally, w=640 and h=480. In at least one embodiment, the output for each key point is a 2D belief map, where pixel values ​​represent the likelihood that the key point projects onto that pixel.

[0051] In one embodiment, the encoder of network 204 includes a convolutional layer of VGG-19 pre-trained on ImageNet. In another embodiment, a ResNet-based encoder can be used. In a further embodiment, the decoder or upsampling component of network 204 is composed of four 2D transposed convolutional layers, each followed by a regular 3×3 convolutional layer and a ReLU activation function. Still further, the output head can be composed of three convolutional layers (3×3, stride=1, padding=1) with ReLU activation with 64, 32, and n channels, respectively. In at least one embodiment, there may be no activation layer after the last convolutional layer. In another embodiment, the encoder network can be trained using an L2 loss function that compares the output belief map to a ground truth belief map, where the ground truth belief map is generated using σ=2 pixels to generate peaks. In at least one embodiment, the use of stereoscopic image pairs can make it possible to fuse the poses estimated by these images, or a point cloud can be computed and the poses determined using processes such as Procrustes analysis or iterative closest point (ICP).

[0052] In at least one embodiment, the belief maps 406 may be provided as input to a peak extraction component 408, or service, that may determine a set of coordinates in two dimensions that represent the locations of relevant robot features. As an option, key point coordinates may be calculated as a weighted average of values ​​around thresholded peaks in each belief map, after first applying Gaussian smoothing to those belief maps to reduce noise effects. This weighted average may allow for sub-pixel accuracy. In at least one embodiment, these two-dimensional coordinates (or pixel locations) may be provided as input to a pose determination module, such as a perspective-n-point (PnP) module 414. This pose determination module may accept as input camera-specific data 410, such as calibration information for the camera, that may be used to account for image artifacts due to lens asymmetry, focal length, principal point, or other such factors.

[0053] In at least one embodiment, the pose determination module may also receive as input information about the forward kinematics 412 of this type of robot to determine possible poses. The kinematics may be used to narrow the search space where only certain feature locations are possible due to the physical configuration or limitations of this type of robot. This information may be analyzed using a PnP algorithm to output a determined camera-to-robot pose. In at least one embodiment, perspective-n-point is used to recover the camera extrinsic factors, assuming that the joint configuration of the robot manipulator is known. Since the base coordinates or other features of the robot may be precisely identified in the camera space or camera coordinate system, this pose information may be used to determine the relative distance and orientation between the camera and the robot. To this end, the neural network 204 may be trained to infer the location of these features, such as by inferring a set of belief maps 406.

[0054] In at least one embodiment, the relative position and orientation information can be used to ensure that the camera coordinate space from the camera's viewpoint is aligned with the robot coordinate space of the robot, both in terms of dimensions and alignment. In at least one embodiment, these coordinates may be correct from the camera coordinate system but not the robot's coordinate system, so that an inaccurate orientation or position of the camera 502 with respect to the robot 504 may give incorrect coordinates for the robot 504 to perform an action. To this end, the relative position and orientation of the camera 502 can be determined with respect to the robot 504. In at least one embodiment, while the relative position of the camera 502 may be sufficient, orientation information may be useful depending on factors such as camera specifics, where asymmetric image properties may affect accuracy if not properly accounted for. This relative position / orientation can be calibrated online (e.g., during the robot's runtime).

[0055] 5 illustrates a robotic grasping system 500 controlled using a robot-to-object pose estimation system, according to one embodiment. The robotic grasping system 500 can be implemented in the context of the embodiments described above.

[0056] As shown, a camera 502 can be used to capture images, potentially in the form of video frames, of an autonomous object such as a robot 504. The camera 502 can be positioned or externally mounted such that the robot 504 is within the field of view 510 of the camera 502 and such that the camera 502 can capture images that may include at least a partial representation of the robot 504, if not a complete view representation. The captured images can be used to assist in providing instructions to the robot 504 to perform a particular task.

[0057] In at least one embodiment, the captured images can be analyzed to determine the location of the object 512 relative to the robot 504, which the robot 504 uses to interact with, such as picking up or modifying the object 512 in some way. In particular, the system 200 of FIG. 2 can be utilized to determine a robot-to-object pose for the purpose of providing precise instructions to the robot 504 or a control system for the robot 504. The robot-to-object pose can also be used for other purposes, such as to help navigate the robot 504 or provide current information about the state of the robot 504. In at least one embodiment, precise position and orientation data between the robot 504 and the object 512 allows the robot 504 to operate robustly in unstructured and dynamic environments while performing tasks such as object grasping and manipulation, human-robot interaction, and collision detection and avoidance. Therefore, it is desirable to determine at least one of the position or orientation of the robot 504 relative to the camera 502 and the position or orientation of the object 512 relative to the camera 502, and then determine the robot-to-object pose.

[0058] As described above, images may be captured by the camera 502 that indicate the current orientation of the robot 504 relative to the camera 502 and the current orientation of the object 512 relative to the camera 502. The robot 504 may have different articulated limbs 508 or components such that the robot 504 may be in different configurations or "poses" with respect to the orientation of the robot 504. In at least one embodiment, different poses of the robot 504 may result in different expressions in the images captured by the camera 502. In at least one embodiment, a single image captured by the camera 502 may be analyzed to determine features of the robot 504 that can be used to determine the pose of the robot 504. The features may correspond to joints or locations where the robot can move or adjust its position or orientation. Furthermore, because the dimensions and kinematics of the robot 504 are known, determining the pose of the robot 504 from the perspective of the camera 502 allows for accurate determination of the distance and orientation of the robot from the camera.

[0059] Machine Learning Deep Neural Networks (DNNs), which involve deep learning models developed on processors, are being used in diverse use cases ranging from self-driving cars to rapid drug discovery, from automatic image captioning in online image databases to smart real-time language translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, and it continuously learns and gets smarter, providing more accurate results in a faster time. A child is initially taught by an adult to correctly identify and classify different shapes, and eventually becomes able to identify shapes without any coaching. Similarly, deep learning or neural learning systems need to be trained on object recognition and classification to become smarter and more efficient in identifying basic objects, occluded objects, etc., while also assigning context to the objects.

[0060] At the simplest level, neurons in the human brain look at the various inputs they receive, assign a level of importance to each of these inputs, and the output is passed on to other neurons for action. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs that represent various features of an object that the perceptron is trained to recognize and classify, and each of these features is assigned a certain weight based on the importance of that feature in defining the object's shape.

[0061] Deep Neural Network (DNN) models contain multiple layers of many connected nodes (e.g., perceptrons, Boltzmann machines, radius-based functions, convolutional layers, etc.) that can be trained with vast amounts of input data to quickly solve complex problems with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different sections and looks for basic patterns such as lines and angles. The second layer assembles the lines and looks for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers generate labels for the input image and identify the model of a specific car brand.

[0062] Once a DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process known as inference. Examples of inference (the process in which a DNN extracts useful information from a given input) include identifying handwritten digits on a check deposited in an ATM machine, identifying images of friends in a photograph, providing movie recommendations to over 50 million users, identifying and classifying different types of vehicles, pedestrians, and road hazards in driverless cars, or translating human speech in real time.

[0063] During training, data is passed through the DNN in the forward propagation phase until a prediction is generated that indicates the label that corresponds to the input. If the neural network does not label an input correctly, the error between the correct label and the predicted label is analyzed, and during the backpropagation phase, the weights for each feature are adjusted until the DNN correctly labels that input and the other inputs in the training data set. Training a complex neural network requires a large amount of parallel computing performance, including floating-point multiplications and additions. Inferring is a less computationally intensive, latency-sensitive process in which a trained neural network is applied to new inputs it has not seen before for image classification, speech translation, and general inference on new information.

[0064] Logic of inference and training As mentioned above, a deep learning or neural learning system needs to be trained to generate inferences from input data. Details regarding inference and / or training logic 615 for a deep learning or neural learning system are provided below in conjunction with Figures 6A and / or 6B.

[0065] In at least one embodiment, the inference and / or training logic 615 may include, without limitation, data storage 601 for storing forward propagation and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used to infer with one or more aspects of the embodiment. In at least one embodiment, the data storage 601 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more aspects of the embodiment while forward propagating input / output data and / or weight parameters during training and / or inference using one or more aspects of the embodiment. In at least one embodiment, any portion of the data storage 601 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache, or system memory.

[0066] In at least one embodiment, any portion of data storage 601 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, data storage 601 may be cache memory, dynamic randomly addressable memory (“DRAM”), static randomly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether data storage 601 is internal or external to a processor, for example, or comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in the inference and / or training of the neural network, or any combination of these factors.

[0067] In at least one embodiment, the inference and / or training logic 615 may include, without limitation, data storage 605 for storing backpropagated and / or output weights and / or input / output data corresponding to neurons or layers of a neural network trained and / or used to infer with one or more embodiments. In at least one embodiment, the data storage 605 stores weight parameters and / or input / output data for each layer of a neural network trained or used in conjunction with one or more embodiments while backpropagating the input / output data and / or weight parameters during training and / or inference using one or more embodiments. In at least one embodiment, any portion of the data storage 605 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 caches of a processor, or system memory. In at least one embodiment, any portion of the data storage 605 may be internal or external to one or more processors, or other hardware logic devices or circuits. In at least one embodiment, data storage 605 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, the choice of whether data storage 605 is internal or external to the processor, for example, or whether it is comprised of DRAM, SRAM, flash, or some other type of storage, may depend on the storage available on-chip versus off-chip, the latency requirements of the training and / or inference functions being performed, the batch size of data used in the inference and / or training of the neural network, or any combination of these factors.

[0068] In at least one embodiment, data storage 601 and data storage 605 may be separate storage structures. In at least one embodiment, data storage 601 and data storage 605 may be the same storage structure. In at least one embodiment, data storage 601 and data storage 605 may be partially the same storage structure and partially separate storage structures. In at least one embodiment, any portion of data storage 601 and data storage 605 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache, or system memory.

[0069] In at least one embodiment, the inference and / or training logic 615 may include, without limitation, one or more arithmetic logic units ("ALUs") 610 for performing logical and / or arithmetic operations based at least in part on or indicated by the training and / or inference code, which may result in activations (e.g., output values ​​from layers or neurons in a neural network) that are stored in activation storage 620 and are a function of input / output and / or weight parameter data stored in data storage 601 and / or data storage 605. In at least one embodiment, the activations stored in activation storage 620 are generated according to linear algebra and / or matrix-based calculations performed by ALU 610 in response to executing instructions or other code, where weight values ​​stored in data storage 605 and / or data 601 are used as operands along with other values, such as bias values, gradient information, momentum values, or other parameters or hyperparameters, any or all of which may be stored in data storage 605 or data storage 601, or in another storage, on-chip or off-chip. In at least one embodiment, ALU 610 is contained within one or more processors or other hardware logic devices or circuits, although in other embodiments ALU 610 may be external to the processors or other hardware logic devices or circuits that use them (e.g., a co-processor). In at least one embodiment, ALU 610 may be included within an execution unit of a processor, or may be otherwise included within an ALU bank accessible by execution units of a processor, either within the same processor or distributed among different processors of different types (e.g., central processing units, graphics processing units, fixed function units, etc.).In at least one embodiment, data storage 601, data storage 605, and activation storage 620 may be in the same processor or other hardware logic device or circuitry, or in another embodiment, they may be in different processors or other hardware logic devices or circuits, or some combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of activation storage 620 may be included with other on-chip or off-chip data storage, including L1, L2, or L3 cache of the processor, or system memory. Additionally, inference and / or training code may be stored with other code accessible to the processor or other hardware logic or circuitry, and may be fetched and / or processed using the fetch, decode, schedule, execute, retire, and / or other logic circuitry of the processor.

[0070] In at least one embodiment, activation storage 620 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, activation storage 620 may be fully or partially internal to or external to one or more processors or other logic circuitry. In at least one embodiment, the choice of whether activation storage 620 is internal or external to a processor, for example, or whether it is comprised of DRAM, SRAM, flash, or some other type of storage, may depend on available on-chip versus off-chip storage, latency requirements of the training and / or inference functions being performed, batch sizes of data used in inference and / or training of the neural network, or any combination of these factors. In at least one embodiment, the inference and / or training logic 615 shown in Figure 6A may be used in conjunction with an application-specific integrated circuit ("ASIC") such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., "Lake Crest") processor from Intel Corp. In at least one embodiment, the inference and / or training logic 615 shown in Figure 6A may be used in conjunction with other hardware such as central processing unit ("CPU") hardware, graphics processing unit ("GPU") hardware, or a field programmable gate array ("FPGA").

[0071] FIG. 6B illustrates inference and / or training logic 615 according to at least one embodiment. In at least one embodiment, the inference and / or training logic 615 may include, without limitation, hardware logic in which computational resources are dedicated to or otherwise used only in conjunction with weight values ​​or other information corresponding to one or more layers of neurons in a neural network. In at least one embodiment, the inference and / or training logic 615 illustrated in FIG. 6B may be used in conjunction with an application specific integrated circuit (ASIC), such as a Tensorflow® processing unit from Google, an inference processing unit (IPU) from Graphcore™, or a Nervana® (e.g., “Lake Crest”) processor from Intel Corporation. In at least one embodiment, the inference and / or training logic 615 illustrated in FIG. 6B may be used in conjunction with other hardware, such as central processing unit (CPU) hardware, graphics processing unit ("GPU") hardware, or field programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 615 may include, without limitation, data storage 601 and data storage 605, which may be used to store weight values ​​and / or other information, including bias values, gradient information, momentum values, and / or other parameters or hyperparameter information. In at least one embodiment shown in FIG. 6B, each of data storage 601 and data storage 605 is associated with dedicated computational resources, such as computation hardware 602 and computation hardware 606, respectively. In at least one embodiment, each of computation hardware 606 includes one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on the information stored in data storage 601 and data storage 605, respectively, with the results stored in activation storage 620.

[0072] In at least one embodiment, each of the data storages 601 and 605 and the corresponding computational hardware 602 and 606 correspond to different layers of a neural network, such that activations resulting from one "storage / computation pair 601 / 602" of data storage 601 and computational hardware 602 are provided as inputs to a next "storage / computation pair 605 / 606" of data storage 605 and computational hardware 606 to reflect the conceptual organization of the neural network. In at least one embodiment, the storage / computation pairs 601 / 602 and 605 / 606 may correspond to two or more layers of the neural network. In at least one embodiment, additional storage / computation pairs (not shown) may be included in the inference and / or training logic 615 after or in parallel with the storage / computation pairs 601 / 602 and 605 / 606.

[0073] Training and deployment of neural networks FIG. 7 illustrates another embodiment for training and deployment of a deep neural network. In at least one embodiment, an untrained neural network 706 is trained using a training data set 702. In at least one embodiment, the training framework 704 is the PyTorch framework, while in other embodiments, the training framework 704 is Tensorflow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training frameworks. In at least one embodiment, the training framework 704 trains the untrained neural network 706 and enables it to be trained using processing resources described herein to generate a trained neural network 708. In at least one embodiment, the weights may be selected randomly or by pre-training using a deep belief network. In at least one embodiment, the training may be performed in a supervised, semi-supervised, or unsupervised manner.

[0074] In at least one embodiment, the untrained neural network 706 is trained using supervised learning, where the training data set 702 includes inputs paired with desired outputs for the inputs, or the training data set 702 includes inputs with known outputs, and the outputs of the neural network 706 are manually scored. In at least one embodiment, the untrained neural network 706 is trained in a supervised manner, processing inputs from the training data set 702 and comparing the resulting outputs to a set of expected or desired outputs. In at least one embodiment, errors are then back-propagated through the untrained neural network 706. In at least one embodiment, the training framework 704 adjusts weights that control the untrained neural network 706. In at least one embodiment, the training framework 704 includes tools to monitor how well the untrained neural network 706 is converging toward a model, such as a trained neural network 708, suitable for generating correct answers, such as in the results 714, based on known input data, such as new data 712. In at least one embodiment, the training framework 704 iteratively trains the untrained neural network 706 while adjusting the weights to refine the output of the untrained neural network 706 using a loss function and a tuning algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework 704 trains the untrained neural network 706 until it reaches a desired accuracy. In at least one embodiment, the trained neural network 708 can then be deployed to implement any number of machine learning operations.

[0075] In at least one embodiment, the untrained neural network 706 is trained using unsupervised learning, where the untrained neural network 706 attempts to train itself using unlabeled data. In at least one embodiment, the training data set 702 for unsupervised learning includes input data without any associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 706 can learn groupings within the training data set 702 and determine how individual inputs relate to the untrained data set 702. In at least one embodiment, unsupervised training can be used to generate a self-organizing map, which is a type of trained neural network 708 that can perform operations useful for reducing the dimensionality of the new data 712. In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which allows for identifying data points in the new data set 712 that deviate from normal patterns in the new data set 712.

[0076] In at least one embodiment, semi-supervised learning may be used, which is a technique in which labeled and unlabeled data are mixed in the training data set 702. In at least one embodiment, the training framework 704 may be used to perform incremental learning, such as by transfer learning techniques. In at least one embodiment, incremental learning allows the trained neural network 708 to adapt to new data 712 without forgetting the knowledge instilled in the network during initial training.

[0077] Data Center 8 illustrates an example data center 800 in which at least one embodiment may be used. In at least one embodiment, the data center 800 includes a data center infrastructure layer 810, a framework layer 820, a software layer 830, and an application layer 840.

[0078] 8, data center infrastructure layer 810 may include a resource orchestrator 812, grouped computing resources 814, and node computing resources ("node CRs") 816(1)-816(N), where "N" represents any positive integer. In at least one embodiment, node CRs 816(1)-816(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors, etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and cooling modules. In at least one embodiment, one or more of the nodes CR 816(1)-816(N) may be a server having one or more of the computing resources described above.

[0079] In at least one embodiment, the grouped computing resources 814 may include separate groups of node CRs housed in one or more racks (not shown), or multiple racks housed in a data center at various graphical locations (also not shown). The separate groups of node CRs in the grouped computing resources 814 may include grouped compute resources, network resources, memory resources, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node CRs including CPUs or processors may be grouped in one or more racks to provide compute resources to support one or more workloads. In at least one embodiment, the one or more racks may also include any number of power modules, cooling modules, and network switches in any combination.

[0080] In at least one embodiment, the resource orchestrator 822 may configure or otherwise control one or more nodes CR 816(1)-816(N) and / or grouped computing resources 814. In at least one embodiment, the resource orchestrator 822 may include a software design infrastructure ("SDI") management entity for the data center 800. In at least one embodiment, the resource orchestrator may include hardware, software, or some combination thereof.

[0081] In at least one embodiment, as shown in FIG. 8 , framework layer 820 includes a job scheduler 832, a configuration manager 834, a resource manager 836, and a distributed file system 838. In at least one embodiment, framework layer 820 may include a framework for supporting software 832 in software layer 830 and / or one or more applications 842 in application layer 840. In at least one embodiment, software 832 or application 842 may each include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure. In at least one embodiment, framework layer 820 may be a type of free and open source software web application framework, such as, but not limited to, Apache Spark® (hereinafter “Spark”), which can use distributed file system 838 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 832 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 800. In at least one embodiment, the configuration manager 834 may be capable of configuring different tiers, such as the software tier 830, as well as the framework tier 820 including Spark and a distributed file system 838 to support large scale data processing. In at least one embodiment, the resource manager 836 may be capable of managing clustered or grouped computing resources that are mapped or allocated to support the distributed file system 838 and the job scheduler 832. In at least one embodiment, the clustered or grouped computing resources may include grouped computing resources 814 in the data center infrastructure tier 810.In at least one embodiment, resource manager 836 may cooperate with resource orchestrator 812 to manage these mappings or allocated computing resources.

[0082] In at least one embodiment, software 832 included in software layer 830 may include software used by at least a portion of nodes CR 816(1)-816(N), grouped computing resources 814, and / or distributed file system 838 of framework layer 820. The one or more types of software may include, but are not limited to, Internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0083] In at least one embodiment, the applications 842 included in the application layer 840 may include one or more types of applications used by at least a portion of the nodes CRs 816(1)-816(N), the grouped computing resources 814, and / or the distributed file system 838 of the framework layer 820. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, Tensorflow, Caffe, etc.), or other machine learning applications used in conjunction with one or more embodiments.

[0084] In at least one embodiment, any of configuration manager 834, resource manager 836, and resource orchestrator 812 may implement any number and type of self-correcting actions based on any amount and type of data obtained in any technically feasible manner. In at least one embodiment, the self-correcting actions may enable data center operators of data center 800 to avoid determining potentially poor configurations and eliminate underutilized and / or underperforming portions of the data center.

[0085] In at least one embodiment, data center 800 may include tools, services, software, or other resources for training one or more machine learning models or predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, in at least one embodiment, a machine learning model may be trained by calculating weight parameters according to a neural network architecture using the software and computing resources described above with respect to data center 800. In at least one embodiment, a trained machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 800 by using weight parameters calculated by one or more techniques described herein.

[0086] In at least one embodiment, the data center may use a CPU, application specific integrated circuit (ASIC), GPU, FPGA, or other hardware to perform training and / or inference using the resources described above. Additionally, one or more of the software and / or hardware resources described above may be configured as a service to enable a user to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0087] Inference and / or training logic 615 is used to perform inference and / or training operations associated with one or more embodiments. In at least one embodiment, inference and / or training logic 615 may be used in the system of FIG. 8 for inference or prediction operations based at least in part on weight parameters calculated using neural network training operations, neural network functionality and / or architecture, or neural network use cases described herein.

[0088] As described herein, a method, computer readable medium, and system for estimating object-to-object pose using externally captured images of the object are disclosed. According to Figures 1-4, an embodiment can provide a neural network usable for performing inferring motion and providing inferred data, as depicted in Figures 6A and 6B, the neural network is stored (partially or entirely) in one or both of data storages 601 and 605 within inference and / or training logic 615. Training and deployment of the neural network can be performed as depicted in Figure 7 and described herein. Distribution of the neural network can be performed using one or more servers of a data center 800, as depicted in Figure 8 and described herein.

Claims

1. identifying an image of a robotic gripping system including a plurality of joints and a target object to be gripped by the robotic gripping system, the image being captured by a camera not attached to the robotic gripping system and the target object, but external to the robotic gripping system and the target object; processing images of the identified target object using a first neural network to estimate a first pose of the target object relative to the camera; estimating a second pose of the robotic gripping system relative to the camera from the inferred positions of the joints using a second neural network operating in parallel with the first neural network, the second neural network generating a plurality of belief maps that infer, from images of the identified robotic gripping system, positions of each of the plurality of joints that operate to grasp the target object; calculating a third pose of the robotic gripping system relative to the target object using the first pose and the second pose; A method comprising:

2. The method of claim 1 , wherein the image captured by the camera is a Red-Green-Blue (RGB) image or a grayscale image.

3. The method of claim 1 , wherein the camera captures one of optical or non-optical wavelengths.

4. The method of claim 1 , wherein the target object is a known object to be grasped by the robotic grasping system.

5. causing the robotic grasping system to grasp the known object using the third pose. The method of claim 4 further comprising:

6. The method of claim 1 , wherein the first pose of the target object relative to the camera includes a three-dimensional (3D) rotation and translation of the target object relative to the camera.

7. The method of claim 1 , wherein the second pose of the robotic gripping system relative to the camera includes a 3D rotation and translation of the robotic gripping system relative to the camera.

8. The method of claim 1 , wherein the second neural network performs an online calibration of the camera.

9. The method of claim 1 , wherein the third pose is a pose of the target object relative to a coordinate frame of the robotic gripping system.

10. The method of claim 1 , wherein only the first neural network is trained on the target object.

11. The method of claim 1 , wherein only the second neural network is trained for the robotic grasping system.

12. refining the first pose; refining the second pose; The method of claim 1 further comprising:

13. refining the first pose comprises: iteratively matching images captured by the camera with a synthetic projection of the model at the first pose, and adjusting parameters of the first pose based on results of the iterative matching. The method of claim 12 , wherein the method is performed by:

14. refining the second pose comprises: iteratively matching images captured by the camera with a synthetic projection of the model at the second pose, and adjusting parameters of the second pose based on results of the iterative matching. The method of claim 12 , wherein the method is performed by:

15. 1. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: identifying an image of a robotic gripping system including a plurality of joints and a target object to be gripped by the robotic gripping system, the image being captured by a camera not attached to the robotic gripping system and the target object, but external to the robotic gripping system and the target object; processing images of the identified target object using a first neural network to estimate a first pose of the target object relative to the camera; estimating a second pose of the robotic gripping system relative to the camera from the inferred positions of the joints using a second neural network operating in parallel with the first neural network, the second neural network generating a plurality of belief maps that infer, from images of the identified robotic gripping system, positions of each of the plurality of joints that operate to grasp the target object; calculating a third pose of the robotic gripping system relative to the target object using the first pose and the second pose; A non-transitory computer readable medium comprising:

16. a first neural network that receives as input images identifying the target object from images of the robotic gripping system and the target object captured by a camera that is not attached to a robotic gripping system including a plurality of joints and a target object to be gripped by the robotic gripping system and that is external to the robotic gripping system and the target object, and processes the images identifying the target object to estimate a first pose of the target object relative to the camera; a second neural network operating in parallel with the first neural network, the second neural network receiving as input an image identifying the robotic gripping system from images of the robotic gripping system and the target object, generating from the image identifying the robotic gripping system a number of belief maps that infer positions of each of the plurality of joints that operate to grasp the target object, and estimating a second pose of the robotic gripping system relative to the camera from the inferred positions of the plurality of joints; a processor that uses the first pose and the second pose to calculate a third pose of the robotic gripping system relative to the target object; A system comprising:

17. The system of claim 16 , wherein the camera is external to the system.

18. The system of claim 16 , wherein the target object is a known object.

19. The system of claim 18 , wherein the processor causes the robotic grasping system to grasp the target object using the third pose.

Images