3D Pose Estimation Using Multiple 2D Cameras

Abstract

To provide a system and method for obtaining a 3D pose of an object using 2D images from a plurality of 2D cameras.SOLUTION: A method based on a system 140 for estimating a 3D pose of an object includes: positioning a first 2D camera to be directed towards the object along a first optical axis; obtaining 2D images 142 of the object by the first 2D camera; and extracting feature points from the 2D images from the first 2D camera using a first feature extraction process. The method also includes: positioning a second 2D camera to be directed towards the object along a second optical axis; obtaining 2D images 144 of the object by the second 2D camera; extracting feature points from the 2D images from the second 2D camera using a second feature extraction process; and estimating a 3D pose 156 of the object using the feature points extracted from the first and second feature extraction processes.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates generally to systems and methods for acquiring a three-dimensional pose of an object using two-dimensional cameras, and more particularly to systems and methods, including certain robotic control applications, for acquiring a three-dimensional pose of an object using multiple two-dimensional cameras and learning-based neural networks. [Background technology]

[0002] Robots perform many tasks, including pick-and-place operations, in which the robot picks up and transports multiple objects from one location, such as a conveyor belt, to another location, such as a collection bin, with the objects at slightly different positions and orientations, known as their 3D pose. Thus, in order for the robot to efficiently pick up objects, it is often necessary for the robot to know the 3D pose of the object. To this end, various techniques for determining the 3D pose of an object use a 3D camera that acquires a 3D image of the object in real time. In this case, the 3D image identifies distance measurements to many points within the camera's field of view, and these measurements are transmitted to the robot controller. These techniques use structured light to create a 3D map of the scene and can identify the object's 3D features to locate the object in the map. For robotic applications, using 3D cameras to determine the 3D pose of an object has proven effective. However, 3D cameras are expensive devices and often operate slower than required.

[0003] One technique known in the art for acquiring the 3D pose of an object for robotic applications instead uses a two-dimensional (2D) camera. This technique acquires a single 2D color image of the scene and processes the image using a fully-convolutional architecture. An algorithm divides the image into a 2D grid containing SXS cells and identifies objects within the cells. The algorithm projects a bounding box around each object in the image and predicts the 2D positions of the box's eight corners. However, this technique for predicting the bounding box corners has limited accuracy and low robustness to occlusion.

[0004] Some of the above processes for determining the 3D pose of an object use artificial intelligence (AI), which is the part of computer science that uses algorithms to allow software applications to learn from an environment and make decisions from it to achieve a particular result. Machine learning is the part of AI that uses software applications that acquire their own knowledge by repeatedly analyzing large amounts of raw input data, extracting patterns from the data and learning to perform a task even if the software application was not specifically programmed to perform the task. Deep learning is a specific type of machine learning that provides better learning performance by representing a particular real-world environment as a hierarchy of concepts of increasing complexity.

[0005] Deep learning typically uses a software structure that includes multiple layers of neural networks that perform nonlinear processing, where each subsequent layer receives the output from the previous layer. The layers typically include an input layer that receives raw data from sensors, multiple hidden layers that extract abstract features from the data, and an output layer that identifies specific things based on feature extraction from the hidden layers. Neural networks include neurons or nodes, each with a "weight" that can be multiplied by the input to the node to obtain a probability that something is correct or not. More specifically, each node has a weight, which is a floating point number, that is multiplied by the input to the node to generate the output of that node, which is a percentage of the input. The weights are first "trained" or set by having the neural network analyze a set of known data under supervised processing and minimizing a cost function to allow the network to obtain the highest probability of obtaining the correct output.

[0006] Deep learning neural networks are frequently used to perform image feature extraction and transformation for visual detection and classification of objects in images. Videos or streams of images can be analyzed by the network, which can identify and classify objects and, through the process, learn to better recognize objects. Thus, with these types of networks, the system can detect certain objects using the same processing configuration and classify them differently based on how the algorithm has learned to recognize them. Summary of the Invention

[0007] The following description discloses and describes a system and method for acquiring a 3D pose of an object using 2D images from multiple 2D cameras. The method includes positioning a first 2D camera to be oriented toward the object along a first optical axis, acquiring a 2D image of the object with the first 2D camera, and extracting feature points from the 2D image from the first 2D camera using a first feature extraction process. The method also includes positioning a second 2D camera to be oriented toward the object along a second optical axis that is angled with the first optical axis, acquiring a 2D image of the object with the second 2D camera, and extracting feature points from the 2D image from the second 2D camera using a second feature extraction process. The method then estimates the 3D pose of the object using the feature points extracted from the first and second feature extraction processes. The first and second feature extraction processes can be a learning-based neural network or a vector field estimation process. Additionally, estimating the 3D pose of the object using feature points extracted from the first and second feature extraction processes can use a perspective n-point algorithm, which compares a 2D projection of a 3D virtual model of the object with a 2D projection of the object having feature locations for the first and second feature extraction processes, and then averaging the two 3D poses.

[0008] Additional features of the present disclosure will become apparent from the following description and claims, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram of a robotic system that includes a robot that picks up an object from a conveyor belt and places it in a bin. The system uses a 2D camera to capture 2D images of the object and a controller that uses the 2D images to calculate the 3D pose of the object. [Diagram 2]FIG. 1 is a flow-type diagram showing the process of acquiring a dataset from an object using a 2D camera for training a neural network. [Diagram 3] FIG. 1 is a diagram of a neural network including multiple nodes, each with adjustable weights. [Figure 4] FIG. 1 is a flow-type diagram illustrating a learning-based neural network process that uses a trained neural network to estimate a 3D pose of an object using a 2D image of the object and the neural network. [Diagram 5] FIG. 5 illustrates a perspective n-point (PnP) process for determining a 3D pose estimate of an object in the process shown in FIG. [Figure 6] FIG. 1 is a diagram of a camera system that uses multiple cameras to image an object. [Figure 7] FIG. 1 is a block diagram of a system for estimating the 3D pose of an object using multiple 2D cameras. [Figure 8] FIG. 8 illustrates a PnP process for estimating a 3D pose of an object in the system shown in FIG. [Figure 9] FIG. 1 is a flow-type diagram illustrating a vector field estimation process for predicting feature points on an object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following description of embodiments of the present disclosure directed to systems and methods using learning-based neural networks to acquire a 3D pose of an object using multiple 2D cameras is merely exemplary in nature and is not intended to limit the invention or its applications or uses. For example, the systems and methods have application in determining the position and orientation of an object grasped by a robot. However, the systems and methods may have other applications.

[0011] FIG. 1 is a diagram of a robotic system 10 including a robot 12 that picks up an object 14 from a conveyor belt 16 and places it in a collection bin 18. The system 10 is intended to represent any type of robotic system that can benefit from the description herein, and the robot 12 can be any robot suitable for that purpose. Each object 14 has a different orientation and position on the conveyor belt 16 relative to the other objects 14 as it moves downstream on the conveyor belt 16. In order for the robot 12 to effectively grasp and pick up the objects 14, the end effector 20 of the robot 12 must be able to be positioned in a proper position and orientation before the end effector 20 grasps the object 14. To accomplish this, the system 10 includes a 2D camera 22 that is positioned at a desired position relative to the object 14 and provides a 2D image to a robot controller 24 that controls the robot 12 to move the end effector 20. As described in more detail below, the robot controller 24 uses algorithms to convert the 2D images from the camera 22 into a 3D pose of the object 14 so that the end effector 20 is properly oriented.

[0012] As will be described below, in order for the controller 24 to accurately determine the 3D pose of the object 14, in one embodiment, a learning-based neural network is used in the algorithms and software executed by the controller 24 to extract features of the object. The neural network must be trained with data acquired from a representative object. FIG. 2 is a flow-type diagram 30 showing a process for acquiring image data from an object 32 identical to the object 14 in the system 10 using a 2D camera 40 used to train the neural network. The object 32 is placed on a marker board 34 having an appropriate number of markers 36 with special detectable designs that are used to identify the ground truth or actual position of the object 32. Only one of the markers 36 on the board 34 is shown, and the offset distance represented by a line 38 between the marker 34 and the object 32 is measured.

[0013] The camera 40 provides one or more 2D images 42, here four images, of the marker board 34 at different angles. The process then performs a projection on the image 42 to determine the rotation and position of one of the markers 36 identified by the coordinate frame 44. The algorithm then adds the previously measured offset between the marker 36 and the object 32 to the rotation and position of the marker 36 to determine the rotation and position of the object 32. Once the true position of the object 32 is determined, the positions of feature points 46, such as corners of the object 32, can be calculated using known feature point extraction processes, where at least four feature points are required. An algorithm, which may use, for example, a Gaussian distribution process, looks at each identified feature point 46 in the image 42 and generates a "heat map" 48 of that point 46, where each heat map 48 determines the likelihood that one of the feature points of the object 32 is present at a particular location, depending on the intensity of the color. Locations in the image 42 where feature points are believed to be present, such as areas 50, are assigned a particular color, such as red. Also, since it is unlikely that a feature point exists away from region 50, the color changes to blue, for example region 52.

[0014] The data provided by the 2D image 42 and the corresponding heat map 48 generated by the process shown in diagram 30 are then used to train a neural network using known AI training schemes and used in the system 10 to identify the 3D pose of the object 14. FIG. 3 is a diagram of a neural network 60 including multiple nodes 62, each with adjustable weights W, which is intended to represent the neural network discussed herein in a schematic manner. The neural network 60 has an input layer 64 that receives individual pixel data from the 2D image, multiple residual block layers 66 and 68 that identify features within the pixel data of the 2D image, and multiple convolutional layers 70, each node 62 of the convolutional layer 70 providing a heat map. It is noted that the convolutional layer 70 and the residual block layers 66 and 68 are standard modules utilized in the field of deep learning. These types of convolutional layers are composed of multiple filter kernels that perform convolution operations on the image input or previous convolutional layers, so that edge, texture, or contour information can be extracted from the input. The residual block layer is a structure that contains two convolutional layers with skip connections, where the output of the second convolutional layer is appended to the input of the first convolutional layer to serve as the block output.

[0015] 4 is a flow-type diagram 80 showing an algorithm running in the controller 24 employing a learning-based neural network 78 that uses a trained neural network to estimate the 3D pose of the object 14 from a 2D image 82 of the object 14 captured by the camera 22. The image 82 is provided to an input layer 84 and multiple successive residual block layers 86 and 88 including a feed-forward loop in the neural network 78 running in the AI ​​software of the controller 24, for feature extraction such as gradients, edges, contours, etc. of possible feature points of the object 14 in the image 82 using a filtering process. The image with the extracted features is provided to multiple successive convolutional layers 90 in the neural network 78, and the possible feature points resulting from the extracted features are defined as a series of heat maps 92, one for each feature point. The heat maps 92 represent the likelihood that the feature point is present on the object 14 based on the color of the heat maps 92. An image 94 is generated using the image 82 of the object 14, which includes feature points 96 for all feature points from all heat maps 92. Each feature point 96 is assigned a confidence value based on the color of that feature point's heat map 92, and feature points 96 that do not have a confidence value above a certain threshold are not used.

[0016] The images 94 are then compared to a nominal or virtual 3D CAD model of the object 14, having the same feature points, in a pose estimation processor 98 to provide an estimated 3D pose 100 of the object 14. One suitable algorithm for comparing the images 94 to the CAD model is known in the art as Perspective n Points (PnP). In general, the PnP process estimates the pose of the object relative to a calibrated camera given a set of n 3D points of the object in the world coordinate frame and the corresponding 2D projections in the images from the camera. The pose includes six degrees of freedom (DOF), consisting of the rotation (roll, pitch, yaw) and 3D translation of the object relative to the coordinate frame of the camera.

[0017] 5 is an illustration 104 showing how the PnP process can be performed in this embodiment to obtain a 3D pose of the object 14. The illustration 104 shows a 3D object 106 representing the object 14 in a ground truth or real-world position. The object 106 is observed by a camera 112 representing the camera 22 and projected as a 2D object image 108 onto a 2D image plane 110. Here, the object image 108 represents the image 94, and the point 102 on the image 108 is a feature point predicted by the neural network 78 and represents the point 96. The illustration 104 also shows a virtual 3D CAD model 114 of the object 14 with a feature point 132 at the same position as the feature point 96, which is randomly placed in front of the camera 112 and projected onto the image plane 110 as a 2D model image 116 that also includes the projected feature point 118. The CAD model 114 is rotated and translated in front of the camera 112, such that the model image 116 is rotated and translated so that the distance between each feature point 118 of the model image 116 and the corresponding feature point 102 of the object image 108 is minimized, i.e., so that the images 116 and 108 are aligned. Once the model image 116 is aligned as closely as possible to the object image 108, the pose of the CAD model 114 relative to the camera 112 becomes the estimated 3D pose 100 of the object 14.

[0018] This analysis is expressed in equation (1) for one of the corresponding feature points between image 108 and image 116, and equation (1) is used for all feature points of images 108 and 116. In equation (1), V i is one of the feature points 132 of the CAD model 114, and v i are the corresponding projected feature points 102 in the model image 116, and a i is one of the feature points 102 in the object image 108, R and T are the rotation and translation of the CAD model 114 relative to the camera 112, respectively, the symbol "'" is the vector transpose, and ∀ refers to any feature point with index i. By solving equation (1) with an optimization solver, the optimal rotation and translation can be calculated and thus the 3D pose 100 of the object 14 is estimated.

[0019]

number

[0020] The above process of using a learning-based neural network to estimate the 3D pose of the object 14 provides a high level of accuracy, but can be improved. For example, a small pixel error in the 2D projection of the object 14 on the image plane 110 in the x-axis or y-axis orthogonal to the optical z-axis of the camera 112 will result in a relatively large error in the z-axis, which creates significant uncertainty in the object's position and reduces the performance of the 3D pose estimation. Therefore, this disclosure also proposes a 3D pose estimation process that reduces the uncertainty in the z-axis by using multiple cameras.

[0021] 6 is a diagram of a camera system 120 including a first camera 122 and a second camera 124 oriented relative to an x, y and z coordinate frame, where the z-axis is along the optical axis of the cameras 122 and 124. An illustrated uncertainty region 126 indicates the limit of accuracy with which the images from the camera 122 can estimate the position of the object 14, and an illustrated uncertainty region 128 indicates the limit of accuracy with which the images from the camera 124 can estimate the position of the object 14. As shown, both cameras 122 and 124 are highly accurate in the x and y axes, but have limited accuracy in the z-axis, with accuracy in all axes depending on the distance between the camera 122 or 124 and the imaged object 14. However, by fusing the 3D pose estimates provided by both cameras 122 and 124, the accuracy of the estimate of the 3D pose of the object 14 in the z-axis is significantly improved, as shown by the uncertainty region 130. In other words, in this illustration, the z-axis of one of the cameras 122 and 124 is the x-axis of the other of the cameras 122 and 124, so the overall uncertainty in the estimation process is the uncertainty in the x-axis direction.

[0022] In this non-limiting embodiment, the cameras 122 and 124 are oriented such that their optical axes are orthogonal to one another and the distance from the cameras 122 or 124 to the object 14 is optimal to provide the smallest uncertainty region 130. However, practical systems may be limited in terms of where the cameras 122 and 124 with their z-axes can be placed relative to one another, the distance between the cameras 122 or 124 and the object 14, and other limiting factors such as lighting and the type of camera used. This may lead to optical axes that are not orthogonal to one another and other limitations. In those systems, it may be desirable to use more than one camera to bring the size of the uncertainty region closer to the minimum uncertainty region 130.

[0023] 7 is a block diagram of a system 140 for estimating the 3D pose of an object 14 using 2D cameras 122 and 124 and a learning-based neural network of the type described above. For each angle of the object 14, camera 122 provides a 2D image 142 and camera 124 provides a 2D image 144. Image 142 is provided to processor 146 and image 144 is provided to processor 148. Processors 146 and 148 generate feature point images 150 and 152, respectively, as well as images 94 from their perspectives. Processors 146 and 148, for example, generate heat maps using input layer 84, multiple successive residual block layers 86 and 88, and multiple successive convolutional layers 90, which are then used to generate images 150 and 152 in the manner described above. Images 150 and 152 are fused in a pose estimation block 154 to generate a more accurately estimated 3D pose 156 of the object 14 than a single camera 22 can provide, the pose estimation process also using the PnP algorithm as described above.

[0024] 8 is an illustration 160 similar to illustration 104, with like elements indicated by the same reference numbers, illustrating the PnP process for image fusion in block 154 described above. In illustration 160, camera 162 represents camera 122 and camera 164 represents camera 124. Illustration 160 shows a 2D model image 166 with feature points 172 that is a projection of CAD model 114 onto the 2D image plane 170 for camera 164, and a 2D object image 168 with feature points 174 that is a projection of 3D object 106 onto the image plane 170 for camera 164. The CAD model 114 is rotated and translated in front of both cameras 162 and 164 to simultaneously obtain an optimal model pose for both cameras 162 and 164, such that feature points 118 of model image 116 on image plane 110 are as close as possible to feature points 102 of object image 108, and feature points 172 of model image 166 on image plane 170 are as close as possible to feature points 174 of object image 168. Once this is achieved, an optimal orientation of the 3D model 114 is obtained, which represents the 3D pose of object 14.

[0025] This analysis is represented by equation (2) for one of the corresponding feature points between the images 108, 116 and 166, 168, and equation (2) is used for all feature points of the images 108, 116 and 166, 168. In equation (2), u i is one of the feature points 172 in the model image 166, and b i are the corresponding feature points 174 in the object image 168.

[0026]

number

[0027] The above-mentioned technique of extracting features from 2D images to estimate the 3D pose of the object 14 using a learning-based neural network is one suitable feature extraction technique, although there are others. In the following description, a feature extraction technique using deep learning and vector field estimation is described that can satisfactorily replace the learning-based technique. As described in more detail below, the vector field estimation process replaces the neural network 78, which includes an input layer 84, multiple successive residual block layers 86 and 88, and multiple successive convolutional layers 90 that generate a heat map 92, with a vector field estimation element. It is emphasized that although a vector field estimation process is described here for obtaining the 3D pose of an object, the process may be applied to any other suitable process that requires feature extraction from an image.

[0028] 9 is a flow-type diagram 180 illustrating a vector field estimation process operating in the controller 24 to generate a 3D pose of the object 14 using a 2D image 82 of the object 14 captured by the camera 22, with like elements designated with the same reference numerals. The image input is a three-channel RGB (Red-Green-Blue) image defined by a three-dimensional matrix N×N×3, where N is the height and width of the image 82. The image 82 is provided to an encoder / decoder neural network 182, which includes an encoder network 184 that extracts possible key features in the image 82, and a decoder network 186 that generates an image from the extracted features and identifies possible feature points of the object 14, as described below. Here, the networks 184 and 186 include multiple neural network layers of the type described above. In particular, in one non-limiting embodiment, the neural network layers of the encoder network 184 include a convolutional layer 188, a batch normalization (BN) layer 190, and a rectified linear unit (ReLu) layer 192, and the neural network layers of the decoder network 186 include a pooling layer 194, an upsampling layer 196, and a softmax layer 198.

[0029] The network 182 processes the three-dimensional matrix of the image 82 and generates from the three channels a one-dimensional (N×N×1) mask image 200 and a two-dimensional (N×N×2) vector field image 202 for each identified feature point. To generate the mask image 200, the network 182 assigns a digital 1 or 0 to each pixel in the image 82 based on whether or not a portion of the object 14 is present at the particular pixel, where pixels in light areas of the mask image 200 have a digital 1 and pixels in dark areas of the mask image 200 have a digital 0. Each vector field image 202 includes a number of arrows, each defined by an x-direction vector value and a y-direction vector value provided to each pixel in the image 82, with the arrows pointing towards one of the feature points of the image 82. Thus, each pixel in the image 82 has the ability to predict the direction to its feature. The values ​​of pixels in the mask image 200 are multiplied by the values ​​of corresponding pixels in the vector field image 202 so that arrows that are not associated with pixels that contain part of the object 14 and do not provide useful directions to feature points are discarded, as shown in vector field extraction image 204.

[0030] Each arrow in the vector field extraction image 204 represents a direction to one of the feature points, but does not indicate the distance to that feature point. Thus, two object pixels p i and p j For all combinations of , the process finds the point x ij Identify the point x ij Vector of v i and v j intersect with each other as shown in the vector intersection image 206. ij In pixel p i and point x ij The distance between pixels p j and point x ij The distance between i and p j Depending on the magnitude of the gradient at, a score is assigned. For example, at pixel p i and point x ij and the distance between pixel p jand point x ij The smaller the distance between pixel p, the higher the score. i and p j The larger the magnitude of the gradient at x, the higher the score. As a result, there are many points x, each of which has a score ij A grid containing most of the points x ij Regions in the image that contain , identify possible locations of interest points, as shown in the point voting image 208.

[0031] The vector field image 202 shows the vector arrows of one possible feature point identified by the network 182. The network 182 also generates a separate vector field image 202 for each possible feature point multiplied by the mask 200. All feature points of all point voting images 208 are combined in a single image, such as image 94 containing feature points 96, with each individual feature point 96 contributed by one of the point voting images 208. The images 94 can then be processed by a PnP process in a pose estimation processor 98 to generate the 3D pose 100 in the manner described above. Additionally, due to the uncertainty along the optical axis of a single camera as described above, multiple cameras can be used, with each of the images from the cameras being processed by the vector field estimation process.

[0032] As mentioned before, the vector field estimation process can be applied to other processes that require the extraction of feature points. For example, one application in robotics is to find the center of an object being picked up by a robot, where the orientation of the object is not required. Another application could be to determine the measurements of an object for quality assurance purposes.

[0033] As will be appreciated by those skilled in the art, some of the various steps and processes discussed herein to describe the present invention may refer to operations performed by a computer, processor, or other electronic computing device that manipulates and / or transforms data using electrical phenomena. These computers and electronic devices may use a variety of volatile and / or non-volatile memories, including non-transitory computer-readable media on which executable programs having various codes or executable instructions executable by the computer or processor are stored. The memory and / or computer-readable media may include memories and other computer-readable media of all forms and types.

[0034] The foregoing description discloses and describes merely exemplary embodiments of the present disclosure. Those skilled in the art will readily recognize from such description, together with the accompanying drawings and claims, that various changes, modifications, and variations can be made without departing from the spirit and scope of the present disclosure, as defined in the following claims. [Configuration 1] 1. A method for estimating a three-dimensional pose of a subject, comprising: positioning a first two-dimensional camera such that the first camera is oriented toward the object along a first optical axis; acquiring a two-dimensional image of the object with the first two-dimensional camera; extracting feature points from the two-dimensional image from the first two-dimensional camera using a first feature extraction process; positioning a second two-dimensional camera such that it is oriented toward the object along a second optical axis that is angled relative to the first optical axis; acquiring a two-dimensional image of the object with the second two-dimensional camera; extracting feature points from the two-dimensional image from the second two-dimensional camera using a second feature extraction process; estimating a 3D pose of the object using the feature points extracted from the first and second feature extraction processes; and The method includes: [Configuration 2] 2. The method of claim 1, wherein the first optical axis and the second optical axis are perpendicular to each other. [Configuration 3] The method of claim 1, wherein estimating the 3D pose of the object using the feature points extracted from the first and second feature extraction processes includes using a perspective n-point algorithm that compares a 2D projection of a 3D virtual model of the object with a 2D projection of the object having feature configurations for the first and second feature extraction processes, and then averaging the two 3D poses. [Configuration 4] 2. The method of claim 1, wherein the first and second feature extraction processes use a learning-based neural network, and the node weights of the neural network are trained using representative objects of the object from a collected training set. [Configuration 5] 5. The method of claim 4, wherein the first and second feature extraction processes use the neural network to extract a plurality of feature points of the object from the two-dimensional image, generate a heat map for each extracted feature point that identifies a probability of the location of the feature point of the object, and convert each heat map to a location of the feature point of the object. [Configuration 6] The method of claim 5, wherein the probability of the location of the feature point in the heat map is represented as a color in the heat map. [Configuration 7] The method of claim 5, wherein the probability of a feature point location of the object is assigned a confidence value that it is a feature. [Configuration 8] 2. The method of claim 1, wherein the first and second feature extraction processes use vector field estimation processes. [Configuration 9] 9. The method of claim 8, wherein the first and second feature extraction processes also use deep learning neural networks. [Configuration 10] 2. The method of claim 1, wherein the method is performed in a robotic system and the object is picked up by the robot. [Configuration 11] 1. A method for estimating a three-dimensional pose of a subject, comprising: positioning a first two-dimensional camera such that the first camera is oriented toward the object along a first optical axis; acquiring a two-dimensional image of the object with the first two-dimensional camera; extracting feature points from the two-dimensional image from the first two-dimensional camera using a first feature extraction process that uses a learning-based neural network whose nodes have weights trained with representative objects of the object from a collected training set; positioning a second two-dimensional camera such that it is oriented toward the object along a second optical axis that is angled relative to the first optical axis; acquiring a two-dimensional image of the object with the second two-dimensional camera; extracting feature points from the two-dimensional image from the second two-dimensional camera using a second feature extraction process that uses a learning-based neural network whose nodes have weights trained with representative objects of the object from a collected training set; estimating a 3D pose of the object using the feature points extracted from the first and second feature extraction processes; and Including, estimating the 3D pose of the object using the feature points extracted from the first and second feature extraction processes includes using a perspective n-point algorithm that compares a 2D projection of a 3D virtual model of the object with a 2D projection of the object having feature locations for the first and second feature extraction processes and then averaging the two 3D poses. [Configuration 12] 12. The method of claim 11, wherein the first optical axis and the second optical axis are perpendicular to each other. [Configuration 13] 12. The method of claim 11, wherein the first and second feature extraction processes use the neural network to extract a plurality of feature points of the object from the two-dimensional image, generate a heat map for each extracted feature point that identifies a probability of the location of the feature point of the object, and convert each heat map to a location of a feature of the object. [Configuration 14] 14. The method of claim 13, wherein the probability of the location of the feature point in the heat map is represented as a color in the heat map. [Configuration 15] 12. The method of claim 11, wherein the probability of a feature point location of the object is assigned a confidence value that it is a feature. [Configuration 16] 1. A system for estimating a three-dimensional pose of a subject, comprising: means for positioning a first two-dimensional camera such that the first camera is oriented toward the object along a first optical axis; means for acquiring a two-dimensional image of the object by the first two-dimensional camera; means for extracting feature points from the two-dimensional image from the first two-dimensional camera using a first feature extraction process; means for positioning a second two-dimensional camera so as to be oriented towards the object along a second optical axis that is angled with respect to the first optical axis; means for acquiring a two-dimensional image of the object by the second two-dimensional camera; means for extracting feature points from the two-dimensional image from the second two-dimensional camera using a second feature extraction process; means for estimating a 3D pose of the object using the feature points extracted from the first and second feature extraction processes; A system having [Configuration 17] 17. The system of configuration 16, wherein the first optical axis and the second optical axis are perpendicular to each other. [Configuration 18] 17. The system of configuration 16, wherein the means for estimating the 3D pose of the object using the feature points extracted from the first and second feature extraction processes uses a perspective n-point algorithm that compares a 2D projection of a 3D virtual model of the object with a 2D projection of the object having feature configurations for the first and second feature extraction processes and then averaging the two 3D poses. [Configuration 19] 17. The system of claim 16, wherein the first and second feature extraction processes use a learning-based neural network, and the node weights of the neural network are trained using representative objects of the object from a collected training set. [Configuration 20] 17. The system of claim 16, wherein the first and second feature extraction processes use vector field estimation processes.

Claims

1. 1. A method for estimating a three-dimensional pose of a subject, comprising: positioning a first two-dimensional camera such that the first camera is oriented toward the object along a first optical axis; acquiring a two-dimensional image of the object with the first two-dimensional camera; extracting feature points from the two-dimensional image from the first two-dimensional camera using a first feature extraction process; positioning a second two-dimensional camera such that the second camera is oriented toward the object along a second optical axis that is angled relative to the first optical axis; acquiring a two-dimensional image of the object with the second two-dimensional camera; extracting feature points from the two-dimensional image from the second two-dimensional camera using a second feature extraction process; estimating a three-dimensional pose of the object using the feature points extracted from the first and second feature extraction processes; and Including, estimating the three-dimensional pose of the object using the feature points extracted from the first and second feature extraction processes includes using a perspective n-point algorithm that compares a two-dimensional projection of a three-dimensional virtual model of the object with a two-dimensional projection of the object having feature configurations for the first and second feature extraction processes and then averaging the two three-dimensional poses.

2. The method of claim 1 , wherein the first optical axis and the second optical axis are orthogonal to each other.

3. 2. The method of claim 1, wherein the first and second feature extraction processes use a learning-based neural network, the weights of the nodes of the neural network being trained using representative objects of the object from a collected training set.

4. 4. The method of claim 3, wherein the first and second feature extraction processes use the neural network to extract a plurality of feature points of the object from the two-dimensional image, generate for each extracted feature point a heat map identifying a probability of a location of the feature point of the object, and convert each heat map to a location of the feature point of the object.

5. The method of claim 4 , wherein the probability of the feature point's location in the heat map is represented as a color in the heat map.

6. The method of claim 4 , wherein the probability of a feature point location of the object is assigned a confidence value that it is a feature.

7. The method of claim 1 , wherein the first and second feature extraction processes use vector field estimation processes.

8. The method of claim 7 , wherein the first and second feature extraction processes also use deep learning neural networks.

9. The method of claim 1 , wherein the method is performed in a robotic system and the object is picked up by a robot.

10. 1. A method for estimating a three-dimensional pose of a subject, comprising: positioning a first two-dimensional camera such that the first camera is oriented toward the object along a first optical axis; acquiring a two-dimensional image of the object with the first two-dimensional camera; extracting feature points from the two-dimensional image from the first two-dimensional camera using a first feature extraction process that uses a learning-based neural network whose nodes have weights trained with representative objects of the object from a collected training set; positioning a second two-dimensional camera such that the second camera is oriented toward the object along a second optical axis that is angled relative to the first optical axis; acquiring a two-dimensional image of the object with the second two-dimensional camera; extracting feature points from the two-dimensional image from the second two-dimensional camera using a second feature extraction process that uses a learning-based neural network whose nodes have weights trained with representative objects of the object from a collected training set; estimating a three-dimensional pose of the object using the feature points extracted from the first and second feature extraction processes; and Including, estimating the three-dimensional pose of the object using the feature points extracted from the first and second feature extraction processes includes using a perspective n-point algorithm that compares a two-dimensional projection of a three-dimensional virtual model of the object with a two-dimensional projection of the object having feature configurations for the first and second feature extraction processes and then averaging the two three-dimensional poses.

11. The method of claim 10 , wherein the first optical axis and the second optical axis are orthogonal to each other.

12. 11. The method of claim 10, wherein the first and second feature extraction processes use the neural network to extract a plurality of feature points of the object from the two-dimensional image, generate for each extracted feature point a heat map identifying a probability of the location of the feature point of the object, and convert each heat map to a feature location of the object.

13. The method of claim 12 , wherein the probability of the feature point's location in the heat map is represented as a color in the heat map.

14. The method of claim 10 , wherein the probability of a feature point location of the object is assigned a confidence value that it is a feature.

15. 1. A system for estimating a three-dimensional pose of a subject, comprising: means for positioning a first two-dimensional camera such that the first camera is oriented toward the object along a first optical axis; means for acquiring a two-dimensional image of the object by the first two-dimensional camera; means for extracting feature points from the two-dimensional image from the first two-dimensional camera using a first feature extraction process; means for positioning a second two-dimensional camera so as to be oriented toward the object along a second optical axis that is angled relative to the first optical axis; means for acquiring a two-dimensional image of the object by the second two-dimensional camera; means for extracting feature points from the two-dimensional image from the second two-dimensional camera using a second feature extraction process; means for estimating a three-dimensional pose of the object using the feature points extracted from the first and second feature extraction processes; having The means for estimating the three-dimensional pose of the object using the feature points extracted from the first and second feature extraction processes uses a perspective n-point algorithm that compares a two-dimensional projection of a three-dimensional virtual model of the object with a two-dimensional projection of the object having feature locations for the first and second feature extraction processes and then averaging the two three-dimensional poses.

16. The system of claim 15 , wherein the first optical axis and the second optical axis are orthogonal to each other.

17. 16. The system of claim 15, wherein the first and second feature extraction processes use a learning-based neural network, the weights of the nodes of the neural network being trained using representative objects of the object from a collected training set.

18. The system of claim 15 , wherein the first and second feature extraction processes use vector field estimation processes.

Images