3d pose estimation by 2d camera

Abstract

A system and method for obtaining the 3D pose of an object using 2D images from a 2D camera and a learning-based neural network. The neural network extracts multiple features from the 2D image of the object and generates a generated heatmap for each of the extracted features, which uses color representation to identify the probability of the location of feature points on the object. The method provides a feature point image including each feature point from the respective heatmaps on the 2D image, and estimates the 3D pose of the object by comparing the feature point image with a 3D virtual CAD model of the object.

Description

Technical Field

[0001] This disclosure generally relates to a system and method for obtaining the 3D pose of an object using a 2D camera, and more specifically, to a system and method for obtaining the 3D pose of an object using a 2D camera and a learning-based neural network, wherein the system and method have a specific application for robot control. Background Technology

[0002] Robots perform multiple tasks including pick-up and placement operations, where they pick up objects and move them from one location (e.g., a conveyor belt) to another (e.g., a collection bin). The position and orientation of the objects, referred to as their 3D pose, vary slightly. Therefore, for a robot to pick up objects effectively, it typically needs to know the object's 3D pose. Various techniques for determining an object's 3D pose for this purpose use 3D cameras that acquire 3D images of the object in real time. These images are identified by distance measurements to numerous points within the camera's field of view, which are then sent to the robot controller. These techniques can construct a 3D map of the scene using structured light and then search for objects in the map by recognizing their 3D features. Using 3D cameras to determine the 3D pose of objects for robotic applications has proven effective. However, 3D cameras are expensive devices and typically operate slower than expected.

[0003] One known technique in the art for obtaining 3D pose of objects for robotic applications employs a 2D camera. This technique captures a single 2D color image of the scene and processes the image using a fully convolutional architecture. An algorithm divides the image into a 2D mesh comprising S×S cells and identifies objects within the cells. The algorithm projects a bounding box around each object in the image and predicts the 2D positions of the eight corners of that box. However, this technique for predicting the corners of the bounding box has limitations in accuracy and low robustness to occlusion.

[0004] Some of the processes mentioned above for determining the 3D pose of an object employ artificial intelligence (AI), a branch of computer science that uses algorithms to allow software applications to learn from their environment and make decisions to achieve specific outcomes. Machine learning is a part of AI, employing software applications that iteratively analyze large amounts of raw input data to extract patterns from the data and allow the software application to learn to perform tasks without being specifically programmed to do so, thereby acquiring its own knowledge. Deep learning is a specific type of machine learning that provides stronger learning performance by representing specific real-world environments as hierarchical structures of increasingly complex concepts.

[0005] Deep learning typically employs a software architecture of several layers of neural networks that perform non-linear processing, where each successive layer receives the output from the layer preceding it. These layers typically include an input layer that receives raw data from sensors, multiple hidden layers that extract abstract features from the data, and an output layer that identifies a specific object based on the feature extraction from the hidden layers. Neural networks consist of neurons or nodes, each with a "weight" that is multiplied by the node's input to obtain the probability of whether something is correct. More specifically, each node has a weight, which is a floating-point number, multiplied by the node's input to generate the node's output, which is a proportion of the input. The neural network is initially "trained" or its weights are set by having it analyze a set of known data under supervised processing and by minimizing a cost function to allow the network to obtain the highest probability of a correct output.

[0006] Deep learning neural networks are frequently used to provide image feature extraction and transformation for the visual detection and classification of objects in images. This involves analyzing video or image streams to identify and classify objects, and learning through this process to better distinguish objects. Therefore, in these types of networks, the system can use the same processing configuration to detect certain objects and classify them differently based on how well the algorithm learns to recognize them. Summary of the Invention

[0007] The following discussion discloses and describes a system and method for obtaining the 3D pose of an object using 2D images from a 2D camera and a learning-based neural network. The neural network extracts multiple features from the 2D image of the object and generates a heatmap for each of the extracted features, which uses color representation to identify the probability of the location of the feature point on the object. The method provides a feature point image comprising multiple feature points from multiple heatmaps of the 2D image, and estimates the 3D pose of the object by comparing the feature point image with a 3D virtual CAD model of the object.

[0008] Additional features of this disclosure will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a robotic system that includes a robot that picks up objects from a conveyor belt and places them in a bin. The system employs a 2D camera for capturing 2D images of the objects and a controller for calculating the 3D pose of the objects using the 2D images.

[0010] Figure 2 This is a flowchart illustrating the process of using a 2D camera to obtain a dataset from objects to train a neural network;

[0011] Figure 3 It is a schematic diagram of a neural network that includes multiple nodes, each with adjustable weights;

[0012] Figure 4 It is a flowchart illustrating a learning-based neural network process that uses a trained neural network to estimate the 3D pose of an object using a 2D image of the object and a neural network.

[0013] Figure 5 This is a schematic diagram illustrating the perspective n-point (PnP) process, which is used to... Figure 4 The process shown determines the 3D pose estimation of the object;

[0014] Figure 6 This is a schematic diagram of a camera system that uses multiple cameras to image an object;

[0015] Figure 7 This is a block diagram of a system for estimating the 3D pose of an object using multiple 2D cameras;

[0016] Figure 8 This is a schematic diagram illustrating the PnP process, which is used to... Figure 7 The system shown estimates the 3D pose of the object; and

[0017] Figure 9 This is a flowchart illustrating the process of vector field estimation for feature points on a prediction object. Detailed Implementation

[0018] The following discussion of embodiments of this disclosure relating to systems and methods for obtaining the 3D pose of an object using a 2D camera and employing learning-based neural networks is merely exemplary in nature and is in no way intended to limit the invention or its applications or uses. For example, the system and method have applications for determining the position and orientation of an object grasped by a robot. However, the system and method may have other applications.

[0019] Figure 1This is a schematic diagram of a robot system 10, which includes a robot 12 that picks up objects 14 from a conveyor belt 16 and places them in a collection bin 18. System 10 is intended to represent any type of robot system that can benefit from the discussion herein, wherein robot 12 can be any robot suitable for this purpose. As the objects 14 move down the conveyor belt 16, each object 14 has a different orientation and position relative to other objects 14 on the conveyor belt 16. In order for robot 12 to effectively grasp and pick up objects 14, it is necessary to be able to position the end effector 20 on robot 12 in the appropriate position and orientation before grasping the object 14. To achieve this, system 10 includes a 2D camera 22 positioned relative to the object 14 at a desired location, which provides a 2D image to a robot controller 24 that controls robot 12 to move the end effector 20. As will be discussed in detail below, robot controller 24 employs an algorithm to convert the 2D image from camera 22 into a 3D pose of the object 14 in order to properly guide the end effector 20.

[0020] As will be discussed, in order for the controller 24 to accurately determine the 3D pose of the object 14, in one embodiment, a learning-based neural network is employed in the algorithm and software for extracting object features running in the controller 24, wherein the neural network needs to be trained with data obtained from representative objects. Figure 2 Flowchart 30 illustrates the process of obtaining image data, which will be used to train a neural network, from an object 32 identical to object 14 in system 10 using a 2D camera 40. The object 32 is placed on a marker plate 34 with an appropriate number of markers 36, and the offset distance between the markers 34 and the object 32, indicated by lines 38, is measured. The markers 36 have a special detectable design that will be used to identify the true or actual position of the object 32, with only one marker 36 shown on the plate 34.

[0021] Camera 40 provides one or more 2D images 42 of the marker board 34 at different angles, here four images. The process then projects the images 42 to determine the rotation and position of one of the individual markers 36 identified by the coordinate frame 44. The algorithm then adds the previously measured offset between the marker 36 and the object 32 to the rotation and position of the marker 36 to determine the rotation and position of the object 32. Once the true position of the object 32 is determined, the positions of feature points 46 on the object 32, such as corners, can be calculated using a known feature point extraction process, requiring at least four feature points. An algorithm employing, for example, a Gaussian distribution process is used to examine each feature point 46 identified in the image 42 and generate a "heatmap" 48 of that point 46, where each heatmap 48 identifies the probability of one of the individual feature points of the object 32 being present at a certain location by color intensity. For example, locations in the image 42 where a feature point is believed to be present, such as in region 50, will be assigned a certain color, such as red, and the color will change to, for example, blue as the probability of a feature point being present decreases in locations away from region 50, such as in region 52.

[0022] Then, the data provided by the 2D image 42 and the corresponding heatmap 48 generated by the process shown in FIG30 are used to train a neural network using a known AI training scheme, which will then be used in system 10 to identify the 3D pose of object 14. Figure 3 This is a schematic diagram of a neural network 60 comprising multiple nodes 62, each node having adjustable weights W, wherein network 60 is intended to generally represent the neural network discussed herein. Neural network 60 includes an input layer 64 receiving individual pixel data from a 2D image, multiple residual block layers 66 and 68 identifying features in the pixel data of the 2D image, and multiple convolutional layers 70, wherein each node 62 in the convolutional layers 70 provides a heatmap. Note that convolutional layers 70 and residual block layers 66 and 68 are standard modules used in the field of deep learning. These types of convolutional layers consist of multiple filtered convolutional kernels that perform convolution operations on the image or the input of a previous convolutional layer, allowing edge, texture, or contour information to be extracted from that input. A residual block layer is a structure containing two convolutional layers with skip connections, wherein the output of the second convolutional layer is added to the input of the first convolutional layer to serve as a block output.

[0023] Figure 4Flowchart 80 illustrates an algorithm running in controller 24 that uses a learning-based neural network 78 to estimate the 3D pose of object 14 based on a 2D image 82 of object 14 captured by camera 22. Image 82 is fed into input layer 84 and multiple consecutive residual block layers 86 and 88, which include feedforward loops in neural network 78 running in AI software within controller 24. This AI software uses a filtering process to extract features, such as gradients, edges, contours, etc., of possible feature points on object 14 in image 82. The image including the extracted features is fed into multiple consecutive convolutional layers 90 in neural network 78, which define the possible feature points obtained from the extracted features as a series of heatmaps 92, one heatmap per feature point, with the heatmaps indicating the likelihood of the feature point's presence on object 14 based on the color in the heatmap 92. Image 94 is generated using image 82 of object 14, which includes multiple feature points 96 from all feature points of all heatmaps 92, wherein each feature point 96 is assigned a confidence value based on the color of the heatmap 92, and wherein feature points 96 whose confidence values ​​are not higher than a certain threshold are not used.

[0024] Then, in the pose estimation processor 98, image 94 is compared with a nominal or virtual 3D CAD model of object 14 having the same feature points to provide an estimated 3D pose 100 for object 14. A suitable algorithm for comparing image 94 with the CAD model is referred to in the art as Perspective n-Point (PnP). Generally, the PnP process estimates the pose of an object relative to a calibrated camera given a set of n 3D points of the object in a world coordinate system and their corresponding 2D projections in an image from the camera. The pose comprises six degrees of freedom (DOF) consisting of rotation (roll, pitch, and yaw) and 3D translation of the object relative to the camera coordinate system.

[0025] Figure 5This is a schematic diagram 104 illustrating how the PnP process is implemented to obtain the 3D pose of object 14 in this example. Schematic diagram 104 shows a 3D object 106 representing object 14 at a real or actual location. Object 106 is observed by camera 112, representing camera 22, and projected onto a 2D image plane 110 as a 2D object image 108, where object image 108 represents image 94, and where point 102 on image 108 is a feature point of representation point 96 predicted by neural network 78. Schematic diagram 104 also shows a virtual 3D CAD model 114 of object 14 with feature point 132 at the same location as feature point 96, which is randomly placed in front of camera 112 and projected onto the 2D image plane 110 as a 2D model image 116 that also includes the projected feature point 118. The CAD model 114 is rotated and translated in front of the camera 112, which in turn rotates and translates the model image 116 in an attempt to minimize the distance between each feature point 118 on the model image 116 and the corresponding feature point 102 on the object image 108, i.e., aligning images 116 and 108. Once the model image 116 is aligned with the object image 108 as optimally as possible, the pose of the CAD model 114 relative to the camera 112 is the estimated 3D pose 100 of the object 14.

[0026] For any one of the corresponding feature points between images 108 and 116, the analysis is described by equation (1), where equation (1) applies to all feature points of images 108 and 116.

[0027]

[0028] Among them, V i It is one of the feature points 132 on the CAD model 114, ν i It is the corresponding projected feature point 102, α in model image 116 i It is one of the feature points 102 on the object image 108, R is the rotation of the CAD model 114 relative to the camera 112, T is the translation of the CAD model 114 relative to the camera 112, and the symbol ' is the vector transpose. This refers to any feature point with index i. By solving equation (1) with an optimization solver, the optimal rotation and translation can be calculated, thus providing an estimate of the 3D pose 100 of object 14.

[0029] The above-described process of estimating the 3D pose of object 14 using a learning-based neural network provides high accuracy. However, improvements can be made. For example, small pixel errors in the 2D projection of object 14 onto the image plane 110 on the x or y axis, which is orthogonal to the optical z-axis of camera 112, can lead to relatively large errors on the z-axis, which can significantly introduce object position uncertainty and degrade 3D pose estimation performance. Therefore, this disclosure also proposes a 3D pose estimation process that reduces z-axis uncertainty by using multiple cameras.

[0030] Figure 6 This is a schematic diagram of a camera system 120 including a first camera 122 and a second camera 124 oriented relative to the x, y, and z coordinate frames, where the z-axis is along the optical axis of cameras 122 and 124. Uncertainty region 126 is shown, describing the accuracy limitations of the image from camera 122 in estimating the position of object 14, and uncertainty region 128 is shown, describing the accuracy limitations of the image from camera 124 in estimating the position of object 14. As shown, cameras 122 and 124 are highly accurate on the x and y axes, but their accuracy on the z-axis is limited, where accuracy on all axes depends on the distance between camera 122 or 124 and the imaged object 14. However, by fusing the 3D pose estimates provided by both cameras 122 and 124, the accuracy of estimating the 3D pose of object 14 on the z-axis is significantly improved, as shown in uncertainty region 130. In other words, for this schematic diagram, the z-axis of one of cameras 122 and 124 is the x-axis of the other camera 122 and 124, such that the general uncertainty of the estimation process is the uncertainty in the x-axis direction.

[0031] In this non-limiting embodiment, cameras 122 and 124 are oriented such that their optical axes are orthogonal to each other and have an optimal distance from either camera 122 or 124 to object 14 to provide a minimum uncertainty region 130. However, real-world systems may be limited by the fact that cameras 122 and 124 can be positioned relative to each other along the z-axis and by the distance between cameras 122 or 124 and object 14, as well as other limiting factors that may lead to non-orthogonal optical axes and other limitations, such as illumination, the type of camera used, etc. In those systems, it may be necessary to employ more than one camera to reduce the size of the uncertainty region to be closer to the minimum uncertainty region 130.

[0032] Figure 7This is a block diagram of a system 140 for estimating the 3D pose of object 14 using 2D cameras 122 and 124 and a learning-based neural network of the type described above. Camera 122 provides a 2D image 142 of object 14 from their respective angles, and camera 124 provides a 2D image 144 of object 14. Image 142 is provided to processor 146, and image 144 is provided to processor 148. Processors 146 and 148 generate feature point images 150 and 152, respectively, that resemble image 94 from their respective angles. Processors 146 and 148 generate heatmaps using, for example, an input layer 84, multiple consecutive residual block layers 86 and 88, and multiple consecutive convolutional layers 90. These heatmaps are then used to generate images 150 and 152 in the manner described above. Images 150 and 152 are fused in a pose estimation block 154, which generates a more accurate estimated 3D pose 156 of object 14 than that provided by a single camera 22. The pose estimation process also uses the PnP algorithm as described above.

[0033] Figure 8 Similar to schematic diagram 104, schematic diagram 160, in which the same elements are identified by the same reference numerals, describes the PnP process for image fusion as discussed in block 154. In schematic diagram 160, camera 162 represents camera 122, and camera 164 represents camera 124. Schematic diagram 160 shows a 2D model image 166 with feature points 172, which is a projection of CAD model 114 onto the 2D image plane 170 of camera 164, and a 2D object image 168 with feature points 174, which is a projection of 3D object 106 onto the image plane 170 of camera 164. The CAD model 114 is rotated and translated in front of cameras 162 and 164 to simultaneously obtain the optimal model pose for both cameras 162 and 164, wherein feature points 118 on the model image 116 on image plane 110 are as close as possible to feature points 102 on object image 108, and feature points 172 on the model image 166 on image plane 170 are as close as possible to feature points 174 on object image 168. Once this is obtained, the optimal orientation of the 3D model 114, representing the 3D pose of object 14, is obtained.

[0034] For each of the corresponding feature points between images 108 and 116 and images 166 and 168, the analysis is described by equation (2), where equation (2) applies to all feature points of images 108 and 116 and images 166 and 168.

[0035]

[0036] Where u i It is one of the feature points 172 on the model image 166, b iIt is the corresponding feature point 174 on the object image 168.

[0037] The technique described above for extracting features from 2D images using a learning-based neural network to obtain a 3D pose estimate of object 14 is a suitable feature extraction technique. Other methods exist. The following discussion describes a feature extraction technique employing deep learning and vector field estimation, which can also replace learning-based techniques with satisfactory results. As will be discussed in detail below, the vector field estimation process replaces the neural network 78, which includes an input layer 84, multiple consecutive residual block layers 86 and 88, and multiple consecutive convolutional layers 90 that generate the heatmap 92, with vector field estimation elements. However, it should be emphasized that although the vector field estimation process is described here for obtaining the 3D pose of an object, this process will be applied to any suitable process requiring feature extraction from an image.

[0038] Figure 9 Flowchart 180 illustrates the vector field estimation process operating in controller 24 for generating a 3D pose of object 14 using a 2D image 82 captured by camera 22, wherein the same elements are identified by the same reference numerals. The image input is a three-channel RGB (red-green-blue) image defined by a three-dimensional matrix N*N*3, where N is the height and width of image 82. Image 82 is provided to encoder / decoder neural network 182, which includes encoder network 184 for extracting possible key features from image 82 and decoder network 186 for generating an image from the extracted features to identify possible feature points on object 14, as described below, wherein both networks 184 and 186 include multiple neural network layers of the type described above. Specifically, in a non-limiting embodiment, the neural network layers in encoder network 184 include convolutional layers 188, batch normalization (BN) layers 190, and rectified linear unit (ReLU) layers 192, and the neural network layers in decoder network 186 include pooling layers 194, upsampling layers 196, and softmax layers 198.

[0039] Network 182 processes the three-dimensional matrix of image 82, generating a one-dimensional (N*N*1) mask image 200 and a two-dimensional (N*N*2) vector field image 202 for each identified feature point from the three channels. To generate the mask image 200, network 182 assigns a number 1 or 0 to each pixel in image 82 based on whether a portion of object 14 is present in a particular pixel, where pixels in bright areas of mask image 200 have the number 1, and pixels in dark areas of mask image 200 have the number 0. Each vector field image 202 includes multiple arrows, each defined by an x-direction vector value and a y-direction vector value provided for each pixel in image 82, wherein the direction of the arrow points toward one of the various feature points in image 82, and thus each pixel in image 82 has the ability to predict the direction of that feature. The values ​​of each pixel in the mask image 200 are multiplied by the values ​​of the corresponding pixels in the vector field image 202, such that those arrows that are not associated with pixels that are part of object 14 and do not provide a useful direction to the feature point are discarded, as shown in the vector field extraction image 204.

[0040] Each arrow in the vector field extraction image 204 provides a direction to one of a plurality of feature points, but does not provide a distance to that feature point. Therefore, for two object pixels p with values ​​in the vector field extraction image 204... i and p j For each combination, the process identifies their vector v i and v j Points x intersecting ij As shown in vector intersection image 206. Based on pixel p... i and point x ij Distance between, pixels p j and point x ij The distance between and the pixel p i and p j The gradient magnitude at each point x ij A score is assigned. For example, pixel p i and point x ij Between and pixel p j and point x ij Smaller distances between pixels result in higher scores, and pixel p i and p j A higher gradient magnitude provides a higher score. The result is a score with many points x. ij A grid, where each point has a score, and the majority of points x in the image are represented by x. ij The area will identify the possible locations of feature points, as shown in point voting image 208.

[0041] Vector field image 202 shows a vector arrow of one of the various possible feature points identified by network 182. Network 182 will also generate a separate vector field image 202 multiplied by mask 200 for each possible feature point. All feature points of all point voting images 208 are combined into a single image, such as image 94 including feature point 96, where each individual feature point 96 is provided by one of the various point voting images 208. Image 94 can then be processed in a PnP process in pose estimation processor 98 to generate 3D pose 100 as described above. Furthermore, due to the uncertainty along the optical axis of a single camera as described above, multiple cameras can be employed, where each image from those cameras will be processed by the vector field estimation process.

[0042] As mentioned, the vector field estimation process can be applied to other processes that require feature point extraction. For example, one application in robotics is finding the center of an object picked up by a robot, where the object's orientation is not required. Another application could be determining object measurements for quality assurance purposes.

[0043] As those skilled in the art will fully understand, the various steps and processes described herein can refer to operations performed by a computer, processor, or other electronic computing device that use electrical phenomena to manipulate and / or transform data. Such computer and electronic devices may employ various volatile and / or non-volatile memories, including non-transient computer-readable media storing executable programs thereon, said executable programs comprising various codes or executable instructions executable by a computer or processor, wherein the memory and / or computer-readable media may include all forms and types of memory and other computer-readable media.

[0044] The foregoing discussion has only disclosed and described exemplary embodiments of this disclosure. Those skilled in the art will readily recognize from such discussion and from the drawings and claims that various changes, modifications, and variations may be made therein without departing from the spirit and scope of this disclosure as defined in the appended claims.

Claims

1. A method for estimating the 3D pose of a real object, the method comprising: A 2D image of the actual object is obtained using a 2D camera; A neural network is used to extract multiple feature points on the actual object from the 2D image; For each of the extracted feature points, generate an actual heatmap, which identifies the probability of the location of the feature point on the actual object; Provide feature point images, which are combinations of feature points from each of the actual heatmaps and the 2D image; as well as The feature point image is used to estimate the 3D pose of the actual object; The method further includes training the neural network by obtaining a dataset, wherein obtaining the dataset includes placing a training object representing the actual object on a marker board including markers, measuring the offset between the markers and the training object on the board, capturing at least one training 2D image of the marker board, identifying the 3D pose of the markers in the training 2D image, adding the offset to the 3D pose of the markers to obtain individual feature points on the training object, generating a separate training heatmap identifying the probability of the location of each feature point, and using the training heatmap to train the neural network.

2. The method of claim 1, wherein estimating the 3D pose of the actual object comprises comparing the feature point image with a 3D virtual model of the actual object.

3. The method according to claim 2, wherein, Estimating the 3D pose of the actual object includes using a perspective n-point algorithm.

4. The method according to claim 1, wherein, The probability of the location of the feature point in the actual heatmap is shown as a color on the actual heatmap.

5. The method according to claim 1, wherein, The probability of the location of the feature point on the actual object is assigned a confidence value that the feature point exists at the location.

6. The method according to claim 1, wherein, Capturing at least one training 2D image includes capturing multiple training 2D images in different orientations.

7. The method according to claim 1, wherein, A Gaussian distribution process is used to generate individual training heatmaps.

8. The method according to claim 1, wherein, Extracting multiple feature points on the actual object includes extracting at least four feature points.

9. The method according to claim 1, wherein, The method is used in a robotic system, and the actual object is being picked up by the robot.

10. A method for estimating the 3D pose of an object picked up by a robot, the method comprising: A 2D image of the object is obtained using a 2D camera; The neural network is used to extract multiple feature points on the object from the 2D image, wherein the weights of each node in the neural network are trained from a collected training set using representative objects of the object; A heatmap is generated for each of the extracted feature points, the heatmap identifying the probability of the location of the feature point on the object, wherein the probability of the location of the feature point in the heatmap is shown as a color on the heatmap; Provide feature point images, which are combinations of feature points from various heatmaps and the 2D image; as well as The 3D pose of the object is estimated by comparing the feature point image with a 3D virtual model of the object using a perspective n-point algorithm. The method further includes training the neural network by obtaining a dataset, wherein obtaining the dataset includes placing a training object representing the object on a marker board including markers, measuring the offset between the markers and the training object on the board, capturing at least one training 2D image of the marker board, identifying the 3D pose of the markers in the training 2D image, adding the offset to the 3D pose of the markers to obtain individual feature points on the training object, generating a separate training heatmap identifying the probability of the location of each feature point, and using the training heatmap to train the neural network.

11. The method according to claim 10, wherein, The probability of the location of the feature point on the object is assigned a confidence value that the feature point exists at the location.

12. The method according to claim 10, wherein, Extracting multiple feature points on the object includes extracting at least four feature points.

13. A system for estimating the 3D pose of an object, the system comprising: A device for obtaining a 2D image of an object using a 2D camera; A means for extracting multiple feature points on an object from a 2D image using a neural network; A device for generating a real heatmap for each of a plurality of extracted feature points, the real heatmap identifying the probability of the location of the feature point on the object; A means for providing feature point images, the feature point images being a combination of individual feature points from each of the actual heatmaps and the 2D image; as well as Apparatus for estimating the 3D pose of the object using the feature point image; The system further includes means for training the neural network by acquiring a dataset, the acquisition of which includes placing a training object representing the object on a marker board including markers, measuring the offset between the markers and the training object on the board, capturing at least one training 2D image of the marker board, identifying the 3D pose of the markers in the training 2D image, adding the offset to the 3D pose of the markers to obtain individual feature points on the training object, generating a separate training heatmap identifying the probability of the location of each feature point, and using the training heatmap to train the neural network.

14. The system according to claim 13, wherein, The device for estimating the 3D pose of the object compares the feature point image with a 3D virtual model of the object.

15. The system according to claim 14, wherein, The device for estimating the 3D pose of the object uses a perspective n-point algorithm.

16. The system according to claim 13, wherein, The probability of the location of the feature point in the actual heatmap is shown as a color on the actual heatmap.

17. The system according to claim 13, wherein, The probability of the location of the feature point on the object is assigned a confidence value that the feature point exists at the location.

18. The system of claim 13, further comprising means for training individual nodes in the neural network from a collected training set using representative objects of the object.

19. The system according to claim 13, wherein, The system is used in robot control, and the object is being picked up by the robot.

Images