3D detection of multiple transparent objects

Abstract

To provide a system and method with which it is possible to acquire the three-dimensional attitude of an object such as a transparent object in a group of objects and enable a robot to take out the object.SOLUTION: A method in a system 30 includes: acquiring the two-dimensional red-green-blue color (RGB) image 32 of an object using a camera; extracting features from the RGB image, allocating labels to pixels in a segmentation image 58 so that objects in the segmentation image have the same label, and executing an image segmentation process using a deep-learning convolutional neural network, thereby generating a segmentation image of the RGB image. The method further includes: separating the segmentation image into a plurality of cutout images 60 including one of the objects; estimating a three-dimensional attitude 72 of each object in each cutout image; and combining the three-dimensional attitudes into a single attitude image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates generally to systems and methods for acquiring three-dimensional poses of objects, and more particularly to a robotic system for acquiring three-dimensional poses of objects that are part of a group of objects, where the system acquires RGB images of the objects, segments the images using image segmentation, crops the segmented images of the objects, and uses a learning-based neural network to acquire the three-dimensional pose of each object in the segmented images. [Background technology]

[0002] Robots perform many tasks, including pick-and-place operations, in which they retrieve and move objects from one location, such as a collection bin, to another location, such as a conveyor belt. Within the bin, the object's position and orientation, known as its three-dimensional pose, can vary slightly. Therefore, in order for a robot to effectively retrieve an object, it often needs to know the object's three-dimensional pose. To determine the three-dimensional pose of an object being retrieved from a bin, some robotic systems employ a three-dimensional camera that generates a two-dimensional red-green-blue (RGB) image of the bin and a two-dimensional grayscale depth map image of the bin. Each pixel in the depth map image has a value that defines the distance from the camera to a particular object. That is, the closer the pixel is to the object, the lower its value. The depth map image identifies distance measurements to points in a point cloud within the camera's field of view. The point cloud is a collection of data points defined by a particular coordinate system, with each point having an x, y, and z value. However, if the object the robot is picking up is transparent, the light will not be accurately reflected from the object's surface, the point cloud generated by the camera will not be valid, and the depth image will not be reliable, making it impossible to reliably identify the object to be picked up.

[0003] U.S. Patent Application No. 16 / 839,274, entitled "3D Pose Estimation from a 2D Camera," filed April 3, 2020, assigned to the assignee of the present application, and incorporated herein by reference, discloses a robotic system for acquiring the 3D pose of an object using 2D images from a 2D camera and a learning-based neural network capable of identifying the 3D pose of a transparent object being extracted. The neural network extracts multiple features on the object from the 2D image and generates a heat map for each extracted feature, which identifies the probability of the feature point's location on the object by color representation. The method provides a feature point image containing the feature points from the heat map on the 2D image, and estimates the 3D pose of the object by comparing the feature point image with a 3D virtual CAD model of the object. In other words, an optimization algorithm is employed to optimally rotate and translate the CAD model so that projected feature points in the model match predicted feature points in the image.

[0004] As previously mentioned, the robotic system of U.S. Patent Application No. 16 / 839,274 predicts multiple feature points on an image of an object the robot is picking. However, if the robot is selectively picking an object from a collection of objects, such as objects in a bin, there will be multiple objects in the image, and each object will have multiple predicted features. This prevents the process from reliably identifying the pose of a single object because rotating the CAD model may cause its predicted feature points to match predicted feature points on a different object. Summary of the Invention

[0005] The following discussion discloses and describes a system and method for acquiring three-dimensional poses of objects to enable a robot to retrieve the objects. The method includes acquiring a two-dimensional red-green-blue (RGB) image of the object using a camera and generating a segmentation image of the RGB image by performing an image segmentation process using a deep learning convolutional neural network to extract features from the RGB image and assign labels to pixels in the segmentation image such that objects in the segmentation image have the same label. The method also includes separating the segmentation image into multiple cropped images, each cropped image including one of the objects, estimating the three-dimensional pose of each object in each cropped image, and combining the three-dimensional poses into a single pose image. The steps of acquiring a color image, generating a segmentation image, separating the segmentation images, estimating the three-dimensional pose of each object, and combining the three-dimensional poses are performed each time the robot retrieves an object from a group of objects.

[0006] Additional features of the present disclosure will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram of a robotic system including a robot that removes an object from a container. [Figure 2] FIG. 2 is a schematic block diagram of a container removal system for removing an object from a container in the robot system shown in FIG. 1. [Figure 3] FIG. 3 is a schematic block diagram of a segmentation module separate from the system shown in FIG. 2. [Figure 4] 1 is a flow diagram illustrating a learning-based neural network process for using a trained neural network to estimate a three-dimensional pose of an object using a two-dimensional segmentation image of the object and the neural network. [Figure 5] FIG. 5 illustrates a Perspective-n-point (PnP) process for determining a 3D pose estimate of the object in the process shown in FIG. 4. [Figure 6] FIG. 1 is a diagram of a segmentation image including multiple categories, each with multiple objects. DETAILED DESCRIPTION OF THE INVENTION

[0008] The following discussion of embodiments of the present disclosure is directed to a robotic system for acquiring a three-dimensional pose of an object within a group of transparent objects, which acquires an RGB image of the object, segments the image using image segmentation, crops the segmented image of the object, and uses a learning-based neural network to acquire the three-dimensional pose of the segmented object. It is merely exemplary in nature and is not intended to limit the invention or its applications or uses in any way. For example, the system and method have application in determining the position and orientation of a transparent object within a group of transparent objects. However, the system and method may have other applications.

[0009] FIG. 1 is a diagram of a robotic system 10 including a robot 12 having an end effector 14 shown individually removing objects 16, such as clear bottles, from a container 18. System 10 is intended to represent any type of robotic system that would benefit from the discussion herein, and robot 12 may be any robot suitable for that purpose. A camera 20 is positioned to capture a top-down image of the container 18 and provide the image to a robot controller 22 that controls the operation of robot 12. Because objects 16 may be transparent, controller 22 cannot rely on a depth map image to determine the location of objects 16 within container 18. Therefore, camera 20 may be a two-dimensional or three-dimensional camera, as only RGB images from camera 20 are used.

[0010] For the robot 12 to effectively grasp and retrieve the object 16, it must be able to position the end effector 14 in the proper position and orientation before grasping the object 16. As discussed in detail below, the robot controller 22 employs an algorithm that enables the robot 12 to retrieve the object 16 without having to rely on an accurate depth map image. More specifically, the algorithm performs an image segmentation process using the different colors of pixels in the RGB image from the camera 20. Image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics. Thus, the segmentation process predicts which pixels belong to which of the objects 16.

[0011] Modern image segmentation techniques may employ deep learning techniques. Deep learning is a specific type of machine learning that provides superior learning performance by representing a specific real-world environment as a hierarchy of increasingly complex concepts. Deep learning typically employs a software configuration that includes multiple layers of neural networks that perform nonlinear processing, where each successive layer receives the output from the previous layer. Generally, the layers include an input layer that receives raw data from sensors, several hidden layers that extract abstract features from the data, and an output layer that identifies specific facts based on the feature extraction from the hidden layers. Neural networks include neurons or nodes, each of which has a "weight" by which the input to the node is multiplied to obtain the probability that a certain fact is correct. More specifically, each node has a weight, which is a floating-point number by which the input to the node is multiplied to generate the output of that node, which is some proportion of the input. The weights are initially "trained" or set by having the neural network analyze a set of known data under supervision and through minimizing a cost function so that the network has the highest probability of the correct output.

[0012] 2 is a schematic block diagram of a container removal system 30 that is part of the controller 22 in the robotic system 10 and operates to remove an object 16 from a container 18. The system 30 receives a two-dimensional RGB image 32 of a top view of the container 18 from the camera 20, where the object 16 is shown in the image 32. The image 32 is provided to a segmentation module 36 that performs an image segmentation process, where each pixel is assigned a specific label, and pixels associated with the same object 16 have the same label.

[0013] FIG. 3 is a schematic block diagram of a module 36 separate from the system 30. The RGB image 32 is provided to a feature extraction module 40, which performs a filtering process to extract important features from the image 32, removing background and noise. For example, the module 40 may include a learning-based neural network to extract gradients, edges, contours, primitive shapes, etc. from the image 32, and the module 40 provides an extracted feature image 44 of the RGB image 32 in a known manner. The feature image 44 is provided to a region proposal module 50, which uses the neural network to analyze the identified features in the image 44 and determine the location of the object 16 within the image 44. In particular, the module 50 includes a trained neural network that provides several bounding boxes, such as 50 to 100 boxes of different sizes, i.e., boxes with various lengths and widths, that are used to determine the probability that the object 16 is present at a particular location within the image 48. In this embodiment, the bounding boxes are all vertical boxes, which helps reduce the complexity of the module 50. The region proposal module 50 employs a sliding search window template, which is well known to those skilled in the art. Here, a search window including all bounding boxes moves over the feature image 44, for example from the top left of the image 44 to the bottom right of the image 44, to search for features that identify the possible presence of one of the objects 16.

[0014] The sliding window search generates a bounding box image 54 containing several bounding boxes 52 each enclosing a predicted object in the image 44. The number of bounding boxes 52 in the image 54 decreases each time the robot 12 removes one of the objects 16 from the container 18. The module 50 parameterizes the center position (x, y), width (w), and height (h) of each box 52 and provides a predicted confidence value between 0% and 100% that the object 16 is present in the box 52. The image 54 is provided to a binary segmentation module 56, which uses a neural network to estimate whether a pixel belongs to the object 16 in each of the bounding boxes 52 and removes background pixels in the boxes 52 that are not part of the object 16. The remaining pixels in the image 54 within each of the boxes 52 are assigned the value of the particular object 16, resulting in a two-dimensional segmentation image 58 that identifies the object 16 by different characteristics, such as color. For this reason, the described image segmentation process is a modified form of deep learning Mask R-CNN (convolutional neural network). The segmented objects in image 58 are then cropped to isolate each of the objects 16 identified in image 58 as a cropped image 60 having only one of the objects 16.

[0015] Each of the cropped images 60 is then sent to a separate 3D pose estimation module 70 which performs 3D pose estimation of the object 16 in that image 60 to obtain an estimated 3D pose 72 in a manner similar to that of U.S. patent application Ser. No. 16 / 839,274. Figure 4 is a flow chart 80 illustrating the algorithm operating in module 70, which employs a learning-based neural network 78 that uses a trained neural network to estimate the 3D pose of the object 16 in a particular cropped image 60. The image 60 is provided to an input layer 84 and multiple successive residual block layers 86 and 88, which include a feed-forward loop in the neural network 78 operating in AI software within controller 22 that uses a filtering process to provide feature extraction, such as gradients, edges, contours, etc., of potential feature points on the object 16 in the image 60. The image containing the extracted features is provided to multiple successive convolutional layers 90 within the neural network 78, which define possible feature points obtained from the extracted features as a series of heat maps 92, one for each feature point. The series of heat maps indicates the likelihood of where the feature point is located on the object 16 based on the color of the heat maps 92. An image 94 is generated using the image 60 of the object 16, including feature points 96 for all feature points from all heat maps 92, where each feature point 96 is assigned a confidence value based on the color of the heat map 92 for that feature point, and those feature points 96 that do not have a confidence value above a certain threshold are not used.

[0016] The image 94 is then compared in a pose estimation processor 98 with a tentative or virtual 3D CAD model of the object 16 having the same feature points to provide an estimated 3D pose 72 of the object 14. One suitable algorithm for comparing the image 94 with the CAD model is known in the art as Perspective-n-point (PnP). In general, the PnP process estimates the pose of the object relative to a calibrated camera given a set of n 3D points of the object in a world coordinate system and the corresponding 2D projections of the 3D points in images from the camera 20. The pose includes six degrees of freedom (DOF), consisting of the rotation (roll, pitch, and yaw) and 3D translation of the object relative to the camera coordinate system.

[0017] FIG. 5 is a diagram 100 illustrating how PnP processing can be implemented in this example to obtain a three-dimensional pose of object 16. Diagram 100 shows a three-dimensional object 106 representing object 16 in a ground-based or real-world position. Object 106 is observed by camera 112, representing camera 20, and projected as a two-dimensional object image 108 onto a two-dimensional image plane 110, where object image 108 represents image 94 and point 102 on image 108 is a feature point predicted by neural network 78 representing point 96. Diagram 100 also shows a virtual three-dimensional CAD model 114 of object 16 randomly placed in front of camera 112 and projected onto plane 110 as a two-dimensional model image 116, also including projected feature point 118, with feature point 104 in the same position as feature point 96. The CAD model 114 is rotated and translated in front of the camera 112, which rotates and translates the model image 116 in an attempt to minimize the distance between each of the feature points 118 on the model image 116 and the corresponding feature points 102 on the object image 108, i.e., to align the images 116 and 108. Once the model image 116 is aligned as best as possible with the object image 108, the pose of the CAD model 114 relative to the camera 112 is the estimated 3D pose 72 of the object 16.

[0018] This analysis is expressed by equation (1) for one of the corresponding feature points between image 108 and image 116, where equation (1) is used for all feature points in image 108 and image 116.

number

[0019] All 3D poses 72 are combined into a single image 74, and the robot 12 selects one of the objects 16 to remove. Once the robot 12 has removed an object 16 and moved on, the camera 20 acquires a new image of the container 18 and removes the next object 16. This process continues until all objects 16 have been removed.

[0020] The above discussion describes identifying the three-dimensional pose of an object, i.e., a clear bottle, within a group of objects having the same type or category of objects. However, the above process has application to identifying the three-dimensional pose of an object within a group of objects having different types or categories of objects. This is illustrated by the segmentation image 124 shown in Figure 6, which includes a segmented object 126 of one category, i.e., a bottle, and a segmented object 128 of another category, i.e., a mug.

[0021] As will be appreciated by those skilled in the art, some of the various steps and processes discussed herein to explain the present disclosure may refer to operations performed by a computer, processor, or other electronic computing device that manipulates and / or transforms data using electrical phenomena. These computers and electronic devices may employ various volatile and / or nonvolatile memories, including non-transitory computer-readable media on which executable programs containing various code or executable instructions that can be executed by the computer or processor are stored. Here, memory and / or computer-readable media may include all forms and types of memory and other computer-readable media.

[0022] The foregoing discussion discloses and describes merely exemplary embodiments of the present disclosure. Those skilled in the art will readily recognize from such discussion and the accompanying drawings and claims that various changes, modifications, and variations can be made without departing from the spirit and scope of the disclosure, as defined in the following claims. [Aspect 1] 1. A method for acquiring a three-dimensional pose of an object in a group of objects, the method comprising: capturing a two-dimensional red-green-blue (RGB) image of the object using a camera; generating a segmentation image of the RGB image by performing an image segmentation process that extracts features from the RGB image and assigns labels to pixels in the segmentation image such that objects in the segmentation image have the same label; Separating the segmentation image into a plurality of cropped images, each cropped image including one of the objects; estimating the 3D pose of each object in each cropped image; combining the three-dimensional poses into a single pose image; A method comprising: [Aspect 2] 2. The method of embodiment 1, wherein generating the segmentation image includes using a deep learning Mask R-CNN (convolutional neural network). [Aspect 3] The method of aspect 1, wherein generating a segmentation image includes providing a plurality of bounding boxes, aligning the bounding boxes to the extracted features, and providing a bounding box image including a bounding box that encloses the object. [Aspect 4] 4. The method of embodiment 3, wherein generating the segmentation image includes determining a likelihood that an object is present in each bounding box. [Aspect 5] The method of embodiment 3, wherein generating the segmentation image includes removing pixels from each bounding box in the bounding box image that are not associated with the object. [Aspect 6] 2. The method of embodiment 1, wherein generating the segmentation image includes assigning labels to pixels in the segmentation image such that each object in the segmentation image has the same label. [Aspect 7] The method of aspect 1, wherein estimating the three-dimensional pose of each object includes using a neural network to extract a plurality of features on the object from the two-dimensional image; generating a heat map for each of the extracted features that identifies possible locations of feature points on the object; providing a feature point image that combines the feature points from the heat map with the two-dimensional image; and estimating the three-dimensional pose of the object using the feature point image. [Aspect 8] 8. The method of claim 7, wherein estimating the three-dimensional pose of each object includes comparing the feature point image with a three-dimensional virtual model of the object. [Aspect 9] The method of embodiment 8, wherein estimating the three-dimensional pose of each object includes using a perspective-n-point algorithm. [Aspect 10] 2. The method of embodiment 1, wherein the object is transparent. [Aspect 11] 2. The method of embodiment 1, wherein the group of objects includes objects having different shapes. [Aspect 12] 2. The method of embodiment 1, wherein the method is employed in a robotic system and the object is retrieved by a robot. [Aspect 13] 1. A method for acquiring a three-dimensional pose of a transparent object in a group of transparent objects to enable a robot to pick said object, said method comprising: capturing a two-dimensional red-green-blue (RGB) image of the object using a camera; generating a segmentation image of the RGB image by performing an image segmentation process using a deep learning convolutional neural network to extract features from the RGB image and assign labels to pixels in the segmentation image such that objects in the segmentation image have the same label; Separating the segmentation image into a plurality of cropped images, each cropped image including one of the objects; estimating the 3D pose of each object in each cropped image; combining the three-dimensional poses into a single pose image, wherein acquiring a color image, generating a segmentation image, separating the segmentation image, estimating the three-dimensional pose of each object, and combining the three-dimensional poses are performed each time the robot picks an object from the collection of objects; A method comprising: [Aspect 14] The method of aspect 13, wherein generating a segmentation image includes providing a plurality of vertically aligned bounding boxes having the same orientation, aligning the bounding boxes to the extracted features using a sliding window template, providing a bounding box image including bounding boxes surrounding the object, determining a likelihood that the object is present in each bounding box, removing pixels from each bounding box that are not associated with the object, and identifying a central pixel of each object within the bounding box. [Aspect 15] 14. The method of claim 13, wherein estimating the three-dimensional pose of each object includes: extracting a plurality of features on the object from a two-dimensional image using a neural network; generating a heat map for each of the extracted features that identifies possible locations of feature points on the object; combining the feature points from the heat map with the two-dimensional image to provide a feature point image; and estimating the three-dimensional pose of the object using the feature point image by comparing the feature point image with a three-dimensional virtual model of the object. [Aspect 16] 16. The method of claim 15, wherein estimating the three-dimensional pose of each object includes using a perspective-n-point algorithm. [Aspect 17] 14. The method of claim 13, wherein the camera is a two-dimensional camera or a three-dimensional camera. [Aspect 18] 1. A robotic system for acquiring a three-dimensional pose of an object in a group of objects, the system comprising: a camera providing a two-dimensional red-green-blue (RGB) image of the object; a deep learning convolutional neural network that generates a segmented image of the object by performing an image segmentation process that extracts features from the RGB image and assigns labels to pixels in the segmented image such that each object in the segmented image has the same label; and means for separating the segmentation image into a plurality of cropped images, each cropped image including one of the objects; means for estimating the three-dimensional pose of each object in each cropped image; means for combining the three-dimensional poses into a single pose image; A system comprising: [Aspect 19] The system of aspect 18, wherein the deep learning neural network provides multiple vertically aligned bounding boxes having the same orientation, aligns the bounding boxes to the extracted features using a sliding window template, provides a bounding box image including bounding boxes surrounding the object, determines the likelihood that the object is present in each bounding box, removes pixels from each bounding box that are not associated with the object, and identifies a center pixel of each object within the bounding box. [Aspect 20] The system of aspect 18, wherein the means for estimating the three-dimensional pose of each object uses a neural network to extract a plurality of features on the object from a two-dimensional image and identify the possible locations of feature points on the object, generates a heat map for each of the extracted features, provides a feature point image that combines the feature points from the heat map with the two-dimensional image, and estimates the three-dimensional pose of the object using the feature point image by comparing the feature point image with a three-dimensional virtual model of the object.

Claims

1. 1. A method for acquiring a three-dimensional pose of an object in a group of objects, the method comprising: capturing a two-dimensional red-green-blue (RGB) image of the object using a camera; generating a segmentation image of the RGB image by performing an image segmentation process that extracts features from the RGB image and assigns labels to pixels in the segmentation image such that objects in the segmentation image have the same label; Separating the segmentation image into a plurality of cropped images, each cropped image including one of the objects; estimating the three-dimensional pose of each object in each cropped image; combining the three-dimensional poses into a single pose image; the camera is a single two-dimensional camera that captures two-dimensional images; Generating the segmentation image includes using a deep learning Mask R-CNN (convolutional neural network); estimating the three-dimensional pose of each object includes using a neural network to extract a plurality of features on the object from the two-dimensional image; generating a heat map for each of the extracted features that identifies possible locations of feature points on the object; combining the feature points from the heat map with the two-dimensional image to provide a feature point image; and estimating the three-dimensional pose of the object using the feature point image.

2. 2. The method of claim 1 , wherein generating a segmentation image comprises providing a plurality of bounding boxes, aligning the bounding boxes to the extracted features, and providing a bounding box image comprising a bounding box that encloses the object.

3. The method of claim 2 , wherein generating a segmentation image includes determining the likelihood that an object is present in each bounding box.

4. The method of claim 2 , wherein generating a segmentation image comprises removing pixels from each bounding box in the bounding box image that are not associated with an object.

5. The method of claim 1 , wherein generating a segmentation image includes assigning labels to pixels in the segmentation image such that each object in the segmentation image has the same label.

6. The method of claim 1 , wherein estimating the three-dimensional pose of each object comprises comparing the feature point image with a three-dimensional virtual model of the object.

7. The method of claim 6 , wherein estimating the three-dimensional pose of each object includes using a perspective-n-point algorithm.

8. The method of claim 1 , wherein the object is transparent.

9. The method of claim 1 , wherein the group of objects includes objects having different shapes.

10. The method of claim 1 , wherein the method is employed in a robotic system and the object is retrieved by a robot.

11. 1. A method for acquiring a three-dimensional pose of a transparent object in a group of transparent objects to enable a robot to pick said object, comprising: the image segmentation process includes using a deep learning convolutional neural network; 2. The method of claim 1, wherein combining the three-dimensional poses into a single pose image comprises acquiring a color image, generating a segmentation image, separating the segmentation image, and estimating the three-dimensional pose of each object, wherein combining the three-dimensional poses is performed each time the robot picks an object from the collection of objects.

12. 12. The method of claim 11, wherein generating a segmentation image comprises: providing a plurality of vertically aligned bounding boxes having the same orientation; aligning the bounding boxes to the extracted features using a sliding window template; providing a bounding box image comprising bounding boxes that enclose the object; determining a likelihood that the object is present in each bounding box; removing pixels from each bounding box that are not associated with the object; and identifying a center pixel of each object within the bounding box.

13. The method of claim 11 , wherein estimating the three-dimensional pose of each object includes using a perspective-n-point algorithm.

14. 1. A robotic system for acquiring a three-dimensional pose of an object in a group of objects, the system comprising: a camera providing a two-dimensional red-green-blue (RGB) image of the object; a deep learning convolutional neural network that generates a segmented image of the object by performing an image segmentation process that extracts features from the RGB image and assigns labels to pixels in the segmented image such that each object in the segmented image has the same label; and means for separating the segmentation image into a plurality of cropped images, each cropped image including one of the objects; means for estimating the three-dimensional pose of each object in each cropped image; means for combining the three-dimensional poses into a single pose image; the camera is a single two-dimensional camera that captures two-dimensional images; Generating the segmentation image includes using a deep learning Mask R-CNN (convolutional neural network); The means for estimating the three-dimensional pose of each object includes extracting a plurality of features on the object from a two-dimensional image using a neural network and identifying possible locations of feature points on the object; generating a heat map for each of the extracted features; providing a feature point image that combines the feature points from the heat map with the two-dimensional image; and estimating the three-dimensional pose of the object using the feature point image by comparing the feature point image with a three-dimensional virtual model of the object.

15. 15. The system of claim 14, wherein the deep learning neural network provides multiple vertically aligned bounding boxes having the same orientation, aligns the bounding boxes to the extracted features using a sliding window template, provides a bounding box image including a bounding box that encloses the object, determines a likelihood that the object is present in each bounding box, removes pixels from each bounding box that are not associated with the object, and identifies a center pixel of each object within the bounding box.

Images