Object bin picking with rotation compensation

Abstract

To provide a system and method for identifying an object picked up by a robot without relying on an accurate depth map image.SOLUTION: A method includes: obtaining a 2D red-green-blue (RGB) color image and a 2D depth map image of objects using a 3D camera, where pixels in the depth map image are assigned a value identifying a distance from the camera to the objects; generating a segmentation image of the objects using a deep learning neural network that performs an image segmentation process that extracts features from the RGB image, assigns a label to the pixels so that objects in the segmentation image have the same label and rotates the object using the orientation of the object in the segmented image; and identifying a location for picking up the object using the segmentation image and the depth map image and rotating the object when the object is picked up.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates generally to systems and methods for identifying objects to be picked up by a robot, and more particularly to systems and methods for identifying objects, such as transparent objects, to be picked up by a robot from a storage container of objects. The methods include: -La An image segmentation process is employed that uses a neural network to extract features from the image captured by the camera, identify the x and y coordinates of pixels in the extracted feature image, and identify the orientation of objects in the segmented image. [Background technology]

[0002] Robots perform many tasks, including pick-and-place operations. Robots pick up objects from one location, such as a storage bin, and move them from there to another location, such as a conveyor belt. To identify objects to be picked up from a storage bin, some robotic systems employ a 3D camera that generates a 2D red-green-blue (RGB) color image of the bin and a 2D grayscale depth map image of the bin, where each pixel in the depth map image has a value that defines the distance from the camera to a particular object. That is, the closer a pixel is to the object, the lower its value. The depth map image identifies distance measurements to points in a point cloud within the camera's field of view. A point cloud is a collection of data points defined by a particular coordinate system, with each point having an x, y, and z value.

[0003] For this purpose, there are two types of point cloud analysis: model-free point cloud analysis and model-based point cloud analysis. Model-free point cloud analysis involves identifying point clusters in the point cloud in a direction perpendicular to the camera and segmenting the clusters relative to each other, with each point cloud segment being identified as an object. Model-based point cloud analysis involves generating a template from a CAD model of the object and then searching for the template in the point cloud. However, if the object being picked up by the robot is transparent, light propagates through the object and is not effectively reflected from the object's surface. Therefore, the point cloud generated by the camera is not an effective representation of the object, and the depth map image is unreliable and cannot properly identify the object being picked up. Summary of the Invention

[0004] In the following discussion, a system and method for identifying an object to be picked up by a robot from a storage bin of objects is disclosed and described. The method includes using a 3D camera to acquire a 2D red-green-blue (RGB) color image of the object and a 2D depth map image, where pixels in the depth map image are assigned values ​​that identify the distance of the object from the camera. The method uses a convolutional neural network to perform an image segmentation process that extracts features from the RGB image and assigns labels to pixels such that objects in the segmented image have the same label to measure the orientation of the object. -LaThe method uses a color network to generate a segmented image of the object. Generating the segmented image includes providing multiple bounding boxes of various sizes and orientations, aligning the bounding boxes to extracted features using a sliding window template, providing a bounding box image that encloses the object and includes the bounding boxes, measuring the probability that an object is present within each bounding box, and identifying a center pixel of each object within the bounding box. The method then identifies a location for picking up the object using the segmented image and the depth map image, and rotates the object using the object's orientation in the segmented image. The color image acquisition, depth map image acquisition, segmented image generation, and identification of a location for picking up the object are performed each time an object is picked up from a group of objects by the robot.

[0005] Additional features of the present disclosure will become apparent from the following description and claims, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram of a robotic system including a robot picking up an object from a storage container. [Figure 2] FIG. 2 is a schematic block diagram of a bin picking system for picking up objects from storage containers in the robotic system of FIG. [Figure 3] FIG. 3 is a schematic block diagram of a segmentation module separate from the system shown in FIG. 2 that provides image segmentation. [Figure 4] FIG. 4 is a schematic block diagram of a segmentation module of the type of FIG. 3 that also employs object rotation correction. [Figure 5] FIG. 5 is a schematic block diagram of a bin-picking system of the type shown in FIG. 2 employing the segmentation module shown in FIG. 4 to provide object rotation compensation. DETAILED DESCRIPTION OF THE INVENTION

[0007] The following discussion of embodiments of the present disclosure relates to systems and methods for identifying objects picked up by a robot from a storage bin of objects, the methods including: -La An image segmentation process is employed that uses a neural network to extract features from an RGB image, identify the x and y coordinates of pixels in the extracted feature image, and identify the orientation of objects in the segmented image. The following discussion is merely exemplary and is not intended to limit the invention or its applications or uses. For example, the system and method have application for identifying transparent objects to be picked up by a robot. However, the system and method may have other applications.

[0008] FIG. 1 is a diagram of a robotic system 10 including a robot 12 having an end effector suction cup 14, which is shown picking up an object 16, e.g., a clear bottle, from a storage container 18. System 10 is intended to represent any type of robotic system that can utilize the teachings herein, and robot 12 can be any robot suitable for said purpose. A 3D camera 20 is positioned to take a top-to-bottom image of storage container 18 and provide it to a robot controller 22 that controls the movement of robot 12. Because object 16 may be transparent, controller 22 may not rely on the depth map image provided by camera 20 to identify the location of object 16 within storage container 18. In this non-limiting embodiment, robot 12 places object 16 onto a conveyor 24, perhaps in a particular direction, e.g., directly above.

[0009] As discussed in more detail below, the robot controller 22 employs an algorithm that enables the robot 12 to pick up the object 16 without having to rely on an accurate depth map image. More specifically, the algorithm performs an image segmentation process using the various colors of the pixels in the RGB image from the 3D camera 20. Image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics. Thus, the segmentation process predicts which pixels belong to which object 16.

[0010] Modern image segmentation methods can employ deep learning technology. Deep learning is a special type of machine learning that provides greater learning performance by representing a particular real-world environment as an increasingly complex hierarchy of concepts. Deep learning typically involves several layers of neural networks that perform nonlinear processing. -La The software architecture includes a neural network, where each successive layer receives the output from the previous layer. Generally, the layers include an input layer that receives raw data from sensors, multiple hidden layers that extract abstract features from the data, and an output layer that identifies specific objects based on feature extraction from the hidden layers. -La A neural network contains neurons or nodes, each with a "weight" that is multiplied by the input to the node to get the probability that something is correct or not. More specifically, each node has a weight, a floating-point number, that is multiplied by the input to the node to generate an output for that node (which is some fraction of the input). The weights are first calculated by minimizing a cost function under supervision to allow the network to obtain the most likely correct output, based on a known data set. -La The data is "trained" or configured by having the network analyze it.

[0011] 2 is a schematic block diagram of a bin picking system 30 that is part of the controller 22 in the robotic system 10 that operates to pick up objects 16 from storage bins 18. The system 30 receives a 2D (two-dimensional) RGB image 32 of the top surface of the storage bin 18 and a 2D depth map image 34 of the top surface of the storage bin 18 from the camera 20, but the depth map image 34 may be unreliable because the object 16 may be transparent. The image 32 is provided to a segmentation module 36 that performs an image segmentation process, where each pixel in the image 32 is assigned a specific label, and pixels associated with the same object 16 have the same label.

[0012] 3 is a schematic block diagram of module 36 separate from system 30. RGB image 32 is provided to feature extraction module 42, which performs a filtering process to remove background and noise and extract important features from image 32. For example, module 42 may be a learning-based neural network extractor that extracts gradients, edges, contours, primitives, etc. from image 32. -La The feature image 44 is provided to a region proposal module 50, which may include a neural network. -La The module 50 uses a trained neural network to analyze identified features in the image 44 and determine the location of the object 16 in the image 44. In particular, the module 50 uses a trained neural network to provide a large number of bounding boxes of different sizes, e.g., 50-100 boxes with various lengths and widths, that are used to identify the probability that the object 16 is present at a particular location in the image 44. -La In this embodiment, the bounding boxes are all vertical boxes, which helps reduce the complexity of module 50. Region proposal module 50 employs a sliding search window template, well known to those skilled in the art, in which a search window containing all of the bounding boxes moves across feature image 44, for example, from the top left of image 44 to the bottom right of image 44, looking for features that identify the possible presence of one of objects 16.

[0013] The sliding window search generates a bounding box image 52 containing multiple bounding boxes 54, each enclosing a predicted object in the image 44; the number of bounding boxes 54 in the image 52 decreases as the robot 12 removes an object 16 from the storage bin 18. The module 50 parameterizes the center location (x, y), width (w), and height (h) of each box 54 to indicate a predicted certainty, ranging from 0% to 100%, that the object 16 resides within the box 54. The image 52 is generated based on the number of bounding boxes 54 in the image 44. -La The resulting image is then fed to a binary segmentation module 56 which uses a convolutional neural network to estimate whether a pixel belongs to an object 16 within each of the bounding boxes 54 and to estimate background pixels within the boxes 54 that are not part of the object 16. The remaining pixels in the image 52 for each of the boxes 54 are assigned a value related to the particular object 16, such that a 2D segmented image 58 is generated that identifies the object 16 by various indicia such as color. The image segmentation process as described above is implemented using a deep learning Mask R-CNN (Convolutional Neural Network). -La It is a modified form of the 'network'.

[0014] The 2D segmented image 58 is then provided to a center pixel module 60, which determines which segmented object in the image 58 has the highest probability of being the object and provides the x-y coordinates of the center pixel for the selected object 16. The identified center pixel of the selected object 16, along with the depth map image 34, is provided to a Cartesian coordinate module 62, which calculates the x-yz Cartesian coordinates of the center pixel of the object 16. The depth map image 34 estimates or predicts the location of each pixel in the real world, although z-depth predictions are not very reliable. The x-yz coordinates of the center pixel of the object 16 are then used in a grip position module 64 to identify an x-yz grip position for positioning the suction cup 14. The x-y coordinates of the center pixel of the selected object 16 are also provided to a grip direction module 66, which determines the grip direction, i.e., roll, pitch, and yaw, of the suction cup 14 based on the ray direction from the camera 20 to the center pixel of the object 16, using, for example, a pinhole camera model known to those skilled in the art. The movement of the suction cup 14 when picking up the object 16 moves along and aligns with the ray direction. Because the depth map image 34 cannot show the exact distance between the camera 20 and the object 16, the ray direction from the camera 20 needs to be measured.

[0015] The gripping position of the suction cup 14 and the gripping direction of the suction cup 14 are combined in a gripping pose module 68 to provide a gripping pose of the suction cup 14. The gripping pose includes the x, y, and z coordinates of the suction cup 14 and the yaw, pitch, and roll positions to provide the approach direction of the suction cup 14 to the object 16 along the ray direction. A collision check module 70 determines whether the gripping pose will cause the suction cup 14 to collide with the bottom of the storage container 18 along the calculated ray direction. The distance to the bottom of the storage container 18 is known. If the gripping pose will cause a collision with the storage container 18 along the ray direction, a safety buffer is generated in a safety buffer module 72 to restrict the movement of the robot 12 along the ray direction.

[0016] As the suction cup 14 moves along the light beam direction, it eventually comes into contact with the object 16 to be picked up. The contact detection module 74 detects that the vacuum cup 14 has come into contact with the object 16 by detecting a pressure difference in the vacuum. The robot 12 either stops its movement and picks up the object 16 at the pickup module 76, or reaches a buffer distance, in which case the object 16 is not picked up. Once the object 16 is picked up and moved by the robot 12, the camera 20 takes the next image of the storage container 18 to pick up the next object 16. The next image does not include the object 16 that was already picked up. This process continues until all of the objects 16 have been picked up from the storage container 18.

[0017] The technique for picking up an object 16 from a storage container 18 discussed above simply measures the center pixel of the object 16 in the image 58 as the location where the suction cup 14 will grasp the object 16, and the robot 12 moves along a line to or approaches the center point of the object 16 to pick it up. However, because the robot 12 does not know the orientation of the object 16 around the center point, it can only place the object 16 in a random orientation once it is picked up. In other words, the segmentation module 36 only identifies the group of pixels that make up the object 16, but does not identify the orientation or rotation of the object 16. However, there may be applications where it is preferable to orient the object 16 in a particular manner, such as aligning the objects 16 in the same orientation on the conveyor 24, in which case the robot 12 needs to change its orientation or rotate the object 16 after picking it up. For this type of robotic system, not only do we need to measure the centers of the objects 16 to be picked up, but we also need to measure the orientation of the objects 16 to be picked up so that when the robot 12 places the objects 16 on the conveyor 24, the robot 12 can rotate the objects 16 to align them in the desired direction. In this way, all of the objects 16 can be aligned in the same direction on the conveyor 24, or even placed upright. Measuring the orientation of the objects 16 requires more complexity than just measuring the centers of the objects 16, and therefore requires more knowledge. -La Requires network training.

[0018] 4 is a schematic block diagram of a segmentation module 80 that is similar to segmentation module 36 but also identifies the orientation of the object 16 being picked up, with elements similar to module 36 being identified with the same reference numerals. Image 32 may be obtained from a 2D camera, in which case depth image 34 is not required and the distance of object 16 may be known in advance. Module 80 includes feature extraction module 42 that performs a filtering process to extract significant features from image 32 to provide feature image 44, as described above. However, in this embodiment, feature image 44 is not the input of module 50. -LaComplex deep learning networks such as Mask R-CNN are much more complex than neural networks. -La It is a global network, and there are many more -La The region proposal module 80 performs a sliding window search by moving or sliding a search window template containing bounding boxes across the feature image 44, e.g., from the top left of the image 44 to the bottom right of the image 44, to align one of the bounding boxes with the object 16 in the feature image 44. However, because there are many more boxes with various orientations, the region proposal module 82 can predict the orientation of the object 16 it identifies in the image 44. The output of module 82 is a bounding box image 84 with bounding boxes 86 indicating the orientation of object 16. Thus, module 82 parameterizes the center location (x, y), width (w), and height (h) of each box 86, as well as the orientation (θ), to indicate a predicted confidence value ranging from 0% to 100% that object 16 is within box 86. In this process, although bounding boxes 86 are rectangular, the angle of boxes 86 allows the boxes to fit snugly around object 16, so the process for removing background pixels performed by binary segmentation module 56 is not necessary.

[0019] 5 is a schematic block diagram of a bin-picking system 90 of the type shown in FIG. 2 that employs rotation compensation to enable the robot 12 to rotate the object 16 so that it is oriented in a particular direction after it has been picked up, with similar elements to system 30 being identified with the same reference numerals. In this embodiment, module 36 is replaced with module 80. System 90 further includes an object rotation module 92 that receives a rotation parameter θ from segmentation module 80 and that is used to rotate the object 16 so that it is oriented in a particular direction after it has been picked up. Body 1 6, lift it to a particular height away from other objects 16, and then rotate object 16 by a desired amount measured by a rotation parameter θ in all x, y, and z directions.

[0020] As will be appreciated by those skilled in the art, some of the various steps and processes discussed herein to explain the present disclosure can refer to operations performed by a computer, processor, or other electronic computing device that manipulates and / or transforms data using electrical phenomena. These computers and electronic devices can employ a variety of volatile and / or non-volatile memory, including non-transitory computer-readable media on which are stored executable programs, including various codes or executable instructions that can be executed by the computer or processor. Memory and / or computer-readable media can include all forms and types of memory and other computer-readable media.

[0021] The foregoing discussion discloses and describes merely exemplary embodiments of the present disclosure. Those skilled in the art will readily recognize from such discussion and the accompanying drawings and claims that various changes, modifications, and variations can be made without departing from the spirit and scope of the disclosure, as defined in the following claims.

Claims

1. 1. A method for picking an object from a group of objects, the method comprising: capturing an image of the object using a camera; generating a segmented image of the object by performing an image segmentation process including: using a neural network to extract features from the image to provide an extracted features image; identifying x and y coordinates of pixels in the extracted features image; and identifying an orientation of the object in the extracted features image; using the segmented image to identify a location for picking up the object; and rotating the object using the orientation of the object in the segmented image; the object is transparent, 1. A method in which generating a segmented image includes identifying a central pixel of each object within a bounding box, and identifying a location for picking up the object includes identifying a central pixel of one of the objects in the segmented image and calculating x-y-z coordinates of the central pixel using the location of the central pixel and a depth map image.

2. 2. The method of claim 1, wherein the neural network is a Deep Learning Mask R-Convolutional Neural Network (CNN).

3. 2. The method of claim 1 , wherein generating a segmented image comprises providing a plurality of bounding boxes, aligning the bounding boxes to the extracted features, and providing a bounding box image comprising a bounding box that encloses the object.

4. The method of claim 3 , wherein the bounding boxes have different sizes, and multiple bounding boxes of each size are oriented in different directions.

5. The method of claim 4 , wherein the bounding boxes of the same size in different orientations are oriented at 0°, 30°, 45°, 90°, 120°, 150° and 180° with respect to each other.

6. The method of claim 3 , wherein aligning the bounding boxes comprises using a sliding window template.

7. The method of claim 3 , wherein generating the segmented image includes measuring the probability that an object is present within each bounding box.

8. The method of claim 1 , wherein generating a segmented image comprises assigning labels to pixels of the segmented image such that each object in the segmented image has the same label.

9. The method of claim 1 , wherein determining a location for picking up the object includes determining an x-y-z grip position for picking the object.

10. 2. The method of claim 1, wherein identifying a location for picking up the object comprises identifying a grasp direction for picking up the object defined by a ray direction from the camera to the center pixel.

11. The method of claim 10 , wherein the light ray direction is measured by a pinhole camera model.

12. The method of claim 10 , wherein identifying a location for picking up the object includes determining a grasp pose for picking up the object using a grasp position and the grasp orientation.

13. The method of claim 12 , wherein the object is picked up by a robot and the grasp pose determines a pose of a robot end effector.

14. 1. A method for picking up a transparent object from a group of transparent objects using a robot, the method comprising: acquiring a 2D red-green-blue (RGB) color image of the object using a 3D camera; acquiring a 2D depth map image of the object using the 3D camera, wherein pixels in the depth map image are assigned values ​​that identify a distance of the object from the camera; generating a segmented image of the object by performing an image segmentation process using deep learning to extract features from the RGB image and assign labels to pixels in the segmented image so that each object in the segmented image has the same label, and to measure the orientation of the object; identifying locations for picking up the object using the segmented image and the depth map image; rotating the object using the orientation of the object in the segmented image, wherein acquiring a color image, acquiring a depth map image, generating a segmented image, identifying a location for picking up the object, and rotating the object are performed each time an object is picked up from the group of objects by the robot; Including, 10. The method of claim 1, wherein generating a segmented image includes providing a plurality of bounding boxes having the same size and a plurality of bounding boxes of the same size in different orientations, aligning the bounding boxes to the extracted features using a sliding window template, providing a bounding box image including bounding boxes that enclose the objects, measuring a probability that an object is present in each bounding box, and identifying a center pixel of each object in the bounding box.

15. 15. The method of claim 14, wherein identifying a location for picking up the object comprises: identifying a central pixel of one of the objects in the segmented image having the highest probability that the object is present; calculating x-y-z coordinates of the central pixel using the central pixel and the depth map image; measuring an x-y-z grasp position of the robot for picking up the object; identifying a grasp direction of the robot for picking up the object defined by a ray direction from the camera to the central pixel using a pinhole camera model; and measuring a grasp pose of the robot for picking up the object using the grasp position and the grasp direction.

16. 1. A robotic system for picking up an object from a group of objects using a robot, the system comprising: a camera providing an image of the object; a deep learning neural network that generates a segmented image of the object by performing an image segmentation process that extracts features from the RGB image and assigns labels to pixels in the segmented image such that each object in the segmented image has the same label; means for identifying a location for picking up the object using the segmented image; means for rotating the object using the orientation of the object in the segmented image, wherein the steps of acquiring an image, generating a segmented image, identifying a location for picking up the object, and rotating the object are performed each time an object is picked up from the group of objects by the robot; and Equipped with the object is transparent, The deep learning neural network provides multiple bounding boxes having the same orientation, aligns the bounding boxes to the extracted features using a sliding window template, provides a bounding box image including bounding boxes that enclose the objects, measures the probability that an object is present in each bounding box, and identifies a center pixel of each object in the bounding box.

Images