Method and system for picking a target box from a group of multiple boxes

Abstract

A system and method for identifying boxes to be picked up by a robot from a stack of multiple boxes. The method includes using a 3D camera to acquire a 2D red-green-blue (RGB) color image of the boxes and a 2D depth map image of the boxes, wherein each pixel in the depth map image is assigned a value identifying the distance from the camera to the boxes. The method generates a segmented image of the boxes by performing an image segmentation process that extracts multiple features from the RGB image and the depth map image, combines the extracted features in the image, and assigns a label to each pixel in the feature image so that each box in the segmented image has the same label. The method then uses the segmented image to identify a location for picking up the box.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for identifying boxes to be picked by a robot from a stack of multiple boxes, and more particularly, to systems and methods for identifying boxes to be picked by a robot from a stack of multiple boxes, wherein the methods employ image segmentation processing that assigns a label to each pixel in an image of the stack. Background Art

[0002] Robots perform a wide range of commercial tasks, including pick-and-place operations, in which robots pick up objects and move them from one location to another. For example, a robot can pick up a box from a pallet and place it on a conveyor belt, where the robot may employ an end effector with a suction cup to hold the box. In order for a robot to pick up a box efficiently, the robot needs to know the width, length, and height of the box it is picking up, which are input into the robot controller before the pick-and-place operation. However, boxes on the same pallet often have different sizes, making it inefficient to input the box dimensions into the robot during a pick-and-place operation. These boxes may also be placed side by side at the same height, making it challenging to distinguish between separate boxes and a single, large box. Currently, no robotic system can determine the dimensions of a box it is about to pick up in real time.

[0003] In a known robotic pick and place system, the width, length, and height of the box to be picked up by the robot are first input into the system. A 3D camera acquires a top-down image of a stack of multiple boxes and generates a 2D red-green-blue (RGB) color image of the multiple boxes and a 2D grayscale depth map image of the multiple boxes, where each pixel in the depth map image has a value that defines the distance from the camera to a specific box, that is, the closer the pixel is to the object, the lower its value. The robot controller provides a series of projected templates based on the width and length of the box, each template having dimensions for a specific distance between the camera and the box. During the search process, the template for the distance of the box provided by the depth map image is moved around the color image so that when the template matches or aligns with the box in the color image, the robot controller will know the location of the box and will use this location to define the center of the box to control the robot to pick up the box. Summary of the Invention

[0004] The following discussion discloses and describes a system and method for identifying boxes to be picked up by a robot from a stack of multiple boxes. The method includes using a 3D camera to obtain a 2D red-green-blue (RGB) color image of the multiple boxes and a 2D depth map image of the multiple boxes, wherein each pixel in the depth map image is assigned a value identifying the distance from the camera to the multiple boxes. The method generates a segmented image of the multiple boxes by performing an image segmentation process that extracts multiple features from the RGB image and the depth map image, combines the extracted multiple features in the images, and assigns a label to each pixel in the feature image so that each box in the segmented image has the same label. The method then uses the segmented image to identify a location for picking up a box.

[0005] Additional features of the present disclosure will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is an illustration of a robotic system including robots that pick boxes from pallets and place them on a conveyor belt;

[0007] Figure 2 is a schematic block diagram of a mixed size tray unloading system. Figure 1 A portion of the robotic system shown;

[0008] Figure 3 is with Figure 2 A schematic block diagram of the separated segmentation modules of the system shown;

[0009] Figure 4 is a stacked top-down RGB image with three layers of boxes;

[0010] Figure 5 yes Figure 4 A top-down depth map image of the stack of boxes shown in ;

[0011] Figure 6 is a histogram showing the count peaks of the bins in the stack, with depth on the horizontal axis and count on the vertical axis;

[0012] Figure 7 yes Figure 4 a cropped top-down RGB image of the top layer of the stack shown in ; and

[0013] Figure 8 yes Figure 4 A cropped top-down RGB image of the middle layer of the stack shown in . DETAILED DESCRIPTION

[0014] The following discussion of embodiments of the present disclosure relates to systems and methods for identifying boxes to be picked up by a robot from a stack of multiple boxes, wherein the methods employ image segmentation processing that assigns a label to each pixel in an image of the stack. This discussion is merely exemplary in nature and is in no way intended to limit the invention or its applications or uses. For example, the systems and methods have application in identifying boxes to be picked up by a robot. However, the systems and methods may have other applications.

[0015] Figure 1 is an illustration of a robotic system 10 that includes a robot 12 having an end effector 14 configured to pick up a plurality of boxes 16 from a stack 18 of boxes 16 located on a pallet 20 and place them on a conveyor belt 22. System 10 is intended to represent any type of robotic system that may benefit from the discussion herein, wherein robot 12 may be any robot suitable for the purpose. A 3D camera 24 is positioned to capture top-down 2D RBG images and depth map images of the stack 18 of boxes 16 and provide them to a robot controller 26 that controls the movement of robot 12. The boxes 16 may have different orientations on the pallet 20, may be stacked in multiple layers on the pallet 20, and may have different sizes.

[0016] As will be discussed in detail below, the robot controller 26 employs an algorithm that determines the dimensions of each box 16 that the robot 12 will be picking up without having the length, width, and height of the box 16 pre-entered into the controller 26 and without having to generate a projection template of the box 16. More specifically, the algorithm performs an image segmentation process that assigns a label to each pixel in the image so that pixels with the same label share certain characteristics. Thus, the segmentation process predicts which pixel belongs to which box among the plurality of boxes 16, with different labels representing different boxes 16.

[0017] Modern image segmentation techniques can employ deep learning. Deep learning is a specific type of machine learning that offers greater learning performance by representing specific real-world environments as a hierarchy of increasingly complex concepts. Deep learning typically employs a software architecture consisting of a neural network with several layers that perform nonlinear processing, with each successive layer receiving the output of the previous layer. Typically, these layers include an input layer that receives raw data from sensors, multiple hidden layers that extract abstract features from this data, and an output layer that identifies specific objects based on the features extracted from the hidden layers. A neural network comprises multiple neurons or nodes, each with a "weight" that is multiplied by the node's input to produce a probability of whether an object is correct. More specifically, each node has a weight—a floating-point number that is multiplied by the node's input to generate an output that is a certain proportion of the input. The weights are initially "trained" or set by having the neural network analyze a set of known data under supervision and minimize a cost function that maximizes the probability of the network achieving the correct output. Deep learning neural networks are often used to provide image feature extraction and transformation for visual detection and classification of objects in images, where a video or image stream can be analyzed by the network to identify and classify objects and, through this process, learn to better recognize objects. Thus, in these types of networks, the system can use the same processing configuration to detect certain objects and classify them differently based on how the algorithm learns to recognize the objects.

[0018] Figure 2 is a schematic block diagram of a mixed-size depalletizing system 30 that is part of the controller 26 in the robotic system 10 that operates to pick the plurality of boxes 16 from the pallet 20. The system 30 receives a 2D RGB image 32 of a top view of the plurality of boxes 16 located on the pallet 20 and a 2D depth map image 34 of a top view of the plurality of boxes 16 from the camera 24. The images 32 and 34 are provided to a segmentation module 36 that performs an image segmentation process in which each pixel is assigned a label, and in which pixels associated with the same box 16 have the same label.

[0019] Figure 3is a schematic block diagram of a module 36 separate from system 30. RGB image 32 is provided to feature extraction module 40, and depth map image 34 is provided to feature extraction module 42, which perform filtering to extract features from images 32 and 34. For example, modules 40 and 42 may include learning-based neural networks that extract gradient edges, contours, basic shapes, and the like from images 32 and 34, and provide an extracted feature image 44 of RGB image 32 and an extracted feature image 46 of depth map image 34 in a known manner. Images 44 and 46 are aligned into a feature stack image 48 to provide a 3D augmentation matrix. Feature stack image 48 is provided to region proposal module 50, which uses a neural network to analyze the identified features in RGB image 32 and depth map image 34 to determine the locations of the plurality of boxes 16 in image 48. Specifically, module 50 includes a trained neural network that provides a plurality of bounding boxes of varying sizes, such as 50 to 100 boxes, i.e., boxes of varying lengths and widths, which are used to identify the probability of a box 16 being present at a particular location in image 48. The region proposal module 50 uses a sliding search window template well known to those skilled in the art, in which a search window including all bounding boxes moves on the feature stack image 48, for example, from the upper left corner 48 of the image to the lower right corner of the image 48, looking for a feature that may be present in the multiple boxes 16.

[0020] The sliding window search generates a bounding box image 54 comprising a plurality of bounding boxes 52, each of which encloses the predicted object in the image 48, wherein the number of bounding boxes 52 in the image 54 is reduced each time the robot 12 removes one of the plurality of boxes 16 from the stack 18. Module 50 parameterizes the center location (x, y), width (w), and height (h) of each box 52 and provides a predicted confidence value between 0% and 100% that the box is present in the box 52. The image 54 is provided to a binary segmentation module 56, which uses a neural network to estimate whether the pixels in each bounding box 52 belong to a box 16 to eliminate background pixels in the box 52 that are not part of the object 16. The remaining pixels in the image 54 in each box 52 are assigned a value for a specific box 16, resulting in a 2D segmented image 58 that identifies the box 16 by different labels (such as color). The image segmentation process described is a modified form of the deep learning Mask R-CNN (convolutional neural network).

[0021] The 2D segmented image 58 is then provided to a center pixel module 60, which determines which segmented box in the image 58 has the highest confidence value for being a box and provides the xy coordinates of the center pixel of the selected box 16. The identified center pixel of the selected box 16 is provided to a Cartesian coordinate module 62 along with the depth map image 34, which calculates the xyz Cartesian coordinates of the center pixel of that box 16, where the depth map image 34 knows the real-world location of each pixel. The xyz coordinates of the center pixel of that box 16 are then used to identify the xyz grasp position in the grasp position module 64 for placing the end effector 14. The grasp position of the end effector 14 and the known vertical orientation of the box 16 from the orientation module 66 determine the grasp pose or orientation of the end effector 14 in the grasp pose module 68, wherein the grasp pose includes the xyz coordinates and yaw, pitch, and roll of the end effector 14 to provide the approach orientation of the end effector 14 to the box 16, and wherein other orientations of the box 16 may be provided, such as the normal orientation of the box surface instead of the vertical orientation. The robot motion is executed at the pick module 70 to pick up the box 16. The robot 12 then signals the camera 24 to provide a new RGB image and depth map image in which the previously extracted box 16 has been removed from the stack 18. This process continues until all boxes 16 have been picked.

[0022] As the plurality of boxes 16 are removed from the stack 18 by the robot 12 , boxes 16 in lower levels of the stack 18 may be partially exposed to the camera 24 and segmented by the segmentation module 36 . Figure 4 84 is a top-down RGB image 80 of a stack 82 of multiple boxes 84 illustrating this, where box 86 in the top layer 88 partially covers box 90 in the middle layer 92 and box 94 in the bottom layer 96, where boxes 90 and 94 may be segmented by the segmentation module 36 and the robot 12 may attempt to pick them up before box 86 has been removed. To the camera 24, the partially segmented boxes may appear to be complete boxes, but because portions of the boxes are occluded, the correct centers of the boxes cannot be determined and the orientation of the end effector 14 may not be correctly positioned to pick up the boxes.

[0023] As will be discussed in detail below, this problem is addressed by using the depth map image of the stack 18 to crop the top layer of boxes 84 and segmenting only those boxes in the top layer until they are removed from the stack 82 by the robot 12 . Figure 5 is a depth map image 98 of the stack 82 where the boxes 84 in a particular layer of layers 88 , 92 , and 96 have the same shading.

[0024] Figure 6is a histogram with depth on the horizontal axis and counts on the vertical axis, where a separate "bin" is defined for each centimeter of depth from camera 24, for example. Each pixel in the depth map image 98 is assigned a value based on its distance from camera 24, and each pixel in the depth map image 98 is a count and each count is assigned to a bin based on its value. Thus, pixels for bin 84 in a particular layer will be assigned to the same bin or surrounding bins. The histogram shows a peak 100 of counts for the top layer 88 at a depth of 70 cm, a peak 102 of counts for the middle layer 92 at a depth of 90 cm, and a peak 104 of counts for the bottom layer 96 at a depth of 105 cm, where peak 106 at zero cm from camera 24 includes counts from pixels that could not be measured and are eliminated, and peak 108 at 120 cm from camera 24 is ground level and is also eliminated.

[0025] The controller 26 identifies the closest peak as the top layer of bins 84, which is peak 100, and crops all bins associated with that peak or surrounding values ​​from the image, as shown in FIG. Figure 7 Now, there are no boxes in the cropped image that are covered by other boxes, and the segmentation process can be performed on the cropped image in the manner described above. Once all boxes 84 have been picked up in the cropped image, the controller 26 will generate another histogram in which the closest peak will now be the peak 102 of the middle layer of boxes 84, as shown in FIG. Figure 8 Shown cropped.

[0026] As will be fully understood by those skilled in the art, the several and various steps and processes discussed herein for describing the present disclosure may refer to operations performed by a computer, processor, or other electronic computing device that uses electrical phenomena to manipulate and / or transform data. Those computers and electronic devices may employ various volatile and / or non-volatile memories, including non-transitory computer-readable media having stored thereon executable programs including various codes or executable instructions capable of being executed by a computer or processor, wherein the memories and / or computer-readable media may include all forms and types of memories and other computer-readable media.

[0027] The foregoing discussion discloses and describes only exemplary embodiments of the present disclosure. Those skilled in the art will readily recognize from such discussion and from the accompanying drawings and claims that various changes, modifications and variations may be made therein without departing from the spirit and scope of the present disclosure as defined in the appended claims.

Claims

1. A method for picking an object box from a group of a plurality of boxes, the method comprising: acquiring 2D red-green-blue (RGB) color images of the plurality of boxes using a 3D camera; acquiring, using the 3D camera, a 2D depth map image of the plurality of boxes, wherein each pixel in the depth map image is assigned a value identifying a distance from the camera to the plurality of boxes; generating segmented images of the plurality of boxes by performing an image segmentation process that extracts a plurality of features from the RGB image and the depth map image, combines the plurality of extracted features in the images, and assigns a label to each pixel in the segmented image such that each box in the segmented image has the same label; as well as using the segmented image to identify a location for picking up a box; The method also includes separating a top layer of the plurality of boxes stacked on top of other boxes in the group of the plurality of boxes from the color image before generating the segmented image of the plurality of boxes, wherein generating the segmented image of the plurality of boxes includes generating a segmented image of the top layer of the plurality of boxes.

2. The method according to claim 1, wherein Generating segmented images involves using deep learning Mask R-CNN (Convolutional Neural Network).

3. The method according to claim 1, wherein Generating a segmented image includes providing a plurality of bounding boxes, aligning the plurality of bounding boxes with the extracted plurality of features, and providing a bounding box image including a plurality of bounding boxes surrounding the object box.

4. The method according to claim 3, wherein: Generating a segmented image involves determining the probability that the object box exists within each bounding box.

5. The method according to claim 3, wherein Generating the segmented image includes removing pixels not associated with an object box from each bounding box in the bounding box image.

6. The method according to claim 1, wherein Generating the segmented image includes assigning labels to pixels in the segmented image such that each box in the segmented image has the same label.

7. The method of claim 1 , wherein when a box is picked from the group of the plurality of boxes, a lower layer of the stack of the plurality of boxes becomes the top layer of the plurality of boxes.

8. The method of claim 1 , wherein separating the top layer comprises counting pixels having similar depth values ​​in the depth map image, and identifying the top layer by a group of pixels having depth values ​​indicating that they are closest to the camera.

9. The method according to claim 8, wherein The histogram is used to identify groups of pixels with similar depth values.

10. The method according to claim 1, wherein Generating the segmented image includes identifying a center pixel of each object in the plurality of bounding boxes, and wherein identifying a location for picking an object includes identifying a center pixel of one of the plurality of objects in the segmented image, and calculating an xyz coordinate of the center pixel using the location of the center pixel and the depth map image.

11. The method of claim 10, wherein identifying a position for picking up the box comprises using the xyz coordinates and the pitch, yaw, and roll positions to determine a grasping pose for picking up the box.

12. The method of claim 11, wherein the box is picked up by a robot and the grasping pose determines a pose of an end effector of the robot.

13. A method for picking a box of an object from a group of a plurality of boxes using a robot, the method comprising: acquiring 2D red-green-blue (RGB) color images of the plurality of boxes using a 3D camera; acquiring, using the 3D camera, a 2D depth map image of the plurality of boxes, wherein each pixel in the depth map image is assigned a value identifying a distance from the camera to the plurality of boxes; generating segmented images of the plurality of boxes by performing an image segmentation process using a deep learning convolutional neural network, the image segmentation process extracting a plurality of features from the RGB image and the depth map image, combining the plurality of extracted features in the images, and assigning a label to each pixel in the segmented image such that each box in the segmented image has the same label; as well as identifying a location for picking up a box using the segmented image, wherein acquiring the color image, acquiring the depth map image, generating the segmented image, and identifying a location for picking up the box are performed each time the robot picks up a box from the group of the plurality of boxes; The method also includes separating a top layer of the plurality of boxes stacked on top of other boxes in the group of the plurality of boxes from the color image before generating the segmented image of the plurality of boxes, wherein generating the segmented image of the plurality of boxes includes generating a segmented image of the top layer of the plurality of boxes.

14. The method according to claim 13, wherein Generating a segmented image includes providing a plurality of bounding boxes, aligning the plurality of bounding boxes with the extracted plurality of features, providing a bounding box image including a plurality of bounding boxes surrounding a plurality of object boxes, determining a probability that an object box exists in each bounding box, and removing pixels not associated with the object box from each bounding box in the bounding box image.

15. The method of claim 13 , further comprising separating the top layers of the plurality of boxes from the color image before generating the segmented image of the plurality of boxes by counting pixels having similar depth values ​​in the depth map image and identifying the top layers of the plurality of boxes by a group of pixels having depth values ​​indicating that they are closest to the camera, the top layers being stacked on top of other boxes in the group of the plurality of boxes, wherein Groups of pixels having similar depth values ​​are identified via a histogram, and wherein generating a segmented image of the plurality of bins comprises generating a segmented image of the top layer of the plurality of bins.

16. The method according to claim 15, wherein When a box is picked from the group of the plurality of boxes, a lower layer of the stack of the plurality of boxes becomes the top layer of the plurality of boxes.

17. A robotic system for picking a box of an object from a group of a plurality of boxes using a robot, the system comprising: a 3D camera that provides 2D red-green-blue (RGB) color images and 2D depth map images of the plurality of boxes; a deep learning convolutional neural network that generates segmented images of the plurality of boxes by performing an image segmentation process that extracts a plurality of features from the RGB image and the depth map image, combines the plurality of extracted features in the images, and assigns a label to each pixel in the segmented image such that each box in the segmented image has the same label; as well as means for identifying a location for picking up a box using the segmented image, wherein acquiring the color image, acquiring the depth map image, generating the segmented image, and identifying the location for picking up the box are performed each time the robot picks up a box from the group of the plurality of boxes; The system further includes a separation unit for separating the top layers of the plurality of boxes from the color image before generating the segmented image of the plurality of boxes by counting pixels having similar depth values ​​in the depth map image and identifying the top layers of the plurality of boxes by a group of pixels having depth values ​​indicating that they are closest to the camera, the top layers being stacked on top of other boxes in the group of the plurality of boxes, the group of pixels having depth values ​​indicating that they are closest to the camera, wherein the plurality of groups of pixels are identified by a histogram, each group of pixels having similar depth values, and wherein generating the segmented image of the plurality of boxes includes generating a segmented image of the top layers of the plurality of boxes.

18. The system according to claim 17, wherein: The image segmentation process provides a plurality of bounding boxes, aligns the plurality of bounding boxes with the extracted plurality of features, provides a bounding box image comprising a plurality of bounding boxes surrounding an object box, determines a probability that the object box exists in each bounding box, and removes pixels not associated with the object box from each bounding box in the bounding box image.

Images