System and method for learning and executing robotic grasps

Abstract

A method for learning robotic grasping of objects includes inputting images of scenes with objects in random configurations from a training dataset to a grasping neural network having an encoder-decoder architecture. Color frames of the images are input to a first encoder and corresponding depth frames are input to a second encoder. Features extracted from the first and second encoders are fused via a third encoder to produce a fusion encoding. The fusion encoding is inputted to first and second decoders to respectively produce a grasp map output and a segmentation map output. The first and second decoders are jointly trained in a supervised process using a common loss function defined by a grasp loss of the first decoder and a segmentation loss of the second decoder. During the training, the grasp map output of the first decoder is guided by the segmentation map output of the second decoder.

Description

SYSTEM AND METHOD FOR LEARNING AND EXECUTING ROBOTIC GRASPSTECHNICAL FIELD

[0001] The present disclosure relates generally to the field of robotics in industrial automation, and more specifically to systems and methods for learning and executing robotic grasps utilizing neural networks.BACKGROUND

[0002] Robotic bin picking of unknown objects is a crucial component in advancing automation in warehouses and manufacturing lines. Recent advances in computer vision and deep learning have enabled this technology to be deployed at large scales for picking objects of all geometries and in any arrangement. Camera systems, such as RGB-D cameras, may collect both color pictures and depth maps or point-clouds of bins with objects in random configurations. The camera input may then be transferred to deep neural networks (“grasping neural networks”) that have been trained to compute grasping locations or “pick points” based on said input.

[0003] The success of a robotic bin picking solution may be measured in customer focused KPIs that include number of operator interventions, and successful picks per hour. It is desirable for the number of operator resets to be minimized and picks per hour to be maximized to allow for continuous running of the deployed system and to maximize the throughput of goods. To enable this, however, it may be necessary to ensure that a grasping neural network not only delivers a valid pick point, but that the pick point is also centered regardless of geometry and is located on an object that is totally un-occluded (not blocked by any other object). This may not be an easy task because of potential issues such as ‘double picks’ (when a pick point is on the boundary between two objects leading to over-picking) and ‘popouts’ (when a pick point is on an object that is occluded and might fling another object out of the bin when being picked at high speeds). Furthermore, there can be a near infinite variation in object configurations that must be handled along with the fact that a grasping neural network is often deployed at the edge on resource constrained devices. Lastly, there can be a chance that an input image frame could be of poor quality. All these factors must be considered when designing an end-to-end grasping neural network whose role is to output the best possible pick point given the extremely high variability in object type and arrangement.SUMMARY

[0004] Aspects of this disclosure provide methods, systems, and computer program products that address and overcome one or more of the above-described technical challenges. The present disclosure is directed to an end-to-end deep learning-based grasping pipeline that can provide robust, centered and un-occluded pick points on unknown objects in random configurations and that can be run efficiently on low compute platforms, such as Edge devices.

[0005] A first aspect of this disclosure provides a computer-implemented method for learning robotic grasping of objects. The method comprises inputting images from a training dataset to a grasping neural network comprising an encoder-decoder architecture. Each image depicts a scene including one or more objects placed in random configurations. The inputting includes inputting color frames of the images to a first encoder and inputting depth frames of the corresponding images to a second encoder. The method further comprises, for each image, fusing features extracted from layers of the first and second encoders via a third encoder to produce a fusion encoding. The method further comprises feeding the fusion encoding to first and second decoders to respectively produce a grasp map output depicting pixel-wise grasp confidence values identifying grasping points for objects in the scene and a segmentation map output depicting pixel- wise segmentation of objects in the scene. The grasping neural network is trained by jointly training the first and second decoders based on respective ground truth information for the first and second decoders and a common loss function defined by a grasp loss of the first decoder and a segmentation loss of the second decoder. The grasp map output of the first decoder is guided by the segmentation map output of the second decoder during the training.

[0006] A second aspect of this disclosure provides a method for executing a robotic grasp. The method comprises acquiring a camera image of a pick scene including one or more objects. The method further comprises inputting the acquired camera image to a grasping neural network trained by the method described above. The inputting includes inputting a color frame derived from the camera image to the first encoder and inputting a depth frame derived from the camera image to the second encoder. The method further comprises estimating a pick point from a grasp map output produced by the first decoder based on the input camera image. The method further comprises outputting the estimated pick point to a robot controller to control a robotic end effector to execute the grasp.

[0007] Further aspects of this disclosure provide a system and a computer program product embodying the described method.

[0008] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0010] FIG. 1 schematically illustrates an autonomous system configured for executing a robotic grasp for a bin picking application, according to one or more embodiments.

[0011] FIG. 2 illustrates an architecture of a grasping neural network according to one or more embodiments.

[0012] FIG. 3 illustrates an example of modal object masks computed from an image.

[0013] FIG. 4 illustrates an example of computation of a weight map to generate ground truth information for an input image.

[0014] FIG.5, FIG. 6 and FIG. 7 illustrate grasp maps and segmentation maps indicating pick points output by the grasping neural network under different example scenarios.

[0015] FIG. 8 illustrates a scenario where a pick point estimation for a poor grasp is informed by a combination of a grasp map and a segmentation map.

[0016] FIG. 9 illustrates a computing environment within which embodiments of this disclosure may be implemented.DETAILED DESCRIPTION

[0017] Various technologies are described herein that are directed to robotic grasping of objects in industrial applications. The objects may be placed in a bin, or otherwise disposed, such as on a table. Without loss of generality, the proposed methodology is described in the context of a robotic bin picking application. The term “bin”, as used herein, refers to a container or other structure (e.g., a tray, tote, pallet, carton, etc.) capable of receiving physical objects. A robotic bin picking application typically involves controlling a robot having a robotic arm with end effector to grasp objects individually from a pile of objects disposed in a bin. The objects may be of the same or assorted types, and may, for example, be disposed in random configurations or poses in the bin.

[0018] To execute a robotic grasp, an optimal grasping location or pick point may be computed from a camera image of the pick scene that includes the bin with the objects using a grasping neural network. Grasping neural networks are typically convolutional, such that given an input image (typically including a depth frame), the network can output a grasp map that assigns each pixel of the input image with some type of grasp score indicative of a confidence of grasp. Based on the grasp map, an optimal grasp location (“pick point”) may be computed, typically by applying an ‘argmax’ operator on the grasp map, i.e., selecting the pixel with the highest grasp confidence.

[0019] The proposed methodology provides an improved deep learning-based grasping pipeline that can provide robust, centered and un-occluded pick points on unknown objects in random configurations and that can be run efficiently on low compute platforms, such as Edge devices. This objective is achieved by redesigning the grasping neural network architecture to truly understand the semantic and geometric properties of a pick scene and all the objects in it. The pipeline uses a color frame and a depth frame as input to the grasping neural network. The grasping neural network according to the proposed methodology is configured to output not only a grasp map (as described above) but also a semantic segmentation map for all the objects in the scene. While semantic segmentation has been commonplace for many years, its complementary effect on producing a strong grasp map to the present requirements is of great benefit.

[0020] According to the proposed methodology, the grasping neural network includes an encoder-decoder architecture. The network is trained by inputting images of scenes with objects in random configurations from a training dataset. Color frames of the images are input to a firstencoder and corresponding depth frames are input to a second encoder. Features extracted layerwise from the first and second encoders are fused via a third encoder to produce a fusion encoding. The provision of the third encoder is to learn lay er- wise self-attention, i.e., how a given pixel relates to other pixels in the feature map of the corresponding layer. The fusion encoding may be produced by aggregating features extracted from the last layers of the first, second and third encoders. The fusion encoding is inputted to first and second decoders to respectively produce a grasp map output depicting pixel- wise grasp confidence values identifying grasping points for objects in the scene and a segmentation map output depicting pixel- wise segmentation of objects in the scene. The first and second decoders are jointly trained in a supervised process using a common loss function defined by a grasp loss of the first decoder and a segmentation loss of the second decoder. During the training, the grasp map output of the first decoder is guided by the segmentation map output of the second decoder.

[0021] The grasping neural network according to the proposed methodology can produce highly centered picks on objects that are totally un-occluded for any unknown object. This is testament to the high level of understanding that the grasping neural network has for spatial relationships and boundaries between objects. The multi-head decoder architecture (including “first” and “second” decoders) enables the grasping neural network to learn highly relevant features that allows it to always identify an un-occluded object and then provide a centered grasp no matter what the object is. Also, due to the addition of the segmentation decoding head (“second” decoder), it may be possible to handle ambiguities in the grasp map better, since the segmentation map can be relied upon to guide the right choice of pick point through post-processing. In some embodiments, the choice of loss function, coupled with the architecture, may allow the grasping neural network to have the ability to compensate for prediction uncertainty. In some embodiments, the choice of weighting used for the class and boundary weighting for each input image can greatly contribute to the training accuracy of the grasping neural network. Furthermore, the proposed architecture is computationally efficient and can function with a single camera, allowing it to be run on resource constrained hardware platforms, such as Edge devices.

[0022] Aspects of the proposed methodology may be embodied as software executable by a processor. In some embodiments, aspects of the disclosed methodology may be suitably integrated into commercial artificial intelligence (Al)-based automation software products, such as SIMATIC Robot Pick Al™ developed by Siemens AG, among others.

[0023] Turning now to the drawings, FIG. 1 illustrates an autonomous system 100 configured for performing robotic bin picking according to one or more embodiments. In the following description, unless otherwise specified, the term “system” refers to the autonomous system 100. The system 100 may be implemented in a factory setting. In contrast to conventional automation, autonomy gives each asset on the factory floor the decision-making and self-controlling abilities to act independently in the event of local issues. The system 100 may comprise one or more robots, such as the robot 102, which may be controlled by a computing system 104 to execute one or more industrial tasks within a physical environment such as a shopfloor. Examples of industrial tasks include assembly, transport, or the like.

[0024] The computing system 104 may comprise an industrial PC, or any other computing device, such as a desktop or a laptop, or an embedded system, among others. The computing system 104 can include one or more processors configured to process information and / or control various operations associated with the robot 102. The processor(s) may include a one or more CPUs, GPUs, microprocessors, or any hardware devices suitable for executing instructions stored on a memory comprising a machine-readable medium. In particular, the one or more processors may be configured to execute an application program, such as an engineering tool, for operating the robot 102.

[0025] To realize autonomy of the system 100, in one embodiment, the application program may be designed to operate the robot 102 to perform a task in a skill-based programming environment. In contrast to conventional automation, where an engineer is usually involved in programming an entire task from start to finish, typically utilizing low-level code to generate individual commands, in an autonomous system as described herein, a physical device, such as the robot 102, is programmed at a higher level of abstraction using skills instead of individual commands. The skills are derived for higher-level abstract behaviors centered on how the physical environment is to be modified by the programmed physical device. Illustrative examples of skills include a skill to grasp or pick up an object, a skill to place an object, a skill to open a door, a skill to detect an object, and so on.

[0026] The application program may generate controller code that defines a task at a high level, for example, using skill functions as described above, which may be communicated to a robot controller 108. From the high-level controller code, the robot controller 108 may generate low-levelcontrol signals for one or more motors for controlling the movement of the robot 102, such as angular position of the robot arms, swivel angle of the robot base, and so on, to execute the specified task. In other embodiments, the controller code generated by the application program may be communicated to intermediate control equipment, such as programmable logic controllers (PLC), which may then generate low-level control commands for the robot 102 to be controlled. Additionally, the application program may be configured to directly integrate sensor data from the physical environment in which the robot 102 operates. To this end, the computing system 104 may comprise a network interface to facilitate transfer of live data between the application program and various sensors, such as camera 122.

[0027] The robot 102 can include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 can include wheels 114 or can otherwise be configured to move within the physical environment 106. The robot 102 can further include an end effector 116 attached to the robotic manipulator 110. The end effector 116 may include a gripper configured to grasp (hold) and pick up an object 118. Examples of end effectors include vacuumbased grippers or suction cups (as shown), antipodal grippers such as fingers or claws, magnetic grippers, among others. The robotic manipulator 110 can be configured to move so as to change the position of the end effector 116, to enable picking and moving objects 118 within the physical environment.

[0028] A robotic bin picking task may involve picking objects 118 in a singulated manner from the bin 120 containing the objects 118 using the end effector 116. The objects 118 may be disposed in random configuration (or poses) within the bin 120. The objects 118 can be of assorted types or of the same type. To accomplish this task, the system 100 may include sensors or cameras that enable the robot 102 to perceive the physical environment. As shown, these sensors may include (among others) a camera 122 for capturing an image of the pick scene that includes, in this case, the bin 120 containing the objects 118. The camera 122 may include, for instance, an RGB-D camera, or a cloud sensor, among others. In some embodiments, the camera 122 may be positioned to capture an image with a top-down view of the bin 120. The camera image may be provided as an input to a computing system, such as the computing system 104, for computing a pick point for the end effector 116 to execute a grasp. The pick point may be defined by coordinates of the end effector 116 in a 3D reference frame 130, as well as a direction of approach, which is typically defined by a normal to the object surface on which the pick point lies. The pick point may be computed using a grasping neuralnetwork according to the proposed methodology. The computed pick point may be outputted to a controller, such as the robot controller 108, to control the end effector 116 to pick an object 118. For example, as described above, the pick point may be output as high-level controller code to the controller, which may therefrom generate low-level commands to control movement of the end effector 116.

[0029] FIG. 2 illustrates an architecture of a grasping neural network (hereinafter “GNN”) 200 according to one or more embodiments. The various components of the GNN, such as encoders 206, 208 and 210, decoders 218, 220, including sub-components thereof, may be implemented by a computing system in various ways as hardware and programming. The programming for the components may take the form of processor-executable instructions stored on non-transitory machine-readable storage mediums and the hardware may include processors to execute those instructions. For example, the programs may run on the computing system 104 shown in FIG. 1. Furthermore, the processing capability may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements.

[0030] To train the GNN 200, a training dataset is utilized that comprises images where each image depicts a pick scene including one or multiple objects disposed in random configurations in a bin or table. In various embodiments, the training dataset may include real-world camera images of a pick scene, or synthetic camera images rendered by a simulation engine using 3D representations (e.g., CAD models) of the objects, bin / table and other physical structures present in the pick scene, or a combination of real-world and synthetic images. A non-limiting example of a commercially available 3D simulation engine suitable for the present application is Process Simulate™ developed by Siemens Industry Software Inc. In some embodiments, the synthetic images of the training dataset may be rendered by randomizing a set of environmental and / or sensor-based parameters. Examples of parameters that can be randomized include one or more of: number and object classes of objects present in the pick scene, position and texture of the objects in the bin, texture of background including the bin, settings of simulation camera, lighting conditions of the scene, etc.

[0031] Each image in the training dataset comprises a color frame and a depth frame. A color frame includes a two-dimensional representation of image pixels, where each pixel includes intensity values for a number of color components. An example of a color frame is an RGB colorframe, which is an image frame including pixel intensity information in red, green and blue color channels. A depth frame or depth map includes a two-dimensional representation of image pixels that contains, for each pixel, a depth value. The depth values correspond to the distance of the surfaces of scene objects from a camera viewpoint. The color frame and the corresponding depth frame of each image are aligned pixel-wise.

[0032] In one embodiment, the images in the training dataset may be acquired (in a physical and / or simulation environment) via an RGB-D camera, which may be configured to acquire an image with red-green-blue (RGB) color and depth (D) channels. Additionally, or alternatively, images in the training dataset can also be acquired (in a physical and / or simulation environment) using a point cloud sensor. A point cloud may include a set of points in a 3D coordinate system that represent a 3D surface or multiple 3D surfaces, where each point position is defined by its Cartesian coordinates in a 3D reference frame, and further by intensity values of color components (e.g., red, green and blue). A point cloud can thus include a colorized 3D representation of all surfaces in the respective scene. The point cloud can be converted into color (RGB) and depth frames by applying a sequence of transforms based on the camera intrinsic parameters. Camera intrinsic parameters are parameters that allow a mapping between pixel coordinates in the 2D image frame and the 3D reference frame. Typically, the camera intrinsic parameters include the coordinates of the principal point or optical center, and the focal length along orthogonal axes.

[0033] Continuing with reference to FIG. 1, the GNN 200 is trained by feeding depth frames 202 of the images as input to a first encoder 206 and feeding color frames 204 of the corresponding images as input to a second encoder 208. The first encoder 206 includes a number of convolutional layers, such as layers 206a, 206b, 206c 206d, which are configured to down-sample each depth frame 202 to learn a first set of features at each layer. The second encoder 208 includes a number of convolutional layers, such as layers 208a, 208b, 208c 208d, which are configured to down-sample each color frame 204 to learn a second set of features at each layer. For each image, features extracted from each layer of the first encoder 206 and the second encoder 208 are fused via a third encoder 210, referred to as a fusion encoder. The fusion encoder 210 comprises a number of convolutional layers 210a, 210b, 210c, 21 Od, which are configured to learn lay er- wise self-attention, i.e., how a given pixel relates to other pixels in the feature map of the corresponding layer. The features extracted from the last layers of the encoders 206, 208, 210 may be suitably aggregated, e.g., using standard computational modules, to produce a fusion encoding 216. In the shownembodiment, these computational modules include a concatenation and addition module 212 and an ASPP (Atrous Spatial Pyramid Pooling) module 214. The shown encoder architecture is illustrative, noting that the number of convolutional layers for each encoder 206, 208, 210 and the computational modules employed is a matter of design choice for one skilled in the art. Furthermore, the encoder architecture may incorporate any number of additional modules not explicitly described herein for brevity.

[0034] The GNN 200 includes two decoding heads, namely a first decoder (grasp decoding head) 218 and a second decoder (segmentation decoding head) 220. The fusion encoding 216 for each image is branched off and fed as an input to both of the decoding heads 218 and 220. Although not shown in FIG. 2, in some embodiments, the input to the first decoder 218 may additionally include a dimension of the gripper or end effector of the robot being used for the task. The dimension of the gripper may be defined by a pixel dimension computed based on the actual gripper dimension (e.g., expressed in a unit of distance measurement) and the depth frame 202.

[0035] The first decoder 218 may include a number of convolutional layers (not explicitly shown) configured to up-sample the fusion coding 216 and output a grasp map 222 depicting pixelwise grasp confidence values identifying grasping points for objects in the scene depicted in the image. The second decoder 220 may also include a number of convolutional layers (not explicitly shown) configured to up-sample the fusion coding 216 and produce as output a segmentation map 224 depicting pixel- wise segmentation of objects in the scene depicted in the image. The shown decoder architecture, which is exemplary, includes a U-Net style decoder that includes skipped connections from the fusion encoder 210 to both the decoders 218 and 220 (illustrated by dashed lines) to better aggregate information.

[0036] To enable training of the first decoder 218 and the second decoder 220, in addition to the images, the training dataset also includes ground truth information which are specifically generated for the respective decoders. The ground truth information for the first decoder 218 may include pixel labels for each pixel of the input image indicative of a grasp confidence value. In various embodiments, the ground truth pixel labels may be assigned based on hand labeling of the images (e.g., by a user locating one or more pick points in the image via a GUI), or may be automatically generated in case of synthetic images. The ground truth information for the second decoder 220 may include pixel labels for each pixel of the input image indicative of whether the pixel belongs to anobject in the scene. In various embodiments, the ground truth pixel labels may be assigned based on hand labeling of the images (e.g., by a user localizing objects in the scene via a GUI), or may be automatically generated in case of synthetic images.

[0037] To achieve a higher computational efficiency during training, the ground truth information for the second decoder 220 may be generated by computing, for each image in the training dataset, a binary segmentation mask which segments all objects in the scene from a background and further defines boundaries separating objects that touch each other. The binary segmentation mask may be generated by assigning binary pixel labels (e.g., 0 and 1) to image pixels depending on whether or not a given pixel belongs to any object in the scene. Generating a binary segmentation mask with defined object boundaries can be technically challenging especially for scenes where objects are disposed in a highly chaotic manner and / or are tightly packed. To overcome this challenge, in one embodiment, the binary segmentation mask for each image may be generated as described below.

[0038] In a first step, individual object masks may be computed for each object in the image. Each individual object mask may be generated by assigning binary pixel labels (e.g., 0 and 1) to image pixels, isolating only those pixels that belong to the particular object. In some embodiments, especially for chaotic scenes, the individual object masks may include modal masks showing only visible pixels of the respective object in the image. This is illustrated in FIG. 3, which shows an input color frame 302 and a corresponding depth frame 304 depicting a highly chaotic and tightly packed bin. The modal object masks 306, 308 and 310 correspond to three different objects, showing only visible pixels of each object.

[0039] Each individual modal mask could be taken and added consecutively on top of an empty map to generate a binary segmentation mask of all the visible objects. However, that would mean one large mask without any distinction between objects especially if they are all touching each other. Hence, in an intermediate step, an erosion operation may be performed to reduce the dimensions (i.e., pixel dimensions) of each object mask. For example, the outer boundary of each object mask may be reduced by 1 pixel (or any defined number of pixels). This would create a separation of 2 pixels for objects that were touching each other. Subsequently, the individual object masks with reduced dimensions may be added on an empty map to create a binary segmentation mask for all objects in the scene, wherein erosion operation creates the boundaries separating objectsthat touch each other.

[0040] An important objective while generating the ground truth information for the second decoder 220 is to enable the GNN 200 to learn boundaries between objects so that the pick point is centered on the object. In some scenarios, a separation of 2 pixels achieved by an erosion operation can be very difficult for a neural network to pick up on during training, where it needs to focus on finding this gap and on separating objects. To address this practical challenge, in some embodiments, the ground truth information for the second decoder 220 may be generated by computing, for each binary segmentation mask, a weight map that assigns higher weights to boundary pixels of each object mask. The extra weight for the boundary can better aid the GNN learn to separate objects.

[0041] In one embodiment, the weight map may be computed as follows. Using the binary segmentation mask the starting point, a distance map may be computed for each object mask in the binary segmentation mask. The boundaries between objects are well-defined consequent to the erosion operation, thereby enabling identifying of each object mask separately on the binary segmentation mask and computing a distance map thereof. First, individual labels are generated from the binary mask using connected component labeling to identify each mask. For each label, the distance map for the given object mask may be computed using Euclidean distance transform, which indicates, for each pixel in the object mask, the shortest distance in pixels to a boundary of the nearest other object mask. The distance maps are then sorted in descending order at the pixel level and the two closest distances are used to calculate the border weight map.

[0042] In some scenarios, e.g., when there is one tiny object in the scene, the GNN could still get a -99% accuracy if it predicted 0s for all pixel labels (i.e., no object). To ensure that the GNN predicts a correct output irrespective of object size relative to the background in the image, class balancing may be desirable. In some embodiments, weights in the weight map may be computed to encode class balancing in the ground truth information.

[0043] FIG. 4 illustrates an example of computation of a weight map to generate ground truth information for an image from the training dataset. The input image comprises a color frame 402 and a depth frame 404. The binary segmentation mask 406 is computed as described above, by (i) determining individual object masks from the input image (ii) performing erosion operation on theinduvial object masks and (iii) adding the object masks subsequent to the erosion operation. As shown, the binary segmentation mask 406 has binary pixel labels segmenting objects (pixel value= 1) from the background (pixel value=0), and which further defines boundaries separating objects that touch each other. Finally, the weight map 408 is computed from the binary segmentation mask 406 as described above, by (i) computing distance maps for each object mask, (ii) aggregating the distance maps and (iii) carrying out post-processing to determine appropriate weighting based on distance. As shown, in contrast to the binary segmentation mask 406, the weight map 408 assigns higher weights to object boundaries (pixel value= 1.2) in relation to the center of the objects (pixel value=0.8). While the boundaries get the highest weighting, in the weight map 408, the pixel values in the object mask (pixel value=0.8) and the background (pixel values=0.4) are non-zero and show a ratio favoring the object mask pixels. This implies that there is a weighting for object masks relative to the background and ensures that a certain degree of class balancing is encoded in the ground truth information.

[0044] Referring again to FIG. 2, having generated the ground truth information, there exists the challenge of training the GNN 200 to achieve at least the stated objectives. The GNN 200 is trained in a supervised learning process using the ground truth information of the respective decoders 218, 220. Logically, if the GNN 200 starts to learn boundaries between objects on a segmentation map output 224, it should learn to separate the grasping points or pick points on the grasp map output 222 too. The trick is to make them strongly correlated so that the grasp map output 222 would be guided by the segmentation map output 224 as the GNN 200 learns during training. To achieve this, the GNN 200 is trained by jointly training the first decoder 218 and the second decoder 220 using a common loss function defined by a grasp loss of the first decoder 218 and a segmentation loss of the second decoder 220. The grasp loss and the segmentation loss are computed based on ground truth information of the first decoder 218 and that of the second decoder 220 respectively. The common loss function may be backpropagated to update parameters (e.g., neural network weights and biases) of both the decoders 218, 220 during the training process.

[0045] In some embodiments, the grasp loss and the segmentation loss may be computed, respectively, as a binary cross-entropy loss of the first decoder 218 and a binary cross-entropy loss of the second decoder 220.

[0046] In some embodiments, the common loss function may be further defined by weightsindicating an uncertainty of predicted outputs of the first decoder 218 and the second decoder 220 respectively. The weights may be designed to control the confidence of the outputs of the respective decoders 218, 220. The uncertainty weighting may ensure that during training, the GNN 200 can learn the two complementary set of features (for grasping and segmentation) at the same rate.

[0047] Taking into consideration the above factors, in one embodiment, the common loss function L may be defined as:where Lgand Lsare binary cross-entropy losses of the first and second decoders 218, 220 respectively, and wgand wsare weights that have been added to the first and second decoders 218, 220 respectively, that represent the uncertainty of the GNN about the particular decoder prediction. Note that in this embodiment, class balancing was conveniently left out in formulating the loss function L, since this is already encoded in the ground truth information during training.

[0048] After the training process, the GNN 200 may be tested, validated, and subsequently deployed on a real-time control system, such as the computing system 104 shown in FIG. 1. In some embodiments, the real-time control system may comprise an Edge device or other computing device(s) operating on a resource-constrained hardware platform. A non-limiting example of a hardware platform where the trained GNN can be suitably deployed is the TM MFP module in the SIMATICTMline of controllers developed by Siemens AG.

[0049] Referring now to FIG. 1, an example of a robotic grasp execution utilizing the trained GNN is described. The camera 122 captures an image of a pick scene including the bin 120 containing objects 118. The camera image is inputted to the trained GNN deployed on the computing system 104. The color frame obtained from the camera image is inputted to the first encoder of the GNN, while the depth frame obtained from the camera image is inputted to the second encoder of the GNN. The GNN may process the color and depth frames of the camera image as described above (see FIG. 2), i.e., using the fusion encoder to produce a fusion encoding feeding into the first and second decoders, to output a grasp map and a segmentation map respectively. The pick point may be estimated utilizing the grasp map output. For example, in some embodiments, the pick point may be computed by applying an argmax operator on the grasp map output, i.e.,determining the pixel with the highest grasp confidence value in the grasp map. The computed pick point may be transformed from the 2D pixel frame of the grasp map to the 3D real-world reference frame 130 based on the depth frame and the camera intrinsic parameters of the camera 122, using a sequence of known transforms. The estimated pick point, which may include X, Y, Z coordinates and a direction of approach, is outputted to the robot controller 108 to control the robotic end effector 116 to execute the grasp.

[0050] FIG. 5-7 illustrate sample outputs of the trained GNN. As can be seen, the trained GNN has a remarkable ability to produce pick points that are always in the center of an object and that are always on the topmost (un-occluded) object. The hypothesis that the segmentation map output would guide the grasp map output has turned out to be accurate.

[0051] FIG. 5 shows the robustness of the GNN output in an extremely difficult scenario with two objects of the same height adjacently placed in a bin. Here, the image 502 depicts the input depth frame, the image 504 depicts the input color frame, the image 506 depicts the grasp map output and the image 508 depicts the segmentation map output. The grasp map 506 has been postprocessed to highlight pixels with grasp confidence values higher than a threshold. As seen, the grasp map 506 has been guided by the segmentation map 508 to produce two distinct pick points Pi and P2, which are separated by the object boundary. The final pick point PF, depicted on the segmentation map 508, represents the pixels with the highest grasp confidence value.

[0052] FIG. 6 illustrates the robustness of the GNN output in a scenario represented by a chaotic arrangement of tightly packed objects in a bin. Here, the image 602 depicts the input depth frame, the image 604 depicts the input color frame, the image 606 depicts the grasp map output and the image 608 depicts the segmentation map output. The grasp map 606 has been post-processed to highlight pixels with grasp confidence values higher than a threshold. As seen, the grasp map 606 has been guided by the segmentation map 608 to produce three distinct pick points Pi, P2 and P3. The final pick point PF, depicted on the segmentation map 608, represents the pixels with the highest grasp confidence value.

[0053] FIG. 7 illustrates the robustness of the GNN output in a tricky scenario where there is one object on top of another object in a bin. Here, the image 702 depicts the input depth frame, the image 704 depicts the input color frame, the image 706 depicts the grasp map output and the image708 depicts the segmentation map output. The grasp map 706 has been post-processed to highlight pixels with grasp confidence values higher than a threshold. As seen, the grasp map 706 has been guided by the segmentation map 708 to produce three distinct pick points Pi, P2 and P3. In particular, the segmentation map 708 has guided the grasp map 706 to produce the pick point Pi on the top object. If it were to be on the bottom object, that would have been a bad prediction, because if picked, it would fling out the topmost object leading to a ‘popout’ situation. The final pick point PF, depicted on the segmentation map 708, represents the pixels with the highest grasp confidence value.

[0054] Furthermore, the architecture of the GNN allows for the ability to deal with ambiguity in the model’s grasp predictions. In some scenarios, the highest grasp confidence value in the grasp map may fail to reach a threshold. If this happens, the GNN may be unable to produce a well- defined pick point on an object in the grasp map. In some embodiments, if the highest grasp confidence value is below a threshold, the estimation of the pick point may be informed by a combination of the grasp map output and the segmentation map output. In this case, the segmentation map output, which has proven to be very reliable, can help guide the choice since the bounds and area of the object are known from this. The final choice of the pick point may be determined, for example, based on a user input.

[0055] FIG. 8 demonstrates a scenario where a pick point estimation for a poor grasp is informed by a combination of a grasp map and a segmentation map. Here, the image 802 depicts the input depth frame, the image 804 depicts the input color frame, the image 806 depicts the grasp map output and the image 808 depicts the segmentation map output. The grasp map 806 has identified the pixel(s) with the highest grasp confidence value pi but fails to show a well-defined pick point since the grasp confidence values of the pixels are below a threshold. In this case, the grasp map 806 may be interpreted in light of the segmentation map 808 to infer the final pick point PF.

[0056] The performance of the proposed GNN architecture on low compute platforms has also been demonstrated. For example, the runtime performance for the above-described GNN turns out to be about 500ms on average, using OPENVino toolkit to port and run on the TM MFP, which only possesses a CPU without any dedicated GPU acceleration. For context, most state-of-the-art vision transformers would not even fit on the memory of this hardware platform. For further context, this hardware platform currently has comparable or lower compute capabilities than a regular laptopCPU, which outlines just how much efficiency the proposed GNN architecture was able to achieve for the level of performance it delivers.

[0057] FIG. 9 illustrates an exemplary computing environment comprising a computing system 902, within which aspects of the present disclosure may be implemented. The computing system 902 may be embodied, for example and without limitation, as an industrial PC with a Linux operating system, for executing real-time control of a physical device, such as a robot.

[0058] As shown in FIG. 9, the computing system 902 may include a communication mechanism such as a system bus 904 or other communication mechanism for communicating information within the computing system 902. The computing system 902 further includes one or more processors 906 coupled with the system bus 904 for processing the information. The processors 906 may include one or more central processing units (CPUs), graphical processing units (GPUs), Al accelerators, or any other processor known in the art.

[0059] The computing system 902 also includes a system memory 908 coupled to the system bus 904 for storing information and instructions to be executed by processors 906. The system memory 908 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 910 and / or random access memory (RAM) 912. The system memory RAM 912 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The system memory ROM 910 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 908 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 906. A basic input / output system 914 (BIOS) containing the basic routines that help to transfer information between elements within computing system 902, such as during start-up, may be stored in system memory ROM 910. System memory RAM 912 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 906. System memory 908 may additionally include, for example, operating system 916, application programs 918, other program modules 920 and program data 922.

[0060] The computing system 902 also includes a disk controller 924 coupled to the system bus 904 to control one or more storage devices for storing information and instructions, such as amagnetic hard disk 926 and a removable media drive 928 (e.g., floppy disk drive, compact disc drive, tape drive, and / or solid state drive). The storage devices may be added to the computing system 902 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0061] The computing system 902 may also include a display controller 930 coupled to the system bus 904 to control a display 932, such as a cathode ray tube (CRT) or liquid crystal display (LCD), among other, for displaying information to a computer user. The computing system 902 includes a user input interface 934 and one or more input devices, such as a keyboard 936 and a pointing device 938, for interacting with a computer user and providing information to the one or more processors 906. The pointing device 938, for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the one or more processors 906 and for controlling cursor movement on the display 932. The display 932 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 938.

[0062] The computing system 902 also includes an I / O adapter 946 coupled to the system bus 904 to connect the computing system 902 to a controllable physical device, such as a robot. In the example shown in FIG. 9, the I / O adapter 946 is connected to robot controller 948. In one embodiment, the robot controller 948 includes, for example, one or more motors for controlling linear and / or angular positions of various parts (e.g., arm, base, etc.) of a robot.

[0063] The computing system 902 may perform a portion or all of the processing steps of embodiments of the disclosure in response to the one or more processors 906 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 908. Such instructions may be read into the system memory 908 from another computer readable storage medium, such as a magnetic hard disk 926 or a removable media drive 928. The magnetic hard disk 926 may contain one or more datastores and data files used by embodiments of the present disclosure. Datastore contents and data files may be encrypted to improve security. The processors 906 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 908. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0064] The computing system 902 may include at least one computer readable storage medium or memory for holding instructions programmed according to embodiments of the disclosure and for containing data structures, tables, records, or other data described herein. The term “computer readable storage medium” as used herein refers to any medium that participates in providing instructions to the one or more processors 906 for execution. A computer readable storage medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto- optical disks, such as magnetic hard disk 926 or removable media drive 928. Non-limiting examples of volatile media include dynamic memory, such as system memory 908. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 904. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0065] The computing environment 900 may further include the computing system 902 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 944. Remote computing device 944 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computing system 902. When used in a networking environment, computing system 902 may include a modem 942 for establishing communications over a network 940, such as the Internet. Modem 942 may be connected to system bus 904 via network interface 945, or via another appropriate mechanism.

[0066] Network 940 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computing system 902 and other computers (e.g., remote computing device 944). The network 940 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networksmay work alone or in communication with each other to facilitate communication in the network940.

[0067] The embodiments of the present disclosure may be implemented with any combination of hardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non-transitory computer-readable storage medium. The computer readable storage medium has embodied therein, for instance, computer readable program instructions for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0068] The computer readable storage medium can include a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network.

[0069] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the disclosure to accomplish the same objectives. Although this disclosure has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the disclosure.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method for learning robotic grasping of objects, comprising: inputting images from a training dataset to a grasping neural network comprising an encoder-decoder architecture, each image depicting a scene including one or more objects placed in random configurations, the inputting comprising inputting color frames of the images to a first encoder and inputting depth frames of the corresponding images to a second encoder, for each image, fusing features extracted from layers of the first and second encoders via a third encoder to produce a fusion encoding, feeding the fusion encoding to first and second decoders to respectively produce a grasp map output depicting pixel- wise grasp confidence values identifying grasping points for objects in the scene and a segmentation map output depicting pixel-wise segmentation of objects in the scene, and the grasping neural network being trained by jointly training the first and second decoders based on respective ground truth information for the first and second decoders and a common loss function defined by a grasp loss of the first decoder and a segmentation loss of the second decoder, whereby the grasp map output of the first decoder is guided by the segmentation map output of the second decoder during the training.

2. The method according to claim 1, wherein the ground truth information for the second decoder is generated by computing, for each image, a binary segmentation mask which segments all objects in the scene from a background and further defines boundaries separating objects that touch each other.

3. The method according to claim 2, wherein the binary segmentation mask is computed by: computing an individual object mask for each object in the image, performing an erosion operation to reduce the dimensions of each object mask by a defined number of pixels, andadding the individual object masks subsequent to the erosion operation to create the binary segmentation mask for all objects, wherein the erosion operation creates the boundaries separating objects that touch each other.

4. The method according to claim 3, wherein the individual object masks are modal masks defined by visible pixels.

5. The method according to any of claims 3 and 4, wherein the ground truth information for the second decoder is generated by computing, for each binary segmentation mask, a weight map assigning higher weights to boundary pixels of each object mask.

6. The method according to claim 5, wherein the weight map is computed based on a computing a distance map for each object mask, the distance map indicating, for each pixel of the object mask, the shortest distance shortest distance in pixels to a boundary of the nearest other object mask.

7. The method according to any of claims 6 and 7, wherein weights in the weight map are computed to encode class balancing in the ground truth information.

8. The method according to any of claims 1 to 7, wherein the loss function is further defined by first and second weights, which are indicative of an uncertainty of predicted outputs of the first decoder and the second decoder respectively.

9. The method according to any of claims 1 to 8, wherein the grasp loss and the segmentation loss of the loss function are computed, respectively, as a binary cross-entropy loss of the first decoder and a binary cross-entropy loss of the second decoder.

10. The metho according to any of claims 1 to 9, wherein the architecture of the grasping neural network includes skipped connections from the third encoder to both of the first and second decoders.

11. A method for executing a robotic grasp, comprising: acquiring a camera image of a pick scene including one or more objects,inputting the acquired camera image to a grasping neural network trained by the method according to any of claims 1 to 10, which comprises: inputting a color frame obtained from the camera image to the first encoder and inputting a depth frame obtained from the camera image to the second encoder, estimating a pick point from a grasp map output produced by the first decoder based on the input camera image, and outputting the estimated pick point to a robot controller to control a robotic end effector to execute the grasp.

12. The method according to claim 11, wherein the pick point is computed by determining a pixel with the highest grasp confidence value in the grasp map output.

13. The method according to any of claims 11 and 12, wherein, if the highest grasp confidence value is below a threshold, the estimation of the pick point is informed by a combination of the grasp map output produced the first decoder and a segmentation map output produced by the second decoder, based on the input camera image.

14. A non-transitory computer-readable storage medium including instructions that, when processed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 13.

15. A system for executing a robotic grasp, comprising: a robot comprising a robotic end effector controllable by a robot controller, a camera configured to capture a camera image of a pick scene including one or more objects, one or more processors, and memory storing instructions executable by the one or more processors to perform a method according to any of claims 11 to 13, to execute a grasp by the robot based on the camera image.

Images