System and method for estimation of object planar dimensions for autonomous handling of objects

Abstract

A method for autonomous handling of an object involves acquiring sensor data to produce a point cloud of a scene including one or more objects at a first location. An object instance is segmented in the point cloud utilizing an instance segmentation mask. The object instance represents an individual object to be transported by a controllable physical device. Using points in the segmented point cloud, a direction of a main plane is determined, which represents a placement surface of the object. A set of points in the segmented point cloud defining an extent of the main plane is projected with respect to a normal of the main plane, to create a 2D image. The 2D image is processed to generate a contour defining an outline of the placement surface, from which planar dimensions of the placement surface are estimated to inform a placement of the object by the physical device.

Description

SYSTEM AND METHOD FOR ESTIMATION OF OBJECT PLANAR DIMENSIONS FOR AUTONOMOUS HANDLING OF OBJECTSTECHNICAL FIELD

[0001] The present disclosure relates generally to the field of autonomy in the execution of industrial automation tasks. Specifically, the described embodiments relate to systems, methods and computer program products for estimation of object planar dimensions for execution of robotic pick and place operation or other autonomous object handling tasks.BACKGROUND

[0002] The fourth industrial revolution aims to drive mass customization to the cost of mass production. This can be enabled by autonomous machines that no longer have to be programmed with detailed instructions, such as way points or manually taught paths, but automatically define their tasks using design information of the product to be produced. Robotic bin picking is one such skill that enables autonomous machines. Traditional robotic picking relies on a teach-based approach, such that operators can predefine the robot poses for pick and drop positions. In the last decade, advances in computer vision and deep learning have enabled flexible robotic bin picking, where pre-teaching pick and drop positions is no longer required. Camera systems, such as RGB-D cameras, collect both color pictures and depth maps or point-clouds of bins with objects in random configurations. The camera input is then transferred to computer vision algorithms or deep neural networks that have been trained to compute grasping points or “pick points” on said input. These methodologies have been proven to work reliably well even without any prior information about the object geometry and color (no CAD data is needed).

[0003] In addition to computing pick points, it is desirable that autonomous systems infer other high-level information of the objects they handle. For example, an autonomous system may need to estimate an object’s footprint, particularly the planar dimensions of the surface on which it is to be eventually placed or dropped, which may allow downstream processes to be performed in an informed manner.

[0004] If an object’s 3D model is known beforehand, this problem is somewhat straightforward. Atraditional computer vision algorithm or ad-hoc-trained neural network can be used to perform object pose estimation. Once the object’s pose is known, the dimensions of the pick surface can be determined readily using the object’s 3D model. If the object’s 3D model is unknown, and an object- agnostic instance segmentation model is used to estimate object segmentation masks, the object’s dimensions can be estimated coarsely based on the entirety of the segmentation mask. However, this may include surfaces other than the picking surface, and in turn may produce erroneous (too large) object dimensions. Another approach is to compute 3D bounding boxes of unknown objects by means of deep neural networks. This approach is data intensive and also computationally intensive, and therefore its usage may not be suitable for high-throughput resource constrained applications. Improved systems are desirable.SUMMARY

[0005] Embodiments of the disclosure address and overcome one or more of the described-herein shortcomings by providing methods, systems, and apparatuses that use instance segmentation masks in combination with 3D point cloud manipulation and 2D image processing to efficiently estimate object planar dimensions for autonomous handling of objects.

[0006] A first aspect of the disclosure provides a computer-implemented method performed by an autonomous system for handling an object. The method comprises acquiring sensor data to produce a point cloud of a scene in a physical environment including one or more objects at a first location. The method further comprises segmenting an object instance in the point cloud utilizing an instance segmentation mask. The object instance represents a first object, from the one or more objects in the scene, to be transported by the autonomous system to a second location. The method further comprises determining a direction of a main plane representing a placement surface of the first object using points in the segmented point cloud. The method further comprises projecting a set of points in the segmented point cloud, which define an extent of the main plane, with respect to a normal of the main plane, to create a 2D image. The method further comprises processing the 2D image to generate a contour defining an outline of the placement surface and therefrom estimating planar dimensions of the placement surface. The estimated planar dimensions of the placement surface are used to determine a placement operation executable by the autonomous system for placing the first object at the second location.

[0007] Further aspects of the disclosure provide autonomous systems and computer program products embodying the above-described method.

[0008] Additional technical features and benefits may be realized through the techniques of the present disclosure. Embodiments and aspects of the disclosure are described in detail herein and are considered a part of the claimed subject matter. For a better understanding, refer to the detailed description and to the drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing and other aspects of the present disclosure are best understood from the following detailed description when read in connection with the accompanying drawings. To easily identify the discussion of any element or act, the most significant digit or digits in a reference number refer to the figure number in which the element or act is first introduced.

[0010] FIG. 1 illustrates an exemplary autonomous system capable of executing a bin picking application, wherein aspects of the present disclosure may be incorporated.

[0011] FIG. 2 illustrates a bin and object configuration showing pick points, in a robotic bin picking application.

[0012] FIG. 3 is a flow chart illustrating a computer-implemented method for estimating planar dimensions of an object for effecting a placement operation by an autonomous system according to disclosed embodiments.

[0013] FIG. 4 illustrates generation of a contour in a 2D representation from a set of points in a 3D point cloud.

[0014] FIG. 5 illustrates a computing environment within which embodiments of the disclosure may be implemented.DETAILED DESCRIPTION

[0015] The present disclosure provides a methodology for efficient and accurate estimation of an object’s planar dimensions that can support performing of end-to-end object handling tasks by anautonomous system. To estimate an object’s planar dimensions, the disclosed methodology uses an instance segmentation mask in combination with 3D point cloud manipulation to determine a main plane defining a placement surface of the object. The main plane may be defined as the plane such that the object footprint is a projection of the object on the main plane. The methodology then reprojects a set of points in the segmented point cloud with respect to the normal of the main plane to create a 2D image and processes the 2D image (e.g., using basic 2D computer vision operations) to generate a contour, from which the object’s planar dimensions are estimated. Based on the estimated planar dimensions, a placement plan may be computed. The computed placement plan may be output to a controller configured to control a physical device for executing a placement of the object at a placement location.

[0016] Unlike known approaches, the disclosed methodology does not rely on deep learning methods that require large amounts of training data and / or prior knowledge of the object’s 3D model. Furthermore, by reprojecting the 3D segmented point cloud into a 2D image to enable processing via basic 2D computer vision operations, the computational cost is significantly reduced, making the solution suitable for high-throughput real-time applications.

[0017] An industrial application of the disclosed methodology involves a robotic pick and place operation. This task comprises operating a robot to autonomously pick an object from a first location (e.g., a bin containing a number of objects) and place it at a second location. The estimated planar dimensions of the object may be used to perform the placement operation in an informed manner, which may include, for example, placing the object in another box, arranging the object in a stack or grid, dropping the object in a chute, among others. The disclosed methodology is, however, not limited to the above-mentioned application. For example, the disclosed methodology may be used to estimate object planar dimensions to inform an induction conveyor system, among other applications.

[0018] Aspects of the disclosed methodology may be embodied as software executable by a processor. In some embodiments, aspects of the disclosed methodology may be suitably integrated into commercial artificial intelligence (Al)-based automation software products, such as SIMATIC Robot Pick Al™ developed by Siemens AG, among others.

[0019] Referring now to FIG. 1, an exemplary autonomous system 100 is illustrated where aspectsof the present disclosure may be embodied. The autonomous system 100 may be implemented, for example, in a factory setting. In contrast to conventional automation, autonomy gives each asset on the factory floor the decision-making and self-controlling abilities to act independently in the event of local issues. The autonomous system 100 comprises one or more controllable physical devices, such as a robot 102. The one or more devices, such as the robot 102, are controllable by a computing system 104 to execute one or more industrial tasks within a physical environment 106. Examples of industrial tasks include assembly, transport, or the like.

[0020] The computing system 104 may comprise an industrial PC, or any other computing device, such as a desktop or a laptop, or an embedded system, among others. The computing system 104 can include one or more processors configured to process information and / or control various operations associated with the robot 102. In particular, the one or more processors may be configured to execute an application program, such as an engineering tool, for operating the robot 102.

[0021] To realize autonomy of the system 100, in one embodiment, the application program may be designed to operate the robot 102 to perform a task in a skill-based programming environment. In contrast to conventional automation, where an engineer is usually involved in programming an entire task from start to finish, typically utilizing low-level code to generate individual commands, in an autonomous system as described herein, a physical device, such as the robot 102, is programmed at a higher level of abstraction using skills instead of individual commands. The skills are derived for higher-level abstract behaviors centered on how the physical environment is to be modified by the programmed physical device. Illustrative examples of skills include a skill to grasp or pick up an object, a skill to place an object, a skill to open a door, a skill to detect an object, and so on.

[0022] The application program may generate controller code that defines a task at a high level, for example, using skill functions as described above, which may be deployed to a robot controller 108. From the high-level controller code, the robot controller 108 may generate low-level control signals for one or more motors for controlling the movement of the robot 102, such as angular position of the robot arms, swivel angle of the robot base, and so on, to execute the specified task. In other embodiments, the controller code generated by the application program may be deployed to intermediate control equipment, such as programmable logic controllers (PLC), which may then generate low-level control commands for the robot 102 to be controlled. Additionally, theapplication program may be configured to directly integrate sensor data from physical environment 106 in which the robot 102 operates. To this end, the computing system 104 may comprise a network interface to facilitate transfer of live data between the application program and the physical environment 106. An example of a computing system suitable for the present application is described hereinafter in connection with FIG. 5.

[0023] Still referring to FIG. 1, the robot 102 can include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 can include wheels 114 or can otherwise be configured to move within the physical environment 106. The robot 102 can further include an end effector 116 attached to the robotic manipulator 110. The end effector 116 can include one or more tools configured to grasp and move an object 118 from a first location to a second location. For example, the end effector 116 may include a vacuum-based gripper (suction cup), among other types. In the shown scenario, objects 118 to be transported are placed in a receiver or bin 120 at the first location. The robotic manipulator 110 can be configured to move so as to change the position of the end effector 116, for example, so as to place or move objects 118 within the physical environment 106.

[0024] A bin picking application involves grasping objects 118, in a singulated manner, from the bin 120, by the robotic manipulator 110, using the end effectors 116. The objects 118 may be arranged in arbitrary poses within the bin 120. The objects 118 can be of assorted types or of the same type. The physical environment 106, which includes the objects 118 placed in the bin 120, is perceived via sensor data generated by one or more sensors 122. The sensor(s) 122 may include, one or more single or multi-modal sensors, for example, RGB sensors, depth sensors, infrared cameras, 3D scanners, among others. In the shown example, the one or more sensors includes at least one camera 122 mounted to the robotic manipulator 110.

[0025] The sensor data may be fed to a grasp detection algorithm deployed on the computing system 104 to determine an optimal grasp to be executed by the robot 102 based on the specified objective and imposed constraints (e.g., dimensions and location of the bin). The grasp detection algorithm may comprise a grasp neural network to compute the grasp for a robot to pick up an object given an input depth image (3D map) of the scene. Grasp neural networks are often convolutional, such that the networks can label each location (e.g., pixel) of the input image with some type of grasp affordance metric, referred to as grasp score. The output of the grasp detectionalgorithm may include an optimal grasp location, referred to as a “pick point.” However, a grasp detection algorithm by itself may not have any notion regarding the presence of an object of interest in the scene from the sensor data. Accordingly, it may be often necessary to additionally deploy an object detection algorithm on the computing system 104 to perform a semantic recognition of the objects 118 in the scene, to be able to select an object 118 to be transported by the robot 102. The object detection algorithm may comprise, among others, a segmentation neural network (e.g., a Mask Recurrent Convolutional Neural Network or Mask R-CNN) that can provide a pixel-wise object recognition output given an input two-dimensional color intensity image, such as a RGB color image.

[0026] Thus, a bin picking application may involve a combination of an object detection algorithm, to localize an object of interest among the objects 118 in the bin 120 given a color intensity image of the scene, and a grasp detection algorithm, to compute optimal grasp locations given a depth image of the scene. FIG. 2 illustrates a bin and object configuration in a robotic bin picking application. As shown, objects 202 in a bin 200 can be detected based on an object detection algorithm and respective pick points 204 for individual objects 202 can be computed using a grasp detection algorithm. In some embodiments, the object detection and grasp detection algorithms may utilize off-the-shelf neural networks that have been validated and tested extensively in similar applications.

[0027] The above-described approach can enable a robot to pick an object from a scene without any preprogramming or prior knowledge of how the object looks like. However, in order to efficiently perform a pick and place operation, in addition to computing pick points, it is desirable for the autonomous system to estimate a footprint of an object to be transported, particularly the planar dimensions of the surface on which it is to be eventually placed or dropped, which may allow downstream processes to be performed in an informed manner. The disclosed methodology is directed to this objective.

[0028] FIG. 3 illustrates an example of a computer-implemented method 300 for estimating planar dimensions of an object for effecting a placement operation by an autonomous system. The method 300 may be implemented in a computing environment in various ways, for example, as hardware and programming. The programming of the blocks 302-310 of the method 300 may take the form of processor-executable instructions stored on non-transitory machine-readable storage mediums andthe hardware may include processors to execute those instructions. For example, the programs may run on an industrial PC or on a smaller device (e.g., a controller) of an autonomous system. Furthermore, the processing capability may be distributed among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements.

[0029] Referring to FIG. 3, block 302 involves acquiring sensor data to produce a point cloud of a scene in a physical environment including one or more objects at a first location. In the example shown in FIG. 1, the scene includes the bin 120 containing objects 118 disposed at the first location. A point cloud is a set of points in a 3D coordinate system that represent a 3D surface, where each point position is defined by its Cartesian coordinates. The point cloud of the scene is thus a 3D representation of all surfaces in the scene where an object to be transported is initially located.

[0030] In one embodiment, the sensor data may comprise a depth image of the scene, which is acquired via a camera, such as the camera 122 in FIG. 1. A depth image is an image or image channel that contains information relating to the distance of the surfaces of scene objects from a viewpoint. The acquired depth image may be converted to a point cloud using intrinsic parameters of the camera. The camera intrinsic parameters are parameters that allow a mapping between pixel coordinates in the 2D image frame and 3D coordinates in the real-world. Typically, the camera intrinsic parameters include the coordinates of the principal point or optical center, and the focal length along orthogonal axes. Using the camera intrinsic parameters, a transformation matrix may be determined that can be used for projecting points in the depth image on to a 3D space, to produce a point cloud representation of the scene. In an alternate embodiment, the point cloud representation of the scene may be produced directly from sensor data generated by a 3D scanner.

[0031] At block 304, an instance segmentation mask is used to segment an object instance in the point cloud. The object instance represents one object, among the objects in the scene, which is selected by the autonomous system to be transported to a second location. Applying the instance segmentation mask on the point cloud of the scene, a segmented point cloud may be produced that represents only the 3D surface of the selected object.

[0032] According to a disclosed embodiment, the acquired sensor data may comprise a two- dimensional color intensity image of the scene, in addition to the depth image. An example of acolor intensity image is a RGB color image, which is an image including pixel intensity information in red, green and blue color channels. The color intensity image and the depth image may be aligned pixel-wise. For example, in some embodiments, a single RGB-D sensor, such as the camera 122 in FIG. 1, can be configured to acquire an image of the scene with RGB color and depth channels. The color intensity image may be provided as input to an instance segmentation algorithm. Instance segmentation essentially includes semantic segmentation and object detection with the added feature of identification of boundaries of the objects at the detailed pixel level. Given the input color intensity image, the instance segmentation algorithm may be used to compute an instance segmentation mask as a 2D pixel mask for each object instance detected in the image, where each instance segmentation mask comprises a set of pixels that denote a particular’ object. The computed instance segmentation masks may be used to segment the corresponding depth image. Examples of Al-based instance segmentation methods that can be applied or adapted for the present purpose include instance segmentation using: “Segment Anything Model” (SAM) developed by Meta Al, “You Look Only Once” (YOLO) model, Mask R-CNN, among others.

[0033] The instance segmentation mask for a selected object may be converted into a 3D point cloud mask by projecting the locations (in pixel coordinates) in the instance segmentation mask on to a 3D space, using the depth information from the segmented depth image and the camera intrinsic parameters, as described above. The 3D point cloud mask may be used to segment the point cloud of the scene to include only the object instance corresponding to the selected object.

[0034] Block 306 involves determining a direction of a main plane representing a placement surface of the selected object, using points in the segmented point cloud. An aim of the disclosed methodology is to estimate dimensions of the object footprint for a placement operation. The main plane may therefore be defined such that the object footprint is a projection of the object on the main plane. The intuition about the main plane is that it is the plane that best represents the object orientation. In many situations, the main plane may be the largest visible upward facing plane of the object. According to disclosed embodiments, the main plane may be determined using a pick point of the object instance, or may be determined using a heuristic.

[0035] According to a first approach, for a robotic pick and place operation, the main plane is determined by determining, in the segmented point cloud, a pick point of the object instance. This approach is based on the intuition that the plane that best represents the object orientation typicallycoincides with the plane on which a pick point will be (“pick plane”). Furthermore, it is recognized that the pick plane would typically correspond to the drop / place plane (i.e., the plane I orientation at which the object is later on dropped for other downstream processes), since objects do not usually get reoriented by the robotic manipulator. Pick point computation is often already part of the workflow in a robotic pick and place application, which can be suitably utilized as an input to methodology according to this approach.

[0036] Consistent with a disclosed embodiment, the pick point of the object instance may be determined from the acquired sensor data using a grasp detection algorithm, as described above. The pick point may be computed by the grasp detection algorithm on a 2D image frame using the instance segmentation mask, and projected on to a 3D space of the segmented point cloud based on the depth information (from depth image) and the camera intrinsic parameters. The computed pick point may thus be indicative of an optimal grasp location in the segmented point cloud.

[0037] In other embodiments, the pick point may be modeled using a key point in the instance segmentation mask. Key point detection can be performed from color intensity images, for example using neural networks, which may be embedded in the instance segmentation model or be a standalone model. Alternatively, non-deep learning methods may be employed to model the pick point. As an example, the centroid of the instance segmentation mask may be used to model the pick point. The key point / centroid computed on the instance segmentation mask may be projected onto a 3D space using the depth information (from depth image) and the camera intrinsic parameters, to locate the pick point in the segmented point cloud.

[0038] Having determined the pick point, a set of neighboring points may be selected around the pick point in the segmented point cloud, to compute a plane equation. The number or the reach of the neighboring points can be determined depending on the application. For example, in a robotic pick and place operation, the number or reach of the neighboring points may be determined based on a size of the end effector of the robot. To illustrate, in the example of FIG. 1, the set of neighboring points may be selected such that the maximum distance from the pick point does not exceed the radius of the suction cup 116. Given the set of neighboring points, a plane equation may be determined, for example using a method least squares, that best fits those points. The plane equation may define the main plane direction.

[0039] According to a second approach, if a pick point is not used or available (e.g., in non-robotic applications), the main plane may be determined based on a heuristic. In this embodiment, in a first step, an iterative search may be performed to determine all significant planes in the segmented point cloud, i.e., all significant planes defining the 3D surface of the segmented point cloud. The main plane may then be selected from among those significant planes using the heuristic.

[0040] To perform the iterative search to determine the significant planes, a Random Sample Consensus (RANSAC) method may be used, among others. For example, starting with a small set of random points, a plane equation may be determined using all those random points by means of least squares or other linear regression methods. Then, it may be checked if any other points in the segmented point cloud also belong to this plane, for example, by performing distance checks or other means. The plane equation may be re-calculated using all the initial random points and the new points, and all of these points (which define “inlier” points) are then removed from the segmented point cloud. The process may be repeated until too few points are left, or a predefined number of planes have been computed. Finally, the main plane may be selected using a heuristic, which may be tuned to the use-case in hand. For example, in most use-cases, the placement surface is desirably largest surface of the object. In this case, the heuristic may include selecting the main plane as the plane with the largest area. In some use-cases, perpendicular planes may be preferred over inclines. In this case, the heuristic may include selecting the main plane as the plane with the most straight normal (e.g., in relation to a reference plane).

[0041] Having determined the main plane, the planar dimensions of the placement surface may be estimated by estimating the area covered by all points in the main plane of the segmented point cloud. To reduce the computation cost associated with processing 3D point clouds, the disclosed methodology involves projecting the 3D point cloud into a 2D image and estimating the object planar dimensions by generating a contour in the 2D image, e.g., using 2D computer vision operations. This makes the solution lightweight and suitable for high-throughput real-time applications with significantly less computational resources.

[0042] Still referring to FIG. 3, at block 308, a set of points in the segmented point cloud is projected with respect to the normal of the main plane, to create a 2D image. To determine the set of points to be projected, first, an extent of the main plane is determined. This step may involve finding all points that belong to the main plane with knowledge of the main plane equation, for example, byusing a clustering method based on a combination of distance to plane, normal classification and other geometric properties. The set of points to be projected may then be obtained by removing all points in the segmented point cloud that do not belong to the main plane.

[0043] In some embodiments, in order to determine the set of points to be projected, outlier points with respect to the main plane that contribute to noisy measurements may be removed, for example, using statistical outlier filters. This step may be performed depending on the object geometry. For example, if the object is deformable (e.g., a bag of chips, garment wrapped in a plastic bag, etc.), this step can be less critical, since there is already significant noise in the point cloud on account of the deformable object geometry. However, in case of rigid objects, this step may be suitably implemented to produce a clean surface.

[0044] The determined set of points in the 3D segmented point cloud may first be projected into a 2D representation (depth image), the transformation being computed using the camera intrinsic parameters. The 3D points in the 2D representation may then be rotated with respect to the normal of the main plane. As a result of the rotation, a 2D image may be produced that has a viewing direction perpendicular to the main plane, i.e., the main plane is aligned with the camera frame of the 2D image. In this manner, camera perspective warping may be removed.

[0045] The above-described step may be illustrated referring to FIG 4. Here, the image 402 represents a 2D projection of a point cloud using camera intrinsic parameters. The image 402 depicts a scene including a bin containing a box lying on its right wall. The image 402 is essentially a depth image, which is a 2D representation of 3D points. That is, each point in the image 402, in addition to x and y coordinates, is further characterized by depth information. The reference numeral 404 refers to the set of all points in the main plane 406 of the object instance (box). The image 408 represents the 2D image resulting from the rotation of the points 404 with respect to the normal of the main plane 406. The image 408 has a viewing direction perpendicular to the main plane 406. That is, the main plane 406 has been rotated such that it is now aligned with the camera frame of the image 408. As illustrated, the main plane 406, which appears trapezoidal in the image 402 due to perspective warping, appears generally rectangular in the image 408 after re-projection of the points 404 in the direction of the main plane normal.

[0046] Continuing with reference to FIG. 3, at block 310, the 2D image is processed to generate acontour that defines an outline of the placement surface. The planar dimensions of the placement surface may be estimated by measuring an area enclosed by such a contour. The processing of the 2D image may involve any operation(s) to obtain an enhanced image otherwise extract useful information to generate the contour. For example, basic 2D computer vision operations, such as infilling, inpainting and opening operations, among others, may be performed to recover missing points or gaps. In some embodiments, missing points or gaps may be recovered by directly fitting a primitive shape to the contour. The fitted primitive shape may be determined such that it encloses the minimum area covered by all points in the contour. The planar dimensions of the placement surface may then be estimated by measuring the area enclosed by the primitive shape.

[0047] Referring again to FIG. 4, in the illustrated example, the contour 410 generated on the 2D image 408 is missing a comer. This may occur, for example, due to faulty depth imaging and / or insufficient points in the point cloud. In this example, the missing corner is recovered by fitting a minimum area rectangle 412 that contains all the points in the contour 410. The dimensions of the main plane may be obtained by simply measuring the length and width of the rectangle 412.

[0048] The estimated planar dimensions of the object placement surface may be used to compute a placement plan. The placement plan may include, among others, the position (coordinates) and / or orientation (rotation) of the transported object at the placement location. The placement plan may be outputted to a controller of a physical device (e.g., robot) for executing a placement of the object at the second location. For example, in a robotic pick and place application, the placement operation may include placing the object in another bin, placing the object in a stack or grid, dropping the object in a chute, etc.

[0049] FIG. 5 illustrates an exemplary computing environment comprising a computing system 502, within which aspects of the present disclosure may be implemented. The computing system 502 may be embodied, for example and without limitation, as an industrial PC with a Linux operating system, for executing real-time control of a physical device, such as a robot.

[0050] As shown in FIG. 5, the computing system 502 may include a communication mechanism such as a system bus 504 or other communication mechanism for communicating information within the computing system 502. The computing system 502 further includes one or more processors 506 coupled with the system bus 504 for processing the information. The processors 506 may includeone or more central processing units (CPUs), graphical processing units (GPUs), Al accelerators, or any other processor known in the ait.

[0051] The computing system 502 also includes a system memory 508 coupled to the system bus 504 for storing information and instructions to be executed by processors 506. The system memory 508 may include computer readable storage media in the form of volatile and / or nonvolatile memory, such as read only memory (ROM) 510 and / or random access memory (RAM) 512. The system memory RAM 512 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The system memory ROM 510 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 508 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 506. A basic input / output system 514 (BIOS) containing the basic routines that help to transfer information between elements within computing system 502, such as during start-up, may be stored in system memory ROM 510. System memory RAM 512 may contain data and / or program modules that are immediately accessible to and / or presently being operated on by the processors 506. System memory 508 may additionally include, for example, operating system 516, application programs 518, other program modules 520 and program data 522.

[0052] The computing system 502 also includes a disk controller 524 coupled to the system bus 504 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 526 and a removable media drive 528 (e.g., floppy disk drive, compact disc drive, tape drive, and / or solid state drive). The storage devices may be added to the computing system 502 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire).

[0053] The computing system 502 may also include a display controller 530 coupled to the system bus 504 to control a display 532, such as a cathode ray tube (CRT) or liquid crystal display (LCD), among other, for displaying information to a computer user. The computing system 502 includes a user input interface 534 and one or more input devices, such as a keyboard 536 and a pointing device 538, for interacting with a computer user and providing information to the one or more processors 506. The pointing device 538, for example, may be a mouse, a light pen, a trackball, or a pointing stick for communicating direction information and command selections to the one or moreprocessors 506 and for controlling cursor movement on the display 532. The display 532 may provide a touch screen interface which allows input to supplement or replace the communication of direction information and command selections by the pointing device 538.

[0054] The computing system 502 also includes an I / O adapter 546 coupled to the system bus 504 to connect the computing system 502 to a controllable physical device, such as a robot. In the example shown in FIG. 5, the VO adapter 546 is connected to robot controller 548. In one embodiment, the robot controller 548 includes, for example, one or more motors for controlling linear and / or angular positions of various pails (e.g., arm, base, etc.) of a robot.

[0055] The computing system 502 may perform a portion or all of the processing steps of embodiments of the disclosure in response to the one or more processors 506 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 508. Such instructions may be read into the system memory 508 from another computer readable storage medium, such as a magnetic hard disk 526 or a removable media drive 528. The magnetic hard disk 526 may contain one or more datastores and data files used by embodiments of the present disclosure. Datastore contents and data files may be encrypted to improve security. The processors 506 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 508. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

[0056] The computing system 502 may include at least one computer readable storage medium or memory for holding instructions programmed according to embodiments of the disclosure and for containing data structures, tables, records, or other data described herein. The term “computer readable storage medium” as used herein refers to any medium that participates in providing instructions to the one or more processors 506 for execution. A computer readable storage medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 526 or removable media drive 528. Non-limiting examples of volatile media include dynamic memory, such as system memory 508. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 504. Transmissionmedia may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0057] The computing environment 500 may further include the computing system 502 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 544. Remote computing device 544 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computing system 502. When used in a networking environment, computing system 502 may include a modem 542 for establishing communications over a network 540, such as the Internet. Modem 542 may be connected to system bus 504 via network interface 545, or via another appropriate mechanism.

[0058] Network 540 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computing system 502 and other computers (e.g., remote computing device 544). The network 540 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 540.

[0059] The embodiments of the present disclosure may be implemented with any combination of hardware and software. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, a non-transitory computer-readable storage medium. The computer readable storage medium has embodied therein, for instance, computer readable program instructions for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0060] The computer readable storage medium can include a tangible device that can retain andstore instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network.

[0061] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the disclosure to accomplish the same objectives. Although this disclosure has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the disclosure.

Claims

CLAIMSWhat is claimed is:

1. A computer- implemented method performed by an autonomous system for handling an object, comprising: acquiring sensor data to produce a point cloud of a scene in a physical environment including one or more objects at a first location, segmenting an object instance in the point cloud utilizing an instance segmentation mask, the object instance representing a first object, from the one or more objects in the scene, to be transported by the autonomous system to a second location, determining a direction of a main plane representing a placement surface of the first object using points in the segmented point cloud, projecting a set of points in the segmented point cloud, which define an extent of the main plane, with respect to a normal of the main plane, to create a 2D image, and processing the 2D image to generate a contour defining an outline of the placement surface and therefrom estimating planar dimensions of the placement surface, wherein the estimated planar dimensions of the placement surface are used to determine a placement operation executable by the autonomous system for placing the first object at the second location.

2. The method according to claim 1, wherein the acquired sensor data comprises a depth image of the scene, and wherein the point cloud is produced by projecting points in the depth image on to a 3D space using camera intrinsic parameters.

3. The method according to claim 2, wherein the acquired sensor data further comprises a color intensity image of the scene, and wherein the segmented point cloud is produced by: using the color intensity image as input to an instance segmentation algorithm to compute the instance segmentation mask as a 2D pixel mask, and using the depth image and the camera intrinsic parameters to convert the 2D pixel mask into a 3D point cloud mask.

4. The method according to any of claims 1 to 3, wherein the autonomous system comprises a robot with an end effector configured for a pick and place operation, and wherein the main plane is determined by: determining, in the segmented point cloud, a pick point of the object instance, and computing a plane equation by selecting neighboring points around the pick point in the segmented point cloud.

5. The method according to claim 4, wherein the pick point of the object instance is determined from the acquired sensor data using a grasp detection algorithm and is indicative of an optimal grasp location in the segmented point cloud.

6. The method according to any of claims 4 and 5, wherein the end effector comprises a suction cup.

7. The method according to any of claims 4 to 6, wherein the number or reach of the neighboring points are determined based on a size of the end effector.

8. The method according to any of claims 1 to 3, wherein the main plane is determined by performing an iterative search to determine all significant planes in the segmented point cloud and selecting the main plane from the significant planes based on a heuristic.

9. The method according to claim 8, wherein the heuristic comprises determining the main plane as the plane with the largest area in the segmented point cloud.

10. The method according to any of claims 1 to 9, wherein the set of points that are projected to create the 2D image are obtained by removing points in the segmented point cloud that do not belong to the main plane.

11. The method according to any of claims 1 to 10, wherein creating the 2D image comprises: projecting the set of points in the segmented point cloud into a 2D representation using camera intrinsic parameters, and rotating the points in the 2D representation with respect to the normal of the main plane, such that the resultant 2D image has a viewing direction perpendicular to the main plane.

12. The method according to any of claims 1 to 11, wherein processing the 2D image comprises generating the contour from the projected points by infilling, or inpainting, or opening operation, or combinations thereof.

13. The method according to any of claims 1 to 12, wherein processing the 2D image fitting a primitive shape to the contour to therefrom compute the planar dimensions.

14. A non-transitory computer- readable storage medium including instructions that, when processed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 13.

14. An autonomous system configured for handling an object, comprising: a sensor configured to generate sensor data for producing a point cloud of a scene in a physical environment including one or more objects at a first location, a controllable physical device, one or more processors, and memory storing instructions executable by the one or more processors to: segment an object instance in the point cloud utilizing an instance segmentation mask, the object instance representing a first object, from the one or more objects in the scene, to be transported by the autonomous system to a second location, determine a direction of a main plane representing a placement surface of the first object using points in the segmented point cloud, project a set of points in the segmented point cloud, which define an extent of the main plane, with respect to a normal of the main plane, to create a 2D image,process the 2D image to generate a contour defining an outline of the placement surface and therefrom estimating planar dimensions of the placement surface, determine a placement operation for placing the first object at the second location by the controllable physical device based on the estimated planar dimensions of the placement surface.

Images