System and method for estimating object planar dimensions for autonomous manipulation of objects

Abstract

A method for autonomously manipulating objects involves acquiring sensor data to produce a point cloud of a scene including one or more objects located at a first position. Object instances are segmented in the point cloud with instance segmentation masks. The object instances represent individual objects to be transported by a controllable physical device. Using points in the segmented point cloud, a direction of a principal plane is determined, the principal plane representing a placement surface of the objects. A set of points in the segmented point cloud defining a range of the principal plane are projected with respect to a normal of the principal plane to create a 2D image. The 2D image is processed to generate a contour defining an outline of the placement surface, from which a planar dimension of the placement surface is estimated to inform placement of the objects by the physical device.

Description

Technical Field

[0001] This disclosure generally relates to the field of autonomy in the execution of industrial automation tasks. Specifically, the described embodiments relate to systems, methods, and computer program products for estimating the planar dimensions of objects to perform robot pick-and-place operations or other autonomous object manipulation tasks. Background Technology

[0002] The Fourth Industrial Revolution aims to drive mass customization at the cost of mass production. This can be achieved through autonomous machines that no longer require programming with detailed instructions (such as waypoints or manually taught paths), but instead automatically define their tasks using design information about the product to be produced. Robotic box picking is one such skill that enables autonomous machines. Traditional robotic picking relies on a teach-based approach, allowing the operator to predefine the robot's pose for pick-up and drop-off locations. Over the past decade, advances in computer vision and deep learning have enabled flexible robotic box picking where pre-teaching of pick-up and drop-off locations is no longer necessary. Camera systems, such as RGB-D cameras, collect both color images and depth maps or point clouds of boxes with objects in random configurations. The camera input is then fed to a computer vision algorithm or deep neural network trained to compute the gripping point, or "pick-up point," on that input. These methods have proven to work reliably even without any prior information about the object's geometry and color (no CAD data is required).

[0003] In addition to calculating pick-up points, autonomous systems are expected to infer other high-level information about the objects they manipulate. For example, an autonomous system may need to estimate the space occupied by the object, particularly the planar dimensions of the surface on which it will eventually be placed or deployed, which can enable downstream processes to be executed in an informed manner.

[0004] If the 3D model of the object is known beforehand, the problem is somewhat simple. Object pose estimation can be performed using traditional computer vision algorithms or specially trained neural networks. Knowing the object's pose, the dimensions of the picking surface can be easily determined using the object's 3D model. If the object's 3D model is unknown, and an object segmentation mask is estimated using an object-independent instance segmentation model, the object's size can be roughly estimated based on the overall segmentation mask. However, this may include surfaces other than the picking surface, potentially leading to incorrect (oversized) object dimensions. Another approach is to compute the 3D bounding box of the unknown object using a deep neural network. This method is data-intensive and computationally intensive, and therefore its use may not be suitable for high-throughput, resource-constrained applications. An improved system is desired. Summary of the Invention

[0005] Embodiments of this disclosure address and overcome one or more of the disadvantages described herein by providing methods, systems, and apparatus for efficiently estimating the planar dimensions of an object for autonomous object manipulation using instance segmentation masks combined with 3D point cloud manipulation and 2D image processing.

[0006] A first aspect of this disclosure provides a computer-implemented method executed by an autonomous system for manipulating an object. The method includes: acquiring sensor data to generate a point cloud of a scene in a physical environment, the scene including one or more objects located at a first location. The method further includes segmenting object instances in the point cloud using an instance segmentation mask. The object instances represent a first object among the one or more objects in the scene to be transported by the autonomous system to a second location. The method further includes: using points in the segmented point cloud to determine the orientation of a principal plane representing a placement surface of the first object. The method further includes: projecting a set of points in the segmented point cloud defining the extent of the principal plane about a normal to the principal plane to create a 2D image. The method further includes: processing the 2D image to generate a contour defining the shape of the placement surface, and thereby estimating the planar dimensions of the placement surface. The estimated planar dimensions of the placement surface are used to determine a placement operation that can be performed by the autonomous system to place the first object at the second location.

[0007] Other aspects of this disclosure provide autonomous systems and computer program products embodying the above-described methods.

[0008] Additional technical features and benefits can be achieved through the techniques disclosed herein. Embodiments and aspects of this disclosure are described in detail herein and are considered part of the claimed subject matter. For a better understanding, reference is made to the specific embodiments and accompanying drawings. Attached Figure Description

[0009] The foregoing and other aspects of this disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings. For ease of identification of any element or action being discussed, the highest digit or multiple highest digits in the reference numerals refer to the drawing number in which the element or action is first introduced.

[0010] Figure 1 An exemplary autonomous system capable of performing box-picking applications is shown, wherein various aspects of this disclosure may be incorporated.

[0011] Figure 2 This illustrates the structure of a box and object that displays the pickup point in a robotic box-picking application.

[0012] Figure 3 This is a flowchart illustrating a computer-implemented method for estimating the planar dimensions of an object for placement operations performed by an autonomous system, according to a disclosed embodiment.

[0013] Figure 4 This illustrates the generation of a 2D representation of a contour from a set of points in a 3D point cloud.

[0014] Figure 5 A computing environment in which embodiments of the present disclosure can be implemented is shown. Detailed Implementation

[0015] This disclosure provides a method for efficiently and accurately estimating the planar dimensions of an object, enabling autonomous systems to perform end-to-end object manipulation tasks. To estimate the planar dimensions of an object, the disclosed method uses instance segmentation masks combined with 3D point cloud manipulation to determine a principal plane defining the placement surface of the object. The principal plane can be defined as the plane such that the space occupied by the object is its projection onto the principal plane. The method then reprojects a set of points in the segmented point cloud with respect to the normals of the principal plane to create a 2D image and processes the 2D image (e.g., using basic 2D computer vision operations) to generate a contour from which the planar dimensions of the object are estimated. Based on the estimated planar dimensions, a placement plan can be calculated. The calculated placement plan can be output to a controller configured to control physical devices to perform object placement at the placement location.

[0016] Unlike known methods, the disclosed method does not rely on deep learning approaches that require large amounts of training data and / or prior knowledge of the 3D model of the object. Furthermore, by reprojecting the 3D segmented point cloud onto a 2D image, enabling processing via basic 2D computer vision operations, computational costs are significantly reduced, making the solution suitable for high-throughput real-time applications.

[0017] The industrial applications of the disclosed method involve robotic pick-and-place operations. This task involves operating a robot to autonomously pick up objects from a first location (e.g., a box containing multiple objects) and place them at a second location. The estimated planar dimensions of the objects can be used to perform the placement operation in an informed manner, which may include, for example, placing the objects into another box, arranging the objects in a stack or grid, or dropping the objects into a chute. However, the disclosed method is not limited to the applications described above. For example, the disclosed method can be used to estimate the planar dimensions of objects to inform a sensing conveyor system, etc.

[0018] Aspects of the disclosed methods can be embodied as software executable by a processor. In some embodiments, aspects of the disclosed methods can be suitably integrated into automated software products based on commercial artificial intelligence (AI), such as SIMATIC Robot Pick AI™ developed by Siemens.

[0019] Now for reference Figure 1An exemplary autonomous system 100 is illustrated, in which various aspects of this disclosure may be embodied. The autonomous system 100 can be implemented, for example, in a factory setting. Compared to conventional automation, autonomy empowers each asset on the factory floor with decision-making and self-control capabilities to act independently in the event of localized problems. The autonomous system 100 includes one or more controllable physical devices, such as robot 102. One or more devices (such as robot 102) can be controlled by a computing system 104 to perform one or more industrial tasks within a physical environment 106. Examples of industrial tasks include assembly, transportation, etc.

[0020] The computing system 104 may include an industrial PC or any other computing device, such as a desktop or laptop computer or an embedded system. The computing system 104 may include one or more processors configured to process information and / or control various operations associated with the robot 102. In particular, the one or more processors may be configured to execute applications for operating the robot 102, such as engineering tools.

[0021] To achieve autonomy for system 100, in one implementation, the application may be designed to operate robot 102 to perform tasks within a skill-based programming environment. Unlike conventional automation, where engineers typically participate in programming the entire task from start to finish (usually using low-level code to generate individual commands), in autonomous systems as described herein, physical devices such as robot 102 are programmed at a higher level of abstraction using skills rather than individual commands. These skills are derived to achieve higher-level abstract behaviors centered on how the physical device modifies the physical environment. Illustrative examples of skills include skills for grasping or picking up objects, placing objects, opening doors, detecting objects, and so on.

[0022] The application can generate controller code that defines tasks at a high level, for example, using skill functions as described above. This controller code can be deployed to robot controller 108. From the high-level controller code, robot controller 108 can generate low-level control signals for one or more motors to control the movement of robot 102, such as the angular position of the robot arm, the rotation angle of the robot base, etc., thereby performing the specified task. In other embodiments, the controller code generated by the application can be deployed to an intermediate control device, such as a programmable logic controller (PLC), which can then generate low-level control commands for the robot 102 to be controlled. Additionally, the application can be configured to directly integrate sensor data from the physical environment 106 in which robot 102 operates. For this purpose, computing system 104 may include a network interface for facilitating the transmission of real-time data between the application and the physical environment 106. Examples of computing systems suitable for this application are incorporated herein by reference. Figure 5 Describe it.

[0023] Still referencing Figure 1 The robot 102 may include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 may include wheels 114 or may otherwise be configured to move within the physical environment 106. The robot 102 may also include an end effector 116 attached to the robotic manipulator 110. The end effector 116 may include one or more tools configured to grasp an object 118 and move it from a first position to a second position. For example, the end effector 116 may include a vacuum-based gripper (suction cup) or similar type. In the illustrated scenario, the object 118 to be transported is placed in a receiver or container 120 at the first position. The robotic manipulator 110 may be configured to move to change the position of the end effector 116, for example, to place or move the object 118 within the physical environment 106.

[0024] The bin-picking application involves using an end effector 116 via a robotic manipulator 110 to grasp an object 118 from a bin 120 in a unidirectional manner. The object 118 can be arranged in any orientation within the bin 120. The object 118 can be of various types or the same type. The physical environment 106, including the object 118 placed in the bin 120, is perceived via sensor data generated by one or more sensors 122. The sensors 122 can include one or more single-modal or multi-modal sensors, such as RGB sensors, depth sensors, infrared cameras, 3D scanners, etc. In the illustrated example, the one or more sensors include at least one camera 122 mounted to the robotic manipulator 110.

[0025] Sensor data can be fed into a grasping detection algorithm deployed on computing system 104 to determine the optimal grasp to be performed by robot 102 based on a specified target and imposed constraints (e.g., the size and position of a bin). The grasping detection algorithm may include a grasping neural network for computing the robot's grasp of an object given an input depth image (3D map) of a scene. The grasping neural network is typically convolutional, allowing the network to label each location (e.g., pixel) of the input image using some type of grasp availability metric (called a grasp score). The output of the grasping detection algorithm may include the optimal grasp location, referred to as the "pickup point." However, the grasping detection algorithm itself may not have any concept of the presence of an object of interest in the scene from the sensor data. Therefore, it may typically be necessary to deploy an additional object detection algorithm on computing system 104 to perform semantic recognition on objects 118 in the scene, thereby enabling the selection of objects 118 to be transported by robot 102. Object detection algorithms can include segmentation neural networks (e.g., Mask R-CNN), which can provide pixel-by-pixel object recognition output given an input two-dimensional color intensity image (such as an RGB color image).

[0026] Therefore, bin picking applications may involve a combination of object detection algorithms and grasping detection algorithms, whereby the object detection algorithm is used to locate the object of interest in the object 118 in the bin 120 given a color intensity image of the scene, and the grasping detection algorithm is used to calculate the optimal grasping position given a depth image of the scene. Figure 2 The diagram illustrates a bin and object configuration in a robotic bin picking application. As shown, objects 202 in bin 200 can be detected based on an object detection algorithm, and the corresponding pick-up points 204 for each object 202 can be calculated using a grasping detection algorithm. In some implementations, the object detection algorithm and the grasping detection algorithm can utilize off-the-shelf neural networks that have been extensively validated and tested in similar applications.

[0027] The methods described above enable robots to pick up objects from a scene without any pre-programming or prior knowledge of the objects' appearance. However, to perform pick-up and placement operations efficiently, in addition to calculating the pick-up point, the autonomous system is expected to estimate the space occupied by the object to be transported, especially the planar dimensions of the surface on which it will ultimately be placed or deployed. This allows downstream processes to be executed in an informed manner. The disclosed method addresses this objective.

[0028] Figure 3An example of a computer-implemented method 300 for estimating the planar dimensions of an object for placement operations performed by an autonomous system is shown. Method 300 can be implemented in various ways within a computing environment, e.g., as hardware and programming. The programming of blocks 302-310 of method 300 can take the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware can include a processor for executing these instructions. For example, the program can run on an industrial PC or a smaller device (e.g., a controller) of an autonomous system. Furthermore, processing power can be distributed among multiple system components, such as distributed among multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements.

[0029] refer to Figure 3 Box 302 relates to acquiring sensor data to generate a point cloud of a scene in a physical environment, the scene including one or more objects located at a first location. Figure 1 In the example shown, the scene includes a box 120 containing an object 118 positioned at a first location. A point cloud is a set of points in a 3D coordinate system representing a 3D surface, where the position of each point is defined by its Cartesian coordinates. Therefore, the point cloud of a scene is a 3D representation of all the surfaces on which the object to be transported initially resides in the scene.

[0030] In one implementation, sensor data may include a depth image of the scene, transmitted via a camera (such as...). Figure 1 The depth image is acquired by camera 122. A depth image is an image or image channel containing information related to the distances of the surfaces of objects in the scene from the viewpoint. The acquired depth image can be converted into a point cloud using camera intrinsics. Camera intrinsics are parameters that enable mapping between pixel coordinates in a 2D image coordinate system and 3D coordinates in the real world. Typically, camera intrinsics include the coordinates of the principal point or optical center, and the focal length along orthogonal axes. Using the camera intrinsics, a transformation matrix can be determined, which can be used to project points in the depth image onto 3D space to produce a point cloud representation of the scene. In an alternative implementation, the point cloud representation of the scene can be generated directly from sensor data generated by a 3D scanner.

[0031] In box 304, an object instance in the point cloud is segmented using an instance segmentation mask. This object instance represents one of the objects in the scene, selected by the autonomous system for transport to a second location. Applying the instance segmentation mask to the point cloud of the scene produces a segmented point cloud that represents only the 3D surface of the selected object.

[0032] According to the disclosed embodiments, the acquired sensor data may include, in addition to the depth image, a two-dimensional color intensity image of the scene. An example of a color intensity image is an RGB color image, which is an image including pixel intensity information in the red, green, and blue channels. The color intensity image and the depth image can be pixel-aligned. For example, in some embodiments, a single RGB-D sensor (such as...) Figure 1 The camera (122) can be configured to acquire an image of the scene with RGB color channels and depth channels. The color intensity image can be provided as input to the instance segmentation algorithm. Instance segmentation essentially includes semantic segmentation and object detection, adding features to identify the boundaries of objects at a detailed pixel level. Given the input color intensity image, the instance segmentation algorithm can be used to compute an instance segmentation mask as a 2D pixel mask for each object instance detected in the image, where each instance segmentation mask includes a set of pixels representing a specific object. The computed instance segmentation mask can be used to segment the corresponding depth image. Examples of AI-based instance segmentation methods that can be applied or are suitable for this purpose include instance segmentation using models such as the SegmentAnything Model (SAM) developed by Meta AI, the You Look Only Once (YOLO) model, Mask R-CNN, etc.

[0033] The instance segmentation mask of the selected object can be converted into a 3D point cloud mask by projecting the positions (represented in pixel coordinates) in the instance segmentation mask onto 3D space using depth information from the segmented depth image and camera intrinsics as described above. The 3D point cloud mask can be used to segment the point cloud of a scene to include only the object instances corresponding to the selected object.

[0034] Box 306 relates to using points in the segmented point cloud to determine the orientation of a principal plane representing the placement surface of the selected object. The goal of the disclosed method is to estimate the size of the object's footprint for placement operations. Therefore, a principal plane can be defined such that the object's footprint is the projection of the object onto the principal plane. Intuitively, the principal plane is the plane that best represents the object's orientation. In many cases, the principal plane can be the object's largest visible upward-facing plane. According to the disclosed implementation, the principal plane can be determined using the pick point of the object instance, or it can be determined using a heuristic method.

[0035] According to the first approach, for robotic pick-and-place operations, the master plane is determined by identifying the pick-up points of object instances in the segmented point cloud. This approach is based on the intuition that the plane best representing the object's orientation typically coincides with the plane containing the pick-up points (the "pick-up plane"). Furthermore, it is recognized that the pick-up plane will generally correspond to the delivery / placement plane (i.e., the plane / orientation on which the object is subsequently delivered for use in other downstream processes), since objects are generally not reoriented by the robot manipulator. Pick-up point calculation is typically already part of the workflow in robotic pick-and-place applications and can be appropriately used as input according to this approach.

[0036] Consistent with the disclosed implementation, the pick point of an object instance can be determined from the acquired sensor data using the grasp detection algorithm described above. The pick point can be calculated by the grasp detection algorithm using an instance segmentation mask in a 2D image coordinate system and projected onto the 3D space of the segmented point cloud based on depth information (from a depth image) and camera intrinsic parameters. Therefore, the calculated pick point can indicate the optimal grasping position in the segmented point cloud.

[0037] In other implementations, pick points can be modeled using keypoints in an instance segmentation mask. Keypoint detection can be performed from color intensity images, for example using neural networks, which can be embedded in the instance segmentation model or stand-alone models. Alternatively, non-deep learning methods can be used to model pick points. As an example, pick points can be modeled using the centroid of the instance segmentation mask. Keypoints / centroids computed on the instance segmentation mask can be projected into 3D space using depth information (from a depth image) and camera intrinsics to locate pick points in the segmented point cloud.

[0038] After the pick-up point is determined, a set of neighboring points can be selected around the pick-up point in the segmented point cloud to compute the plane equations. The number or range of neighboring points can be determined depending on the application. For example, in robot pick-and-place operations, the number or range of neighboring points can be determined based on the dimensions of the robot's end effector. To illustrate, in Figure 1 In the example, the set of neighboring points can be selected such that the maximum distance from the pick point does not exceed the radius of suction cup 116. Given this set of neighboring points, the plane equation can be determined, for example, using the least squares method, which best fits these points. The plane equation can define the orientation of the principal plane.

[0039] According to the second method, if pick-up points are not used or cannot be obtained (e.g., in non-robotic applications), the principal plane can be determined based on a heuristic method. In this implementation, in the first step, an iterative search can be performed to determine all salient planes in the segmented point cloud, i.e., all salient planes defining the 3D surface of the segmented point cloud. The principal plane can then be selected from these salient planes using a heuristic method.

[0040] To perform an iterative search to determine salient planes, methods such as Random Sampling Consensus (RANSAC) can be used. For example, starting with a small set of random points, the plane equation can be determined using all these random points with the aid of least squares or other linear regression methods. Then, it can be checked whether any other points in the segmented point cloud also belong to the plane, for example, by performing a distance check or other means. The plane equation can be recalculated using all the initial random points and new points, and then all these points (those within their defined "inside") are removed from the segmented point cloud. This process can be repeated until too few points remain, or a predetermined number of planes have been computed. Finally, a heuristic can be used to select the principal plane, which can be tailored to the use case at hand. For example, in most use cases, it is desirable to place the surface as the largest surface of the object. In this case, the heuristic could include selecting the principal plane as the plane with the largest area. In some use cases, a vertical plane may be preferred over an inclined plane. In this case, the heuristic could include selecting the principal plane as the plane with the straightest normal (e.g., relative to a reference plane).

[0041] After determining the principal plane, the planar dimensions of the placement surface can be estimated by estimating the area covered by all points in the principal plane of the segmented point cloud. To reduce the computational costs associated with processing 3D point clouds, the disclosed method involves projecting the 3D point cloud onto a 2D image and estimating the planar dimensions of the object by generating a contour in the 2D image, for example, using 2D computer vision operations. This makes the solution lightweight and suitable for high-throughput real-time applications while significantly reducing computational resources.

[0042] Still referencing Figure 3 In box 308, a set of points in the segmented point cloud is projected onto the normals of the principal plane to create a 2D image. To determine the set of points to be projected, the extent of the principal plane is first determined. This step can involve finding all points belonging to the principal plane given its equation, for example, by using a clustering method based on a combination of distances to the plane, normal classification, and other geometric properties. The set of points to be projected can then be obtained by removing all points in the segmented point cloud that do not belong to the principal plane.

[0043] In some implementations, to determine a set of points to be projected, outliers that contribute to noisy measurements about the principal plane can be removed, for example, using a statistical outlier filter. This step can be performed depending on the object's geometry. For example, if the object is deformable (e.g., a bag of chips, clothing wrapped in a plastic bag, etc.), this step may be less critical because significant noise already exists in the point cloud due to the geometry of deformable objects. However, in the case of rigid objects, this step can be appropriately implemented to produce a clean surface.

[0044] A set of points identified in the 3D segmented point cloud can first be projected onto a 2D representation (depth image), a transformation calculated using camera intrinsics. Then, the 3D points in the 2D representation can be rotated about the normal to the principal plane. As a result of this rotation, a 2D image with a viewing direction perpendicular to the principal plane can be produced; that is, the principal plane is aligned with the camera coordinate system of the 2D image. In this way, camera perspective distortion can be eliminated.

[0045] The above steps can be referred to Figure 4 Here, image 402 represents a 2D projection of a point cloud using camera intrinsics. Image 402 depicts a scene including a box containing a container located on its right side wall. Image 402 is essentially a depth image, which is a 2D representation of 3D points. That is, each point in image 402 is further characterized by depth information in addition to its x and y coordinates. Reference numeral 404 refers to the set of all points in the principal plane 406 of the object instance (the box). Image 408 represents a 2D image generated by rotating point 404 about the normal of principal plane 406. Image 408 has a viewing direction perpendicular to principal plane 406. That is, principal plane 406 has been rotated so that it is now aligned with the camera coordinate system of image 408. As shown, principal plane 406, which appears trapezoidal in image 402 due to perspective distortion, appears roughly rectangular in image 408 after point 404 is reprojected in the direction of the principal plane normal.

[0046] Continue to refer to Figure 3 In box 310, the 2D image is processed to generate a contour defining the shape of the placement surface. The planar dimensions of the placement surface can be estimated by measuring the area enclosed by such a contour. The processing of the 2D image can involve any operation to obtain an enhanced image or otherwise extract useful information to generate the contour. For example, basic 2D computer vision operations, such as fill, repair, and open operations, can be performed to recover missing points or gaps. In some implementations, missing points or gaps can be recovered by directly fitting the original shape to the contour. The fitted original shape can be determined such that it covers the minimum area enclosing all points in the contour. The planar dimensions of the placement surface can then be estimated by measuring the area enclosed by the original shape.

[0047] Refer again Figure 4 In the example shown, the contour 410 generated on the 2D image 408 is missing a corner. This could be due to depth imaging failure and / or insufficient points in the point cloud. In this example, the missing corner is recovered by fitting a minimum area rectangle 412 containing all points of contour 410. The dimensions of the principal plane can be obtained by simply measuring the length and width of rectangle 412.

[0048] The estimated planar dimensions of the object placement surface can be used to calculate a placement plan. The placement plan can include the position (coordinates) and / or orientation (rotation) of the transported object at the placement location. The placement plan can be output to the controller of a physical device (e.g., a robot) for performing object placement at a second location. For example, in robot pick-and-place applications, placement operations can include placing an object in another bin, placing an object in a stack or grid, or dropping an object into a chute.

[0049] Figure 5 An exemplary computing environment, including computing system 502, in which various aspects of this disclosure can be implemented is shown. Computing system 502 can be implemented, for example, but not limited to, as an industrial PC equipped with a Linux operating system, for performing real-time control of physical equipment such as robots.

[0050] like Figure 5 As shown, the computing system 502 may include communication mechanisms, such as a system bus 504 or other communication mechanisms for transmitting information within the computing system 502. The computing system 502 also includes one or more processors 506 coupled to the system bus 504 for processing information. The processor 506 may include one or more central processing units (CPUs), graphics processing units (GPUs), AI accelerators, or any other processor known in the art.

[0051] The computing system 502 also includes a system memory 508, which is coupled to the system bus 504 for storing information and instructions to be executed by the processor 506. The system memory 508 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 510 and / or random access memory (RAM) 512. The system memory RAM 512 may include other dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). The system memory ROM 510 may include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). Additionally, the system memory 508 may be used to store temporary variables or other intermediate information during instruction execution by the processor 506. A basic input / output system 514 (BIOS) containing basic routines may be stored in the system memory ROM 510, which facilitates the transfer of information between elements within the computing system 502, such as during startup. The system memory RAM 512 may contain data and / or program modules that are immediately accessible and / or currently operated by the processor 506. System memory 508 may additionally include, for example, an operating system 516, application programs 518, other program modules 520, and program data 522.

[0052] The computing system 502 also includes a disk controller 524 coupled to the system bus 504 to control one or more storage devices for storing information and instructions, such as magnetic hard disks 526 and removable media drives 528 (e.g., floppy disk drives, optical disk drives, tape drives, and / or solid-state drives). Storage devices can be added to the computing system 502 using appropriate device interfaces such as Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or FireWire.

[0053] The computing system 502 may also include a display controller 530 coupled to a system bus 504 to control a display 532, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computing system 502 includes a user input interface 534 and one or more input devices (such as a keyboard 536 and a clicking device 538) for interacting with the computer user and providing information to one or more processors 506. For example, the clicking device 538 may be a mouse, light pen, trackball, or pointing stick for transmitting directional information and command selections to one or more processors 506 and for controlling cursor movement on the display 532. The display 532 may provide a touchscreen interface that allows input to supplement or replace the transmission of directional information and command selections by the clicking device 538.

[0054] The computing system 502 also includes an I / O adapter 546, which is coupled to the system bus 504 to connect the computing system 502 to a controllable physical device, such as a robot. Figure 5 In the example shown, I / O adapter 546 is connected to robot controller 548. In one embodiment, robot controller 548 includes one or more motors, for example, for controlling the linear and / or angular positions of various parts of the robot, such as the arm, base, etc.

[0055] The computing system 502 can perform some or all of the processing steps of embodiments of the present disclosure in response to one or more processors 506 executing one or more sequences of one or more instructions contained in memory (such as system memory 508). Such instructions can be read into system memory 508 from another computer-readable storage medium (such as magnetic hard disk 526 or removable media drive 528). Magnetic hard disk 526 may contain one or more data repositories and data files used by embodiments of the present disclosure. The contents of the data repositories and data files can be encrypted to improve security. Processor 506 can also be employed in a multiprocessing device to execute one or more sequences of instructions contained in system memory 508. In alternative embodiments, hard-wired circuitry can be used instead of or in combination with software instructions. Therefore, the embodiments are not limited to any particular combination of hardware circuitry and software.

[0056] The computing system 502 may include at least one computer-readable storage medium or memory for storing instructions programmed according to embodiments of the present disclosure and for containing data structures, tables, records, or other data described herein. As used herein, the term "computer-readable storage medium" means any medium involved in providing instructions to one or more processors 506 for execution. Computer-readable storage media may take many forms, including but not limited to non-transitory media, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical discs, solid-state drives, magnetic disks, and magneto-optical discs, such as magnetic hard disk 526 or removable media drive 528. Non-limiting examples of volatile media include dynamic memory, such as system memory 508. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including wires constituting system bus 504. Transmission media may also take the form of acoustic waves or light waves, such as acoustic waves or light waves generated during radio wave and infrared data communications.

[0057] The computing environment 500 may also include a computing system 502 that operates in a networked environment using logical connections to one or more remote computers, such as a remote computing device 544. The remote computing device 544 may be a personal computer (laptop or desktop), mobile device, server, router, network PC, peer-to-peer device, or other common network node, and typically includes many or all of the elements described above with respect to the computing system 502. When used in a networked environment, the computing system 502 may include a modem 542 for establishing communication over a network 540, such as the Internet. The modem 542 may be connected to the system bus 504 via a network interface 545 or via another suitable mechanism.

[0058] Network 540 can be any network or system generally known in the art, including the Internet, intranet, local area network (LAN), wide area network (WAN), metropolitan area network (MAN), direct connection or a series of connections, cellular telephone network, or any other network or medium capable of facilitating communication between computing system 502 and other computers (e.g., remote computing device 544). Network 540 can be wired, wireless, or a combination thereof. Wired connections can be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections can be implemented using Wi-Fi, WiMAX and Bluetooth, infrared, cellular networks, satellite, or any other wireless connection methodology generally known in the art. Additionally, several networks can operate independently or communicate with each other to facilitate communication within network 540.

[0059] Embodiments of this disclosure can be implemented using any combination of hardware and software. Additionally, embodiments of this disclosure can be included in an article of manufacture having, for example, a non-transitory computer-readable storage medium (e.g., one or more computer program products). For example, a computer-readable storage medium may contain computer-readable program instructions implemented therein for providing and facilitating embodiments of this disclosure. This article of manufacture may be included as part of a computer system or sold separately.

[0060] Computer-readable storage media may include tangible devices that can retain and store instructions for use by an instruction execution device. Computer-readable storage media may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. The computer-readable program instructions described herein may be downloaded from the computer-readable storage medium to a corresponding computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device.

[0061] The systems and processes illustrated in the accompanying drawings are not exclusive. Other systems, processes, and menus can be derived from the principles of this disclosure to achieve the same objectives. Although this disclosure has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the present design can be made by those skilled in the art without departing from the scope of this disclosure.

Claims

1. A computer-implemented method executed by an autonomous system for manipulating an object, comprising: Acquire sensor data to generate a point cloud of a scene in a physical environment, the scene including one or more objects located at a first location; The point cloud is segmented using an instance segmentation mask, and each object instance represents the first object among one or more objects in the scene that needs to be transported to the second location by the autonomous system. The orientation of the principal plane representing the placement surface of the first object is determined using the points in the segmented point cloud; Project a set of points in the segmented point cloud that define the range of the principal plane onto the normal of the principal plane to create a 2D image; as well as The 2D image is processed to generate a contour that defines the shape of the placement surface, and thereby estimate the planar dimensions of the placement surface; The estimated planar dimensions of the placement surface are used to determine the placement operation that can be performed by the autonomous system to place the first object in the second position.

2. The method according to claim 1, wherein, The acquired sensor data includes a depth image of the scene, wherein the point cloud is generated by projecting points in the depth image into 3D space using camera intrinsics.

3. The method according to claim 2, wherein, The acquired sensor data also includes a color intensity image of the scene, wherein the segmented point cloud is generated in the following manner: Using the color intensity image as input to the instance segmentation algorithm, an instance segmentation mask is computed as a 2D pixel mask; and The 2D pixel mask is converted into a 3D point cloud mask using the depth image and the camera intrinsics.

4. The method according to any one of claims 1 to 3, wherein, The autonomous system includes a robot with an end effector configured for pick-up and place operations, wherein the main plane is defined by the following: Determine the pick point of the object instance in the segmented point cloud; and The plane equation is calculated by selecting neighboring points around the pick point in the segmented point cloud.

5. The method according to claim 4, wherein, The pick point of the object instance is determined from the acquired sensor data using a grasp detection algorithm and indicates the optimal grasp position in the segmented point cloud.

6. The method according to any one of claims 4 and 5, wherein, The end effector includes a suction cup.

7. The method according to any one of claims 4 to 6, wherein, The number or range of neighboring points is determined based on the size of the end effector.

8. The method according to any one of claims 1 to 3, wherein, The principal plane is determined by performing an iterative search to identify all salient planes in the segmented point cloud and selecting the principal plane from the salient planes based on a heuristic method.

9. The method according to claim 8, wherein, The heuristic method includes determining the principal plane as the plane with the largest area in the segmented point cloud.

10. The method according to any one of claims 1 to 9, wherein, The set of points projected to create the 2D image is obtained by removing points in the segmented point cloud that do not belong to the principal plane.

11. The method according to any one of claims 1 to 10, wherein, Creating the 2D image includes: The points in the segmented point cloud are projected into a 2D representation using camera intrinsic parameters; and Rotate the points in the 2D representation about the normal of the principal plane such that the resulting 2D image has a viewing direction perpendicular to the principal plane.

12. The method according to any one of claims 1 to 11, wherein, Processing the 2D image includes generating the contour from the projected points by a fill operation, or a repair operation, or an open operation, or a combination thereof.

13. The method according to any one of claims 1 to 12, wherein, Processing the 2D image includes fitting the original shape to the contour to calculate the planar dimensions.

14. A non-transitory computer-readable storage medium comprising instructions that, when processed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 13.

15. An autonomous system configured for manipulating an object, comprising: A sensor configured to generate sensor data for producing a point cloud of a scene in a physical environment, the scene including one or more objects located at a first location; Controllable physical devices; One or more processors; as well as The memory stores instructions executable by the one or more processors for: The point cloud is segmented using an instance segmentation mask, and each object instance represents the first object among one or more objects in the scene that needs to be transported to the second location by the autonomous system. The orientation of the principal plane representing the placement surface of the first object is determined using the points in the segmented point cloud; Project a set of points in the segmented point cloud that define the range of the principal plane onto the normal of the principal plane to create a 2D image; The 2D image is processed to generate a contour that defines the shape of the placement surface, and the planar dimensions of the placement surface are estimated therefrom. Based on the estimated planar dimensions of the placement surface, a placement operation is determined for placing the first object in the second position by the controllable physical device.

Images