Object identification for robotic systems

Abstract

Systems, apparatuses and methods pertaining to robotic equipment interfacing with objects in an operating environment are provided. In some embodiments, a robotic system comprises a robot including an object interfacing mechanism and a robot controller, a detection system to collect information from an operating environment, and a control circuit. The control circuit: obtains a 3-dimensional (3D) point cloud of the operating environment; generates a depth map image; segments the depth map image into image segments; selects a target segment; identifies a point cloud segment corresponding to the target segment; selects a planar surface in the point cloud segment; encompasses points of the planar surface; selects one of corners of the 3D bounding box as an alignment point; and provides a coordinate for the selected corner to the robot controller to align the object interfacing mechanism to a position of the coordinate.

Description

OBJECT IDENTIFICATION FOR ROBOTIC SYSTEMSRelated Application^)

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 627,208, filed January 31, 2024, which is incorporated by reference in its entirety herein.Technical Field

[0002] This invention relates generally to a robot system, and specifically to a robot system with automated object identification.Background

[0003] Robots can be used to unload freight from trailers. Robots can use vision / perception systems to guide the robot to remove the freight from the trailer. However, freight can shift during transit such that freight may not be at expected locations, and vision / perception systems can be slow and inaccurate.Brief Description of the Drawings

[0004] Disclosed herein are embodiments of systems, apparatuses and methods pertaining to robotic equipment interfacing with objects in an operating environment. This description includes drawings, wherein:

[0005] FIG. 1 is a block diagram of a robotic system in accordance with some embodiments.

[0006] FIG. 2 is a flowchart depicting a method for use with a robotic system in accordance with some embodiments.

[0007] FIG. 3A depicts an example operating environment of a robot in accordance with some embodiments.

[0008] FIG. 3B illustrates a 3D point cloud of the operating environment of FIG. 3A in accordance with some embodiments.

[0009] FIG. 3C illustrates a depth map image generated based on the 3D point cloud of FIG. 3B in accordance with some embodiments.

[0010] FIG. 3D illustrates a segmented depth map image including a plurality of image segments and a picking zone in accordance with some embodiments.

[0011] FIG. 3E illustrates a target segment selected among image segments of FIG. 3D in accordance with some embodiments.

[0012] FIG. 3F illustrates a 3D bounding box on the 3D point cloud of FIG. 3B in accordance with some embodiments.

[0013] FIG. 3G illustrates an alignment point and three axes of a coordinate frame in accordance with some embodiments.

[0014] FIG. 4 is a flowchart depicting a method to select the target segment among the image segments in accordance with some embodiments.

[0015] FIGS. 5A and 5B illustrate examples of encompassing and encompassed segments in accordance with some embodiments.

[0016] FIGS. 6A and 6B illustrate an example box in open and closed configurations in accordance with some embodiments.

[0017] FIGS. 7A and 7B illustrate flap segments encompassed within a box segment of FIG. 5A in accordance with some embodiments.

[0018] FIG. 8 is a flowchart of a method to select an alignment point in accordance with some embodiments.

[0019] Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions and / or relative positioning of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments. Also, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various embodiments. Certain actions and / or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. The terms and expressions used herein have the ordinary technical meaning as is accorded to such terms and expressions by persons skilled in the technical field as set forth above except where different specific meanings have otherwise been set forth herein.Detailed Description

[0020] Generally speaking, pursuant to various embodiments, systems, apparatuses, devices, and methods are provided herein useful to handle, pick, manipulate, load, and / or unloadobjects such as freight, cases, and boxes in a retail environment. In some embodiments, a robotic system may comprise a robot including an object interfacing mechanism and a robot controller, a detection system configured to collect information from an operating environment of the robot, and a control circuit communicatively coupled to the robot and the detection system, the control circuit configured to obtain, from the detection system, a 3 -dimensional (3D) point cloud of the operating environment of the robot, generate, based on the 3D point cloud, a depth map image, segment the depth map image into image segments, select a target segment among the image segments, identify a point cloud segment corresponding to the target segment, select a planar surface in the point cloud segment, encompass, with a 3D bounding box, points of the planar surface, select one of comers of the 3D bounding box as an alignment point, and provide a coordinate for the selected comer of the 3D bounding box to the robot controller to cause the robot controller to align the object interfacing mechanism to a position of the coordinate.

[0021] To automate a robot system, an object perception system may be used to identify objects and the positions of objects and control the robotic movement in an operating environment. The object perception system may use a pre-trained model. However, to generate the pre-trained model, an extensive library of training data (training images) and a model training process are needed, which may not be desirable or available.

[0022] Systems and methods in accordance with some embodiments described herein may cost-effectively identify / determine locations of the objects for picking to determine the movement / operation of a robot in an operating environment. Systems and methods according to the disclosure may also provide efficient vision perception for a robotic system without requiring extensive pre-training of a model by the user or a third-party. In some embodiments, systems and methods according to this disclosure may reduce operational risks in workspaces in which robots operate.

[0023] Various embodiments and examples of systems, devices, apparatus, and methods are described herein. FIGS. 1-8 are provided to illustrate various embodiments. It is noted that when describing certain embodiments, certain features may be shown in one or more of FIGS. 1- 8.

[0024] FIG. 1 is a block diagram of a robotic system 100 in accordance with some embodiments. The system 100 may include a control circuit 102 and a memory 104, a detectionsystem 112, and a robot 120. The control circuit 102 may be communicatively coupled to the robot 120 and the detection system 112.

[0025] The robot 120 may be capable of handling, manipulating, and transporting objects such as boxes, cases, cartons, or other items in a robot operating environment. The operating environment may be, but is not limited to, a commercial product facility such as a warehouse, a fulfdlment center, a distribution center, a shipping container, a trailer, a vehicle, a working station and so on. The robot 120 may include a robot controller 122 configured to control the operation and / or motion of the robot 120. Perception systems and methods described herein may be utilized by a variety of robotic devices. The robot 120 may include, but is not limited to, an articulating arm (a robotic arm), an industrial variant of an articulating arm, a bi-pedal robot, a humanoid robot, a wheeled robot, a tracked robot, an airborne autonomous mobile robot, a parallel link robot, or any combination thereof. In some embodiments, the robot 120 may include a multi-axis robotic arm. In some embodiments, the robotic arm may be a custom design robotic arm with two or more axes. In some embodiments, the robotic arm may have four or six axes. In some embodiments, the robotic arm includes multiple arm segments that are pivotally, rotatably, and / or statically attached.

[0026] In some embodiments, the robot 120 may include an object interfacing mechanism 124 that may interface with, manipulate, and handle objects and / or freight. The object interfacing mechanism 114 may include, but is not limited to, an end effector (e.g., an end of arm tool) that may be coupled to a robotic arm, a gripper, a pallet fork, other types of payload interface mechanism, and / or any combination thereof. Some embodiments incorporate the some or all of U.S. Patent Application No. 63 / 536,608 entitled DEVICES AND METHODS FOR OBJECT MANIPULATION, and / or some or all of U.S. Patent Application No. 63 / 536,609, entitled SYSTEMS AND METHODS FOR FRIEGHT MANIPULATION, which are each incorporated herein by reference in their entirety.

[0027] The detection system 112 may be configured to collect information from the robot operating environment. The detection system 112 may collect detection data 106 of the operating environment. In some embodiments, the detection data 106 may include image data of the field of view of the detection system 112. In some embodiments, the detection system 112 may include, but not limited to, a 2-dimentional (2D) camera, a 3D camera, an RGB-D sensor, an LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, a microphone, or any combination thereof.The detection system 112 may scan or capture data from the operating environment in real-time or near real-time during the operation of the robot 120. The detection system 112 may be mounted / coupled to the robot 120 or may be part of the robot 120. In some embodiments, the detection system 112 may be separate from the robot 120. In some embodiments, the control circuit 102 operably couples to the detection system 112 to receive information from the detection system 112. In some embodiments, the control circuit 102 may store the information (e.g., detection data 106) received from the detection system 112 in the memory 104.

[0028] In some embodiments, the memory 104 may include a volatile and / or non-volatile memory. In some embodiments, the memory 104 may include a random-access memory (RAM). The memory 104 may serve to store computer instructions that, when executed by the control circuit 102, cause the control circuit 102 to behave as described herein. In some embodiments, the memory 104 may serve, for example, to non-transitorily store computer instructions. As used herein, this reference to "non-transitorily" will be understood to refer to a non-ephemeral state for the stored contents (and hence excludes when the stored contents merely constitute signals or waves) rather than volatility of the storage media itself and hence the may include both non-volatile memory (such as read-only memory (ROM) as well as volatile memory (such as an erasable programmable read-only memory (EPROM).

[0029] The memory 104 may provide storage of the detection data 106 and one or more modules 108. The detection data 106 may be collected via the detection system 112 and may include, but not limited to, a 2D image, color, depth information, 3D point cloud. The detection data 106 may include any data used to conduct the steps, actions, and / or functions described herein. The memory 104 may also store data generated during the use of the system 110. The modules 108 may include codes executable by the control circuit 102 and, when executed by the control circuit 102, cause the control circuit 102 to perform steps, actions, and / or functions described herein. The modules 108 may include, but is not limited to, a point cloud accumulation module, a point cloud filtering module, and an image segmentation module. In some embodiments, the modules 108 may further include a distance calculation module, an inverse kinematics module, a motion planning module. In some embodiments, the distance calculation module, the inverse kinematics module, and the motion planning module may be stored in the robot controller 122 and / or the memory 104. In some embodiments, the modules 108 may further include a systemsafety module and a fleet management module. In some embodiments, one or more of the modules 108 may employ machine learning to improve capabilities for object identification, object manipulation, and ancillary tasks such as navigating and / or exceptional handling and so on. In some embodiments, the detection data 106 captured during the operating may be used to train the modules 108 that employ the machine learning.

[0030] In some embodiments, the control circuit 102 may operably couple to the memory 104. The control circuit 102 may access the memory 104 and execute the codes stored in the memory 104. In some embodiments, the memory 104 may be integral to the control circuit 102 or may be physically discrete in whole or in part from the control circuit 102 as desired. This memory 104 may also be local with respect to the control circuit 102 (where, for example, both share a common circuit board, chassis, power supply, and / or housing) or may be partially or wholly remote with respect to the control circuit 102 (where, for example, the memory 104 is physically located in another housing or remotely location). In some embodiments, the memory 104 may be distributed in multiple locations.

[0031] The control circuit 102 is configured, for example by using corresponding programming and / or using the modules 108 stored in memory 104, to carry out and / or send signals to carry out one or more of the steps, actions, and / or functions described herein. The control circuit 102 may comprise structure that includes at least one (and typically many) electrically-conductive paths (such as paths comprised of a conductive metal such as copper or silver) that convey electricity in an ordered manner, which path(s) will also typically include corresponding electrical components (both passive (such as resistors and capacitors) and active (such as any of a variety of semiconductor-based devices) as appropriate) to effect one or more of the steps, actions, and / or functions described herein. The control circuit 102, for example, may comprise a fixed-purpose hard-wired hardware platform (including but not limited to an application-specific integrated circuit (ASIC) (which is an integrated circuit that is customized by design for a particular use, rather than intended for general-purpose use), a field-programmable gate array (FPGA), and the like) or can comprise a partially or wholly-programmable hardware platform (including but not limited to microcontrollers, microprocessors, and the like).

[0032] FIG. 2 is a flowchart depicting a method 200 for use with a robotic system in accordance with some embodiments. The method 200 may be performed using the system 100 inaccordance with the approaches described above. Although the method 200 is mainly illustrated with the system 100, the method 200 may also be performed with a system differently configured.

[0033] FIG. 3A depicts an example operating environment 302 of the robot 120 controlled via the steps described with reference to FIG. 2 herein in accordance with some embodiments. In FIG. 3A, the operating environment 302 includes a trailer 304 and a plurality of objects 306 (e.g., boxes) stacked in the trailer 304. FIGS. 3B-3G depict example data obtained and generated during use with a robotic system 100 in accordance with some embodiments. FIG. 3B illustrates a 3D point cloud 308 of the operating environment 302. FIG. 3C illustrates the depth map image 310 generated based on the 3D point cloud 308. FIG. 3D illustrates a segmented depth map image 312 including a plurality of image segments 314 and a picking zone 315. FIG. 3E illustrates a target segment 316 selected among the image segments 314. FIG. 3F illustrates a 3D bounding box 318 on the 3D point cloud 308. FIG. 3G illustrates the alignment point 320 and three axes 322a, 322b, 322c of the coordinate frame.

[0034] In step 202, the control circuit 102 may obtain, from the detection system 112, detection data 106 from the operating environment of the robot 120. The detection data 106 may include 3D image information about the operating environment 302. The 3D image information may include depth information describing depth values for various positions in the field of view of the detection system 112. In some embodiments, the control circuit 102 may obtain, from the detection system 112 and with a point cloud accumulation module, a 3D point cloud 308 for the operating environment 302. In some embodiments, the 3D point cloud 308 may be collected via a LiDAR sensor. In some embodiments, the 3D point cloud 308 obtained in step 202 may be as viewed from the detection system 112. For example, when the 3D point cloud 308 is obtained via the LiDAR sensor, the 3D point cloud may be as viewed from the LiDAR sensor. In some embodiments, the detection data 106 may further include other types of data collected via the detection system 112. For example, the detection data 106 may include a 2D image of the operating environment 302 collected via a 2D camera. In some embodiments, the control circuit 102 may crop, with the point cloud filtering module, the 3D point cloud 308 to remove points for one or more non-pickable objects. For example, the control circuit 102 may crop the 3D point cloud 308 to remove points for a wall, a floor, and / or a ceiling when the wall, the floor, and / or the ceiling areincluded in the 3D point cloud 308. For example, the 3D point cloud 308 in FIG. 3B is cropped to remove the walls, the ceiling, and the floor of the trailer 304 of the operating environment 302.

[0035] In step 204, the control circuit 102 may generate a depth map image 310 based on the 3D point cloud 308 obtained in step 202. In some embodiments, the depth map image 310 may be generated by taking each specific point of the 3D point cloud 308 and assigning a value (e.g., darkness or color) for each point based on its distance from the reference point. In some embodiments, the depth map image 310 generated in step 204 is as viewed from the robot 120. In some embodiments, the depth map image 310 may be generated as viewed from the front of the robot 120. In some embodiment, the depth map image 310 may be aligned with a base frame of the robot 120.

[0036] In step 206, the control circuit 102 may segment, with the image segmentation module, the depth map image 310 into image segments 314. The image segmentation module may include a segmentation algorithm. In some embodiment, each segment 314 may include a matrix of true and false values for each pixel in the depth map image 310. In the matrix of each segment 314, the true value may indicate that the pixel of the true value is part of the segment 314 and the false value may indicate that the pixel of the false value is not part of the segment 314.

[0037] In step 208, the control circuit 102 selects a target segment 316 among the image segments 314. FIG. 4 illustrates a method to select the target segment 316 among the image segments 314 in step 208 in accordance with some embodiments.

[0038] Referring to FIG. 4, in step 402, the control circuit 102 may consider confidence scores for each of the image segments 314. The confidence scores may be determined by the image segmentation module (e.g., segmentation algorithm) when segmenting the depth map image 310 into the image segments 314. A higher confidence score may represent a higher possibility that the segment represents a surface of the object 306 (e.g., a box or case) in the operation environment 302. In some embodiments, the confidence scores may include prediction scores and / or stability scores. The prediction scores may be based on the segmentation algorithm’s own prediction of the quality of the segmentation masks. The stability scores may be based on a measurement of the quality of the segmentation masks.

[0039] In step 404, the control circuit 102 may sort the image segments 314 in size decreasing order. In step 406, the control circuit 102 may identify one or more segments partiallyor fully encompassed within and / or overlapped with another segment. In some embodiments, the control circuit 102 may identify a unique / real object based on the image segments 314 at least by identifying encompassed or overlapping segments. For example, when two segments occupy the same portion of the image (e.g., when two segments share true value(s) for the same pixel(s)), only one of them may correspond to the shape of a real object (e.g., case / box). In some embodiments, the control circuit 102 may use the result of step 404 in identifying one or more segments encompassed within and / or overlapped with another segment. For example, control circuit 102 may start identifying a segment encompassed within and / or overlapped with another segment based on the order of the largest segment to the smallest segment. For instance, the control circuit 102 may evaluate, from the largest segment to the smallest segment, each segment below (i.e., smaller segments than) the current turn segment to determine whether the current turn segment encompasses any one or more of the segments below the current turn segment. In some embodiments, when true values of the matrix of the smaller segment are shared with true values for the same location of the matrix of the larger segment, the control circuit 102 may determine that the smaller segment is encompassed and / or overlapped with the larger segment. The control circuit 102 may iterate down the list of segments sorted in size decreasing order and skip any image segments that have already been determined to be within a larger segment. In some embodiments, the control circuit 102 may identify a segment fully encompassed within another segment. For example, when a smaller segment shares all true values at the same matrix location with a larger segment, then the control circuit 102 may determine that the smaller segment is fully encompassed within the larger segment.

[0040] FIGS. 5A-5B illustrate examples of encompassing and encompassed segments. The segment 502 in FIG. 5A is a box segment representing a single box. The segment 504 in FIG. 5B is a group segment representing a group of three boxes including the box corresponding to the box segment 502. The segment 504 in FIG. 5B is larger than the segment 502 in FIG. 5A and the segment 502 shares all true value for the same positions of the matrix with the segment 504. In FIGS. 5A-5B, the smaller segment 504 is fully encompassed within the larger segment 502.

[0041] When the control circuit 102 identifies any segment that is fully encompassed within a larger segment, in step 408 the control circuit 102 may determine a flap probability for each of the segments fully encompassed within a larger segment. FIGS. 6A-6B are simplifiedillustrations of an example box 600 in open and closed configurations in accordance with some embodiments. FIGS. 7A-7B are illustrations to describe the flap segments 702, 704 encompassed within a box segment 502 in FIG. 5 A. In some embodiments, objects in the field of view of the detection system 112 may include a box 600 having a pair of flaps 602 extending from the side of the box 600 and configured to be folded down to close the remaining opening of the box 600. In the closed configuration, the pair of flaps 602 may form a surface of the box 600 and there may be a gap 604 between the pair of flaps 602 in the closed configuration. Because of the gap 604 between the pair of flaps 602, the segmentation algorithm may segment the image for a box 600 into two flap segments 702, 704. However, the flap segments 702, 704 may not properly represent the object for picking (e.g., the box 600) because the flap segments 702, 704 represent a part (e.g., a half) of the surface of the box 600.

[0042] As illustrated in FIG. 5A and 7A-7B, the pair of flap segments 702, 704 may be encompassed within the box segment 502. In some embodiments, in determining the flap probability of a smaller segment encompassed within a larger segment, the control circuit 102 may consider a size and a location of the smaller segment relative to the larger segment. In some embodiments, in determining the flap probability of a smaller segments encompassed within a larger segment, the control circuit 102 may determine whether there exists another segment encompassed within the larger segment and consider the existence of another segment encompassed within the larger segment. In the event that the control circuit 102 identifies another segment encompassed within the larger segment, the control circuit 102 may further consider a degree of similarity and / or symmetry between the smaller segment and the other segment encompassed within the larger segment.

[0043] In step 410, the control circuit 102 may mark, based on the flap possibility, one of the smaller segment and the larger segment encompassing the smaller segment as an error segment (a non-box segment). When the flap probability determined in step 408 for the smaller segment is the same as or exceeds a threshold value, the control circuit 102 may mark the smaller segment as an error segment. When the flap probability determined in step 408 for the smaller segment encompassed within the larger segment is below the threshold value, the control circuit 102 may mark the larger segment as an error segment.

[0044] In step 412, the control circuit 102 may exclude the error segments from the image segments 314 for the selection of the target segment 316. The remaining image segments after exclusion of the error segments may be referred as candidate segments for selection of the target segment.

[0045] In step 414, the control circuit 102 may select a segment among the candidate segments as the target segment 316. In some embodiments, the control circuit 102 may select the target segment 316 among the image segments that do not have another box above them. In some embodiment, the control circuit 102 may re-order the remaining segments (candidate segments) based on the position from the highest to the lowest and select a segment at the highest position within the picking zone 315 as the target segment 316. The picking zone 315 may correspond to an area of objects that is able to be picked by the robot 120. In some embodiments, the control circuit 102 may consider, in determining the picking zone 315, the nature and type of the robot 120 and the object interfacing mechanism 124, the position of the segments 314 relative to the robot 120 and the object interfacing mechanism 124, the positions of the segments 314 relative to one another, the position of the segments 314 relative to the non-pickable objects in the operating environment 302, and so on.

[0046] Referring back to FIG. 2, in step 210, the control circuit 102 may identify a point cloud segment corresponding to the target segment 316 selected in step 208. The control circuit 102 may identify the point cloud segment based on a portion of the 3D point cloud 308 that corresponds to the target segment 316. In some embodiments, the control circuit 102 may identify the point cloud segment corresponding to the target segment 316 by identifying each point of the 3D point cloud 308 corresponding to each pixel included in the target segment 316.

[0047] In step 212, the control circuit 102 may find and select a planar surface in the cloud point segment. In some embodiments, the planar surface may be the largest planar surface in the cloud point segment. In some embodiments, the control circuit 102 may transmit the information of the planar surface (e.g., the size, direction, and position of the planar surface) to the robot controller 122 and the robot controller 122 may use the planar surface selected in step 212 as a pick surface with which the object interfacing mechanism 124 interfaces when the robot 120 picks the object corresponding to the target segment 316 with the object interfacing mechanism 124. In some embodiments, the control circuit 102 may find and select the largest planar surface in thecloud point segment in step 212. In some embodiments, the control circuit 102 may transmit the information of the largest planar surface (e.g., the size, direction, and position of the largest planar surface) to the robot controller 122 and the robot controller 122 may use the largest planar surface selected in step 212 as a pick surface.

[0048] In step 214, the control circuit 102 may encompass, with a 3D bounding box 318, points of the planar surface selected in step 212. In some embodiments, the 3D bounding box 318 may be the smallest 3D bounding box that can encompass all points of the planner surface. In some embodiments, each edge of the 3D bounding box 318 may be parallel to the three axes 322a, 322b, 322c of the coordinate frame, and the length of each edge of the 3D bounding box may be adjustable depending on the planar surface to find a smallest bounding box encompassing all points of the planner surface. In some embodiments, the bounding box may correspond to or be based on the edges of the segment boundaries associated with the selected object.

[0049] In step 216, the control circuit 102 may select one of the eight corners of the 3D bounding box 318 as an alignment point 320. The control circuit 102 may select the alignment point 320 among the eight corners of the 3D bounding box 318 based on the location of the target segment 316 in the operating environment 302 and the moving path of the robot 120 and the object interfacing mechanism 124 toward the alignment point 320.

[0050] FIG.8 illustrates an example method to select the alignment point 320 among the eight comers of the 3D bounding box 318. In step 802, the control circuit 102 may select one of the eight corners of the 3D bounding box 318 as a candidate comer. The control circuit 102 may select the candidate comer based on the location of the target segment 316 in the operating environment 302. In some embodiment, the control circuit 102 may select the candidate comer based on a rule. For example, the rule may include, but is not limited to (1) selecting a lower left corner of the bounding box 318 as a candidate corner when the location of the target segment 316 is on the left side of the operating environment 302; (2) selecting a lower right comer of the bounding box 318 as a candidate corner when the location of the target segment 316 is on the right side of the operating environment 302; and (3) selecting an upper right or upper left corner of the bounding box 318 as a candidate comer when the location of the target segment 316 is on or adjacent to the floor of the operating environment 302.

[0051] In step 804, the control circuit 102 may simulate, with a 3D design model of the robot 120 overlay ed in the 3D point cloud 308 to verify a picking scenario with the selected candidate corner with the object interfacing mechanism 124 aligned to the selected candidate corner. To verify the picking scenario, the control circuit 102 may detect any potential risks that may occur during operation of robot 120 according to the picking scenario. The risks may include, but are not limited to, a collision of the robot with other objects (e.g., ceiling or sides of the trailer) and the stack of objects tilting or falling. In some embodiments, in the simulation, the control circuit 102 may consider the non-pickable objects such a wall, a floor, and / or a ceiling that exists within the operating environment 302. The information of the non-pickable objects may be detected and marked via the detection system 112. In some embodiments, the non-pickable objects may be pre-defined by a user.

[0052] In step 806, when the control circuit 102 does not detect a potential risk during the picking scenario associated with the candidate comer, the control circuit 102 may determine the candidate corner as the alignment point 320.

[0053] When the control circuit 102 detects a potential risk during the picking scenario associated with the candidate corner, in step 808 the control circuit 102 may select a next candidate corner and repeat steps 804 and 808 until the control circuit 102 find a candidate corner with acceptable or no risk. In some embodiments, when the control circuit 102 detects a potential risk for all eight corners of the bounding box 318 (e.g., when the control circuit 102 does not find a corner of the 3D bounding box with no risk), the control circuit 102 may select a comer with a minimum risk among the eight corners of the 3D bounding box 318 as the alignment point 320. In other approaches, when the control circuit 102 detects a potential risk for all eight corners of the 3D bounding box 318, the control circuit 102 may select another segment as a target segment 316 and may repeat steps 210 to 216 in FIG. 2.

[0054] In step 218, the control circuit 102 may provide a coordinate for the comer selected as the alignment point 320 to the robot controller 122 to cause the robot controller 122 to align the object interfacing mechanism 124 to the position of the coordinate. The coordinate may be a three- dimensional coordinate system and the coordinate for the alignment point 320 may be indicated as (a, b, c). In some embodiment, the three-dimensional coordinate system may be a Cartesian coordinate system. In some embodiments, the robot controller 122 and the detection system 112may use a common coordinate frame. In some embodiments, the control circuit 102 may generate the common coordinate frame by calculating an offset of the detection system-based frame and the robot-based frame, using a calibration grid.

[0055] In step 220, the robot controller 122 may align the object interfacing mechanism 124 to the position of the coordinate to pick the object corresponding to the target segment 316. In some embodiments, the robot controller 122 and the detection system 112 use the common coordinate frame, and the robot controller 122 may move the object interfacing mechanism 124 to the position of the coordinate received from the control circuit 102 without converting it to a coordinate of another coordinate frame.

[0056] In step 222, the robot controller 122 may cause the robot 120 to pick, with the object interfacing mechanism 124, and move the picked object to an object-release location. In some embodiments, the object-release location may be a pre-designated location or be determined by the system 100 during the operation of the system 100.

[0057] In some embodiments, the flow may continue after step 222, going back to step 202. Steps 202 to 222 may be repeated as many times as necessary. In some embodiments, steps 202 to 218 may be conducted simultaneously to step 220 and / or step 222. For example, while the robot controller 122 controls and / or operates the robot 120 in steps 220 to 222, the control circuit 102 may identify the next object to be picked and a position to which the object interfacing mechanism may be aligned to pick the next object according to steps 202 to 216 and may provide the coordinate of the position to the robot 120 according to step 218.

[0058] In some embodiments, a robotic system may include a robot including an object interfacing mechanism and a robot controller, a detection system configured to collect information from an operating environment of the robot, and a control circuit communicatively coupled to the robot and the detection system, the control circuit configured to obtain, from the detection system, a 3-dimensional (3D) point cloud of the operating environment of the robot, generate, based on the 3D point cloud, a depth map image, segment the depth map image into image segments, select a target segment among the image segments, identify a point cloud segment corresponding to the target segment, select a planar surface in the point cloud segment, encompass, with a 3D bounding box, points of the planar surface, select one of corners of the 3D bounding box as an alignment point, and provide a coordinate for the selected corner of the 3D bounding box to the robotcontroller to cause the robot controller to align the object interfacing mechanism to a position of the coordinate.

[0059] In some embodiments, a method for use with a robotic system may include obtaining, from a detection system, a 3 -dimensional (3D) point cloud of an operating environment of a robot, generating, based on the 3D point cloud, a depth map image, segmenting the depth map image into image segments, selecting a target segment among the image segments, identifying a point cloud segment corresponding to the target segment, selecting a planar surface in the point cloud segment, encompassing, with a 3D bounding box, points of the planar surface, selecting one of corners of the 3D bounding box as an alignment point, and providing a coordinate for the selected comer of the 3D bounding box to a robot controller to cause the robot controller to align an object interfacing mechanism to a position of the coordinate.

[0060] Those skilled in the art will recognize that a wide variety of other modifications, alterations, and combinations can also be made with respect to the above described embodiments without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

Claims

CLAIMSWhat is claimed is:

1. A robotic system comprising: a robot including an object interfacing mechanism and a robot controller; a detection system configured to collect information from an operating environment of the robot; and a control circuit communicatively coupled to the robot and the detection system, the control circuit configured to: obtain, from the detection system, a 3 -dimensional (3D) point cloud of the operating environment of the robot; generate, based on the 3D point cloud, a depth map image; segment the depth map image into image segments; select a target segment among the image segments; identify a point cloud segment corresponding to the target segment; select a planar surface in the point cloud segment; encompass, with a 3D bounding box, points of the planar surface; select one of corners of the 3D bounding box as an alignment point; and provide a coordinate for the selected comer of the 3D bounding box to the robot controller to cause the robot controller to align the object interfacing mechanism to a position of the coordinate.

2. The robotic system of claim 1, wherein the 3D point cloud is as viewed from the detection system and the depth map image is as viewed from the robot.

3. The robotic system of claim 1, wherein the control circuit is configured to select the target segment among the image segments at least by: identifying an error segment among the image segments; and excluding the error segment from the image segments for the selection of the target segment.

4. The robotic system of claim 3, wherein the control circuit is configured to identify the error segment at least by considering confidence scores for each of the image segments.

5. The robotic system of claim 3, wherein the control circuit is configured to identify the error segment at least by: identifying a first segment fully encompassed within a second segment; determining a flap probability of the first segment; and marking, in the event that the flap probability is the same as or exceeds a threshold, the first segment as the error segment, and in the event that the flap probability is below the threshold, mark the second segment as the error segment.

6. The robotic system of claim 5, wherein the control circuit is configured to identify the error segment at least by sorting, before identifying the first segment fully encompassed within the second segment, the image segments in size decreasing order.

7. The robotic system of claim 5, wherein the control circuit is configured to consider, in determining the flap probability of the first segment, a size and a location of the first segment relative to the second segment.

8. The robotic system of claim 5, wherein the control circuit is configured to consider, in determining the flap probability of the first segment, existence of another segment encompassed within the second segment.

9. The robotic system of claim 8, wherein the control circuit is configured to consider, in the event that the control circuit identify the another segment encompassed within the second segment, a degree of a similarity and / or symmetry between the first segment and the another segment in determining the flap probability of the first segment.

10. The robotic system of claim 1 , wherein the control circuit is configured to select the target segment among the image segments at least by selecting a segment at a highest position within a picking zone as the target segment.

11. The robotic system of claim 1, wherein the control circuit is configured to identify the point cloud segment based on a portion of the 3D point cloud that corresponds to the target segment.

12. The robotic system of claim 1, wherein the control circuit is configured to select the one of the corners of the 3D bounding box as the alignment point at least by: selecting a candidate corner based on a location of the target segment in the operating environment; simulating, with a 3D design model of the robot overlayed in the 3D point cloud to detect a potential risk, a picking scenario where the object interfacing mechanism is aligned to the selected candidate corner; and selecting, in response to detection of the potential risk, a next candidate corner.

13. The robotic system of claim 1, wherein the robot controller and the detection system use a common coordinate frame.

14. The robotic system of claim 1, wherein the control circuit is further configured to crop the 3D point cloud to remove points for a wall, a floor, and / or a ceiling.

15. The robotic system of claim 1, wherein the detection system comprises a 2- dimentional (2D) camera, a 3D camera, an RGB-D sensor, a LiDAR (Light Detection and Ranging) sensor, a line-scanning laser, or any combination thereof.

16. A method for use with a robotic system, the method comprising: obtaining, from a detection system, a 3 -dimensional (3D) point cloud of an operating environment of a robot;generating, based on the 3D point cloud, a depth map image; segmenting the depth map image into image segments; selecting a target segment among the image segments; identifying a point cloud segment corresponding to the target segment; selecting a planar surface in the point cloud segment; encompassing, with a 3D bounding box, points of the planar surface; selecting one of corners of the 3D bounding box as an alignment point; and providing a coordinate for the selected corner of the 3D bounding box to a robot controller to cause the robot controller to align an object interfacing mechanism to a position of the coordinate.

17. The method of claim 16, wherein selecting the target segment among the image segments comprises: identifying an error segment among the image segments; and excluding the identified error segment from the image segments for the selection of the target segment.

18. The method of claim 17, wherein identifying the error segment comprises considering confidence scores for each of the image segments.

19. The method of claim 17, wherein identifying the error segment comprises: identifying a first segment fully encompassed within a second segment; determining a flap probability of the first segment; and marking, in the event that the flap probability is the same as or exceeds threshold, the first segment as the error segment, and in the event that the flap probability is below a threshold, mark the second segment as the error segment.

20. The method of claim 16, wherein selecting the target segment among the image segments comprises selecting a segment at a highest position within a picking zone as the target segment.