System, method, and computer program product

Abstract

SOLUTION: A robotic singulation system is disclosed. In various embodiments, sensor data including image data associated with a workspace is received. The sensor data is used to generate a three-dimensional view of at least a portion of the workspace, the three-dimensional view including boundaries of a plurality of items present in the workspace. A grasp strategy is determined for each of at least some of items and, for each grasp strategy, a corresponding probability of grasp success is computed. The grasp strategies and corresponding probabilities of grasp success are used to determine and implement a plan to autonomously operate a robotic structure to pick one or more items from a workplace and to place each item singly in a corresponding location in a singulation conveyance structure.SELECTED DRAWING: Figure 5B

Description

[Technical field]

[0001] CROSS-REFERENCE TO OTHER APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 926,165, entitled "COORDINATING MULTIPLE ROBOTS TO MEET WORKFLOW AND AVOID CONFLICT," filed on October 25, 2019, which is incorporated by reference for all purposes, and to U.S. Provisional Patent Application No. 62 / 993,579, entitled "SINGULATION OF ARBITRARY MIXED ITEMS," filed on March 23, 2020, which is incorporated by reference for all purposes. [Background technology]

[0002] Parcel distribution centers and other distribution centers may receive any mixture of items having various sizes, dimensions, shapes, weights, stiffness, and / or other attributes, often in a random and random mix. Each item may bear machine-readable information (such as text and / or optically or otherwise encoded information) that may be read by a machine and utilized, for example, to route the item through an automated sorting / routing system and / or process. To read the information for a given item, in a typical approach, the items are separated from one another by a process known as "singulation."

[0003] Typically, singulation has been performed manually by human workers: mixed items arrive at a work station, e.g., by a chute or other conveyance, and a set of one or more human workers each manually separates the items and places them in a space defined for a single item, such as on a conveyor belt. For each item, its destination (or at least the next transportation leg) is determined by machine reading information on the item, and the item is routed to a bag, bin, container, or other receptacle associated with the next leg and / or a destination associated with the next leg, such as a delivery vehicle or staging area.

[0004] Manual singulation processes can be labor intensive and inefficient: for example, upstream workers may fill many of the single-item spots, leaving downstream human workers with few locations to place the singulated items; overall throughput can be suboptimal.

[0005] Using robots to perform singulation is a difficult challenge due to the jumble of items arriving at the workstations and the dynamic flow of items at and across each workstation, as a result it can be difficult to automatically identify, grasp, and separate (singulate) items using robotic arms and end effectors. [Brief description of the drawings]

[0006] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0007] [Figure 1] 1 is a flow chart illustrating one embodiment of a process for receiving, sorting, and transporting items for distribution and delivery.

[0008] [Figure 2A] FIG. 1 illustrates one embodiment of a robotic singulation system.

[0009] [Figure 2B] FIG. 1 illustrates one embodiment of a multi-station robotic singulation system.

[0010] [Figure 3A] 1 is a flow chart illustrating one embodiment of a process for picking and placing items for sorting.

[0011] [Figure 3B] 1 is a flow chart illustrating one embodiment of a process for picking and placing items for sorting.

[0012] [Figure 4A] FIG. 13 illustrates the calculation and display of normal vectors in one embodiment of a robotic singulation system.

[0013] [Figure 4B] 11 is a flow chart illustrating one embodiment of a process for processing image data to identify and calculate normal vectors for an item.

[0014] [Figure 5A] 1 is a flow chart illustrating an embodiment of a process for determining a plan for picking and placing an item using image data.

[0015] [Figure 5B] FIG. 1 is a block diagram illustrating an embodiment of a hierarchical scheduling system in an embodiment of a robot singulation system.

[0016] [Figure 5C] 1 is a flow chart illustrating one embodiment of a process for scheduling and controlling resources that include a robotic singulation system.

[0017] [Figure 6A]FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6B] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6C] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system.

[0018] [Figure 6D] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6E] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6F] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system.

[0019] [Figure 7A] 1 is a flow chart illustrating one embodiment of a process for modeling the flow of items for picking and placing items.

[0020] [Figure 7B] 1 is a flow chart illustrating one embodiment of a process for modeling the flow of items for picking and placing items.

[0021] [Figure 7C] 1 is a flow chart illustrating one embodiment of a process for modeling the flow of items for picking and placing items.

[0022] [Figure 8A] FIG. 2 is a block diagram illustrating a front view of one embodiment of a suction-based end effector.

[0023] [Figure 8B]FIG. 8B is a block diagram showing the bottom view of the suction-based end effector 802 of FIG. 8A.

[0024] [Figure 8C] FIG. 8B is a block diagram illustrating a front view of an example of multi-item grasping using the suction-based end effector 802 of FIG. 8A.

[0025] [Figure 8D] FIG. 8B is a block diagram illustrating a bottom view of an example of multiple item grasping using the suction-based end effector 802 of FIG. 8A.

[0026] [Figure 9] 1 is a flow chart illustrating one embodiment of a process for picking and placing an item using a robotic arm and end effector.

[0027] [Figure 10A] FIG. 1 illustrates one embodiment of a robotic singulation system.

[0028] [Figure 10B] A close-up view of a multi-view sensor array including cameras (or other sensors) 1010, 1012, and 1014 of FIG. 10A.

[0029] [Figure 10C] 1 is a flow chart illustrating one embodiment of a process for grasping and scanning an item.

[0030] [Figure 11] FIG. 1 illustrates an embodiment of a multi-station robotic singulation system incorporating one or more human singulation workers.

[0031] [Figure 12] 1 is a flow chart illustrating one embodiment of a process for detecting and correcting placement errors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments or any other form the present invention may take may be referred to herein as techniques. In general, the order of steps of a disclosed process may be altered within the scope of the present invention. Unless otherwise noted, components such as a processor or memory described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0033] A detailed description of one or more embodiments of the present invention will now be given with reference to the drawings illustrating the principles of the present invention. The present invention will be described in connection with such embodiments, but is not limited to any of them. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0034] A robotic system for performing singulation and / or sorting is disclosed. In various embodiments, the robotic system includes a robotic arm and end effectors that are used to pick items from a source pile / stream and place the items on a partitioned conveyor or similar conveyance to be sorted and routed for transport to a downstream (e.g., final destination / physical) destination. In some embodiments, the picked items are placed one-by-one in a nearby bin or other container. In some embodiments, multiple robots are coordinated to maximize overall throughput. In various embodiments, one or more robots may be used at the singulation station. The system may include multiple stations. Human personnel may be deployed at one or more stations. The robotic system may be configured in various embodiments to request (request) assistance from a human operator, e.g., by remote operation of the robotic arm, manual task completion, etc., to handle items that the robot cannot handle by fully automated processing and / or items dropped by the robot, etc.

[0035] Courier companies, postal services, delivery services, large retail stores or distributors, and other businesses and government agencies that handle, transport, and deliver items between various locations typically receive numerous items from various source locations, each of which is delivered to a corresponding one of various destination locations.

[0036] Machines exist for handling, sorting, and routing items, but to utilize machine readers and sorting equipment, items may need to be spaced apart from one another and / or oriented in a particular way so that their labels or tags can be read by the machines. Such spacing and orientation may need to be accomplished during the process of "guiding" items to the sorting / routing equipment, and may be performed in connection with a "sorting" or "sorting" process (e.g., a process in which items to be delivered to various locations are sorted by general destination (e.g., region, state, city, zip code, street, house number, etc.)).

[0037] Machine readers (such as radio frequency (RF) tag readers, optical code readers, etc.) may require that items be spaced apart from one another (a process also known as "singulation") so that the tag or code can be reliably read and so that the system can associate the resulting information with specific items (such as items at a particular location on a conveyor or other structure or means).

[0038] In a typical induction / sortation process in a baggage sorting operation, for example, individual bags may be picked from a bulk pile and placed onto a moving conveyor or tilt-tray sorting system. In most facilities, this type of induction is entirely manual.

[0039] A typical manual baggage guidance / sorting process may include one or more of the following: A chute carrying unsorted packages descends onto a sorting table adjacent to a conveyor-based sorting system A worker's job is to "singulate" items onto a conveyor or tray-based sortation system Workers ensure that all packages introduced into the sorting equipment have their shipping barcode (or other optical code, electronic tag, etc.) oriented in a way that makes it readable for sorting (this orientation is typically determined by the facility's scanning equipment). Wait for empty trays or slots to pass and ensure that only one package is placed in each slot or tray

[0040] In a typical manual induction / sortation process, packages of various shapes and sizes arrive in large quantities at a manual (or machine) feed chute in various orientations; the packages may have different dimensions, shapes, stiffness, packaging, etc.; typically, human workers remove packages from the chute and deposit them at stations where each worker works to place the packages one by one in an open section or otherwise defined portion of the conveyor; and finally, many workers, one at a time at the stations, deposit the singulated packages at a location on one or more conveyors to facilitate downstream machine processing, such as code or tag reading and automated sorting actions based thereon (e.g., routing each package to a location within the facility associated with the destination to which the package is to be delivered), which may include further sorting (e.g., to a more targeted location within the facility) and / or packing / loading the packages for further shipment (e.g., a truck or airplane to a further destination for further sorting and delivery, loading onto a truck for regional delivery, etc.).

[0041] FIG. 1 is a flow chart illustrating one embodiment of a process for receiving, sorting, and transporting items for distribution and delivery. In the illustrated example, process 100 begins with an induction process 102 in which items are provided to one or more work stations for singulation by a singulation process 104. In various embodiments, the singulation process 104 is at least partially automated by a robotic singulation system as disclosed herein. The singulation process 104 receives a pile or stream of heterogeneous items by the induction process 102 and provides a stream of singulated items to a sorting / routing process 106. For example, the singulation process 104 may place items one by one onto a partitioned conveyor or other structure that feeds the items one by one to a sorting / routing device. In some embodiments, the items are placed in an orientation in which the label or tag can be read by a downstream reader configured to read the routing (e.g., destination address) information and use the routing information to sort the items into a corresponding destination, such as a pile, bin, or other set of items bound for the same next intermediate and / or final destination. Once sorted, groups of items bound for a common next / final destination are processed by transportation processing 108. For example, the items may be placed into containers and loaded onto delivery or shipping trucks or other vehicles, etc., for delivery to the next / final destination.

[0042]

[0023] Figure 2A illustrates one embodiment of a robotic singulation system. In various embodiments, the singulation process 104 of Figure 1 is performed, at least in part, by a robotic singulation system (such as system 200 of Figure 2A).

[0043] In various embodiments, a robotic system with one or more robotic arms performs the singulation / guiding. In the example shown in FIG. 2A, the system 200 includes a robotic arm 202 with a suction-based end effector 204. In the illustrated example, the end effector 204 is a suction-based end effector, but in various embodiments, one or more other types of end effectors may be used in the singulation systems disclosed herein, including but not limited to pinch-based end effectors or other types of actuated grippers. In various embodiments, the end effector may be actuated by one or more of suction, pneumatic, pneumatic, hydraulic, or other actuation. The robotic arms 202 and 204 are configured to be used to pick up parcels or other items arriving via a chute or bin 206 and place each item in a corresponding position on a partitioned conveyor 208. In this example, items are fed into the chute 206 from an inlet end 210. For example, one or more human and / or robotic workers may supply items to the intake end 210 of the chute 206 either directly or via a conveyor or other electromechanical structure configured to supply items to the chute 206.

[0044] In the illustrated example, one or more of the robotic arm 202, end effector 204, and conveyor 208 are coordinately operated by a control computer 212. In various embodiments, the control computer 212 includes a vision system that is used to determine individual items and their orientation based on image data provided by image sensors (such as 3D cameras 214 and 216 in this example). The vision system generates outputs that are used by the robotic system to determine a strategy for grasping individual items and placing each item in a corresponding available defined location for machine identification and sorting.

[0045] In various embodiments, the robotic systems disclosed herein include and / or perform one or more of the following, for example, by operation of a control computer (such as control computer 212): Computer vision information is generated by merging data from one or more of multiple sensors, including 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors, to generate a three-dimensional view of a workspace including one or more sorting stations. The robotics system coordinates the actions of multiple robots to avoid collisions, getting in each other's way, and competing with another robot to pick up the same item and / or place an item in the same destination location (e.g., a partitioned section of a conveyor). The robotics system coordinates the actions of multiple robots to ensure that every item is placed once per slot / position. For example, if robot A drops an item, the system tasks robot B with picking it up; an item that is placed but improperly oriented can be picked up and adjusted or moved to another position by the same or another robot; if more than one item falls into a single destination slot, a robot at a downstream station picks one of them off the conveyor and places it in its position; etc. The system continuously updates the motion plans for each and every robot to maximize overall throughput. If two robots are tasked with independently retrieving the same item, the system will randomly select one robot to take that item, while the other robot transitions to the next item (e.g., identify, select, decide on a grasping strategy, pick, move according to plan, place). Conveyor movement and / or speed controlled as needed to eliminate empty positions and maximize robot productivity (throughput) In the event of a misplaced or dropped item, the system assigns the robot, or a human worker if necessary, to pick up the item and either return it to the pile from which the retrieval robot itself came, or, if available or more optimal, place it in the next free slot on the conveyor. An upstream robot that is controlled to intentionally leave some slots open for the downstream robot to place items on the conveyor. Failures that cannot be fixed by the same or another robot result in an alert being issued to obtain human (or other robot) intervention to resolve.

[0046] In various embodiments, the arbitrary mixture of items to be singulated may include luggage, packages, and / or letters of various shapes and sizes. Some items may be standard packages and one or more of their attributes may be known, while others may be unknown. Image data is used to distinguish the individual items (e.g., by image segmentation) in various embodiments. The boundaries of partially occluded items may be evaluated, for example, by recognizing the item as a standard or known type and / or by extending the visible item boundaries to a logical evaluation range (e.g., extrapolating such that two edges meet at an occluded corner). In some embodiments, the degree of overlap (i.e., occlusion by other items) is evaluated for each item, and the degree of overlap is taken into account when selecting the next item to attempt to grasp. For example, for each item, a score may be calculated to evaluate the probability of successful grasping, and in some embodiments, the score depends at least in part on the degree of overlap / occlusion by other items. An item that is less occluded, for example, may be more likely to be selected, other considerations being equal.

[0047] In various embodiments, multiple 3D and / or other cameras may be used to generate image data, a 3D view of the scene may be generated, and / or in some embodiments a combination of cameras may be used to view the scene from different angles, e.g., for a workspace and / or one or more particular items within the workspace, the camera that is least occluded is selected and utilized to grasp and move the item or items.

[0048] The multiple cameras serve many purposes in various embodiments. First, they provide a richer, full 3D view of the scene. Second, they work together to minimize errors due to package gloss when light reflecting off the package and entering the camera interferes with the camera's operation, in which case another camera in a different position provides a backup. In some embodiments, they can be selectively triggered by a predictive vision algorithm that determines which camera has the best viewing angle and / or the lowest error rate to pick a particular package, so that each package is seen by the optimal camera. In some embodiments, one or more cameras are mounted on an operating base, and the system can change the position and orientation of the base to provide a more optimal perception (e.g., view) of the packages.

[0049] Another purpose served by the camera is to detect any kind of unforeseen error in the robot operation or any disturbance to the environment. The cameras placed on the robot and in the environment have different error and accuracy profiles. The camera on the robot may be more accurate because it is fixed firmly on the robot, but may be slower to use because the robot needs to slow down or stop when using. The camera in the environment is substantially faster because it has a stable view and the robot can multitask and do other things while the camera is taking pictures. However, if someone moves or shakes the camera stand, the camera will be out of sync with the robot, causing many errors. Combining images from the robot camera and the non-robotic camera (occasionally or in the event of a package mis-packaging) makes it possible to detect if the robot is out of sync with the non-robotic camera, in which case the robot can take corrective action and become more robust. In some embodiments, the camera may not be fixedly mounted on the robot arm, and in some such embodiments, a gyro and / or accelerometer on the camera may be used to filter or compensate for the movement of the mounting base.

[0050] 2A , in the illustrated example, the system 200 further comprises an on-demand teleoperator 218 available to a human operator 220 to remotely operate one or more of the robotic arm 202, the end effector 204, and the conveyor 208. In some embodiments, the control computer 212 is configured to attempt to grasp and place an item in a fully automated mode. However, if the control computer 212 determines that there is no (further) strategy available to grasp one or more items after attempting to operate in a fully automated mode, in various embodiments, the control computer 212 sends an alert to obtain assistance from a human operator remotely, for example, by the human operator 220 using the teleoperator 218. In various embodiments, the control computer 212 uses image data from a camera (such as cameras 214 and 216) to provide a visual representation of the scene to the human operator 220 to facilitate teleoperation. For example, the control computer 212 may display a view of the pile of items in the chute 206. In some embodiments, a segmentation process is performed by the control computer 212 on the image data generated by the cameras 214 and 216 to determine item / object boundaries. Masking techniques may be used to highlight individual items, for example using different colors. An operator 220 may use a visual display of the scene to identify the items to be grasped and use a teleoperated device 218 to control the robotic arm 202 and end effector 204 to pick the items from the chute 206 and place each item in a corresponding location on the conveyor 208. In various embodiments, once the items for which human intervention was prompted are placed on the conveyor, the system 200 resumes fully automated operation. In various embodiments, during human intervention, the robotic system observes the human worker (e.g., completing a task manually, completing a task using the robotic arm and end effector remotely) and attempts to learn strategies to (better) complete the task in autonomous mode in the future.For example, the system may learn a strategy for grasping an item by observing the location of grasps on an item as a human worker grasps the item and / or by memorizing how a human worker has utilized a robotic arm and end effector to remotely grasp an item.

[0051] FIG. 2B illustrates an embodiment of a multi-station robotic singulation system. In the illustrated example, the robotic singulation system of FIG. 2A is expanded to include multiple singulation stations. Specifically, in addition to the robotic arm 202 configured to pick items from the chute 206 and place each item in a corresponding available and / or assigned location on the partitioned conveyor 208, the system shown in FIG. 2B includes three additional stations, namely, robotic arms 230, 232, and 234, positioned and configured to pick / place items from chutes 236, 238, and 240, respectively. Additional cameras 224 and 226 are included in addition to cameras 214 and 216 to provide a 3D view of the full scene, including each of the four stations / chutes 206, 236, 238, and 240, and the conveyor 208.

[0052] In various embodiments, the control computer 212 coordinates the operation of the four robotic arms 202, 236, 238, and 240 and associated end effectors with the conveyor 208 to pick / place items from the chutes 206, 236, 238, and 240 onto the conveyor 208 in a manner that maximizes the overall throughput of the system.

[0053] In the example shown in FIG. 2B, each station has one robotic arm, however, in various embodiments, two or more robots may be deployed at a single station and operated under the control of an associated control computer (such as control computer 212 in the example shown in FIG. 2B) to avoid the robots interfering with each other's operations and movements and to maximize overall throughput, such as by avoiding and / or managing contention to pick and place the same items.

[0054] In various embodiments, a scheduler coordinates the operations of multiple robots (e.g., one or more robots operating at each of multiple stations) to achieve a desired throughput without conflicts between the robots (e.g., one robot placing an item in a location assigned by the scheduler to another robot).

[0055] In various embodiments, the robotic systems disclosed herein coordinate the actions of multiple robots to pick items, one at a time, from a source bin or chute and place the items in an assigned location on a conveyor or other device to move the items to the next stage of machine identification and / or sorting.

[0056] In some embodiments, multiple robots may pick from the same chute or other source receptacle. In the example shown in FIG. 2B, for example, robot arm 202 may be configured to pick from chute 206 or chute 236. Similarly, robot arm 230 may pick from chute 236 or chute 238, and robot arm 232 may pick from chute 238 or chute 240. In some embodiments, two or more robot arms configured to pick from the same chute may have different end effectors. The robot singulation system disclosed herein may select the robot arm best suited to pick and singulate a given item. For example, the system may determine which robot arm can reach the item and select the robot arm with the best suited end effector and / or other attributes to successfully grasp the item.

[0057] Although a fixed robot arm is shown in FIG. 2B, in various embodiments, one or more robots may be mounted on a movable transport device, such as a robot arm mounted on a chassis configured to move along rails, tracks, or other guides, or a robot arm mounted on a movable cart of a chassis. In some embodiments, robotic means actuators other than robot arms may be used. For example, an end effector may be mounted on and configured to move along a rail, and the rail may be configured to move on one or more axes perpendicular to the rail to enable the end effector to be moved to pick, translate, and place items as disclosed herein.

[0058] FIG. 3A is a flow chart illustrating one embodiment of a process for picking and placing items for sortation. In various embodiments, process 300 of FIG. 3A is performed by a control computer (such as control computer 212 of FIGS. 2A and 2B). In the illustrated example, at step 302, items to be picked are identified from a chute or other source or bin through which items are received at the singulation station. In some embodiments, image data from one or more cameras is used, for example by a vision system or module that includes the control computer, to generate a 3D view of the pile or stream of items. A segmentation process is performed to determine the boundaries and orientation of the items. At step 304, a robotic arm is used to pick the items from the chute. Items may be picked one at a time, and in some embodiments, multiple items may be gripped at one time. At step 306, the one or more gripped items are each moved to a corresponding spot on a partitioned conveyor. If there are more items, further iterations of steps 302, 304, and 306 are performed, and so on, until step 308 determines that the item to be picked and placed is no longer present in the chute (or other bin or source).

[0059] FIG. 3B is a flow chart illustrating an embodiment of a process for picking and placing items for sorting. In various embodiments, the process of FIG. 3B implements step 304 of process 300 of FIG. 3A. In the illustrated example, at step 320, a plan and / or strategy for grasping one or more items is determined. In various embodiments, the plan / strategy is determined at least in part based on one or more of image data indicating, for example, the size, extent, and orientation of the items, and attributes that may be known, determined, and / or estimated about the items, such as by categorizing the items by size and / or item type. For example, in the context of a package delivery service, a particular standard package may be used. A weight range or other information may be known for each standard package type. Additionally, in some embodiments, strategies for grasping items may be learned over time, for example, by the system noting and recording the success or failure of previous attempts to grasp similar items (e.g., same spec item / package type; similar shape, stiffness, dimensions; same or similar shape; same or similar material; location and orientation relative to other items in the pile; degree of overlap of items, etc.).

[0060] In step 322, the system attempts to grasp one or more items using the strategy determined in step 320. For example, an end effector of a robotic arm may be moved to a position adjacent the items to be grasped according to the determined strategy, and the end effector may be operated according to the determined strategy to attempt to grasp the items.

[0061] At step 324, a determination is made as to whether the grip was successful. For example, image data and / or force (weight), pressure, proximity, and / or other sensor data may be used to determine whether the item was successfully gripped. If successful, the item is moved to the conveyor at step 326. If unsuccessful, processing returns to step 320 where a new strategy for gripping the item (if available) is determined.

[0062] In some embodiments, if the system fails to grasp the item after a predetermined and / or set number of attempts, or if the system is unable to determine a further strategy for grasping the item, the system transitions to identifying and grasping another item, if possible, and / or sends an alert to obtain assistance, such as from a human operator.

[0063] 4A is a diagram illustrating the calculation and display of normal vectors in one embodiment of a robotic singulation system. In various embodiments, a control computer (such as control computer 212 in FIGS. 2A and 2B ) constituting the robotic singulation system disclosed herein determines the item's boundary and normal vectors, and generates and displays a visualization of the item's boundary and normal vectors.

[0064] In the illustrated example, a 3D scene visualization representation 400 includes the output of a vision system implemented in various embodiments. In this example, a normal vector to an item surface is shown. The normal vector is used in some embodiments to determine a strategy for grasping the item. For example, in some embodiments, the robot has a suction-type end effector, and the normal vector is used to determine the angle and / or orientation of the end effector to maximize the chances of successful suction-based grasping.

[0065] In some embodiments, a vision system is used to distinguish and differentiate items by object type, which may be indicated in the visualization 400 by highlighting each object with a color corresponding to its type. In some embodiments, the object type information is or may be used to determine and / or modify the item's grasping strategy, such as by increasing suction pressure or gripping force, reducing movement speed, using a robot with a specific required end effector, payload capacity, etc.

[0066] In some embodiments, additional information not shown in FIG. 4A may be displayed. For example, in some embodiments, for each item, the best grasping strategy and associated grasping success probability is determined and displayed adjacent to the item. The displayed information may be utilized in some embodiments to monitor system operation in a fully automated operating mode and / or to allow a human operator to intervene and quickly gain a view of the scene and available grasping strategies and probabilities, for example for purposes of operating a robotic singulation system remotely.

[0067] 4B is a flow chart illustrating one embodiment of a process for processing image data to identify and calculate normal vectors for items. In various embodiments, process 420 may be performed to generate and display a visual representation of a 3D view of a scene (such as visualization representation 400 of FIG. 4A). In various embodiments, process 420 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B).

[0068] In the illustrated example, at step 422, 3D image data is received from one or more cameras (such as cameras 214, 216, 224, and / or 226 of FIGS. 2A and 2B, cameras mounted on a robot arm and / or end effector, etc.). In various embodiments, image data from the multiple cameras is merged to generate a composite 3D view of a scene, such as a pile or stream of items from which items are to be picked. At step 424, a segmentation process is performed to determine item boundaries in 3D space. At step 426, for each item for which a sufficient and sufficiently unobstructed view has been obtained, the geometric center and normal vector of each of the item's one or more surfaces (e.g., the largest and / or most exposed (e.g., not obscured by overlapping other items) surfaces) are determined.

[0069] In various embodiments, the segmentation data determined in step 424 is used to generate a corresponding mask layer for each of the plurality of items. Each mask layer is used in various embodiments to generate and display a visual representation of the scene in which the respective item is highlighted. In various embodiments, each item is highlighted with a color corresponding to the item's object type and / or other attributes, such as weight, classification, softness, deformability, unpickability (e.g., broken packaging, porous surfaces, etc.).

[0070] 5A is a flow chart illustrating one embodiment of a process for determining a plan for picking and placing an item using image data. In various embodiments, the process 500 of FIG. 5A is performed by a computer (such as the control computer 212 of FIGS. 2A and 2B).

[0071] In the illustrated example, image data is received at step 502. The image data may be received from multiple cameras, such as one or more 3D cameras. The image data may be merged to generate a 3D view of the workspace, such as a pile or stream of items in a chute or other container. Segmentation, object type identification, and normal vector calculations are performed at step 504. In some embodiments, the segmentation data is used to determine item boundaries and generate item-specific mask layers, as described above.

[0072] At step 506, a gripping strategy is determined for as many items as possible given the current view of the workspace. For each item, one or more gripping strategies are determined, and for each strategy, a probability that the strategy will lead to successful gripping of the item is determined. In some embodiments, the gripping strategy is determined based on item attributes (item type, size, estimated weight, etc.). In some embodiments, determining the gripping strategy includes determining whether to attempt to pick up one or multiple items, whether to utilize all end effector actuators (e.g., whether to utilize only one suction cup or set of cups or more than one), and / or selecting a gripping technique and corresponding speed combination from the set of calculated gripping technique+speed combinations under consideration (e.g., one suction cup at 50% speed or two suction cups at 80% speed). For example, in some embodiments, a decision may be made based on the package type to utilize one suction cup at 50% speed (moving at 50% speed instead of 80% to avoid dropping it), such as to grip a small item that may be heavy or difficult to grip securely with one suction cup (two suction cups cannot be utilized), whereas a larger item that is relatively light or easier to grip securely with two suction cups may, in contrast, be gripped with two cups and moved at a higher speed.

[0073] In some embodiments, the gripping strategy may change if the object is moving as part of a flowing pile. For example, according to one strategy, the robot may press down harder to freeze the object in place, while the suction cups may be activated independently in the order they touch the package to ensure a firm and stable grip. The robot also matches its speed with the flowing pile based on vision or depth sensor feedback to ensure that the package does not slip within the pile.

[0074] Another optimization implemented in some embodiments is to change the gripping strategy based on double (or n-) item picks (see also next section), where the robot may adapt its strategy based on whether it picks multiple items in a batch or sequentially while trying to reuse vision data. For example, a robot gripper with four suction cups may utilize two cups to pick up one object and the other two cups to pick up a second object. In this scenario, the robot will slow down and also begin to grip the second object, to avoid collisions between the first object already held and the surrounding pile of objects.

[0075] At step 508, a plan is determined for grasping, moving, and placing items according to the probabilities using the item-specific grasping strategies and probabilities determined at step 506. For example, a plan may be determined for grasping the next n items in a particular order, each according to a grasping strategy and the probability of success determined for that item and strategy.

[0076] In some embodiments, changes in item position, sequence, orientation, and / or flow due to grasping and removal of previously grasped items are taken into account in planning for sequential grasping of multiple items in step 508.

[0077] In various embodiments, the process 500 of Figure 5 is a continuous and ongoing process: as items are picked and placed from the pile, subsequently received image data is processed to identify and determine a strategy to grab more items (steps 502, 504, 506), and a plan for picking / placing the items is updated based on the grab strategy and probabilities (step 508).

[0078] FIG. 5B is a block diagram illustrating an embodiment of a hierarchical scheduling system in an embodiment of a robot singulation system. In various embodiments, the hierarchical scheduling system 520 of FIG. 5B is implemented at least in part on a computer (such as the control computer 212 of FIGS. 2A and 2B). In the illustrated example, the hierarchical scheduling system 520 comprises a global scheduler 522 configured to optimize throughput by coordinating the operation of multiple robot singulation stations and a partitioned conveyor (or similar structure) onto which the robot singulation stations are configured to place items. The global scheduler 522 may be implemented as a processing module or other software running on a computer. The global scheduler manages and coordinates work among the robot singulation stations, at least in part, by monitoring, optionally controlling, and / or otherwise providing input to multiple robot singulation station schedulers 524, 526, 528, and 530.

[0079] Each of the robotic singulation station schedulers 524, 526, 528, and 530 is associated with a corresponding robotic singulation station and each controls and coordinates the movement of one or more robotic arms and associated end effectors to pick up items from a corresponding chute or other item receptacle and place them one-by-one on a partitioned conveyor or similar structure. Each of the robotic singulation station schedulers 524, 526, 528, and 530 is associated with a corresponding set of one or more station sensors 532, 534, 536, and 538, respectively, and each performs automated singulation at that robotic singulation station using sensor data generated by that station's sensors. In some embodiments, each scheduler implements and executes process 500 of FIG. 5A.

[0080] In various embodiments, each of the robot singulation station schedulers 524, 526, 528, and 530 reports one or more of image and / or other station sensor data, object identification, grasping strategy, and success probability data; pick / place planning information; and expected item singulation throughput information to the global scheduler 522. The global schedule 522 is configured to use information received from the robot singulation station schedulers 524, 526, 528, and 530, together with sensor data received from other sensors 540 (such as cameras directed at other parts of the partitioned conveyor and / or workspace not covered by the station sensors or not fully or fully covered), to coordinate work by each robot singulation station, each under the control of its station-specific scheduler 524, 526, 528, or 530, and to control the operation (e.g., speed) of the partitioned conveyor via the conveyor controller 542, in order to optimize (e.g., maximize) the overall singulation throughput of the system.

[0081] In various embodiments, the global scheduler 522 utilizes one or more techniques to optimize the utilization of multiple robots comprising the robot singulation system to perform singulation (e.g., to maximize overall throughput). For example, if there are four robots in a sequence, the lead (or other upstream) robot may be controlled to place packages to leave free slots so that downstream robots do not wait for an empty slot. This approach impacts downstream robots as they wait for unknown / random times due to package flow, etc. As a result, a naive strategy (e.g., the lead robot places every fourth free slot) may not optimize the overall throughput. Sometimes, if the lead robot's packages are not flowing, it may be better for the lead robot to place 2-3 packages in consecutive slots in the sequence, but the overall system makes such decisions with awareness of the state and flow of each station. In some embodiments, the optimal strategy for leaving free slots for downstream robots is based on the predicted demand for free slots by downstream robots (e.g., as a function of their package flow). In some embodiments, information from the local station scheduler is utilized to predict the maximum throughput of each station, as well as to control the conveyor speed and the number of slots left empty by the upstream robot so that the downstream robot can access the empty slots proportional to the rate at which it can (currently) pick / place. In some embodiments, when a partitioned conveyor is full due to some bottleneck in the downstream sortation process, the robotic singulation system disclosed herein may track the pose of each pre-singulated package while pre-singulating one or more packages, for example, in a staging area in or near its corresponding chute.When some free space becomes available on the partitioned conveyor, the system / station moves the pre-singulated packages one by one in rapid succession onto the partitioned conveyor without further visual processing time.

[0082] In some embodiments, the presence of a human collaborating with a robot impacts the placement and multi-robot coordination strategy, as the robot or associated computer vision or other sensor systems must see what the human does and adapt the placement of the robot in real time. For example, if a human takes over a conveyor belt slot that was scheduled to be utilized by a robot, the system must adjust the global and local schedules / plans accordingly. In another example, if a human interferes with a package picked by a robot, causing it to register as not picked, the system adapts to correct that error. Also, if a human corrects an error in a robot's picking (the robot is commanded to place a package in slot A, but mistakenly places it across slot A and adjacent B, and the human places it in slot B, but in the system's memory the package is in slot A), the system must observe the human's behavior and adjust the downstream robot's behavior.

[0083] In various embodiments, the global scheduler 522 may operate a station slower than its maximum possible throughput at a given time. For example, the global scheduler 522 may explicitly instruct a local station scheduler (e.g., 524, 526, 528, or 530) to slow down and / or make fewer slots available to the local station, e.g., explicitly by allocating fewer slots to the station or indirectly, such as by allowing an upstream station to fill more slots.

[0084] 5C is a flow chart illustrating one embodiment of a process for scheduling and controlling resources including a robotic singulation system. In various embodiments, process 560 of FIG. 5C is performed by a global scheduler (such as global scheduler 522 of FIG. 5B).

[0085] In various embodiments, items arrive via a chute and / or conveyor. The chute / conveyor may be fed, for example, by humans, robots, and / or both, and items may arrive in chunks. As items are added to the stream at the input end, they may slide or flow down / along the chute or conveyor toward the end where one or more robotic arms are provided to perform the singulation, for example, by gripping individual items and moving each item, one at a time, to a corresponding one of the item locations on the output conveyor.

[0086] In various embodiments, images from one or more 3D or other cameras are received and processed. The flow of items down / through the chute / conveyor feeding the singulation station is modeled based on the image data. The model is used to predict moments of relatively low or no movement, at which time the image data is used to identify and grasp the items. In some embodiments, the flow model is used to ensure that the robotic arm is not in a position to block one or more cameras during moments of relatively low or no flow. In some embodiments, the system is configured to detect areas of relatively low or no flow within a larger flow. For example, the flow may be analyzed within different predefined areas of a chute or other source conveying device or container. The system may pick items for singulation from areas of low / stable flow and / or no flow, and may avoid areas where less stable flow occurs for the time being.

[0087] In some embodiments, the system identifies different areas in the flow of items and guides the robot to pick items from such areas accordingly. In some embodiments, the system utilizes a flow model, calculated using sequential images from multiple cameras mounted on and around the chute, to predict the movement of items on / through the chute or other source conveying device or container. The flow model is used to predict the future location of one or more items as they flow down the chute or other conveying structure. Each time, an image is captured and / or a grasp is attempted at the time / location where the item is predicted to be based on the flow model. In some embodiments, real-time segmentation is performed. For example, segmentation is performed in the range of 30-40 milliseconds and / or other rates as fast as the 3D camera frame rate. In some embodiments, the results of the real-time segmentation are used to track and / or model the flow of individual items through the chute. The future location of the item is predicted based on the item-specific model / movement, and a plan and strategy to grasp the item at that future location and time is determined and executed autonomously. In some embodiments, the location of the target item is continuously updated. In some embodiments, the trajectory of the robotic arm may be updated in real-time based on the updated predicted future position of the item as the arm moves to grasp the item.

[0088] In some embodiments, if the observed flow of items is faster than the threshold speed, the system waits a set amount of time (e.g., 100 milliseconds) and checks the image data and / or flow speed again. The system may repeat such waits and / or vary their length until a sufficiently stable flow condition is observed to have a high probability of successful picking. In some embodiments, different areas in the chute may have different speed thresholds that are tolerated. For example, if an area is away from the actual spot where the robot is picking, a faster movement speed is tolerated in some embodiments, since an unstable flow in that area is not expected to interfere with the operation being performed by the robot at the time. For example, a package rolling at the top of the chute ("at high speed") may not be an issue if the robot is picking from the bottom of the chute. However, if the rolling item is close to the pick area, it may not be acceptable as it may occlude other items or move them around by colliding with them. In some embodiments, the system detects flow by region and / or location relative to the expected pick area, allowing for less stable flow in areas far (sufficiently far) from the area the robot intends to pick next.

[0089] In the example shown in FIG. 5C, at step 562, the overall throughput, the observed and / or estimated (locally scheduled) throughput of the local singulation station, and the overall and station-specific error rates are monitored. At step 564, the conveyor speed and local station speeds are adjusted to maximize the net amount of overall throughput or error. At step 566, conveyor slots are assigned to each station to maximize the net throughput. In this example, conveyor slots are explicitly assigned / designated, but in some embodiments, station speeds are controlled to ensure that downstream stations have available slots to place items, without (necessarily) pre-assigning specific slots to specific stations. Processing continues (steps 562, 564, 566) while any stations have items yet to be picked / placed (step 568).

[0090] 6A-6C show an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. In various embodiments, the flow of items through the chute or other receptacle is modeled. The flow model is used in various embodiments to determine a strategy for grasping items from the flow. In some embodiments, the modeled and / or observed flow may be used to perform one or more of the following: determining a grasping strategy and / or plan for grasping an item at a future location where the item is predicted to flow; determining a grasping strategy for each of a plurality of items and determining and implementing a plan for grasping a series of items such that each is grasped at a corresponding future location determined based at least in part on the flow model; ensuring that the robot arm is in a position that does not obscure the view of an item at a future instant in time where the item is predicted to be located based on the model and where it is planned to be picked; and waiting, for example, a calculated or predetermined time (based on the model) to allow the flow to become more stable (e.g., move slower, most of the items move in a uniform direction, have minimal or low rates of change in orientation, etc.).

[0091] 6A-6C, in the illustrated example, the flow model shows a currently mostly stable array of items where the model indicates that items 602 and 604 will remain relatively stable / uniform as they are picked from the flow / pile. In various embodiments, the model information shown in Figures 6A-6C, potentially along with other information (e.g., available pick strategies, desired / allocated station throughput, etc.), is used to determine and implement a plan for sequentially picking and placing items 602 and 604 from the location where the model indicates the items are expected to be at the time they are scheduled to be picked.

[0092] 6D-6F show an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. In this example, the model shows that the pile is moving in a relatively slow, even flow, but is disturbed (or observed to be disturbed) when item 602 is picked. In various embodiments, the model information shown in FIGS. 6D-6F may be used to decide to pick item 602 alone, but then wait a short time for the pile / stream to stabilize before determining and implementing a grasping strategy and plan to pick and place other items from the pile / stream. In some embodiments, the stream may be detected to be unstable enough that one or more items are at risk of tipping or otherwise falling out of the chute or other source vessel before they can be picked up and singulated. In some embodiments, in such a situation, the system may be configured to implement a strategy to stabilize the stream using the robotic arm and / or end effector, such as by blocking the bottom of the chute to prevent items from spilling out of the chute, by placing an arm across the stream, or by using an end effector and / or robotic arm to block or slow the stream.

[0093] FIG. 7A is a flow chart illustrating one embodiment of a process for modeling the flow of items to pick and place the items. In various embodiments, process 700 of FIG. 7A is performed by a computer configured to model the flow of items (e.g., through a chute) and use the model to pick and place the items, such as the example described above in connection with FIGS. 6A-6F. In various embodiments, process 700 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B). In some embodiments, process 700 is performed by a robot singulation station scheduler (such as schedulers 524, 526, 528, and 530 of FIG. 5B).

[0094] In the example shown in FIG. 7A, image data is received at step 702. At step 704, segmentation and item (e.g., type) identification is performed. At step 706, the flow of items through a chute or other container is modeled. (In this example, steps 704 and 706 are performed sequentially, but in some embodiments, these steps may be performed in parallel and / or in parallel). At step 708, the model is used to determine a pick strategy and plan for picking the next n items from the pile / stream. At step 710, the plan is executed. If, at step 712, it is determined that the attempt was not completely successful (e.g., one or more items were not grasped successfully, an item was not in the expected position, the flow was disrupted, or was otherwise not as expected), or if additional items have not yet been singulated (step 714), further iterations of steps 702, 704, 706, 708, and 710 are performed, and the next iteration is performed until, at step 714, it is determined that no more items remain to be singulated.

[0095] In various embodiments, the system may take a picture and identify two (or more) objects to pick. The system picks and moves the first one, and then instead of doing a full scene recalculation to find the package to pick, it simply checks if the second package is in the way. If not, the system picks it without doing a full scene recalculation, which typically saves a lot of time. In various embodiments, the time savings are in one or more of the following: sensor reading latency, image acquisition time, system calculation time, subdivision, masking, package pile ordering calculation, and finally control and planning. If the second item is not in the expected location, the system does a full scene recalculation to find the next package to pick.

[0096] In some embodiments, the robot may use vision, depth, or other sensors to identify two graspable packages that are determined by the robot's control algorithm to be far enough apart that they cannot interfere with each other's pick (such as by destabilizing the pile and causing it to collapse or by colliding with other packages). Instead of repeating the sensor processing pipeline after picking the first package, the robot may proceed directly to pick the second package. That said, statistically there is a risk that the control algorithm's prediction may be wrong or that some unforeseen event may have caused the pile to move. In this scenario, the controller may pick more cautiously and use a predicted model of the gripping suction cup actuation (or other predicted sensory models, e.g., force, pressure, and other types of sensing modalities) to test whether the package being picked matches the previous prediction (without having to redo the vision calculations).

[0097] FIG. 7B is a flow chart illustrating one embodiment of a process for modeling the flow of items to pick and place the items. In various embodiments, process 720 of FIG. 7B is performed by a computer configured to model the flow of items (e.g., through a chute) and use the model to pick and place the items, such as the example described above in connection with FIGS. 6A-6F. In various embodiments, process 720 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B). In some embodiments, process 720 is performed by a robot singulation station scheduler (such as schedulers 524, 526, 528, and 530 of FIG. 5B).

[0098] In the example shown in Figure 7B, image data is received and processed at step 722. Items are picked / placed at step 724 based at least in part on the model, e.g., similar to process 700 of Figure 7B. If an indication is received and / or a determination is made at step 726 that the flow has become unstable, the system pauses at step 728, e.g., for a random, predetermined, and / or set interval, after which processing resumes and continues unless / until there are no more items to be picked / placed, after which process 720 ends.

[0099] FIG. 7C is a flow chart illustrating one embodiment of a process for modeling the flow of items to pick and place the items. In various embodiments, process 740 of FIG. 7C is performed by a computer configured to model the flow of items (e.g., through a chute) and use the model to pick and place the items, such as the example described above in connection with FIGS. 6A-6F. In various embodiments, process 740 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B). In some embodiments, process 740 is performed by a robot singulation station scheduler (such as schedulers 524, 526, 528, and 530 of FIG. 5B).

[0100] In the example shown in FIG. 7C, at step 742, a condition is detected where no items are currently available for grasping. For example, the system may have attempted to determine a grasping strategy for items in the pile, but determined that due to flow speed, clutter, orientation, overlap, etc., no items are currently available for which a grasping strategy has a probability of success greater than a predetermined minimum threshold. At step 744, in response to the determination at step 742, the system attempts to use the robotic arm to change the pile / flow conditions to make a grasping strategy available. For example, the robotic arm may be used to gently peck, pull, push, etc., an item or items to a different location in the pile. After each peck, the system may re-evaluate, for example, by recalculating a 3D view of the scene to determine whether a viable grasping strategy has become available. If at step 746, it is determined that a grasp (or multiple grasps, each on a different item) is now viable, autonomous operation is resumed at step 750. Conversely, if after a predetermined number of attempts to change the pile / stream state no viable grasping strategy becomes available, then in step 748 the system receives assistance, for example, from another robot and / or a human worker, the latter via teleoperation and / or manual intervention (such as shuffling items in the pile and / or manually picking / placing items) until the system determines that autonomous operation can resume.

[0101] FIGURE 8A is a block diagram illustrating a front view of one embodiment of a suction-based end effector. In various embodiments, the end effector 802 may be used as an end effector of a robotic arm (such as end effector 204 of FIGURE 2A) that constitutes a robotic singulation system. FIGURE 8B is a block diagram illustrating a bottom view of the suction-based end effector 802 of FIGURE 8A.

[0102] 8A and 8B, in the illustrated example, end effector 802 includes four suction cups 804, 808, 812, and 814. In this example, a vacuum may be applied to suction cups 804 and 812 via hose 806, and a vacuum may be applied to suction cups 808 and 814 via hose 810. In various embodiments, pairs of suction cups (i.e., a first pair including suction cups 804 and 812, and a second pair including suction cups 808 and 814) may be operated independently. For example, a single item (such as a relatively small and / or relatively lightweight item) may be grasped using only one or the other of the suction cup pair. In some embodiments, each pair may be used to grasp a separate item at the same time, allowing for simultaneous pick / place of two (or, in some embodiments, three or more) items.

[0103] Figure 8C is a block diagram illustrating an example of gripping multiple items from a front view using the suction-based end effector 802 of Figure 8A. Figure 8D is a block diagram illustrating an example of gripping multiple items from a bottom view using the suction-based end effector 802 of Figure 8A. In this example, each suction cup pair was independently actuated to grip a corresponding one of the two items shown gripped in the example diagram.

[0104] In various embodiments, more or fewer suction cups and / or more or fewer independently actuated sets of one or more suction cups may be included on a given end effector.

[0105] 9 is a flow chart illustrating one embodiment of a process for picking and placing an item using a robotic arm and end effector. In various embodiments, the process 900 of FIG. 9 may be performed by a computer (such as the control computer 212 of FIGS. 2A and 2B).

[0106] When using a suction-based end effector (such as end effector 802), pushing too hard to ensure suction cup engagement may damage fragile items and / or packaging. Additionally, sensor readings may vary based on package type. For example, suction sensor readings when gripping a cardboard box (where air escapes through the grooves / paper) may be different from a plastic poly bag. In some embodiments, if the package type is known, the sensor readings are evaluated and / or adjusted accordingly. In some embodiments, if a sensor reading associated with successful gripping of a determined type of package is not achieved on initial engagement, additional force and / or suction may be applied to achieve a better grip. Such an approach increases the likelihood of successful gripping and is more efficient than determining that the grip has failed and trying again.

[0107] In the example shown in FIG. 9 , at step 902, the robotic arm is used to approach and attempt to grasp the item according to the determined and selected grasping strategy for grasping the item. At step 904, a determination is made as to whether the grasping of the item was successful. For example, one or more of a force (weight) sensor, a suction / pressure sensor, and image data may be used to determine whether the grasping was successful. If successful, at step 906, the item is assigned and / or moved to the next available slot. If the grasping was determined to be unsuccessful at step 904 and the system determines to make further attempts at step 908 (e.g., a number less than a predetermined maximum number of attempts has not yet been reached, the item has not been determined to be too fragile to apply a greater force, etc.), at step 910, the system adjusts the force applied to engage the item (and / or, in various embodiments, the grasping pose, i.e., one or more of the position and / or orientation of the robotic arm and / or end effector) and attempts to grasp the item again. For example, a robotic arm may be used to press the suction cup against the item with a slightly greater force before activating the suction and / or a stronger suction may be applied.

[0108] If, at step 908, it is determined that no further effort should be made at this time to grasp the item, processing proceeds to step 912 where it is determined whether there are other items that can be grasped (e.g., other items are present in the chute and the system has a grasping strategy available for one or more of them). If so, at step 914 the system moves on to the next item and performs an iteration of step 902 and subsequent steps for that item. If there are no remaining items and / or if no items have a grasping strategy available, at step 916 the system receives assistance, for example, from another robot, from a human via teleoperation, and / or from a human via manual intervention.

[0109] In various embodiments, the number and / or nature of further attempts to grasp the item, as in steps 908 and 910, may be determined by factors such as the item or item type, item and / or package characteristics determined by the item type, item and / or package characteristics determined by inspecting the item with the robotic arm and end effector, item attributes determined by various sensors, etc.

[0110] 10A illustrates one embodiment of a robotic singulation system. In the illustrated example, a robotic singulation station 1000 includes a robotic arm 1002 operated under the control of a control computer (not shown in FIG. 10A ) to pick items (such as item 1006 in the illustrated example) from a chute or other container 1004 and place them one-by-one onto a partitioned conveyor 1008.

[0111] In various embodiments, the robotic singulation station 1000 is controlled by a control computer configured to, at least in certain circumstances, scan addresses or other routing information locally at station 1000 using a multi-view array of sensors, in this example including cameras (or other sensors) 1010, 1012, and 1014.

[0112] FIG. 10B is a close-up view of a multi-view sensor array including cameras (or other sensors) 1010, 1012, and 1014 of FIG. 10A.

[0113] In various embodiments, sensors such as cameras (or other sensors) 1010, 1012, and 1014 are positioned to read the routing information (e.g., text address, optical or other code, etc.) regardless of the orientation of the package placed by the robot on the output conveyor. For example, if the package is placed label-side down, a scanner across which the package is slid and / or swiped reads the label and associates the sorting / routing information with the corresponding location on the output conveyor.

[0114] A challenge in picking objects from a (possibly moving) pile and placing them on a singulation conveyor is that the singulation conveyor needs to associate the package barcode with each package in the conveyor belt slot. Based on the orientation of the package in the bulk pile that the robot is picking from, the robot may find a situation where the barcode is actually facing down. In some embodiments, the robotic singulation system disclosed herein may be used to flip or otherwise flip the item into a position where the barcode or other label or routing information is facing up. In some embodiments, the robotic arm flips the package using its own motion, either by using controlled slippage due to gravity if the end effector is a pinch gripper (e.g., gripping at an end and allowing slippage / gravity to begin to rotate the item before releasing once the flipping motion is initiated), or by using gravity and a controlled release sequence of multiple suction cups to reorient the package (e.g., releasing the suction of the cups at one end of the effector while still applying suction at the other end to begin to rotate the item about the axis about which it is flipped). However, flipping the package before placing it on the conveyor belt may damage the package, may not work, and / or may take a lot of time and may require additional hands (e.g., additional robots), which is expensive. In some embodiments, the systems disclosed herein may recognize that the item is of a type that may require an action to ensure that the label can be read. For example, packages in polyethylene ("poly") or other plastic or similar bags may need to be placed to flatten and / or smooth the package so that a downstream scanner can read the label. In some embodiments, a robotic arm may be used to smooth and / or flatten the package.In some embodiments, such items may be dropped from a height to help flatten the packaging enough for reading. In various embodiments, the robot may flatten the plastic bag by throwing it onto a chute or conveyor prior to picking and / or reading at the station, spreading the bag using actuated suction cups after picking, performing bi-manual (two robotic arms) operations to remove wrinkles in deformable items, etc. In some embodiments, a blower or other mechanism may be used to smooth the package after placement. The blower may be standalone / fixed mounted or integrated into the robotic arm.

[0115] In some embodiments, a multi-axis barcode (or other) scanner or sensor, i.e., a device that can scan top or bottom barcodes, is placed downstream on the conveyor belt 1008. However, this only works if the bottom of the conveyor belt is clear, or there is a sequencing operation where packages are passed over a clear (or open) slot with the barcode reader looking up through the slot.

[0116] In various embodiments, if the package cannot be easily placed on the conveyor 1008 with the label facing up (and therefore easily scanned by an overhead scanner), an array of sensors at the singulation station, such as cameras (or other sensors) 1010, 1012, and 1014 in the example shown in FIGS. 10A and 10B. In some embodiments, if the top of the package is visible, the barcode or other address information may be read as part of a computer vision scanning pipeline and / or with a camera mounted on the robot arm 1002 and / or end effector. If the barcode is on the side or bottom or otherwise hidden, cameras (or other sensors) 1010, 1012, and 1014 are used to scan the package as the robot picks it up and moves it to the conveyor belt or bin. To get this right, in some embodiments the robot modifies its controller and motion planning to ensure that the package is scanned in mid-air, which requires positioning the object in an optimal way for the barcode scanner to see the package, while at the same time constraining the motion path so the object lands in an empty slot. This is especially challenging for barcode scanning on the bottom or side, as the barcode scanners must be placed in an optimized configuration to simplify the motion planning task for the robot and to ensure that the robot can perform the scan and place quickly. The scanner configuration and robot motion planning are dynamically influenced by the package itself, as depending on the package's size and weight the object may need to be positioned differently when the robot attempts to scan the barcode (see also the section on gripping / speed optimization based on package type for examples of how the robot has to adapt its motion). · The height of a package is typically unknown when it is picked from a dense pile, since cameras or other sensors cannot see the bottom of the pile. In this case, the robot uses machine learning or geometric models to predict the height of the object. If the height estimate is too low, the robot will hit the barcode scanner with the object. If the height estimate is too high, the robot will not be able to scan the object. One side of the error spectrum is catastrophic (the item hits the scanner), while the other simply causes a replanning of the trajectory. In various embodiments, the robot control algorithm chooses to play on the safer side and assume the object is higher to avoid a collision. If the object is too far away, the robot can use a human-like rocking motion to gradually move it closer to the barcode scanner so it is scanned.

[0117] FIG. 10C is a flow chart illustrating one embodiment of a process for gripping and scanning an item. In various embodiments, process 1020 of FIG. 10C is performed by a control computer configured to control the operation of a robotic singulation station (such as station 1000 of FIG. 10A). In the illustrated example, at step 1022, a robotic arm is used to approach and grip the item. As the item is approached and / or gripped, an image of the top surface of the item is captured. In various embodiments, a camera pointed at the item / pile, such as a camera mounted near the station and / or on the robotic arm and / or end effector, may be used to capture the image. At step 1024, the image is processed to determine whether an address label or other routing information is on the top surface. If so, at step 1026, the item is placed face up on the conveyor.

[0118] If the label is not on the top surface when grasped (step 1024), the system attempts to find the label using one or more local sensors at step 1028. If the label is found and on a surface such that the robotic singulation system can place the item on the conveyor with the label rotated up, then at step 1030 it is determined that a local scan is not necessary and at step 1026 the package is rotated so that the label is up and placed on the conveyor. Conversely, if the label is not found or the package cannot be rotated so that the label is up, then at step 1030 it is determined that the label should be scanned locally. In such a case, at step 1032 the label is scanned locally using a multi-axis sensor array such as, for example, cameras 1010, 1012, and 1014 of Figures 10A and 10B. The package is placed on a conveyor at step 1034 and, at step 1036, the routing information determined by locally scanning the label is associated with the slot or other demarcated location on the conveyor in which the item was placed.

[0119] Downstream, the routing information determined by locally scanning the item's label at the robotic singulation station is used to sort / route the item to an intermediate or final destination determined at least in part based on the routing information.

[0120] Figure 11 illustrates an embodiment of a multi-station robotic singulation system incorporating one or more human singulation personnel. In the illustrated example, the robotic singulation system 1100 includes elements of the system of Figure 2B, except that at the left-most station (as shown), the robotic arm 202 is replaced by a human operator 1102.

[0121] In some embodiments, the robotic system includes one or more human singulation workers, for example at the last or other downstream station, such as the example shown in Figure 11. The system and / or human workers provide items that are difficult to singulate (e.g., large flexible bags) to the workstations staffed by the human workers. The robotic system leaves slots open to be filled by the human workers.

[0122] In some embodiments, each singulation station includes a location for one or more robots and one or more human workers, e.g., working together at the same station. The location for the human workers provides enough space for the workers to perform the singulation in the out-of-reach area of ​​the robotic arms. In various embodiments, the human workers can augment the throughput of the robotic arms, correct misalignments, incorrect orientations (e.g., for scanning labels), ensure that conveyor slots not filled by upstream robots or other workers are filled, etc.

[0123] Given that the singulation workspace is often limited, the following situations are addressed by the controller (e.g., control computer) in various embodiments:

[0124] If the robot determines that the flow of packages is too fast or that it is not feasible to grasp some of the packages, it will move away, go into safety mode and trigger a message for a human operator to come and help.

[0125] --If the robot determines that a package is difficult to pick (e.g., too crowded to pick, stuck on a corner or edge, has sharp / pointed parts that could damage the end effector's suction cups, is too heavy, etc.), it will walk away, go into safety mode, and trigger a message to a human operator to come and assist.

[0126] Alternatively, the robot may present pick decisions on a software interface to a remote operator who guides the robot to make the correct pick, where the pick is high-level guidance provided by a human (not direct remote control) and is performed by the robot's pick logic described above.

[0127] Alternatively, the robot will shove or push all unpickable packages into a "return or unpickable item chute" which will then send them to a human operator.

[0128] --If the flow of packages is too fast, the robot may trigger a handoff to a human. In this situation, the robot goes into a safety mode to stay out of the way of the human. The control system provides two options: (i) a human operator has enough space and can operate next to the robot to help temporarily pick some packages and reduce the overall flow, or (ii) the human operator physically moves the entire robot out of the way (e.g., down a chute) and performs the pick as normal. In both situations, the computer vision system keeps monitoring the human pick and uses that data to teach the robot to behave better when it encounters that type of situation.

[0129] In some embodiments, a single singulation station may let both human and robotic workers pick and singulate items. Humans may singulate items that humans are trained to pick, such as items that prove difficult to pick and place with autonomous robotic operation. In some embodiments, the system monitors the movements of the human worker and uses the robotic arm to pick, move, and place items on a trajectory that avoids approaching the human worker. Safety protocols ensure that the robot slows or stops its movement if a human gets too close.

[0130] 12 is a flow chart illustrating one embodiment of a process for detecting and correcting placement errors. In various embodiments, the process 1200 of FIG. 12 is performed by a control computer (such as control computer 212 of FIGS. 2A and 2B).

[0131] At step 1202, placement errors are detected. For example, one or more of images processed by a vision system, a force sensor on a robotic arm, a pressure sensor to detect loss of vacuum, etc. may be used to detect that an item has been dropped prior to placement. Alternatively, image data may be processed to determine that an item was not placed in its intended slot on the conveyor, that two items were placed in the same slot, that an item was placed in an orientation such that it cannot be scanned downstream, etc.

[0132] At step 1204, the controller determines a plan for downstream workers (such as another robotic arm and / or a human worker) to correct the detected error. At step 1206, the downstream workers are assigned to correct the error, and at step 1208, the global plan and / or any affected local plans are updated, as necessary, to reflect the utilization of the downstream workers in correcting the error. For example, the downstream workers may no longer be available or may not be immediately available to perform a local task assigned by the local scheduler. In various embodiments, once a resource (e.g., a robotic arm) is assigned to correct the upstream detected error, the local scheduler updates its local plan and assigned tasks to incorporate the assigned error correction task.

[0133] For example, if a robot places two packages in a slot that is meant to hold one, a downstream camera or other sensor can identify the error and share that information with the downstream robot. The downstream robot's control algorithm can deprioritize the pick in real time and instead pick an additional package to place in the empty slot (or into its own pile to be picked later and placed into the slots one by one). In some embodiments, a local barcode scanner can be used to scan the package, allowing the system to ensure that the package barcode is associated with the slot in which the package will ultimately be placed.

[0134] In some embodiments, a quality monitoring system is provided to detect misplacement, dropped items, empty slots expected to receive an item, slots containing more than one item, etc. In some embodiments, if more than one item (or more than other desired and / or expected number) is present in a slot of a compartmented conveyor belt or other output transport device, the quality monitoring system checks whether other slots are empty and takes corrective action to move the correct item from the slot with too many items to the empty slot. For example, a human worker or downstream robot may be tasked with picking the misplaced item from the slot with too many items and placing it in an empty slot.

[0135] In various embodiments, the techniques disclosed herein are used to provide a robotic singulation system that can operate in a fully autonomous mode in most cases.

[0136] Although the above embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. A system comprising: A communication interface; a processor connected to the communication interface; Equipped with The processor, receiving sensor data via the communication interface, the sensor data including image data related to a workspace; generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present within the workspace; determining a grasping strategy for each of at least a portion of the items; determining a probability of grasp success for each grasping strategy, the probability of grasp success being calculated based at least in part on learned data corresponding to past attempts to grasp similar items, the learned data including information regarding (i) a position of a portion of the item relative to one or more other items of the plurality of items, (ii) a degree of overlap of the items, and (iii) an orientation of the items; determining and executing a plan for autonomously operating a robotic structure to pick one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item, one at a corresponding location on a singulation transport structure; A system configured to disturb at least a portion of the plurality of items present in the workspace.

2. 2. The system of claim 1, The processor, determining whether the workspace includes an item whose corresponding grasp success probability meets a predetermined minimum threshold; in response to determining that the workspace does not include an item whose corresponding grasp success probability meets the predetermined minimum threshold, disturbing at least a portion of the plurality of items present in the workspace. The system is configured as follows:

3. 3. The system of claim 2, Disturbing at least a portion of the plurality of items present in the workspace includes the robotic structure disturbing the workspace or the at least a portion of the plurality of items present in the workspace.

4. 4. The system of claim 3, The system, wherein the robotic structure disturbing the workspace or at least a portion of the plurality of items present in the workspace includes using the robotic structure to poke, pull, or push one or more items to a different location in the workspace.

5. 2. The system of claim 1, The system, wherein the processor is configured to update the three-dimensional view, the grasping strategy, and the corresponding grasping success probability in response to an item being picked from the workspace and placed in the corresponding position on the singulation transport structure.

6. 2. The system of claim 1, The system, wherein the processor is configured to update the plan for autonomously operating a robotic structure to pick one or more items from the workspace in response to an item being picked from the workspace and placed in the corresponding position on the singulation transport structure.

7. 2. The system of claim 1, the robotic structure comprises a robotic arm, the robotic arm comprising a suction-based end effector configured to be used to grasp the one or more items, the suction-based end effector comprising two or more independently actuated suction cup sets.

8. 8. The system of claim 7, each suction cup set comprises one or more suction cups, and the processor is configured to simultaneously grasp a corresponding one of the items from the workspace using two or more of the separately actuated suction cup sets.

9. 9. The system of claim 8, said two or more of said independently actuated suction cup sets move said items together to a destination location of a first one of said items.

10. 9. The system of claim 8, one of the set of separately actuated suction cups releases the first item at a destination of the first item.

11. 9. The system of claim 8, at least one other of said separately actuated suction cup sets sequentially places the remaining items one by one into corresponding locations on said singulation transport structure.

12. 2. The system of claim 1, The singulation transport structure includes a partitioned conveyor.

13. 2. The system of claim 1, The processor, Receives sensor data; generating a three-dimensional view of at least a portion of the workspace using the sensor data; For each of at least a portion of items present in the workspace, determining a corresponding grasping strategy, and for each grasping strategy, determining a corresponding grasping success probability; determining and executing a plan for autonomously operating a robotic structure to pick a next one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item, one at a time, at a corresponding location on a singulation transport structure; The system is configured to repeatedly perform the steps of:

14. 14. The system of claim 13, The system, wherein the processor is configured to stop repetitive execution of the process in response to determining that no additional items remain in the workspace.

15. 14. The system of claim 13, The processor is configured to abort operation based at least in part on a determination that no additional items are present that can be grasped autonomously despite one or more items remaining in the workspace.

16. 2. The system of claim 1, The processor is configured to determine the probability of successful grasping based at least in part on one or more of item attributes of the item and a degree of overlap of the item by one or more other items.

17. 2. The system of claim 1, The processor is configured to determine one or more candidate grasping strategies for the item, calculate a probability of grasping success for each candidate grasping strategy, and select a best grasping strategy for the item using the respective calculated grasping success probabilities.

18. 2. The system of claim 1, The system, wherein the workspace comprises a chute or other receptacle, the robotic structure comprises one of a plurality of robotic structures associated with the workspace, and the processor is configured to coordinate operation of the robotic structures to pick and place items from the workspace.

19. 2. The system of claim 1, The system, wherein the workspace includes a first workspace included in a plurality of workspaces associated with the singulation transport structure, each workspace having one or more robot structures associated therewith, and the processor is configured to operate each of the robot structures associated with each workspace to pick items from the workspace and place the items one by one on the singulation transport structure.

20. 20. The system of claim 19, The system, wherein the processor is configured to coordinate operation of each of the robot structures associated with each workspace to maximize overall throughput in placing items one at a time onto the singulation transport structure.

21. 1. A method comprising: receiving sensor data via a communications interface, the sensor data including image data related to a workspace; generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present within the workspace; determining a grip strategy for each item of at least a portion of the items; determining a probability of grasp success for each grasping strategy, the corresponding probability of grasp success being calculated based at least in part on learned data corresponding to past attempts to grasp similar items, the learned data including information regarding (i) a position of a portion of the item relative to one or more other items of the plurality of items, (ii) a degree of overlap of the items, and (iii) an orientation of the items; determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item, one at a corresponding location on a singulation transport structure; A method comprising:

22. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for receiving sensor data via a communications interface, the sensor data including image data related to a workspace; computer instructions for generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present within the workspace; computer instructions for determining a grasping strategy for each item of at least a portion of the items and for each grasping strategy determining a corresponding grasping success probability, the corresponding grasping success probability being calculated at least in part based on learned data corresponding to past attempts to grasp similar items, the learned data including information regarding: (i) a position of the portion of the item relative to one or more other items of the plurality of items; (ii) a degree of overlap of the items; and (iii) an orientation of the item; computer instructions for determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item, one at a corresponding location on a singulation transport structure; A computer program product comprising:

Images