COORDINATION OF MULTIPLE ROBOTS TO COMPLY WITH THE WORKFLOW AND AVOID CONFLICTS.

Abstract

A robotic singulation system is disclosed. In various modalities, sensor data, including image data associated with a workspace, is received. The sensor data is used to generate a three-dimensional view of at least a portion of the workspace, where the three-dimensional view includes the boundaries of a plurality of items present in the workspace. A grasping strategy is determined for each of at least a subset of items, and for each grasping strategy, a corresponding probability of successful grasping is calculated. The grasping strategies and the corresponding probabilities of successful grasping are used to determine and implement a plan for autonomously operating a robotic structure to pick up one or more items from the workspace and place each item individually in a corresponding location on a singulation transport structure.

Description

This application claims priority from U.S. Provisional Patent Application No. 62 / 926,165 entitled COORDINATION OF MULTIPLE ROBOTS TO MEET WORKFLOW AND AVOID CONFLICTS filed on October 25, 2019, which is incorporated herein by reference for all purposes, and claims priority from U.S. Provisional Patent Application No. 62 / 993,579 entitled SINGULATION OF ARBITRARY MIXED ITEMS filed on March 23, 2020, which is incorporated herein by reference for all purposes. BACKGROUND OF THE INVENTION Parcel distribution centers and others may receive an arbitrary mix of items of various sizes, dimensions, shapes, weights, rigidity, and / or other attributes, often in a random, disordered mix. Each item may have machine-readable information, such as text and / or optically or otherwise encoded information, that can be read by machine and used to route the item, for example, through an automated sorting / routing system and / or processing. To read the information from a given item, in a typical approach, the items are separated from each other through a process known as singulation. 5 Typically, human workers have performed the sorting process manually. A combination of items arrives at a workstation, for example, via a hopper or other means of transport, and each of a group of one or more human workers manually separates the items and places them on a conveyor belt or similar device in a space defined for a single item. For each item, its destination (or at least the next transport leg) is determined by automatically reading the item's information, and the item is routed to a destination associated with the next leg, such as a bag, container, or other receptacle and / or a delivery vehicle or picking area associated with the next leg. 2. Manual singulation processes are labor-intensive and can be inefficient. For example, in a later step of the process, a human worker may have few locations in which to place individual items, because workers in previous steps in the process have filled many of the individual item locations. Collective output may be suboptimal. Using robots to perform separation is challenging due to the arrival of a disordered mixture of items at a workstation, the dynamic flow of items at each station and in general, and it can be difficult to automatically identify, grasp and separate (singleize) items using a robotic arm and end effector. BRIEF DESCRIPTION OF THE DRAWINGS The following detailed description and accompanying drawings disclose several embodiments of the invention. Figure 1 is a flowchart illustrating one modality of a process for receiving, sorting, and transporting items for distribution and delivery. 2o Figure 2Ά is a diagram illustrating one modality of a robotic singulation system. Figure 2B is a diagram illustrating one modality of a multi-station robotic singulation system. Figure 3A is a flowchart illustrating one modality of a process for picking and placing items for sorting. Figure 3B is a flowchart illustrating one modality of a process for picking and placing an item for sorting. Figure 4A is a diagram illustrating a normal vector I / O computation and a one-mode display of a singulation robotic system. Figure 4B is a flowchart illustrating one modality of a process for processing image data to identify and calculate normal vectors for items. Figure 5A is a flowchart illustrating one modality of a process for using image data to determine a plan for picking and placing items. Figure 5B is a block diagram illustrating one modality of a hierarchical ordering system in one modality of a robotic singulation system. Figure 5C is a flowchart illustrating one modality of a process for programming and controlling resources included in a robotic singulation system. Figures 6A to 6C illustrate an example of item flow through a feed hopper in one mode of a robotic singulation system. Figures 6D to 6F illustrate an example of 10 item flow through a feed hopper in one mode of a robotic singulation system. Figure 7A is a flowchart illustrating one modality of a process for modeling the flow of items 15 for picking and placing. Figure 7B is a flowchart illustrating one modality of a process for modeling the flow of items for picking and placing. Figure 7C is a flowchart illustrating one modality of a process for modeling the flow of items for picking and placing. Figure 8A is a block diagram illustrating a front view of one type of suction-based end effector. Figure 8B is a block diagram illustrating a bottom view of a suction-based end effector 802 from Figure 8A. Figure 8C is a block diagram illustrating in a front view an example of gripping several io items using the suction-based end effector 802 from Figure 8A. Figure 8D is a block diagram illustrating a bottom-up view of an example of multi-item gripping using the suction-based 802 end effector from Figure 8A. Figure 9 is a flowchart illustrating one modality of a process for picking up and placing items 20 using a robotic arm and end effector. Figure 10A is a diagram illustrating one modality of a robotic singulation system. Figure 10B is a diagram providing a close-up view of the multiview sensor array comprising cameras (or other sensors) 1010, 1012, and 1014 from Figure 10A. Figure 10C is a flowchart illustrating one modality of a process for grasping and examining items. Figure 11 is a diagram illustrating a modality of a multi-station robotic singulation system incorporating one or more human singulation workers. Figure 12 is a flowchart illustrating one modality of a process for detecting and correcting placement errors. DETAILED DESCRIPTION The invention can be implemented in numerous ways, 20 including as a process; apparatus; system; composition of matter; product of a computer program embodied by means of a computer-readable storage medium; and / or a processor, such as that processor configured to execute instructions stored and / or entered by a memory 25 coupled to the processor. In this specification, these implementations, or any other form the invention may take, may be referred to as techniques. In general, within the scope of the invention, the order of the steps in the described processes may be modified. Unless otherwise stated, any component, such as a processor or memory, described as being configured to perform a task, may be implemented as a general component that is temporarily configured to perform the task at a given time, or as a specific component manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as instructions from computer programs. A detailed description of one or more embodiments of the invention is presented below, along with accompanying figures illustrating its principles. The invention is described in relation to such embodiments but is not limited to any one embodiment. The scope of the invention is limited only by the claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Many specific details are set forth in the following description to enable a complete understanding of the invention. These details are presented by way of example, and the invention can be implemented according to the claims without some or all of these specific details. For the sake of clarity, the technical material known in the technical fields related to the invention has not been described in detail so as not to unnecessarily obscure the invention. A robotic system for singulation and / or sorting is disclosed. In various embodiments, a robotic system includes a robotic arm and an end effector used to pick items from a source pile / stream and place them onto a segmented conveyor or similar transport medium for sorting and routing to a final destination (e.g., physical / ultimate destination). In some embodiments, the picked items are individually placed into nearby bins or other receptacles. In some embodiments, multiple robots are coordinated to maximize collective throughput. In various embodiments, one or more robots may be employed at a singulation station. A system may include multiple stations. Human workers may be used at one or more stations.In several ways, the robotic system can be configured to invoke (request) the assistance of a human worker, for example, by teleoperating a robotic arm, manually completing tasks, etc., for example, to handle an item that the robot cannot handle in a fully automated way and / or an item that the robot drops, etc. Parcel carriers, postal services, delivery services, large retailers or distributors, and other companies and government entities that handle, transport, and deliver items to and from various locations typically receive large quantities of items from various origin locations, each to be delivered to one of a variety of destination locations. Machines exist for handling, sorting, and routing items, but to use automated readers and sorting equipment, items may need to be spaced from one another and / or placed in a certain orientation so the machine can read the label. Such spacing and orientation may need to be done during an item induction process at a sorting / routing facility and may be done in connection with a sorting or ordering process—for example, a process by which items to be delivered to various locations are sorted by general destination (e.g., region, state, city, postal code, street, street number, etc.). Automated readers, such as radio frequency (RF) tag readers, optical code readers, etc., 25 may need items to be separated from each other, a process sometimes called singulation, in order to reliably read a tag or code and for the system to associate the resulting information with a specific item, such as an item in a specific location on a conveyor or other structure or instrument. In a typical induction / sorting process in a package sorting operation, for example, individual packages of bulk stacks may be collected and placed on a moving conveyor or tilting tray sorting system. In most installations, this type of induction is entirely manual. A typical manual package induction / classification process may include one or more of the following cases: • A hopper of unsorted packages is filtered onto a sorting table adjacent to a conveyor-based sorting system. • A worker's job is to single out items on the conveyor or in the tray-based sorting system. • Workers ensure that each package fed into the sorter is oriented so that the shipping barcode (or other optical code, electronic label, etc.) can be read for sorting purposes (this orientation is usually determined by the scanning infrastructure in the facility). • Wait for an empty tray or slot to pass and ensure that only one package is placed in each slot or tray. In a typical manual induction / sorting process, manually (or mechanically) fed ramps carry packages of various shapes and sizes in bulk and various orientations. The packages may have different dimensions, shapes, rigidity, packaging, etc. Human workers typically take packages from a hopper feeding a station where each worker is stationed and place them one by one onto a divided or otherwise defined open segment of a conveyor. Finally, multiple workers at a station fill locations on one or more conveyors with individual packages to facilitate subsequent machine processing, such as reading the code or label and taking automated sorting actions based on that, such as routing each package to a location within the facility that is associated with a destination to which the package is to be delivered. The location may involve further sorting (e.g.,e.g., a more specific location of the destination within the facility) and / or pack / load the package for further shipment (e.g., truck or plane to another destination where further sorting and delivery will take place, loading onto a truck for local delivery, etc.). s Figure 1 is a flowchart illustrating one modality of a process for receiving, sorting, and transporting items for distribution and delivery. In the example shown, process 100 begins with an induction process 102, whereby items are fed to one or more workstations for singulation via the singulation process 104. In several modalities, the singulation process 104 is at least partially automated by a robotic singulation system as described in this document. The singulation process 104 receives stacks or streams of different items via the induction process 102 and generates a stream of separated items to a sorting / routing process 106. For example, the sorting process 104 may place items one by one onto a segmented conveyor or other structure that feeds items one by one to a sorting / routing machine.In some configurations, items are oriented so that the label can be read by a reader configured to read routing information (e.g., the destination address) and use that information to sort the item into a corresponding destination, such as a stack, container, or other group of items destined for the same next intermediate and / or final destination. Once sorted, groups of items destined for a common next final destination are processed by a transport process. For example, items may be placed in containers, loaded onto delivery or transport trucks or other vehicles, etc., for delivery to the next / final destination. Figure 2A is a diagram illustrating one modality of a robotic singulation system. In several modalities, the singulation process 104 of Figure 1 is performed at least partially by a robotic singulation system such as the system 200 in Figure 2A. In several embodiments, a robotic system comprising one or more robotic arms performs singulation / induction. In the example shown in Figure 2A, the system 200 includes a robotic arm 202 equipped with a suction-based end effector 204. Although in the example shown the end effector 204 is a suction-based end effector, in several embodiments one or more types of end effector may be used in a singulation system as described herein, including, but not limited to, a gripper-based end effector or other types of gripper actuators. In several embodiments, the end effector may be actuated by one or more suction, air pressure, pneumatic, hydraulic, or other means. The robotic arms 202 and 204 are configured to retrieve packages or other items arriving through the hopper or container 206 and place each item in a corresponding location on the segmented conveyor 208.In this example, items are introduced into hopper 206 from an input end 210. For example, one or more human and / or robotic workers can introduce items into the input end 210 of hopper 206, either directly or via a conveyor or other electromechanical structure configured to introduce items into hopper 206. In the example shown, one or more devices from among the robotic arm 202, the end effector 204, and the conveyor 208 are operated in coordination with the control computer 212. In various modes, the control computer 212 includes a vision system that is used to discern individual items and the orientation of each item based on image data provided by the image sensors, including in this example the 3D cameras 214 and 216. The vision system produces an output that the robotic system uses to determine strategies for grasping individual items and placing each one in a corresponding available defined location for automated identification and sorting, such as a split section of the segmented conveyor 208. In various forms, the robotic system as described in this document includes and / or performs one or more of the following tasks, for example, by operating a control computer such as the 212 control computer: • Computer vision information is generated by combining data from multiple sensors, including one or more 2D cameras, 3D cameras (e.g., RGBD), infrared, and other sensors to generate a three-dimensional view of a workspace that includes one or more sorting stations. • The robotic system coordinates the operation of multiple robots to avoid collisions, getting in each other's way, and competing to pick up the same item and / or place an item in the same destination location (e.g., a segmented part of the conveyor) as another robot. The robotic system coordinates the operation of multiple robots to ensure that all items are placed, and only one per slot / location. For example, if robot A drops an item, the system instructs robot B to pick it up; if an item is placed but with the wrong orientation, the same or another robot picks it up and adjusts it or moves it to another location; if two or more items are in a single destination slot, the subsequent robot station in the process takes one of those items from the conveyor and places it in its own location; and so on. • The system continuously updates the movement plan of each robot and of all robots to maximize collective performance. • If two robots are tasked with obtaining the same item independently, the system randomly chooses one to obtain that item and the other moves on to the next item (e.g., identify, select, determine the gripping strategy, lift, move according to the plan and place). • The conveyor movement and / or speed are controlled as needed to avoid empty locations and to maximize robot productivity (throughput). • In the event that an item is lost or dropped, the system assigns a robot or, if necessary, a human worker to pick it up and place it back into the robot's own retrieval source stack or, if possible or more optimal, into the next slot that opens on the conveyor. • The robots at the beginning of the process are controlled to intentionally leave some slots open 5 so that subsequent robots in the process can place items on the conveyor. • A fault that cannot be corrected by the same or another robot generates an alert seeking human (or robotic) intervention to resolve it. In several modalities, the arbitrary mix of items to be individualized may include packages, boxes, and / or letters of a variety of shapes and sizes. Some may be standard packages for which one or more attributes are known, and where others are unknown. Image data are used, in several modalities, to discern individual items (e.g., through image segmentation). The boundaries of partially occluded items can be estimated, for example, by recognizing an item as a standard or known type and / or by extending the boundaries of visible items to estimated logical extensions (e.g., two edges extrapolated to meet at an occluded corner). In some modalities, a degree of overlap (i.e., occlusion by other items) is estimated for each item, and this degree of overlap is taken into account when selecting the next item to attempt to grasp.For example, a score can be calculated for each item to estimate the probability of successful grasping, and in some modalities, the score is determined at least partially by the degree of overlap / occlusion of other items. Less occluded items are more likely to be selected, all other things being equal. In several modes, multiple 3D cameras and / or other cameras can be used to generate image data. A 3D view of the scene can be generated, and / or in some modes, a combination of cameras is used to view the scene from different angles. The camera that is least occluded, for example, with respect to a workspace, and / or one or more specific items in the workspace are selected and used to grasp and move one or more items. Multiple cameras serve many purposes in various ways. First, they provide a richer, complete 3D view of the scene. Second, they operate in concert to minimize errors due to package glare, as light reflecting off a package and the camera can disrupt its operation; in this case, a backup camera in a different location provides a backup. In some configurations, cameras can be selectively activated by a predictive vision algorithm that determines which camera has the best viewing angle and / or the lowest error rate for picking up a particular package; thus, each package is observed by the optimal camera. In some configurations, one or more cameras are mounted on a driven base, the position and orientation of which the system can change to provide a more optimal perception (e.g., view) of a package. Another purpose of the cameras is to detect any unforeseen errors in the robot's operation or any disruptions in the environment. Cameras mounted on the robot and those in the environment have different error profiles and accuracy levels. Robot cameras can be more accurate because they are rigidly attached to the robot, but their use is slower, as it requires the robot to slow down or stop. Cameras in the environment have a stable view and are effectively faster, since the robot can multitask and perform other tasks while a camera takes a picture. However, if someone moves or shakes the camera mount, it will become out of sync with the robot and cause many errors.The combination of images from robotic and non-robotic cameras (either occasionally or in a lost package) allows the robot to detect whether it is synchronized with the non-robotic cameras, in which case the robot can take corrective action and become more robust. In some configurations, a camera may not be rigidly mounted on a robotic arm, and in some such configurations, gyroscopes and / or accelerometers can be used in the cameras to filter or compensate for movement of the mounting base. With further reference to Figure 2A, in the example shown, the system 200 also includes an on-demand teleoperation device 218 that can be used by a human worker 220 via teleoperation to operate one or more devices from among the robotic arm 202, the end effector 204, and the conveyor 208. In some modalities, the control computer 212 is configured to attempt to grasp and place items in a fully automated mode. However, if after attempting to operate in fully automated mode, the control computer 212 determines that it has no (further) strategies available to grasp one or more items, in several modalities, the control computer 212 sends an alert to request assistance from a human operator via teleoperation, for example, if the human operator 220 uses the teleoperation device 218.In several configurations, the control computer 212 uses image data from cameras such as cameras 214 and 216 to provide a visual representation of the scene to the human operator 220 to facilitate teleoperation. For example, the control computer 212 can display a view of the stack of items on the ramp 206. In some configurations, segmentation processing is performed by the control computer 212 on the image data generated by cameras 214 and 216 to discern item / object boundaries. Masking techniques can be used to highlight individual items, for example, by using different colors. The operator 220 can use the visual display of the scene to identify the items to be picked and use the teleoperation device 218 to control the robotic arm 202 and the end effector 204 to pick the items from the hopper 206 and place each one in a corresponding location on the conveyor 208.In several modes, once the item(s) for which human intervention was requested have been placed on the conveyor, the system resumes fully automated operation. In several modes, in the event of human intervention, the robotic system observes the human worker (e.g., manual completion of the task, completion of the task using a robotic arm and end effector via teleoperation) and attempts to learn a strategy to complete the task (better) in the future autonomously. For example, the system might learn a strategy for grasping an item by observing the locations on the item where the human worker grasps it and / or by remembering how the human worker used the robotic arm and end effector to grasp the item via teleoperation. Figure 2B is a diagram illustrating one embodiment of a multi-station robotic singulation system. In the example shown, the robotic singulation system 5 of Figure 2A has been extended to include a plurality of singulation stations. Specifically, in addition to the robotic arm 202 configured to pick items from hopper 206 and place each item in a corresponding available and / or assigned location on the segmented conveyor 208, the system shown in Figure 2B includes three additional stations: robotic arms 230, 232, and 234 positioned and configured to pick / place items from hoppers 236, 238, and 240, respectively. Additional cameras 224 and 226 are included, in addition to cameras 214 and 216, to provide a 3D view of the entire scene, including each of the four stations / hoppers 206, 236, 238 and 240, as well as conveyor 208. In various modes, the control computer 212 20 coordinates the operation of the four robotic arms 202, 236, 238 and 240 and the associated end effectors, together with the conveyor 208, to pick up / place items from the hoppers 206, 236, 238 and 240 to the conveyor 208 in a manner that maximizes the collective performance of the system. Whereas in the example shown in Figure 2B, each station has one robotic arm, in various modalities two or more robots can be implemented in one station, operated under the control of an associated control computer, 5 such as control computer 212 in the example shown in Figure 2B, in a way that prevents the robots from interfering with each other's operation and movement and maximizes their collective performance, including avoiding and / or managing containment for picking up and placing the same item. In several modalities, a programmer coordinates the operation of a plurality of robots, for example, one or more robots working at each of a plurality of stations, to achieve the desired performance without conflict between robots, such as a robot placing an item in a location that the programmer has assigned to another robot. In various forms, a robotic system, such as the one disclosed in this document, coordinates the operation of multiple robots to pick up items one by one from a source container or conduit and place them in an assigned location on a conveyor or other device to move the items to the next stage of machine identification and / or sorting. In some embodiments, multiple robots can pick from the same hopper or other source receptacle. In the example shown in Figure 2B, for instance, robotic arm 202 can be configured to pick from either hopper 206 or hopper 236. Similarly, robotic arm 230 can pick from either ramp 236 or ramp 238, and robotic arm 232 can pick from either ramp 238 or ramp 240. In some embodiments, two or more robotic arms configured to pick from the same hopper can have different end effectors. A robotic singulation system as described here can select the most suitable robotic arm to pick and separate a given item. For example, the system determines which robotic arms can reach the item and selects one with the most appropriate end effector and / or other attributes to successfully grasp the item. Although stationary robotic arms are shown in Figure 2B, in several configurations one or more robots can be mounted on a mobile transport medium, such as a robotic arm mounted on a chassis configured to move along a rail, track, or other guide, or a robotic arm mounted on a mobile carriage of a chassis. In some configurations, a robotic instrument actuator other than a robotic arm can be used. For example, the end effector can be mounted and configured to move along a rail, and the rail can be configured to move on one or more axes perpendicular to the rail to allow the end effector to move to pick up, move, and place an item as described here. Figure 3A is a flowchart illustrating one modality of a process for picking and placing items for sorting. In several modalities, process 300 in Figure 3A is performed by the control computer, such as the control computer 212 in Figures 2A and 2B. In the example shown, in 302, the items to be picked from a hopper or other source or receptacle through which the items are received at a singulation station are identified. In some 15 modalities, image data from one or more cameras are used, for example, by a vision system or module comprising a control computer, to generate a 3D view of the stack or flow of items. Segmentation processing is performed to determine the boundaries and orientation of the 20 items. In 304, a robotic arm is used to pick items from the hopper. The items can be picked one at a time, or, in some modalities, multiple items can be grasped simultaneously.In 306, the grasped item(s) are each moved to a corresponding point on a segmented conveyor. If there are more items, an additional iteration of steps 302, 304, and 306 is performed, and successive iterations are performed until it is determined in 308 that there are no more items on the ramp (or other receptacle or source) to pick up and place. Figure 3B is a flowchart illustrating one modality of a process for picking and placing an item for sorting. In several modalities, the process in Figure 3B implements step 300 of Figure 3A. In the example shown, step 320 determines a plan and / or strategy for grasping one or more items. In several modalities, the plan / strategy is determined at least in part based on one or more image data points, for example, indicating the size, extent, and orientation of an item, and attributes that can be known, determined, and / or inferred about the item, such as classifying the item by size and / or item type. For example, in the context of a parcel delivery service, certain standard packaging might be used. A range of weights or other information about each type of standard package might be known.In addition, in some modalities, strategies for grasping items can be learned over time, for example, by means of a system that records and records the success or failure of previous attempts to grasp a similar item (e.g., same standard item / packaging type; similar shape, stiffness, dimensions, same or similar shape, same or similar material, position and orientation in relation to other items in the stack, extent of item overlap, etc.). In 322, the system attempts to grasp one or more items using the strategy determined in 320. For example, the end effector of a robotic arm can be moved to a position adjacent to the items to be grasped, according to the determined strategy, and the end effector can be operated according to the determined strategy to attempt to grasp the items. Step 324 determines whether the gripping was successful. For example, image data and / or force (weight), pressure, proximity, and / or other sensor data can be used to determine if the item was successfully gripped. If so, the items are moved to the conveyor in step 326. If not, processing returns to step 320, where a new gripping strategy is determined (if available). In some modes, if after a prescribed and / or configured number of attempts, the system fails to grasp an item, or if the system cannot determine an additional strategy to grasp the item, the system proceeds to identify and grasp another item, if one exists, and / or sends an alert to obtain help, such as from a human worker. Figure 4A is a diagram illustrating a normal vector computation and a display of one mode of a robotic singulation system. In various modes, the item boundaries and normal vectors are determined, and a visualization of the item boundaries and normal vectors is generated and displayed by a control computer comprising a robotic singulation system as described herein, such as the control computer 212 of Figures 2A and 2B. In the example shown, the 3D scene visualization 400 comprises the output of a vision system implemented in various modes. This example displays 15 normal vectors to the surfaces of the items. Normal vectors are used in some modes to determine a gripping strategy. For example, in some modes, the robot has a suction-type end effector, and normal vectors are used to determine the angle and / or orientation of the end effector to maximize the probability of a successful grip using suction. In some configurations, the vision system is used to discern and distinguish items by object type. The object type can be indicated on the display by highlighting each object in a color corresponding to its type. In some configurations, the object type information is used or can be used to determine and / or modify a gripping strategy for the item, such as increasing suction pressure or gripping force, reducing movement speed, using a robot with a particular end effector, weight capacity, etc. In some modes, additional information not shown in Figure 4A can be displayed. For example, in some modes, a best-grasping strategy and its associated probability of successful grasping are determined for each item and displayed next to it. The information displayed in some modes can be used to monitor system operation in a fully automated mode and / or to allow a human operator to intervene and quickly gain an overview of the scene and the available grasping strategies and probabilities, for example, to operate the robotic singulation system remotely. 2. Figure 4B is a flowchart illustrating one modality of a process for processing image data to identify and calculate normal vectors for items. In several modalities, process 420 can be performed to generate and display a visual representation of a 3D view of a scene, such as visualization 400 in Figure 4A. In several modalities, process 420 is performed by a computer, such as the control computer 212 in Figures 2A and 2B. In the example shown, 3D image data from one or more cameras, such as cameras 214, 216, 224, and / or 226 in Figures 2A and 2B, cameras mounted on robotic arms and / or end effectors, etc., is received in step 422. In various modes, image data from multiple cameras is fused to generate a composite 3D view of the scene, such as a stack or flow of items from which items are to be picked. In step 424, segmentation processing is performed to determine the boundaries of the items in 3D space. In step 426, for each item that has a sufficiently clear view, a geometric center and a normal vector are determined for each of one or more surfaces of the item, for example, the largest and / or most exposed (not obscured, for example, by another overlapping item) surface of the item. 2. In several modes, the segmentation data determined in 424 is used to generate a corresponding mask layer for each of a plurality of items. The respective mask layers are used in several modes to generate and display a visual representation of the scene in which the respective items are highlighted. In several modes, each item is highlighted in a color corresponding to an object type and / or other attribute (weight, class, softness, deformability, inability to pick up, for example, a broken package, porous surfaces, etc.) of the item. Figure 5A is a flowchart illustrating one modality of a process for determining a plan for collecting modalities, process 500, of using image data to and placing items. In several instances, Figure 5A implements it using a computer, such as the control computer 212 in Figures 5A and 5A. 2A and 2B. In the example shown, image data is received at 502. Image data can be received from multiple cameras, including one or more 3D cameras. The image data can be merged to generate a 3D view of the workspace, such as a stack or flow of items in a chute or other receptacle. Segmentation, object type identification, and vector normal calculation are performed at 504. In some modes, the segmentation data is used to discern item boundaries and generate item-specific mask layers, as described above. In 506, gripping strategies are determined for as many items as possible given the current view of the workspace. For each item, one or more gripping strategies can be determined, and for each strategy, a probability of successfully gripping the item is determined. In some modalities, a gripping strategy is determined based on item attributes, such as item type, size, estimated weight, etc. In some modalities, determining a gripping strategy includes deciding whether to attempt to pick up one item or multiple items, whether to use all end-effector actuators (e.g., using only one, two, or more suction cups or sets of suction cups), and / or selecting a combined gripping technique and corresponding speed from a set of calculated gripping techniques and speed combinations under consideration (e.g., one suction cup at 50% speed versus two suction cups at 80% speed).For example, in some modalities, a decision can be made based on the type of package to use one suction cup at 50% speed, such as to grasp a smaller item (two suction cups cannot be used) that may be heavy or difficult to grip securely with one suction cup (moving at 50% speed instead of 80%, so it doesn't fall). A larger item that is also relatively lightweight or easier to grip securely with two suction cups, on the other hand, can be gripped with two suction cups and moved more quickly. In some modes, you can change the 5-grab strategy if you move the object as part of a flowing stack. For example, depending on the strategy, the robot might push the object harder to lock it in place, while the suction cups can be activated independently in the order they touch the package to ensure a firm and stable grip. The robot will also match its speed, based on visual feedback or the depth sensor, to the flowing stack to ensure the package doesn't slip. Another optimization implemented in some modalities is changing the gripping strategy based on double item collection (on) (see also the next section) where a robot can adapt its strategy depending on whether it is trying to reuse vision data while selecting 2. Several items in a batch or sequentially. For example, a robot gripper with 4 suction cups might use 2 suction cups to pick up one object and the other 2 to pick up a second object. In this scenario, the robot would reduce its speed and also approach the gripping of the second object in a way that avoids collisions between the first object already being held and the surrounding stack of objects. In 508, the item-specific grasping strategies and probabilities determined in 506 are used to determine a plan for grasping, moving, and placing items according to the probabilities. For example, a plan can be determined for grasping the following items in a specific order, each according to the grasping strategies and corresponding success probabilities determined for that item and strategy. In some modalities, changes in location, arrangement, orientation and / or flow of the 15 items resulting from grasping and removing previously grasped items are taken into account when formulating a plan, in 508, to grasp multiple items in succession. In several modalities, the 500 process of Figure 5 is a 2<) continuous and ongoing process. As items are selected and placed from the pile, subsequently received image data are processed to identify and determine strategies for grasping more items (502, 504 and 506), and a plan for picking / placing items is updated based on grasping strategies and probabilities (508). Figure 5B is a block diagram illustrating one mode of a hierarchical sorting system in one mode of a robotic singulation system. In several 5 modes, the hierarchical programming system 520 of the Figure 5B is at least partially implemented on a computer, such as the control computer 212 of Figures 2A and 2B. In the example shown, the hierarchical scheduling system 520 includes a global scheduler 522 configured to optimize performance by coordinating the operation of a plurality of robotic singulation stations and a segmented conveyor (or similar structure) on which the robotic singulation stations are configured to place items. The global scheduler 522 can be implemented as a processing module or other software entity running on a computer. The global scheduler supervises and coordinates the work among the robotic singulation stations at least in part by monitoring and, as required, controlling and / or providing information to a plurality of robotic singulation station schedulers 524, 526, 528 and 530. Each of the robotic singulation station programmers 524, 526, 528, and 530 is associated with a robotic singulation station and each controls and coordinates the operation of one or more robotic arms and associated end pickers to collect items from a corresponding hopper or other item receptacle and to place them individually onto a segmented conveyor or similar structure. Each of the robotic singulation station programmers 524, 526, 528, and 530 is associated with a corresponding set of one or more station sensors 532, 534, 536, and 538, respectively, and each uses the sensor data generated by its station's sensors to perform automated singulation at its robotic singulation station. In some modalities, each implements and performs process 500 of Figure 5A. In various modes, each of the programmers 524, 526, 528 and 530 of the robotic singulation station informs the global programmer 522 of one or more of the data from the image sensor and / or other station; object identification, gripping strategy and probability of success data; selection / placement plan information; and expected information on item singulation performance.The global program 522 is configured to use the information received from the programmers 524, 526, 528, and 530 of the robotic singulation station, together with the sensor data received from other sensors 540, such as cameras pointing at the segmented conveyor and / or other parts of the workspace not covered or covered well or completely by the station's sensors: to coordinate the work of the respective robotic singulation stations, each under the control of its station-specific programmer 524, 526, 528, or 530, and to control the operation (e.g., speed) of the segmented conveyor via the conveyor controller 542, to optimize (e.g., maximize) the collective singulation performance of the system. In various ways, the global scheduler 522 employs one or more techniques to optimize the use of a plurality of robots comprising the robotic singulation system to perform singulation, for example, to maximize overall throughput. For instance, if there are four robots in sequence, the lead robot (or another higher-level robot) can be controlled to place packages in a way that leaves open slots so that a robot at a lower level is not waiting for an empty slot. This approach has an impact because subsequent robots wait an unknown / random amount of time due to package flow, etc. Therefore, a naive strategy (say, the lead robot placing itself in every four empty slots) may not optimize collective throughput.Sometimes, it may be better for the lead robot to place two or three packages in successive slots sequentially if its packages are not flowing, but generally, the system makes such decisions with knowledge of the state and flow at each station. In some configurations, the optimal strategy for leaving slots open for robots at a lower level is based on an advance request for an open slot by the robot at the lower level (based on its package flow, for example). In some configurations, information from the local station scheduler is used to anticipate the maximum throughput of each station and to control conveyor speeds and how many slots are left empty by robots at a higher level to ensure that robots at a lower level have access to empty slots in proportion to the rate at which they can (at that moment) pick / place.In some configurations, when the segmented conveyor is full due to bottlenecks in the subsequent sorting process, the robotic singulation system as described herein can pre-separate one or more packages, for example, within their corresponding hopper or in a nearby preparation area, while simultaneously tracking the position of each singulated package in advance. Once empty spaces become available on the segmented conveyor, the system / station moves the pre-separated packages to the segmented conveyor individually and in rapid succession, without requiring additional vision processing time. In some scenarios, the presence of humans working alongside robots impacts the positioning and coordination strategy of multiple robots, as the robots, or the associated computer vision or other sensor system, must now also observe what the humans are doing and adapt the robots' positions in real time. For example, if a human takes over a conveyor belt slot that was programmed for a robot, the system must adjust its global and local programs / plans accordingly. In another example, if a human interrupts a robot's selected package and causes it to be registered as unselected, the system adapts to correct the error.Or, if a human corrects a robot's mistakes when picking (the robot was instructed to place a 15 package in slot A, but accidentally placed it over slot A and the adjacent slot B; and the human places it in slot B even though the system memory says the package is in slot A), the system should observe the human action and adjust the robot's subsequent actions. In several ways, global scheduler 522 can cause a station to operate more slowly than its maximum possible throughput at any given time. For example, global scheduler 522 can explicitly instruct the scheduler of the local station (e.g., 524, 526, 528, or 530) to slow down, and / or it can reduce the number of slots available for the local station, either by explicitly allocating fewer slots to the station or indirectly by allowing upstream stations to occupy more slots. Figure 5C is a flowchart illustrating one modality of a process for scheduling and controlling resources within a robotic singulation system. In several modalities, process 560 in Figure 5C is implemented by a global I / O scheduler, such as the global scheduler 522 in Figure 5B. In several configurations, items arrive via a hopper and / or conveyor. The hopper / conveyor is fed, for example, by humans, robots, and / or both, and items may arrive in batches. With each addition to the flow at the inlet end, items may slide or flow down / along the chute or conveyor to an end where one or more robotic arms are positioned to perform the separation, for example, by grasping individual items and moving them one by one to the corresponding single-item location on an outlet conveyor. In several modalities, images from one or more 3D cameras or other cameras are received and processed. The flow of items down / through the chute / conveyor feeding a singulation station is modeled based on the image data. The model is used to predict a moment of relatively small or no movement, and at that moment, the image data is used to identify and grasp an item. In some modalities, the flow model is used to ensure that the robotic arm is not in a position that obscures the camera(s) at the moment of relatively small or no flow. In some modalities, the system is configured to detect regions of relatively small or no flow within a larger flow. For example, the flow within distinct, predefined regions of a chute or other conveying source or receptacle can be analyzed.The system can select items for singulation of areas of scarce / stable and / or zero flow and avoid for the time being regions where a less stable flow occurs. In some configurations, the system identifies distinct areas of item flow and guides the robot to collect the items in those areas. In some configurations, the system predicts the movement of items over / through the hopper or other conveying medium or receptacle, for example, using a flow model calculated using successive images from multiple cameras mounted on and around the hopper. The flow model is used to predict a future position of one or more items as the items flow down the chute or other conveying structure. At each point in time, an image is captured and / or an attempt is made to capture an image at a time / location where the item is expected to be, according to the flow model. In some configurations, real-time segmentation is performed. For example, segmentation is performed at 30–40 milliseconds and / or some other speed that is as fast as the 3D camera's frame rate.In some configurations, real-time segmentation results are used to track and / or model the flow of an individual item through the hopper. A future item position is predicted based on the item's specific model / movement, and a plan and strategy for grasping the item at the future location and time are autonomously determined and executed. In some configurations, the target item's position is continuously updated. In some configurations, a robotic arm trajectory can be updated in real time, while the arm is moving to grasp an item, based on an updated predicted future item position. In some modes, if the observed item flow exceeds a threshold speed, the system waits a predefined amount of time (e.g., 100 milliseconds) and rechecks the image data and / or flow speeds. The system may repeat and / or vary the duration of these waits until a sufficiently stable flow condition is observed to ensure a high probability of successful grasping. In some modes, different areas of the hopper may have different allowable speed thresholds. For example, if an area is far from the actual location where the robot is collecting, a higher movement speed is tolerated in some modes, as the unstable flow in that region is not expected to disrupt the robot's current operation. For instance, a package falling (at high speed) onto the top of the hopper may not be a concern if the robot is collecting from the bottom. However, if the falling items were to approach the picking area, it might not be tolerable, as they could obstruct other objects or move them upon impact. In some modes, the system detects the flow by region and / or location relative to an anticipated picking area and tolerates a less stable flow in the eliminated area (sufficiently distant) from the area or areas where the robot expects to perform the next picking. In the example shown in Figure 5C, in 562, the collective throughput, the observed and / or estimated (locally programmed) throughput of the local singulation station, and the overall and station-specific error rates are monitored. In 564, the conveyor speed and the local station speeds are adjusted to maximize the collective production network or errors. In 566, conveyor slots are assigned to respective stations to maximize net throughput. Although conveyor slots are explicitly assigned in this example, in some modalities, station speeds are controlled to ensure that stations at a lower level have slots available for placing items, without (necessarily) pre-assigning specific slots to specific stations. Processing continues (562, 564, 566) while any station has items remaining to be picked up / placed (568).Figures 6A to 6C illustrate an example of item flow through a feed hopper in one embodiment of a robotic singulation system. In several embodiments, the flow of items through a hopper or other receptacle is modeled. The flow model is used in several embodiments to determine strategies for grasping items from the flow. In some embodiments, the modeled and / or observed flow can be used to perform one or more of the following: to determine a grasping strategy and / or a plan for grasping an item at a future location to which it is expected to flow; to determine grasping strategies for each of a plurality of items; and to determine and implement a plan for grasping a sequence of items, each to be grasped at a corresponding future position determined at least in part based on the flow model.to ensure that a robotic arm is in a position that avoids obscuring the view of an item at a future time in a location where items are anticipated based on the model to be located and from which it is planned to pick; and wait, for example, a calculated (model-based) or prescribed amount of time, or to allow the flow to become more stable (e.g., slower movement, items moving mainly in a uniform direction, minimal change or low rate of change of orientation, etc.). With reference to Figures 6A to 6C, in the example shown, the flow model exhibits a mostly stable arrangement of items that, according to the model, will remain relatively stable / uniform as items 602 and 604 are removed from the flow / stack. In various configurations, the 2<) Model information illustrated in Figures 6A to 6C would be used, potentially with other information (e.g., available grasping strategies, required / allocated station throughput, etc.), to determine and implement a plan to pick up and place items 602 and 604 in succession, each from a location that the model indicates it is expected to be at the time it is scheduled to make the grasp. Figures 6D through 6F illustrate an example of item flow through a feed hopper in one modality of a robotic singulation system. In this example, the model indicates that the stack moves with a relatively slow, steady flow, but will become disorganized (or be observed to become disorganized) once item 602 has been picked. In several modalities, the model information, as shown in Figures 6D through 6F, can be used to determine whether to pick item 602 individually, but then wait for the stack / flow to stabilize before determining and implementing a grasping strategy and a plan for picking and placing other items from the stack / flow. In some modalities, the flow may be detected as unstable enough to risk one or more items falling or flowing out of the trough or other source receptacle before they can be picked and separated.In some modalities, in such a circumstance, the system can be configured to implement a strategy to use the robotic arm and / or end effector to stabilize the flow, for example, by blocking the bottom of the conduit to prevent an item from overflowing, using the end effector and / or robotic arm to block or slow down the flow, for example by placing the arm transversely to the flow, etc. Figure 7A is a flowchart illustrating one modality of a process for modeling the flow of items for picking and placing. In several modalities, the process in Figure 7A is performed by a computer configured to model the flow of items, for example, through a chute, and use the model to pick and place items, as in the examples described earlier in relation to the Figures 6A to 6F. . In several modalities, process 700 is performed by a computer, such as the control computer 212 in Figures 2A and 2B. In some modalities, process 700 is performed by a robotic singulation station programmer, 15 such as programmers 524, 526, 528 and 530 in Figure 5B. In the example shown in Figure 7A, image data is received in 702. In 704, item segmentation and identification (e.g., type) are performed. In 706, the flow of items through a hopper or other receptacle is modeled. (While in this example steps 704 and 706 are performed sequentially, in some modalities these steps are performed and / or can be performed in parallel.) In 708, the model is used to determine 25 grasping strategies and a plan for grasping the next n items from the stack / flow. In 710, the plan is executed. If in 712 it is determined that the attempt is not entirely successful (e.g.If one or more items could not be picked, the item was not in the expected location, the flow was interrupted or was not as expected, or if there are more items to be picked (714), an additional iteration of steps 702, 704, 706, 708 and 710 is performed, and successive iterations are performed until it is determined in 714 that there are no more items to be picked. In several modes, the system can take a picture and identify two (or more) objects to pick up. The system picks up and moves the first object; then, instead of performing a complete recalculation of the scene to find the next package to pick, the system simply observes whether the second package has moved. If not, the system selects it without performing a complete recalculation of the scene, which generally saves a significant amount of time. In various modes, the time savings come from the latency of one or more sensor readouts, image acquisition time, system computation time, resegmentation, masking, package stack sorting calculations, and finally, control and planning. If the second item is not where expected, the system recalculates the entire scene to find the next package to pick up. In some configurations, a robot can use vision, depth, or other sensors to identify two grabable packages that the robot's control algorithm judges to be far enough apart not to interfere with each other's selections (by destabilizing the stack and causing it to flow, or by bumping into the other package, etc.). Instead of repeating the sensor processing pipeline after picking up the first package, the robot can proceed directly to picking up the second package. That said, statistically, there is a risk that the control algorithm's prediction is incorrect or that the stack has shifted due to some unforeseen event.In this scenario, the controller can select more carefully and use a predicted gripping suction cup activation model (or other predicted sensory models, e.g., force, pressure, and other types of sensing modalities) to test whether the selected package matches the previous prediction (without having to redo the vision calculations). 2. Figure 7B is a flowchart illustrating one modality of a process for modeling the flow of items for picking and placing. In several modality, process 720 in Figure 7B is performed by a computer configured to model the flow of items, for example, through a chute, and to use the model for picking and placing items, as in the examples described above in relation to Figures 6A through 6F. In several modality, process 720 is performed by a computer, such as the control computer 212 in Figures 2A and 2B. In some modality, process 720 is performed by a robotic singulation station programmer, such as programmers 524, 526, 528, and 530 in Figure 5B. In the example shown in Figure 7B, image data is received and processed at 722. At 724, items are picked / placed based at least in part on the model, for example, as in process 700 of Figure 7B. If an indication is received at 726 and / or it is determined that the flow has become unstable, the system is stopped at 728, for example, for a random, prescribed, and / or configured interval, after which processing resumes and continues unless / until there are no more items to pick / place, after which process 720 ends. Figure 7C is a flowchart illustrating one modality of a process for modeling the flow of items for picking and placing. In several modalities, the process 740 in Figure 7C is performed by a computer configured to model the flow of items, for example, through a chute, and use the model to pick and place items, as in the examples described earlier in relation to the Figures 6A to 6F. In several modalities, process 740 is performed by a computer, such as the control computer 212 in Figures 2A and 2B. In some modalities, process 740 is performed by a robotic singulation station programmer, such as programmers 524, 526, 528, and 530 in Figure 5B. In the example shown in Figure 7C, a condition is detected in 742 where no item can currently be grasped. For example, the system may have attempted to determine grasping strategies for the items in the stack, but determined that due to flow velocity, disorder, orientation, overlap, etc., there is no item for which a grasping strategy has a probability of success greater than a prescribed minimum threshold. In 744, in response to the determination in 742, the system uses the robotic arm to attempt to change the state of the stack / flow in a way that makes a grasping strategy available. For example, the robotic arm may be used to gently push, pull, shove, etc., one or more items into different positions in the stack.After each push, the system can reassess, for example, by recalculating the 3D view of the scene to determine if a viable gripping strategy is available. If it is determined at 746 that a grip (or multiple grips, each for a different item) has become available, then autonomous operation resumes at 750. Otherwise, if after a prescribed number of attempts to change the stack / flow state a viable gripping strategy is not available, then at 748 the system obtains assistance, for example, from another robot and / or a human worker, the latter through teleoperation and / or manual intervention such as moving items in the stack and / or manually picking / placing items until the system determines that autonomous operation can resume. io Figure 8A is a block diagram illustrating a front view of one embodiment of a suction-based end effector. In various embodiments, the end effector 802 can be used as the end effector of a robotic arm comprising a robotic singulation system 15, such as the end effector 204 of Figure 2A. Figure 8B is a block diagram illustrating a rear view of the suction-based end effector 802 of Figure 8A. 2. Referring to Figures 8A and 8B, in the example shown, the end effector 802 includes four suction cups 804, 808, 812, and 814. In this example, vacuum can be applied to suction cups 804 and 812 through hose 806, and vacuum can be applied to suction cups 808 and 814 through hose 807. 810. In several embodiments, the suction cup pairs can operate independently (i.e., a first pair comprising suction cups 804 and 812, and a second pair comprising suction cups 808 and 814). For example, a single item, such as a smaller and / or lighter item, can be grasped using only one or the other of the suction cup pairs. In some embodiments, each pair can be used to grasp a separate item at the same time, allowing two (or in some embodiments more) items to be picked up / placed simultaneously. Figure 8C is a block diagram illustrating, in a front view, an example of gripping multiple items using the suction-based end effector 802 from Figure 8A. Figure 8D is a block diagram illustrating, in a back view, an example of gripping multiple items using the suction-based end effector 802 from the Figure 8A. In this example, the respective pairs of suction cups have been actuated independently, each to grip one of the two items shown being grasped in the example shown. In various configurations, more or fewer suction cups and / or more or fewer independently actuated assemblies of one or more suction cups can be included in a given end effector. Figure 9 is a flowchart illustrating one embodiment of a process for picking and placing items using a robotic arm and an end effector. In several embodiments, the process 900 in Figure 9 can be implemented by a computer, such as the control computer 212 in Figures 2A and 2B. When using a suction-based end effector, such as the 802 end effector, applying too much pressure to ensure suction cup attachment can damage items and / or fragile packaging. Additionally, sensor readings may vary depending on the type of packaging. For example, suction sensor readings may differ when gripping corrugated cardboard boxes (air leakage from slots / paper) compared to plastic bags. In some models, if the package type is known, the sensor readings are evaluated and / or adjusted accordingly. In some models, if the sensor reading associated with a successful grip of a given package type is not achieved on initial attachment, additional force and / or suction can be applied to achieve a better grip. This approach increases the likelihood of a successful grip and is more efficient than determining that the grip has failed and starting over. In the example shown in Figure 9, at step 902 a robotic arm is used to approach and attempt to grasp an item according to a predetermined and selected grasping strategy. At step 324, it is determined whether the grasp was successful. For example, one or more force (weight) sensors, suction / pressure sensors, and image data can be used to determine if the grasp was successful. If so, at step 906 the item is moved to its assigned slot and / or the next available slot. If it is determined at step 904 that the grasp was unsuccessful, and the system determines at step 908 that a new attempt should be made (for example, fewer than the prescribed maximum number of attempts yet to be reached, the item is not determined to be too fragile to apply more force, etc.).Then, at step 910, the system adjusts the force applied to engage the item (and / or, in various modes, one or more of the gripping positions, i.e., the position and / or orientation of the robotic arm and / or the end effector) and attempts to grasp the object again. For example, the robotic arm can be used to push the suction cups into the item with a slightly greater force before engaging the suction, and / or a greater suction force can be applied. If it is determined in step 908 that no further effort should be made at this time to grasp the item, the process advances to step 912-25, where it is determined whether other items are available to grasp (e.g., other items are present in the hopper and the system has a grasping strategy for one or more of them). If so, in step 914 the system moves on to the next item and performs an iteration of step 902 and the following steps 5 with respect to that item. If no item remains for which a grasping strategy exists, then in step 916 the system obtains assistance, for example, from another robot, a human via teleoperation, and / or a human via manual intervention. 1(1 In several modalities, the number and / or nature of additional attempts to grasp an item, as in steps 908 and 910, can be determined by factors such as the item or item type, the characteristics of the item and / or packaging determined by probing the item with the robotic arm and end effector, the attributes of the items determined by a variety of sensors, etc. 2() Figure 10A is a diagram illustrating one modality of a robotic singulation system. In the example shown, the robotic singulation station 1000 includes a robotic arm 1002 that operates under the control of a control computer (not shown in Figure 10A) to pick up items from a hopper or other receptacle 1004, such as item 1006 in the example shown, and place it individually onto conveyor 1008. In several modalities, the robotic singulation station 5 1000 is controlled by a control computer configured to use a multi-view sensor array, comprising cameras (or other sensors) 1010, 1012 and 1014, in this example, to scan addresses or other local routing information, at station 1000, at least under certain circumstances. Figure 10B is a diagram providing a close-up view of the multiview sensor array comprising cameras (or other sensors) 1010, 1012, and 1014 of Figure 10A. In several configurations, sensors such as cameras (or other sensors) 1010, 1012, and 1014 are positioned to read routing information (e.g., text address, optical code, or other) regardless of the orientation of the package as it is placed by the robot on the outfeed conveyor. For example, if the package label is positioned under a scanner through which the package slides, the scanner reads the label and associates the sorting / routing information with the corresponding location on the outfeed conveyor. The challenge of picking up items from stacks (which may be flowing) and placing them on a singulation conveyor is that the singulation conveyor needs to associate a package barcode with each package in a conveyor belt slot. Depending on the orientation of the package in the bulk stack from which the robot is picking, the robot may encounter a situation where the barcode is facing down. In some embodiments, the robotic singulation system as described herein can be used to move or otherwise flip an item to a position where the barcode or other label or routing information is facing up. In some embodiments, the robotic arm uses its movements to flip the package by gravity-controlled sliding, if the end effector is a gripper (e.g., a grabber).To grasp one end and allow the item to begin rotating by sliding / gravity before releasing it, or a controlled sequence of suction release on multiple suction cups and gravity to reorient a package (e.g., releasing suction on the suction cups at one end of the effector while continuing to apply suction at the other end, to initiate rotation about an axis around which the item will be turned). However, turning the package over before placing it on the conveyor belt can damage the package, may not work, and / or may be time-consuming and require another hand (e.g., another robot), which is costly. In some embodiments, a system such as the one described herein can recognize that an item is of a type that may require actions to ensure that the label can be read.For example, it may be necessary to place a package in a polyethylene (poly) or other plastic bag in such a way that the packaging is flat and / or smoothed so that downstream scanners can read the label. In some embodiments, a robotic arm may be used to smooth and / or flatten the packaging, for example, after placement. In some embodiments, the item may be dropped from a prescribed height to help flatten the packaging sufficiently for reading. In several embodiments, the robot may flatten plastic bags by dropping them into the hopper or conveyor before picking them up and / or reading them at the station, using actuated suction cups to stretch the bags after picking them up, performing bimanual operations (two robotic arms) to unwrinkle deformable items, etc. In some embodiments, a blower or other mechanism may be used to smooth the package after placement.The blower can be mounted independently / stationary or integrated with / in the robotic arm. In some configurations, a multi-axis barcode scanner or sensor (or other device) is placed further along the 1008 conveyor belt run, capable of scanning barcodes on the top or bottom. However, this only works if the conveyor belt has a transparent bottom or if there is a sequencing operation in which the package passes through a transparent (or open) slot with a barcode reader facing upward through the slot. In several modes, if a package cannot be easily placed on conveyor 1008 with the label facing upwards so that it can be easily scanned by a top scanner, followed by a sensor array at the singulation station, such as cameras (or other sensors) 1010, 1012, and 1014 in the example shown in Figures 10A and 10B. In some modes, if the top of the package is visible, a barcode or other address information can be read as part of the computer vision scanning process and / or using a camera mounted on the robotic arm. 1002 and / or the end effector. If the barcode is on the side or bottom or is covered, cameras (or other sensors) 1010, 1012 and 1014 are used to scan the package as the robot picks it up and moves it to the conveyor belt or bin. 5 • To do this correctly, in some modes, the robot modifies its controller and motion plan to ensure that the package is scanned on the fly, which may require positioning the object in an optimal way to be seen by a barcode scanner while simultaneously restricting the path of movement so that the object lands in an empty slot. • This is particularly difficult for scanning barcodes on the bottom or sides, as the barcode readers must be positioned in an optimized configuration to simplify the robot's motion planning task and ensure that the robot can scan and place quickly. • The configuration of the scanners and the robot's movement plan are dynamically affected by the package itself, as the size and weight of the package may require the object to be positioned differently as the robot attempts to scan it with a barcode (see also the grip / speed optimization based on package type section for examples of how the robot should adapt its movement). The height of a package is generally unknown when it is removed from a dense stack, as cameras or other sensors cannot see the bottom of the stack. In this case, the robot uses a geometric or machine learning model to predict the object's height. If the estimated height is too low, the robot will hit the barcode scanner with the object. If the estimated height is too high, the robot will be unable to scan the object. One side of the error spectrum is catastrophic (the item hits the scanner), while the other simply results in a replanned trajectory. In several modes, the robot's control algorithm prefers the safer side and assumes the object is taller to avoid collisions. If the object is too far away, the robot can gradually move it closer to the barcode scanner with a human-like jerking motion to allow it to be scanned. Figure 10C is a flowchart illustrating one mode of a process for grasping and examining items. In several modes, process 1020 in Figure 10C is performed by the control computer configured to control the robotic singulation station, such as station 1000 in Figure 10A. In the example shown, the robotic arm is used at 1022 to approach and grasp an item. An image of the item's top surface is captured as the item is approached and / or grasped. In various modes, a camera pointed at the item / stack, such as a camera mounted near the station and / or on the robotic arm and / or end effector, can be used to capture the image. At step 1024, the image is processed to determine if the address tag or other routing information is on the top surface. If so, at step 1026 the item is placed on the conveyor with that side facing up. If the label is not on the top surface as it is grasped (1024), in 1028 the system uses one or more local sensors to attempt to locate the label. If the label is found and is located on a surface such that the robotic separation system can place the item on the conveyor with the label facing upwards, then in 1030 it is determined that a local scan is not necessary, and the package is rotated and placed on the conveyor in 1026 so that the label faces upwards. If, on the other hand, the label cannot be found or the package cannot be rotated to place it, then it is determined in 1030 that the label must be scanned locally. In such a case, in 1032 the label is scanned locally, for example, using a multi-axis sensor array such as the cameras 1010, 1012, and 1014 of the Figures 10A and 10B. The package is placed on the conveyor belt in 1034, and in 1036 the routing information determined by scanning the label locally is associated with the slot or other segmented location on the conveyor where the item was placed. ,S Later in the process, the routing information determined by locally scanning the item's label at the robotic singulation station is used to classify / route the item to a determined intermediate or final destination based at least in part on the routing information. Figure 11 is a diagram illustrating one modality of a multi-station robotic singulation system that incorporates one or more human singulation workers. In the example shown, the robotic singulation system 1100 includes the elements of the system in Figure 2B, except that at the leftmost station (as shown), the robotic arm 202 has been replaced by a human worker 1102. In some configurations, a robotic system includes one or more human single-unit workers, for example, at a last or other subsequent station, as in the example shown in Figure 11. The system and / or the human workers deliver hard-to-separate items (e.g., large flexible bags) to the workstation with the human worker. The robotic system leaves gaps open for the human worker to fill. In some configurations, each singulation station includes space for one or more robots and one or more human workers, for example, to work together at the same station. The human worker's location provides sufficient space for them to perform the singulation in an area where the robotic arm will not extend. In several configurations, the human worker can increase the robotic arm's throughput, correct incorrect placements or orientation (for example, when scanning a label), ensure that conveyor slots not already occupied by a previous robot or another worker are filled, and so on. Given that the singulation workspace is usually limited, the controller (e.g., the control computer) handles the following situations in 2 ​If the robot decides that the flow of packets is too fast or that some packets are not feasible to grab, it moves aside, activates safety mode, and sends a message to a human operator to come and help it. If the robot decides that the packages are difficult to pick up (e.g., too bunched together to pick up, or stuck in a corner or edge, have sharp / pointed features that could damage the 5 suction cups of the end effector, are too heavy, etc.), it moves aside, activates safety mode, and sends a message to a human operator to come and help. Alternatively, it presents a selection decision on an iO software interface to a remote operator who guides the robot to make the correct selection; the selection is a high-level guidance provided by a human (not direct teleoperation) and executed by the robot's selection logic above. Alternatively, the robot pushes all uncollectible packages into a return hopper or uncollectible items hopper, which then directs them to a human handler. 2. If the package flow is too high, the robot may trigger a human backpedal. In this situation, the robot enters safety mode, moving out of the human's path. The control system is prepared for two options: (i) a human operator has sufficient space and can work alongside the robot to temporarily assist in picking some packages and reduce the overall flow; or (ii) a human operator physically moves the entire robot out of the way to an unobstructed location (e.g., under the hopper) and performs the picking normally. In both situations, the computer vision system continues to monitor the human operator's picking and uses that data to help the robot learn to perform better in that type of situation. In some configurations, a single sorting station can have both human and robotic workers picking and sorting items. Humans can select items they are trained to pick, such as items known to be difficult to pick and place, for autonomous robotic operation. In some configurations, the system monitors the human worker's movement and uses the robotic arm to pick, move, and place items along paths that avoid getting too close to the human worker. Safety protocols ensure that the robot slows down or stops if it gets too close to the human. Figure 12 is a flowchart illustrating one mode of a process for detecting and correcting placement errors. In several modes, process 1200 in the Figure is performed by the control computer, such as control computer 212 in Figures 2A and 2B. In 1202, a placement error is detected. For example, one or more images processed by the vision system, force sensors on the robotic arm, pressure sensors that detect a loss of vacuum, etc., can be used to detect that an item has fallen before being placed. Or, the image data can be processed to determine that an item was not placed in the intended slot on the conveyor, or that two items were placed in the same slot, or that an item was placed in an orientation that will prevent it from being scanned later, etc. In step 1204, the controller determines a plan for a downstream worker, such as another robotic arm and / or a human worker, to correct the detected error. In step 1206, the downstream worker is assigned to correct the error, and in step 1208, the affected global and / or local plans are updated, as needed, to reflect the downstream worker's use for error correction. For example, the downstream worker may no longer be available, or may soon be available, to perform a local task assigned by the local scheduler. In several modes, when the resource (for example, a robotic arm) is assigned to correct a previously detected error in the process, the local scheduler will update their local plan and assigned tasks to incorporate the assigned error correction task. In another example, if a robot places two packages in a slot that is supposed to hold one package, a rear camera or other sensor can identify the error and share the information with a subsequent robot. The subsequent robot's control algorithm can then deprioritize the package in real time and instead pick it up and place it in an empty slot (or in its own stack for later retrieval and placement in a subsequent slot). In some configurations, a local barcode scanner can be used to scan the package, allowing the system to ensure that the package's barcode is associated with the slot in which it is ultimately placed. In some embodiments, a quality control system is provided to detect misplaced items, dropped items, empty slots expected to contain an item, slots containing more than one item, etc. In some embodiments, if there are more than a certain number of items (or another required and / or expected number) in a slot on the segmented conveyor belt or other outgoing transport, the quality control system checks to determine if any other slots are empty and takes corrective action to move the correct item from the slot with too many items to the empty slot. For example, a human worker or a subsequent robot may be tasked with picking up the stray item from the slot with too many items and placing it in the empty slot. In various modalities, the techniques described in this document are used to provide a robotic io singulation system capable of operating in most cases in a fully autonomous mode. Although the above embodiments have been described in some detail to facilitate understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The embodiments described are illustrative and not restrictive.

Claims

1. A system comprising: a communication interface; and a processor coupled to the communication interface and configured to: receive sensor data through the communication interface, wherein the sensor data includes image data associated with a workspace; use the sensor data to generate a three-dimensional view of at least a portion of the workspace, wherein the three-dimensional view includes the boundaries of a plurality of items present in the workspace; determine for each of at least a subset of items a corresponding grasping strategy and for each grasping strategy a corresponding probability of grasping success;and use grasping strategies and corresponding grasping success probabilities to determine and implement a plan to autonomously operate a robotic structure to pick up one or more items from the workplace and place each item individually into a corresponding location on a singulation transport structure 5.; 2. The system of claim 1, wherein the robotic structure comprises a robotic arm.

3. The system of claim 2, wherein the robotic arm comprises an end effector configured to be used for gripping one or more items.

4. The system of claim 3, wherein the final effector 15 comprises a suction-based end effector.

5. The system of claim 4, wherein the suction-based end effector includes two or more sets of suction cups that are separately actuated, each set includes one or more suction cups, and 2o the processor is configured to use one or more sets to grasp a given item.

6. The system of claim 5, wherein the processor is configured to use two or more of the 25 separately actuated suction cup assemblies, each to simultaneously grasp one of the corresponding items from the work area, move the items together to a target location of the first item, and successively place the items each individually into a corresponding location in the single-carrying structure.

7. The system of claim 1, wherein the singulation transport structure comprises a segmented belt.

8. The system of claim 1, wherein the processor is configured to iteratively receive sensor data, use the sensor data to generate a three-dimensional view of at least a portion of the workspace, determine for each of at least a subset of items present in the workspace a corresponding grasping strategy and for each grasping strategy a corresponding probability of grasping success, and use the grasping strategies and the corresponding probabilities of grasping success to determine and implement a plan to autonomously operate a robotic structure to select one or more items from the workspace and place each item individually in a corresponding location on a single-carrying structure.

9. The system of claim 8, wherein the processor is configured to interrupt the iterative execution of the enumerated steps in response to a determination that no more items remain present in the workspace.

10. The system of claim 8, wherein the processor is configured to interrupt operation based at least in part on the determination that no other item can be grasped autonomously even though one or more items remain in the workspace.

11. The system of claim 10, wherein the processor is configured to request human intervention based at least in part on said determination that no other item can be grasped autonomously even though one or more items remain in the workspace.

12. The system of claim 1, wherein the processor is configured to determine the probabilities of grip success based at least in part on one or more attributes of an article and a degree of overlap of article 20 with one or more articles.

13. The system of claim 1, wherein the processor is configured to determine one or more candidate gripping strategies for an item, calculate a 25 probability of gripping strategy success for each candidate gripping strategy, and use the respective calculated probabilities of gripping strategy success to select a best gripping strategy for the item.

14. The system of claim 1, wherein the workspace 5 comprises a hopper or other receptacle, the robotic structure comprises one of a plurality of robotic structures associated with the workspace, and the processor is configured to coordinate the operation of the robotic structures for picking up and placing items from the workspace 15. The system of claim 1, wherein the workspace comprises a first workspace included within a plurality of workspaces associated with the singulation transport structure, wherein each workspace has one or more associated robotic structures, and wherein the processor is configured to operate the respective robotic structures associated with each workspace to pick up workspace items and place them individually into the singulation transport structure, as indicated. 2(1 16. The system of claim 15, wherein the processor is configured to coordinate the operation of the respective robotic structures associated with each workspace to maximize the collective performance of the individual placement of items in the singulation transport structure.

17. The system of claim 16, wherein the processor is further configured to control the operation of the separation transport structure to maximize the collective performance of the individual placement of articles in the singulation transport structure.

18. The system of claim 15, wherein the processor is further configured to detect an error in the placement of an item in the singulation transport structure and to take response measures.

19. The system of claim 18, wherein the response action includes the allocation of a robotic structure associated with a back workspace to correct the error.

20. A method comprising: receiving sensor data through the communication interface, wherein the sensor data includes image data associated with a workspace; using the sensor data to generate a three-dimensional view of at least a portion of the workspace, wherein the three-dimensional view includes the boundaries of a plurality of items present in the workspace; determining for each of at least a subset of items a corresponding grasping strategy and for each grasping strategy a corresponding probability of grasping success; and using the grasping strategies and the corresponding probabilities of grasping success to determine and implement a plan for autonomously operating a robotic structure to pick up one or more items from the workspace and place each item individually in a corresponding location on a single-carrying structure.

21. A product of a computer program contained on a non-transient, computer-readable medium comprising computer instructions for: 2o receiving sensor data through the communication interface, wherein the sensor data includes image data associated with a workspace; using the sensor data to generate a three-dimensional view of at least a portion of the workspace, wherein the three-dimensional view includes the boundaries of a plurality of items present in the workspace; determining for each of at least a subset of items a corresponding grasping strategy and for each grasping strategy a corresponding probability of grasping success;and use the grasping strategies and corresponding io probabilities of grasping success to determine and implement a plan to autonomously operate a robotic structure to pick up one or more items from the workplace and place each item individually into a corresponding location on a single-carrying structure.

Images