Systems, methods and computer program products

Abstract

SOLUTION: A robotic singulation system is disclosed. In various embodiments, sensor data including image data associated with a workspace is received. The sensor data is used to generate a three-dimensional view of at least a portion of the workspace, the three-dimensional view including boundaries of a plurality of items present in the workspace. A grasp strategy is determined for each of at least some of items and, for each grasp strategy, a corresponding probability of grasp success is computed. The grasp strategies and corresponding probabilities of grasp success are used to determine and implement a plan to autonomously operate a robotic structure to pick one or more items from a workplace and to place each item singly in a corresponding location in a singulation conveyance structure.SELECTED DRAWING: Figure 5B

Description

[Technical Field]

[0001] CROSS-REFERENCE TO OTHER APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 926,165, entitled "COORDINATING MULTIPLE ROBOTS TO MEET WORKFLOW AND AVOID CONFLICT," filed October 25, 2019, which provisional patent application is incorporated herein by reference for all purposes, and to U.S. Provisional Patent Application No. 62 / 993,579, entitled "SINGULATION OF ARBITRARY MIXED ITEMS," filed March 23, 2020, which provisional patent application is incorporated herein by reference for all purposes. [Background technology]

[0002] Parcel distribution centers and other distribution centers may receive an arbitrary mixture of items, often in a random and random manner, having various sizes, dimensions, shapes, weights, stiffness, and / or other attributes. Each item may have machine-readable information (such as text and / or optically or otherwise encoded information) that can be read by a machine and utilized, for example, to route the item through an automated sorting / routing system and / or process. To read the information for a given item, in a typical approach, the items are separated from one another by a process known as "singulation."

[0003] Typically, singulation has been performed manually by human workers. Mixed items arrive at a work station, for example, by chute or other conveyance, and a set of one or more human workers each manually separates the items and places them in spaces defined for single items, such as on a conveyor belt. For each item, its destination (or at least its next transportation leg) is determined by machine reading information on the item, and the item is routed to a bag, bin, container, or other receptacle associated with the next leg and / or a destination associated with the next leg, such as a delivery vehicle or staging area.

[0004] Manual singulation processes can be labor-intensive and inefficient. For example, upstream workers may fill many of the single-item spots, leaving downstream human workers with few locations to place the singulated items. Overall throughput can be suboptimal.

[0005] Using robots to perform singulation is a difficult challenge due to the jumble of items arriving at workstations, the dynamic flow of items at and across each workstation, and the fact that it can be difficult to automatically identify, grasp, and separate (singulate) items using robotic arms and end effectors. [Brief explanation of the drawings]

[0006] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0007] [Figure 1] 1 is a flow chart illustrating one embodiment of a process for receiving, sorting, and transporting items for distribution and delivery.

[0008] [Figure 2A] FIG. 1 illustrates one embodiment of a robotic singulation system.

[0009] [Figure 2B] FIG. 1 illustrates one embodiment of a multi-station robotic singulation system.

[0010] [Figure 3A] 1 is a flow chart illustrating one embodiment of a process for picking and placing items for sorting.

[0011] [Figure 3B] 1 is a flow chart illustrating one embodiment of a process for picking and placing items for sorting.

[0012] [Figure 4A] FIG. 10 illustrates the calculation and display of normal vectors in one embodiment of a robotic singulation system.

[0013] [Figure 4B] 10 is a flow chart illustrating one embodiment of a process for processing image data to identify and calculate normal vectors for items.

[0014] [Figure 5A] 10 is a flow chart illustrating an embodiment of a process for determining a plan for picking and placing an item using image data.

[0015] [Figure 5B] FIG. 1 is a block diagram illustrating an embodiment of a hierarchical scheduling system in an embodiment of a robot singulation system.

[0016] [Figure 5C] 1 is a flow chart illustrating one embodiment of a process for scheduling and controlling resources including a robotic singulation system.

[0017] [Figure 6A]FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6B] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6C] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system.

[0018] [Figure 6D] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6E] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. [Figure 6F] FIG. 1 illustrates an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system.

[0019] [Figure 7A] 1 is a flow chart illustrating an embodiment of a process for modeling the flow of items for picking and placing items.

[0020] [Figure 7B] 1 is a flow chart illustrating an embodiment of a process for modeling the flow of items for picking and placing items.

[0021] [Figure 7C] 1 is a flow chart illustrating an embodiment of a process for modeling the flow of items for picking and placing items.

[0022] [Figure 8A] FIG. 1 is a block diagram illustrating the front view of one embodiment of a suction-based end effector.

[0023] [Figure 8B]FIG. 8B is a block diagram showing the bottom view of the suction-based end effector 802 of FIG. 8A.

[0024] [Figure 8C] FIG. 8B is a block diagram illustrating a front view of an example of multi-item grasping using the suction-based end effector 802 of FIG. 8A.

[0025] [Figure 8D] FIG. 8B is a block diagram illustrating a bottom view of an example of multi-item grasping using the suction-based end effector 802 of FIG. 8A.

[0026] [Figure 9] 1 is a flow chart illustrating one embodiment of a process for picking and placing an item using a robotic arm and end effector.

[0027] [Figure 10A] FIG. 1 illustrates one embodiment of a robotic singulation system.

[0028] [Figure 10B] 10B is a close-up view of a multi-view sensor array including cameras (or other sensors) 1010, 1012, and 1014 of FIG. 10A.

[0029] [Figure 10C] 10 is a flow chart illustrating one embodiment of a process for grasping and scanning an item.

[0030] [Figure 11] FIG. 1 illustrates an embodiment of a multi-station robotic singulation system incorporating one or more human singulation workers.

[0031] [Figure 12] 1 is a flow chart illustrating one embodiment of a process for detecting and correcting placement errors. DETAILED DESCRIPTION OF THE INVENTION

[0032] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0033] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0034] A robotic system for performing singulation and / or sortation is disclosed. In various embodiments, the robotic system comprises a robotic arm and end effectors used to pick items from a source pile / stream and place them on a partitioned conveyor or similar transport vehicle to be sorted and routed for transportation to a downstream (e.g., final destination / physical) destination. In some embodiments, the picked items are placed one by one in a nearby bin or other container. In some embodiments, multiple robots are coordinated to maximize overall throughput. In various embodiments, one or more robots may be used at a singulation station. The system may include multiple stations. Human workers may be deployed at one or more stations. In various embodiments, the robotic system may be configured to request (require) assistance from a human worker, e.g., by remotely operating a robotic arm, manually completing tasks, etc., to handle items that the robot cannot handle through fully automated processing and / or items dropped by the robot.

[0035] Courier companies, postal services, delivery services, large retail stores or distributors, and other businesses and government agencies that handle, transport, and deliver items between various locations typically receive numerous items from various source locations, each to be delivered to a corresponding one of various destination locations.

[0036] Although machines exist for handling, sorting, and routing items, utilizing machine readers and sorting equipment may require items to be spaced apart and / or oriented in a particular way so that their labels or tags can be read by the machines. Such spacing and orientation may need to be achieved during the process of "guiding" items to the sorting / routing equipment, and may be performed in connection with a "sorting" or "sorting" process (e.g., a process in which items to be delivered to various locations are sorted by general destination (e.g., region, state, city, zip code, street, house number, etc.)).

[0037] Machine readers (such as radio frequency (RF) tag readers, optical code readers, etc.) may require items to be spaced apart from one another (a process also known as "singulation") so that the tags or codes can be reliably read and so that the system can associate the resulting information with specific items (such as items at specific locations on a conveyor or other structure or means).

[0038] In a typical induction / sorting process in a baggage sorting operation, for example, individual bags may be picked from a bulk pile and placed onto a moving conveyor or tilt-tray sorting system. In most facilities, this type of induction is entirely manual.

[0039] A typical manual baggage guidance / sorting process may include one or more of the following: A chute carrying unsorted packages descends onto a sorting table adjacent to the conveyor sorting system The worker's job is to "singulate" items onto a conveyor or tray-based sortation system Workers ensure that all packages introduced into the sorting equipment are oriented so that the shipping barcode (or other optical code, electronic tag, etc.) can be read for sorting (this orientation is typically determined by the facility's scanning equipment). Wait for an empty tray or slot to pass, ensuring only one package is placed in each slot or tray

[0040] In a typical manual guidance / sortation process, packages of various shapes and sizes arrive in large quantities at a manual (or machine) feed chute in various orientations; the packages may have different dimensions, shapes, stiffness, packaging, etc.; typically, human workers remove packages from the chute and deposit them at stations where each worker works, placing the packages one by one in an open bay or otherwise defined portion of the conveyor; and finally, multiple workers, one at a time at each station, deposit the singulated packages at locations on one or more conveyors to facilitate downstream machine processing, such as code or tag reading and based automated sorting actions (e.g., routing each package to a location within the facility associated with the destination to which the package is to be delivered), which may include further sorting (e.g., to a more targeted location within the facility) and / or packing / loading of the packages for further shipment (e.g., to a truck or plane to a further destination for further sorting and delivery, loading onto a truck for regional delivery, etc.).

[0041] FIG. 1 is a flowchart illustrating one embodiment of a process for receiving, sorting, and transporting items for distribution and delivery. In the illustrated example, process 100 begins with induction process 102, in which items are provided to one or more work stations for singulation by singulation process 104. In various embodiments, singulation process 104 is at least partially automated by a robotic singulation system as disclosed herein. Singulation process 104 receives a stack or stream of heterogeneous items by induction process 102 and provides a stream of singulated items to sorting / routing process 106. For example, singulation process 104 may place items one by one onto a partitioned conveyor or other structure that feeds the items one by one into the sorting / routing device. In some embodiments, items are placed with their labels or tags oriented so that they can be read by a downstream reader configured to read routing (e.g., destination address) information and use the routing information to sort the items into corresponding destinations, such as stacks, bins, or other sets of items bound for the same next intermediate and / or final destination. Once sorted, groups of items bound for a common next / final destination are processed by transport processing 108. For example, the items may be placed into containers and loaded onto delivery or transport trucks or other vehicles, etc., for delivery to the next / final destination.

[0042] 2A illustrates one embodiment of a robotic singulation system. In various embodiments, the singulation process 104 of FIG. 1 is performed, at least in part, by a robotic singulation system (such as system 200 of FIG. 2A).

[0043] In various embodiments, a robotic system with one or more robotic arms performs the singulation / guiding. In the example shown in FIG. 2A , system 200 includes a robotic arm 202 with a suction-based end effector 204. In the illustrated example, end effector 204 is a suction-based end effector, but in various embodiments, one or more other types of end effectors may be used in the singulation systems disclosed herein, including, but not limited to, a pinch-based end effector or other types of actuated grippers. In various embodiments, the end effector may be actuated by one or more of suction, pneumatic, air, hydraulic, or other actuation. Robotic arm 202 and End Effector 204 is configured to be used to take packages or other items arriving via chute or bin 206 and place each item in a corresponding position on partitioned conveyor 208. In this example, items are fed into chute 206 from inlet end 210. For example, one or more human and / or robotic workers may feed items into inlet end 210 of chute 206 directly or via a conveyor or other electromechanical structure configured to feed items into chute 206.

[0044] In the illustrated example, one or more of robotic arm 202, end effector 204, and conveyor 208 are coordinately operated by control computer 212. In various embodiments, control computer 212 includes a vision system used to determine individual items and their orientation based on image data provided by image sensors (such as 3D cameras 214 and 216 in this example). The vision system generates output used by the robotic system to determine a strategy for grasping individual items and placing each item in a corresponding available defined location for machine identification and sorting.

[0045] In various embodiments, the robotic systems disclosed herein, for example, by operation of a control computer (such as control computer 212), include and / or perform one or more of the following: Computer vision information is generated by merging data from one or more of multiple sensors, including 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors, to generate a three-dimensional view of the workspace including one or more sorting stations. The robotic system coordinates the actions of multiple robots to avoid collisions, getting in each other's way, and competing with another robot to pick up the same item and / or place the item in the same destination location (e.g., a partitioned section of a conveyor). The robotics system coordinates the actions of multiple robots to ensure that every item is placed once per slot / position. For example, if robot A drops an item, the system tasks robot B with picking it up; an item that is placed but improperly oriented can be picked up and adjusted or moved to another position by the same or another robot; if more than one item falls into a single destination slot, a robot at a downstream station can pick one of them from the conveyor and place it in its position; etc. The system continuously updates the motion plans for each robot and for all robots to maximize overall throughput. When two robots are tasked with independently retrieving the same item, the system randomly selects one robot to take that item, while the other robot moves on to the next item (e.g., identifies, selects, decides on a grasping strategy, picks, moves according to the plan, and places). Eliminate empty positions to maximize robot productivity (throughput) For this reason, Conveyor movement and / or speed Control as needed. In the event of a misplaced or dropped item, the system assigns the item to a robot, or a human worker if necessary, to pick it up and place it back in the pile from which the retrieval robot itself came, or, if available or more optimal, place it in the next free slot on the conveyor. - Some slots are intentionally left empty for downstream robots to place items on the conveyor. Up Flow Robot Control . Failures that cannot be fixed by the same or another robot result in an alert being issued to obtain human (or other robot) intervention to resolve.

[0046] In various embodiments, the arbitrary mixture of items to be singulated may include luggage, packages, and / or letters of various shapes and sizes. Some items may be standard packages, one or more attributes of which may be known, while others may be unknown. Image data, in various embodiments, is used to identify individual items (e.g., by image segmentation). The boundaries of partially occluded items may be evaluated, for example, by recognizing the item as a standard or known type and / or by extending the visible item boundaries to a logical evaluation range (e.g., extrapolating so that two edges meet at an occluded corner). In some embodiments, the degree of overlap (i.e., occlusion by other items) is evaluated for each item, and the degree of overlap is taken into account when selecting the next item to attempt grasping. For example, a score may be calculated for each item to assess the probability of successful grasping, and in some embodiments, the score depends at least in part on the degree of overlap / occlusion by other items. An item that is less occluded may, for example, be more likely to be selected, other considerations being equal.

[0047] In various embodiments, multiple 3D and / or other cameras may be used to generate image data. A 3D view of the scene may be generated, and / or in some embodiments, a combination of cameras may be used to view the scene from different angles, e.g., with respect to a workspace and / or one or more particular items within the workspace, the camera that is least occluded is selected and utilized to grasp and move the item(s).

[0048] Multiple cameras serve many purposes in various embodiments. First, they provide a richer, full 3D view of the scene. Second, they work together to minimize errors due to package shine when light reflecting off the package interferes with the camera's operation; in this case, another camera in a different position provides backup. In some embodiments, they can be selectively triggered by a predictive vision algorithm that determines which camera has the best viewing angle and / or lowest error rate for picking a particular package, so that each package is viewed by the optimal camera. In some embodiments, one or more cameras are mounted on an operating base, and the system can change the position and orientation of the base to provide a more optimal perception (e.g., view) of the package.

[0049] Another purpose served by the camera is to detect any kind of unforeseen error in the robot's operation or any disturbance to the environment. Cameras placed on the robot and in the environment have different error and accuracy profiles. A camera on the robot may be more accurate because it is firmly fixed to the robot, but may be slower to use because the robot must slow down or stop to use it. A camera in the environment is substantially faster because it has a stable view and the robot can multitask and do other things while the camera is taking pictures. However, if someone moves or shakes the camera stand, the camera will get out of sync with the robot, causing many errors. Combining images from the robot camera and a non-robotic camera (occasionally or in the event of a packaging error) makes it possible to detect whether the robot is out of sync with the non-robotic camera, in which case the robot can take corrective action and become more robust. In some embodiments, the camera need not be firmly mounted on the robot arm; in some such embodiments, a gyro and / or accelerometer on the camera may be used to filter or compensate for movement of the mounting base.

[0050] 2A , in the illustrated example, system 200 further includes an on-demand teleoperator 218 available to a human operator 220 to remotely operate one or more of robotic arm 202, end effector 204, and conveyor 208. In some embodiments, control computer 212 is configured to attempt to grasp and place items in a fully automated mode. However, if, after attempting to operate in a fully automated mode, control computer 212 determines that no further strategies are available for grasping one or more items, in various embodiments, control computer 212 sends an alert to remotely obtain assistance from a human operator 220, for example, using teleoperator 218. In various embodiments, control computer 212 uses image data from cameras (such as cameras 214 and 216) to provide a visual representation of the scene to human operator 220 to facilitate teleoperation. For example, control computer 212 may display a view of the pile of items in chute 206. In some embodiments, a segmentation process is performed by control computer 212 on image data generated by cameras 214 and 216 to determine item / object boundaries. Masking techniques may be used to highlight individual items, for example, using different colors. An operator 220 may use a visual representation of the scene to identify items to be grasped and, using a teleoperated device 218, control robotic arm 202 and end effector 204 to pick the items from chute 206 and place each item in a corresponding location on conveyor 208. In various embodiments, once the item that prompted human intervention is placed on the conveyor, system 200 resumes fully automated operation. In various embodiments, during human intervention, the robotic system observes the human worker (e.g., completing a task manually, completing a task using a teleoperated robotic arm and end effector) and attempts to learn strategies for (better) completing tasks in autonomous mode in the future.For example, the system may learn a strategy for grasping an item by observing the location of grasps on an item when a human worker grasps the item and / or by memorizing how a human worker has utilized a robotic arm and end effector to remotely grasp an item.

[0051] FIG. 2B illustrates one embodiment of a multi-station robotic singulation system. In the illustrated example, the robotic singulation system of FIG. 2A has been expanded to include multiple singulation stations. Specifically, in addition to robotic arm 202 configured to pick items from chute 206 and place each item in a corresponding available and / or assigned location on partitioned conveyor 208, the system shown in FIG. 2B includes three additional stations, namely, robotic arms 230, 232, and 234, positioned and configured to pick / place items from chutes 236, 238, and 240, respectively. In addition to cameras 214 and 216, additional cameras 224 and 226 are included to provide a 3D view of the full scene, including each of the four stations / chutes 206, 236, 238, and 240, and conveyor 208.

[0052] In various embodiments, the control computer 212 controls the four robotic arms 202, 236, 238, and 240 to pick and place items from the chutes 206, 236, 238, and 240 onto the conveyor 208 in a manner that maximizes the overall throughput of the system. 0 ,twenty three 2 , and ,2 34 , and associated end effector motion is coordinated with the conveyor 208.

[0053] In the example shown in FIG. 2B, each station has one robotic arm, but in various embodiments, two or more robots are deployed at a single station, and the robots may interact with each other, such as by avoiding and / or managing contention to pick and place the same item. operation and move 2B) so as to avoid interfering with the other components and maximize overall throughput.

[0054] In various embodiments, a scheduler coordinates the operations of multiple robots (e.g., one or more robots operating at each of multiple stations) to achieve a desired throughput without conflicts between the robots (e.g., one robot placing an item in a location assigned by the scheduler to another robot).

[0055] In various embodiments, the robotic systems disclosed herein coordinate the actions of multiple robots to pick items one by one from a source bin or chute and place the items in an assigned location on a conveyor or other device to move the items to the next stage of machine identification and / or sorting.

[0056] In some embodiments, multiple robots may pick from the same chute or other source container. In the example shown in FIG. 2B , for example, robot arm 202 may be configured to pick from chute 206 or chute 236. Similarly, robot arm 230 may pick from chute 236 or chute 238, and robot arm 232 may pick from chute 238 or chute 240. In some embodiments, two or more robot arms configured to pick from the same chute may have different end effectors. The robotic singulation system disclosed herein may select the robot arm best suited to pick and singulate a given item. For example, the system may determine which robotic arm can reach the item and select the robotic arm with the best suited end effector and / or other attributes to successfully grasp the item.

[0057] While a fixed robot arm is shown in FIG. 2B , in various embodiments, one or more robots may be mounted on a movable transport device, such as a robot arm mounted on a chassis configured to move along rails, tracks, or other guides, or a robot arm mounted on a movable cart on a chassis. In some embodiments, robotic means actuators other than robot arms may be used. For example, an end effector may be mounted on and configured to move along a rail, which may be configured to move on one or more axes perpendicular to the rail to enable the end effector to be moved to pick, translate, and place items as disclosed herein.

[0058] 3A is a flow chart illustrating one embodiment of a process for picking and placing items for sortation. In various embodiments, process 300 of FIG. 3A is performed by a control computer (such as control computer 212 of FIGS. 2A and 2B). In the illustrated example, in step 302, items to be picked are identified from a chute or other source or bin through which items are received at the singulation station. In some embodiments, image data from one or more cameras generates a 3D view of the pile or stream of items. For A segmentation process is performed to determine the boundaries and orientation of the items. At step 304, a robotic arm is used to pick items from the chute. Items may be picked one at a time, and in some embodiments, multiple items may be grasped at once. At step 306, one or more grasped items are each moved to a corresponding spot on the partitioned conveyor. If more items are present, further iterations of steps 302, 304, and 306 are performed, and so on, until step 308 determines that there are no more items to be picked and placed in the chute (or other receptacle or source).

[0059] FIG. 3B is a flowchart illustrating one embodiment of a process for picking and placing items for sorting. In various embodiments, the process of FIG. 3B implements step 304 of process 300 of FIG. 3A. In the illustrated example, at step 320, a plan and / or strategy for grasping one or more items is determined. In various embodiments, the plan / strategy is determined based at least in part on one or more of image data indicating, for example, the size, extent, and orientation of the items and attributes that can be known, determined, and / or estimated about the items, such as by categorizing the items by size and / or item type. For example, in the context of a package delivery service, specific standard packages may be used. Weight ranges or other information may be known for each standard package type. Additionally, in some embodiments, strategies for grasping items may be learned over time, for example, by the system noting and recording the success or failure of previous attempts to grasp similar items (e.g., same standard item / package type; similar shape, stiffness, dimensions; same or similar shape; same or similar material; position and orientation relative to other items in the pile; degree of overlap of items, etc.).

[0060] In step 322, the system attempts to grasp one or more items using the strategy determined in step 320. For example, an end effector of a robotic arm may be moved to a position adjacent the items to be grasped according to the determined strategy, and the end effector may be operated according to the determined strategy to attempt to grasp the items.

[0061] At step 324, a determination is made as to whether the grip was successful. For example, image data and / or force (weight), pressure, proximity, and / or other sensor data may be used to determine whether the item was successfully gripped. If successful, the item is moved to the conveyor at step 326. If unsuccessful, processing returns to step 320, where a new strategy for gripping the item (if available) is determined.

[0062] In some embodiments, if the system fails to grasp the item after a predetermined and / or set number of attempts, or if the system cannot determine a further strategy for grasping the item, the system transitions to identifying and grasping another item, if possible, and / or sends an alert to obtain assistance, such as from a human operator.

[0063] 4A is a diagram illustrating the calculation and display of normal vectors in one embodiment of a robotic singulation system. In various embodiments, a control computer (such as control computer 212 in FIGS. 2A and 2B ) comprising the robotic singulation system disclosed herein determines the item's boundary and normal vectors, and generates and displays a visualization of the item's boundary and normal vectors.

[0064] In the illustrated example, a 3D scene visualization representation 400 includes the output of a vision system implemented in various embodiments. In this example, normal vectors to the item surface are shown. The normal vectors are used in some embodiments to determine a strategy for grasping the item. For example, in some embodiments, the robot has a suction-type end effector, and the normal vectors are used to determine the angle and / or orientation of the end effector to maximize the chances of successful suction-based grasping.

[0065] In some embodiments, a vision system is used to distinguish and differentiate items by object type, which may be indicated in visualization 400 by highlighting each object with a color corresponding to its type. In some embodiments, object type information is or may be used to determine and / or modify the item's grasping strategy, such as by increasing suction pressure or gripping force, reducing movement speed, using a robot with a specific required end effector, weight capacity, etc.

[0066] In some embodiments, additional information not shown in FIG. 4A may be displayed. For example, in some embodiments, for each item, the best grasping strategy and associated probability of grasping success is determined and displayed adjacent to the item. The displayed information may be utilized in some embodiments to monitor system operation in a fully automated operating mode and / or to allow a human operator to intervene and quickly gain a view of the scene and available grasping strategies and probabilities, for example, for purposes of operating a robotic singulation system remotely.

[0067] 4B is a flow chart illustrating one embodiment of a process for processing image data to identify and calculate normal vectors for items. In various embodiments, process 420 may be performed to generate and display a visual representation of a 3D view of a scene (such as visualization 400 of FIG. 4A). In various embodiments, process 420 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B).

[0068] In the illustrated example, in step 422, 3D image data is received from one or more cameras (such as cameras 214, 216, 224, and / or 226 of FIGS. 2A and 2B, cameras mounted on a robotic arm and / or end effector, etc.). In various embodiments, image data from multiple cameras is merged to generate a composite 3D view of a scene, such as a pile or stream of items from which items are to be picked. In step 424, a segmentation process is performed to determine item boundaries in 3D space. In step 426, , ten For each item for which an unobscured view is obtained, the geometric center and normal vector of each of the item's one or more surfaces (e.g., the largest and / or most exposed (e.g., not obscured by overlapping other items) surfaces of the item) are determined.

[0069] In various embodiments, the segmentation data determined in step 424 is used to generate a corresponding mask layer for each of the plurality of items. Each mask layer, in various embodiments, is used to generate and display a visual representation of the scene in which the respective item is highlighted. In various embodiments, each item is highlighted with a color corresponding to the item's object type and / or other attributes, such as weight, classification, softness, deformability, unpickability (e.g., broken packaging, porous surface, etc.).

[0070] 5A is a flow chart illustrating one embodiment of a process for determining a plan for picking and placing an item using image data. In various embodiments, the process 500 of FIG. 5A is performed by a computer (such as the control computer 212 of FIGS. 2A and 2B).

[0071] In the illustrated example, image data is received at step 502. The image data may be received from multiple cameras, such as one or more 3D cameras. The image data may be merged to generate a 3D view of the workspace, such as a pile or stream of items in a chute or other container. At step 504, segmentation, object type identification, and normal vector calculation are performed. In some embodiments, the segmentation data is used to determine item boundaries and generate item-specific mask layers, as described above.

[0072] In step 506, a grasping strategy is determined for as many items as possible given the current view of the workspace. For each item, one or more grasping strategies are determined, and for each strategy, a probability that the strategy will lead to successful grasping of the item is determined. In some embodiments, the grasping strategy is determined based on item attributes (item type, size, estimated weight, etc.). In some embodiments, determining the grasping strategy includes determining whether to attempt pickup of one or multiple items, whether to utilize all end effector actuators (e.g., whether to utilize only one suction cup or set of cups or more than one), and / or selecting a grasping technique and corresponding speed combination from the set of calculated grasping technique+speed combinations under consideration (e.g., one suction cup at 50% speed or two suction cups at 80% speed). For example, in some embodiments, a decision may be made based on package type to utilize one suction cup at 50% speed (moving at 50% speed instead of 80% to avoid dropping it), such as to grip small items that may be heavy or difficult to grip securely with one suction cup (two suction cups cannot be utilized), whereas larger items that are relatively light or easier to grip securely with two suction cups may be gripped with two cups and moved at a higher speed.

[0073] In some embodiments, the gripping strategy may change if the object is moving as part of a flowing pile. For example, according to one strategy, the robot may press down harder to freeze the object in place, while the suction cups may be activated independently in the order they touch the package to ensure a firm and stable grip. The robot also matches its speed with the flowing pile based on visual or depth sensor feedback to ensure the package does not slip within the pile.

[0074] Another optimization implemented in some embodiments is modifying the grasping strategy based on double (or n-) item picks (see also the next section), where the robot may adapt its strategy based on whether it is picking multiple items in a batch or sequentially while attempting to reuse vision data. For example, a robot gripper with four suction cups may use two cups to pick up one object and the other two cups to pick up a second object. In this scenario, the robot will slow down and also begin grasping the second object, avoiding collisions between the first object already being held and the surrounding pile of objects.

[0075] At step 508, a plan is determined for grasping, moving, and placing items according to probability using the item-specific grasping strategies and probabilities determined at step 506. For example, a plan may be determined for grasping the next n items in a particular order, each according to a grasping strategy and the probability of success determined for that item and strategy.

[0076] In some embodiments, changes in item position, arrangement, orientation, and / or flow due to the grasping and removal of previously grasped items are taken into account when planning in step 508 for sequential grasping of multiple items.

[0077] 5 is a continuous and ongoing process. As items are being picked and placed from the pile, subsequently received image data is processed to identify and determine a strategy for grasping more items (steps 502, 504, 506), and a plan for picking / placing the items is updated based on the grasping strategy and probabilities (step 508).

[0078] FIG. 5B is a block diagram illustrating an embodiment of a hierarchical scheduling system in an embodiment of a robotic singulation system. In various embodiments, the hierarchical scheduling system 520 of FIG. 5B is implemented at least in part on a computer (such as the control computer 212 of FIGS. 2A and 2B ). In the illustrated example, the hierarchical scheduling system 520 includes a global scheduler 522 configured to optimize throughput by coordinating the operation of multiple robotic singulation stations and the partitioned conveyors (or similar structures) onto which the robotic singulation stations are configured to place items. The global scheduler 522 may be implemented as a processing module or other software running on a computer. The global scheduler manages and coordinates work among the robotic singulation stations, at least in part, by monitoring, controlling as needed, and / or otherwise providing input to multiple robotic singulation station schedulers 524, 526, 528, and 530.

[0079] Each of robotic singulation station schedulers 524, 526, 528, and 530 is associated with a corresponding robotic singulation station and each controls and coordinates the movement of one or more robotic arms and associated end effectors to pick up items from a corresponding chute or other item receptacle and place them one-by-one onto a partitioned conveyor or similar structure. Each of robotic singulation station schedulers 524, 526, 528, and 530 is associated with a corresponding set of one or more station sensors 532, 534, 536, and 538, respectively, and each performs automated singulation at that robotic singulation station using sensor data generated by that station's sensors. In some embodiments, each scheduler implements and performs process 500 of FIG. 5A .

[0080] In various embodiments, each of the robot singulation station schedulers 524, 526, 528, and 530 reports one or more of image and / or other station sensor data, object identification, grasping strategy, and success probability data; pick / place planning information; and expected item singulation throughput information to the global scheduler 522. The global schedule 522 is configured to use information received from the robot singulation station schedulers 524, 526, 528, and 530, together with sensor data received from other sensors 540 (such as cameras directed at other parts of the partitioned conveyor and / or workspace not covered or not fully or fully covered by the station sensors), to coordinate work by each robot singulation station, each under the control of its station-specific scheduler 524, 526, 528, or 530, and to control the operation (e.g., speed) of the partitioned conveyor via the conveyor controller 542, in order to optimize (e.g., maximize) the overall singulation throughput of the system.

[0081] In various embodiments, the global scheduler 522 utilizes one or more techniques to optimize the utilization of multiple robots comprising the robot singulation system to perform singulation (e.g., to maximize overall throughput). For example, if there are four robots in a sequence, the lead (or other upstream) robot may be controlled to place packages to leave empty slots so that downstream robots do not have to wait for empty slots. This approach impacts downstream robots because they wait for unknown / random periods of time due to package flow, etc. As a result, a naive strategy (e.g., the lead robot places every fourth empty slot) does not optimize overall throughput. Sometimes, if the lead robot's packages are not flowing, it may be better for the lead robot to place two or three packages in consecutive slots in the sequence, but the overall system makes such decisions while being aware of the status and flow of each station. In some embodiments, the optimal strategy for leaving empty slots for downstream robots is based on the downstream robots' predicted demand for empty slots (e.g., as a function of their package flow). In some embodiments, information from the local station scheduler is used to predict the maximum throughput of each station and to control the conveyor speed and the number of slots left empty by the upstream robot so that the downstream robot can access empty slots proportionally to the rate at which it can (currently) pick / place. In some embodiments, when a partitioned conveyor is full due to some bottleneck in the downstream sortation process, the robotic singulation system disclosed herein may track the pose of each pre-singulated package while pre-singulating one or more packages, for example, in a corresponding chute or nearby staging area.When some free space becomes available on the partitioned conveyor, the system / station moves the pre-singulated packages one by one in rapid succession onto the partitioned conveyor without further visual processing time.

[0082] In some embodiments, the presence of humans collaborating with robots impacts placement and multi-robot coordination strategies, as the robots or associated computer vision or other sensor systems must observe what the humans are doing and adapt the robot's placement in real time. For example, if a human takes over a conveyor belt slot that was scheduled to be used by a robot, the system must adjust the global and local schedules / plans accordingly. In another example, if a human interferes with a robot's picking, causing it to register as not picked, the system adapts to correct the error. Also, if a human corrects a robot's picking error (the robot was commanded to place a package in slot A but mistakenly placed it across slot A and adjacent slot B, and the human placed it in slot B, but the system's memory says the package is in slot A), the system must observe the human's behavior and adjust the behavior of downstream robots.

[0083] In various embodiments, the global scheduler 522 may operate a station slower than its maximum possible throughput at a given time. For example, the global scheduler 522 may explicitly instruct a local station scheduler (e.g., 524, 526, 528, or 530) to slow down and / or may make fewer slots available to the local station, for example, explicitly by allocating fewer slots to the station or indirectly, such as by allowing an upstream station to fill more slots.

[0084] 5C is a flow chart illustrating one embodiment of a process for scheduling and controlling resources including a robotic singulation system. In various embodiments, process 560 of FIG. 5C is performed by a global scheduler (such as global scheduler 522 of FIG. 5B).

[0085] In various embodiments, items arrive via a chute and / or conveyor. The chute / conveyor may be fed, for example, by humans, robots, and / or both, and items may arrive in batches. As items are added to the stream at the input end, they may slide or flow down / along the chute or conveyor toward the end where one or more robotic arms are provided to perform singulation, for example, by grasping individual items and moving each item one-by-one to a corresponding item location on an output conveyor.

[0086] In various embodiments, images from one or more 3D cameras or other cameras are received and processed. The flow of items down / through a chute / conveyor feeding the singulation station is modeled based on the image data. The model is used to predict moments of relatively low or no movement, during which the image data is used to identify and grasp the items. In some embodiments, the flow model is used to ensure that the robotic arm is not positioned to block one or more cameras during moments of relatively low or no flow. In some embodiments, the system is configured to detect areas of relatively low or no flow within a larger flow. For example, flow may be analyzed within different predetermined areas of a chute or other source conveying device or container. The system may pick items for singulation from areas of low / steady flow and / or no flow, and temporarily avoid areas experiencing less stable flow.

[0087] In some embodiments, the system identifies different areas in the stream of items and guides the robot to pick items from those areas accordingly. In some embodiments, the system utilizes a flow model calculated using sequential images from multiple cameras mounted on and around the chute to identify items on a chute or other source conveying device or container. items, or chutes or other source conveying devices or containers Predicting the movement of passing items. A flow model is used to predict the future position of one or more items as they flow down a chute or other conveying structure. Each time, an image is captured and / or a grasp attempt is made at the time / location where the item is predicted to be based on the flow model. In some embodiments, real-time segmentation is performed. For example, segmentation is performed in the range of 30-40 milliseconds and / or other rates as fast as the 3D camera frame rate. In some embodiments, the results of the real-time segmentation are used to track and / or model the flow of individual items through the chute. The future position of the item is predicted based on the item-specific model / movement, and a plan and strategy for grasping the item at that future position and time is determined and executed autonomously. In some embodiments, the position of the target item is continuously updated. In some embodiments, the trajectory of the robotic arm may be updated in real time based on the item's updated predicted future position as the arm moves to grasp the item.

[0088] In some embodiments, if the observed flow of items is faster than the threshold speed, the system waits a set time (e.g., 100 milliseconds) and checks the image data and / or flow speed again. The system may repeat and / or vary the length of such waits until a sufficiently stable flow condition is observed to have a high probability of successful grasping. In some embodiments, different areas of the chute may have different acceptable speed thresholds. For example, if an area is far from the actual picking spot of the robot, a faster movement speed is acceptable in some embodiments, since an unstable flow in that area is not expected to interfere with the operation being performed by the robot at the time. For example, a package rolling at the top of the chute (“at high speed”) may not be a problem if the robot is picking from the bottom of the chute. However, if the rolling item is close to the pick area, it may be unacceptable because it may block other items or move them around by colliding with them. In some embodiments, the system detects flow by region and / or location relative to the expected pick area, allowing for less stable flow in areas far (sufficiently far) from the area where the robot intends to pick next.

[0089] In the example shown in FIG. 5C , in step 562, the overall throughput, the observed and / or estimated (locally scheduled) throughput of the local singulation station, and the overall error rate and station-specific error rate are monitored. In step 564, the conveyor speed and local station speeds are adjusted to maximize the net amount of overall throughput or error. In step 566, conveyor slots are assigned to each station to maximize the net throughput. While in this example, conveyor slots are explicitly assigned / designated, in some embodiments, station speeds are controlled to ensure that downstream stations have available slots in which to place items, without (necessarily) pre-assigning specific slots to specific stations. Processing continues (steps 562, 564, 566) while any stations have items yet to be picked / placed (step 568).

[0090] 6A-6C show an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. In various embodiments, the flow of items through a chute or other container is modeled. The flow model, in various embodiments, is used to determine a strategy for grasping items from the flow. In some embodiments, the modeled and / or observed flow may be used to perform one or more of the following: determining a grasping strategy and / or plan for grasping an item at a future location where the item is predicted to flow; determining and implementing a plan for grasping a series of items, each of which is grasped at a corresponding future location determined at least in part based on the flow model, determining a grasping strategy for each of a plurality of items; ensuring that the robot arm is positioned at a future moment so as not to obscure the view of an item where the item is predicted to be located based on the model and where it is planned to be picked; and waiting, for example, a calculated or predetermined time (based on the model) to allow the flow to become more stable (e.g., moving slower, most of the items moving in a uniform direction, minimal or low rate of orientation change, etc.).

[0091] 6A-6C, in the illustrated example, the flow model shows a currently mostly stable arrangement of items where the model indicates that items 602 and 604 will remain relatively stable / uniform as they are picked from the flow / pile. In various embodiments, the model information shown in FIGS. 6A-6C, potentially along with other information (e.g., available pick strategies, desired / assigned station throughput, etc.), is used to determine and implement a plan for sequentially picking and placing items 602 and 604 from the locations where the model indicates the items are expected to be at the time they are scheduled to be picked.

[0092] 6D-6F show an example of the flow of items through a feeder chute in one embodiment of a robotic singulation system. In this example, the model shows that the pile is moving in a relatively slow, even flow but is disturbed (or observed to be disturbed) when item 602 is picked. In various embodiments, the model information shown in FIGS. 6D-6F can be used to decide to pick item 602 alone but then wait a short time for the pile / stream to stabilize before determining and implementing a grasping strategy and plan to pick and place other items from the pile / stream. In some embodiments, the stream can be detected to be sufficiently unstable that one or more items are at risk of tipping or otherwise falling out of the chute or other source container before they can be picked up and singulated. In some embodiments, in such a situation, the system can be configured to implement a strategy to stabilize the stream using the robotic arm and / or end effector, such as by blocking the bottom of the chute to prevent items from overflowing the chute, or by placing an arm across the stream to block or slow the stream using the end effector and / or robotic arm.

[0093] FIG. 7A is a flow chart illustrating one embodiment of a process for modeling the flow of items for picking and placing the items. In various embodiments, process 700 of FIG. 7A is performed by a computer configured to model the flow of items (e.g., through a chute) and pick and place the items using the model, such as the example described above in connection with FIGS. 6A-6F. In various embodiments, process 700 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B). In some embodiments, process 700 is performed by a robot singulation station scheduler (such as schedulers 524, 526, 528, and 530 of FIG. 5B).

[0094] In the example shown in FIG. 7A, image data is received in step 702. In step 704, segmentation and item (e.g., type) identification is performed. In step 706, the flow of items through a chute or other container is modeled. (In this example, steps 704 and 706 are performed sequentially, but in some embodiments, these steps may be performed in parallel and / or in parallel.) In step 708, the model is used to determine a grab strategy and plan for grabbing the next n items from the pile / stream. In step 710, the plan is executed. If, at step 712, it is determined that the attempt was not completely successful (e.g., one or more items were not grasped successfully, an item was not in the expected position, the flow was disrupted, or was otherwise not as expected), or if additional items have not yet been singulated (step 714), further iterations of steps 702, 704, 706, 708, and 710 are performed, and the next iteration is performed until step 714 determines that no more items remain to be singulated.

[0095] In various embodiments, the system may take a photograph and identify two (or more) objects to pick. The system picks and moves the first one, and then, instead of doing a full scene recalculation to find the package to pick, simply checks to see if the second package is in the way. If not, the system picks it without performing a full scene recalculation, which typically saves a lot of time. In various embodiments, the time savings are in one or more of: sensor reading latency, image acquisition time, system calculation time, subdivision, masking, package pile ordering calculation, and finally control and planning. If the second item is not in the expected location, the system does a full scene recalculation to find the next package to pick.

[0096] In some embodiments, the robot may use vision, depth, or other sensors to identify two graspable packages that the robot's control algorithm determines are far enough apart that they cannot interfere with each other's pick (such as by destabilizing the stack and causing it to collapse or by colliding with other packages). Instead of repeating the sensor processing pipeline after picking the first package, the robot may proceed directly to pick the second package. That said, statistically, there is a risk that the control algorithm's prediction may be incorrect or that some unforeseen event may have caused the stack to move. In this scenario, the controller may pick more conservatively and use a predicted model of grasper suction cup actuation (or other predicted sensory models, e.g., force, pressure, and other types of sensing modalities) to test whether the package being picked matches the previous prediction (without having to redo the vision calculations).

[0097] FIG. 7B is a flow chart illustrating one embodiment of a process for modeling the flow of items for picking and placing the items. In various embodiments, process 720 of FIG. 7B is performed by a computer configured to model the flow of items (e.g., through a chute) and pick and place the items using the model, such as the example described above in connection with FIGS. 6A-6F. In various embodiments, process 720 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B). In some embodiments, process 720 is performed by a robot singulation station scheduler (such as schedulers 524, 526, 528, and 530 of FIG. 5B).

[0098] In the example shown in Figure 7B, image data is received and processed in step 722. In step 724, an item is received, e.g., as shown in Figure 7B. ASimilar to process 700 of step 720, items are picked / placed based at least in part on the model. If an indication is received and / or a determination is made in step 726 that the flow has become unstable, the system may pause in step 728, e.g., for a random, predetermined, and / or set interval, after which the process resumes and continues unless / until there are no more items to be picked / placed, after which process 720 ends.

[0099] FIG. 7C is a flow chart illustrating one embodiment of a process for modeling the flow of items for picking and placing the items. In various embodiments, process 740 of FIG. 7C is performed by a computer configured to model the flow of items (e.g., through a chute) and pick and place the items using the model, such as the example described above in connection with FIGS. 6A-6F. In various embodiments, process 740 is performed by a computer (such as control computer 212 of FIGS. 2A and 2B). In some embodiments, process 740 is performed by a robot singulation station scheduler (such as schedulers 524, 526, 528, and 530 of FIG. 5B).

[0100] In the example shown in FIG. 7C , a condition in which no graspable items are currently available is detected at step 742. For example, the system may have attempted to determine a grasp strategy for items in the pile but determined that due to flow speed, clutter, orientation, overlap, etc., no items currently have a grasp strategy available with a probability of success greater than a predetermined minimum threshold. At step 744, in response to the determination at step 742, the system uses the robotic arm to attempt to change the pile / flow conditions to make a grasp strategy available. For example, the robotic arm may be used to gently poke, pull, push, etc., an item or items into a different position in the pile. After each poke, the system may reevaluate, for example, by recalculating a 3D view of the scene to determine whether a viable grasp strategy is available. If at step 746 it is determined that a grasp (or multiple grasps, each on a different item) is available, autonomous operation resumes at step 750. Conversely, if after a predetermined number of attempts to change the pile / stream state no viable grasping strategy becomes available, then in step 748 the system seeks assistance, for example, from another robot and / or a human worker, the latter via teleoperation and / or manual intervention (such as shuffling items in the pile and / or manually picking / placing items) until the system determines that autonomous operation can resume.

[0101] Figure 8A is a block diagram illustrating the front of one embodiment of a suction-based end effector. In various embodiments, the end effector 802 may be used as an end effector of a robotic arm (such as end effector 204 of Figure 2A) that constitutes a robotic singulation system. Figure 8B is a block diagram illustrating the bottom of the suction-based end effector 802 of Figure 8A.

[0102] 8A and 8B , in the illustrated example, end effector 802 includes four suction cups 804, 808, 812, and 814. In this example, a vacuum may be applied to suction cups 804 and 812 via hose 806, and a vacuum may be applied to suction cups 808 and 814 via hose 810. In various embodiments, pairs of suction cups (i.e., a first pair including suction cups 804 and 812 and a second pair including suction cups 808 and 814) may be operated independently. For example, a single item (such as a relatively small and / or relatively lightweight item) may be grasped using only one or the other of the suction cup pair. In some embodiments, each pair may be used to grasp a separate item simultaneously, allowing for simultaneous pick / place of two (or, in some embodiments, three or more) items.

[0103] Figure 8C is a block diagram illustrating an example of gripping multiple items from the front using the suction-based end effector 802 of Figure 8A. Figure 8D is a block diagram illustrating an example of gripping multiple items from the bottom using the suction-based end effector 802 of Figure 8A. In this example, each suction cup pair was independently actuated to grip a corresponding one of the two items shown gripped in the example.

[0104] In various embodiments, more or fewer suction cups and / or more or fewer independently actuated sets of one or more suction cups may be included on a given end effector.

[0105] 9 is a flow chart illustrating one embodiment of a process for picking and placing an item using a robotic arm and end effector. In various embodiments, the process 900 of FIG. 9 may be performed by a computer (such as the control computer 212 of FIGS. 2A and 2B).

[0106] When using a suction-based end effector (such as end effector 802), pressing too hard to ensure suction cup engagement may damage fragile items and / or packaging. Additionally, sensor readings may vary based on package type. For example, suction sensor readings may be different when gripping a cardboard box (where air escapes through the grooves / paper) than when gripping a plastic polythene bag. In some embodiments, if the package type is known, the sensor readings are evaluated and / or adjusted accordingly. In some embodiments, if a sensor reading associated with successful gripping of a determined type of package is not achieved on initial engagement, additional force and / or suction may be applied to achieve a better grip. Such an approach increases the likelihood of successful gripping and is more efficient than determining that the grip failed and trying again.

[0107] In the example shown in FIG. 9 , in step 902, the robotic arm is used to approach and attempt to grasp the item according to the determined and selected grasping strategy for grasping the item. In step 904, a determination is made as to whether the item was successfully grasped. For example, one or more of a force (weight) sensor, a suction / pressure sensor, and image data may be used to determine whether the grasp was successful. If successful, in step 906, the item is assigned to and / or moved to the next available slot. If it is determined in step 904 that the grasp was not successful and the system determines to make additional attempts in step 908 (e.g., because the number of attempts has not yet been less than a predetermined maximum number of attempts, the item has not been determined to be too fragile to apply a greater force, etc.), in step 910, the system adjusts the force applied to engage the item (and / or, in various embodiments, one or more of the grasp pose, i.e., the position and / or orientation of the robotic arm and / or end effector) and attempts to grasp the item again. For example, a robotic arm may be used to press the suction cup against the item with a slightly greater force before activating the suction and / or a stronger suction may be applied.

[0108] If step 908 determines that no further effort should be made to grasp the item at this time, processing proceeds to step 912, where it is determined whether there are other items available for grasping (e.g., other items are present in the chute and the system has a grasping strategy available for one or more of them). If so, then at step 914 the system moves on to the next item and performs an iteration of step 902 and subsequent steps for that item. If there are no remaining items and / or no items for which a grasping strategy is available, then at step 916 the system receives assistance, for example, from another robot, from a human via teleoperation, and / or from a human via manual intervention.

[0109] In various embodiments, the number and / or nature of further attempts to grasp the item, as in steps 908 and 910, may be determined by factors such as the item or item type, item and / or package characteristics determined by the item type, item and / or package characteristics determined by examining the item with the robotic arm and end effector, item attributes determined by various sensors, etc.

[0110] 10A illustrates one embodiment of a robotic singulation system. In the illustrated example, robotic singulation station 1000 includes a robotic arm 1002 operated under the control of a control computer (not shown in FIG. 10A ) to pick items (such as item 1006 in the illustrated example) from a chute or other container 1004 and place them one-by-one onto a partitioned conveyor 1008.

[0111] In various embodiments, the robotic singulation station 1000 is controlled by a control computer configured to, at least in certain circumstances, scan addresses or other routing information locally at station 1000 using a multi-view array of sensors, in this example including cameras (or other sensors) 1010, 1012, and 1014.

[0112] FIG. 10B is a close-up view of the multi-view sensor array including cameras (or other sensors) 1010, 1012, and 1014 of FIG. 10A.

[0113] In various embodiments, sensors such as cameras (or other sensors) 1010, 1012, and 1014 are positioned to read routing information (e.g., text address, optical or other code, etc.) regardless of the orientation of the package placed by the robot on the output conveyor. For example, if the package is placed label-side down, a scanner across which the package is slid and / or swiped reads the label and associates sorting / routing information with the corresponding location on the output conveyor.

[0114] A challenge in picking objects from a (possibly moving) pile and placing them on a singulation conveyor is that the singulation conveyor must associate the package barcode with each package in the conveyor belt slot. Based on the orientation of the package within the bulk pile the robot is picking from, the robot may discover that the barcode is actually facing down. In some embodiments, the robotic singulation system disclosed herein may be used to flip or otherwise flip the item so that the barcode or other label or routing information is facing up. In some embodiments, the robotic arm flips the package using its own motion if the end effector is a pinch gripper, either by using controlled slippage due to gravity (e.g., gripping at an end and allowing slippage / gravity to begin rotating the item before releasing once the flipping motion is initiated), or by using gravity and a controlled release sequence of multiple suction cups to reorient the package (e.g., releasing the suction of the cups at one end of the effector while still applying suction at the other end to begin rotating the item about the axis about which it will be flipped). However, flipping a package before placing it on a conveyor belt can damage the package, can be unsuccessful, and / or can take a lot of time and require additional hands (e.g., another robot), which is expensive. In some embodiments, the systems disclosed herein can recognize that an item is of a type that may require an action to ensure the label can be read. For example, packages in polyethylene ("poly") or other plastic or similar bags may need to be placed to flatten and / or smooth the package so that a downstream scanner can read the label. In some embodiments, a robotic arm may be used to smooth and / or flatten the package.In some embodiments, such items may be dropped from a predetermined height to help flatten the packaging sufficiently for reading. In various embodiments, the robot may flatten the plastic bag by throwing it onto a chute or conveyor before picking and / or reading at the station, spreading the bag using actuated suction cups after picking, performing bi-manual (two robotic arms) operations to remove wrinkles from deformable items, etc. In some embodiments, a blower or other mechanism may be used to smooth the package after placement. The blower may be freestanding / fixed, attached to or integrated into the robotic arm.

[0115] In some embodiments, a multi-axis barcode (or other) scanner or sensor, i.e., a device that can scan top or bottom barcodes, is placed downstream on the conveyor belt 1008. However, this only works if the bottom of the conveyor belt is clear, or if there is a sequencing operation where packages are passed over a clear (or open) slot with the barcode reader looking up through the slot.

[0116] In various embodiments, if the packages cannot be easily placed on the conveyor 1008 with the label facing up (and therefore easily scanned by an overhead scanner), an array of sensors at the singulation station, such as cameras (or other sensors) 1010, 1012, and 1014 in the example shown in FIGS. 10A and 10B, may be used to scan the labels of the packages. is usedIn some embodiments, if the top of the package is visible, the barcode or other address information may be read as part of a computer vision scanning pipeline and / or using a camera mounted on the robotic arm 1002 and / or end effector. If the barcode is on the side or bottom or is otherwise hidden, cameras (or other sensors) 1010, 1012, and 1014 are used to scan the package as the robot picks it up and moves it onto a conveyor belt or bin. To get this right, in some embodiments, the robot modifies its controller and motion planning to ensure that the package is scanned in mid-air, which requires positioning the object in an optimal way for the barcode scanner to see the package, while also constraining the motion path so the object lands in an empty slot. This is particularly challenging for bottom or side barcode scanning, as the barcode scanners must be placed in an optimized configuration to simplify the motion planning task for the robot and ensure that the robot can quickly perform the scan and place. The scanner configuration and robot motion planning are dynamically influenced by the package itself, as package size and weight may require the robot to position the object differently when attempting a barcode scan (see also the section on grip / speed optimization based on package type for examples of how the robot must adapt its motion). Because cameras or other sensors cannot see the bottom of the pile, the height of a package is typically unknown when it is picked from a dense pile. In this case, the robot uses machine learning or geometric models to predict the object's height. If the height estimate is too low, the robot will hit the barcode scanner with the object. If the height estimate is too high, the robot will not be able to scan the object. One side of the error spectrum is catastrophic (the item hits the scanner), while the other simply causes the trajectory to be replanned. In various embodiments, the robot control algorithm chooses to be on the safe side and assume the object is higher to avoid a collision. If the object is too far away, the robot can use a human-like rocking motion to gradually move it closer to the barcode scanner so the object is scanned.

[0117] FIG. 10C is a flow chart illustrating one embodiment of a process for grasping and scanning an item. In various embodiments, process 1020 of FIG. 10C is performed by a control computer configured to control the operation of a robotic singulation station (such as station 1000 of FIG. 10A). In the illustrated example, at step 1022, a robotic arm is used to approach and grasp an item. As the item is approached and / or grasped, an image of the top surface of the item is captured. In various embodiments, a camera pointed at the item / pile, such as a camera mounted near the station and / or on the robotic arm and / or end effector, may be used to capture the image. At step 1024, the image is processed to determine whether an address label or other routing information is on the top surface. If so, at step 1026, the item is placed face up on a conveyor.

[0118] If the label is not on the top surface when grasped (step 1024), the system attempts to find the label using one or more local sensors in step 1028. If the label is found and on a surface that allows the robotic singulation system to place the item on the conveyor with the label rotated up, it is determined in step 1030 that a local scan is not necessary, and the package is rotated so that the label is facing up and placed on the conveyor in step 1026. Conversely, if the label is not found or the package cannot be rotated so that the label is facing up, it is determined in step 1030 that the label should be scanned locally. In such a case, the label is scanned locally in step 1032 using a multi-axis sensor array, such as cameras 1010, 1012, and 1014 of Figures 10A and 10B. The package is placed on a conveyor at step 1034 and the routing information determined by locally scanning the label is associated at step 1036 with the slot or other demarcated location on the conveyor where the item was placed.

[0119] Downstream, the routing information determined by locally scanning the item's label at the robotic singulation station is used to sort / route the item to an intermediate or final destination determined at least in part based on the routing information.

[0120] Figure 11 illustrates an embodiment of a multi-station robotic singulation system incorporating one or more human singulation personnel. In the illustrated example, robotic singulation system 1100 includes elements of the system of Figure 2B, except that at the left-most station (as shown), robotic arm 202 is replaced by a human operator 1102.

[0121] In some embodiments, the robotic system includes one or more human singulation workers, for example, at the last or other downstream station, as in the example shown in Figure 11. The system and / or human workers provide items that are difficult to singulate (e.g., large flexible bags) to the workstations staffed by the human workers. The robotic system leaves slots open to be filled by the human workers.

[0122] In some embodiments, each synth Gyu A singulation station provides a location for one or more robots and one or more human workers to work together, e.g., at the same station. The location for the human worker provides sufficient space for the worker to perform singulation in the out-of-reach area of ​​the robot arm. In various embodiments, the human worker can augment the throughput of the robot arm, correct misalignments, incorrect orientations (e.g., for scanning labels), ensure that conveyor slots not filled by upstream robots or other workers are filled, etc.

[0123] Given that the singulation workspace is often limited, the following situations are addressed by the controller (e.g., control computer) in various embodiments:

[0124] If the robot determines that the flow of packages is too fast or that grasping some packages is not feasible, it will step away, go into safety mode, and trigger a message for a human operator to come and help.

[0125] --If the robot determines that a package is difficult to pick (e.g., too crowded to pick, stuck on a corner or edge, has sharp / pointed parts that could damage the end effector suction cups, is too heavy, etc.), it will move away, go into safety mode, and trigger a message to a human operator to come and assist.

[0126] Alternatively, the robot may present pick decisions on a software interface to a remote operator who guides the robot to make the correct pick, with the pick being high-level guidance provided by a human (not direct remote control) and performed by the robot's pick logic described above.

[0127] Alternatively, the robot will shove or push all unpickable packages into a "return or unpickable item chute" which will then send them to a human operator.

[0128] If the flow of packages is too fast, the robot may trigger a human handover. In this situation, the robot transitions to a safety mode to stay out of the way of the human. The control system provides two options: (i) a human operator has enough space and can operate next to the robot to help temporarily pick some packages and reduce the overall flow, or (ii) the human operator physically moves the entire robot out of the way (e.g., down a chute) and performs the pick as normal. In both situations, the computer vision system continues to monitor the human pick and uses that data to teach the robot to behave better when encountering that type of situation.

[0129] In some embodiments, a single singulation station may allow both human and robotic workers to pick and singulate items. Humans may singulate items that humans are trained to pick, such as items that prove difficult for autonomous robotic operation to pick and place. In some embodiments, the system monitors the movements of the human worker and uses the robotic arm to pick, move, and place items on a trajectory that avoids the human worker. Safety protocols ensure that the robot slows or stops its movement if a human gets too close.

[0130] Figure 12 is a flow chart illustrating one embodiment of a process for detecting and correcting placement errors. In various embodiments, process 1200 of Figure 12 is performed by a control computer (such as control computer 212 of Figures 2A and 2B).

[0131] In step 1202, placement errors are detected. For example, one or more of images processed by a vision system, force sensors on a robotic arm, pressure sensors detecting loss of vacuum, etc. may be used to detect that an item has been dropped before being placed. Alternatively, image data may be processed to determine that an item was not placed in the intended slot on the conveyor, that two items were placed in the same slot, that an item was placed in an orientation that prevents it from being scanned downstream, etc.

[0132] In step 1204, the controller determines a plan for downstream workers (e.g., another robotic arm and / or a human worker) to correct the detected error. In step 1206, the downstream workers are assigned to correct the error, and in step 1208, the global plan and / or any affected local plans are updated, as necessary, to reflect the downstream workers' use in correcting the error. For example, the downstream workers may no longer be available or may be immediately available to perform a local task assigned by the local scheduler. In various embodiments, once a resource (e.g., a robotic arm) is assigned to correct the upstream detected error, the local scheduler updates its local plan and assigned tasks to incorporate the assigned error correction task.

[0133] For example, if a robot places two packages in a slot that's meant to hold one, a downstream camera or other sensor can identify the error and share that information with the downstream robot. The downstream robot's control algorithm can, in real time, deprioritize the pick and instead pick an additional package to place in the empty slot (or in its own pile to be picked later and placed into the slots one by one). In some embodiments, a local barcode scanner can be used to scan the package, allowing the system to ensure that the package barcode is associated with the slot in which the package will ultimately be placed.

[0134] In some embodiments, a quality monitoring system is provided to detect misplacement, dropped items, empty slots expected to receive items, slots containing more than one item, etc. In some embodiments, if more than one item (or more than other desired and / or expected number) is present in a slot of a compartmented conveyor belt or other output transport device, the quality monitoring system checks whether other slots are empty and takes corrective action to move the correct item from the slot with too many items to the empty slot. For example, a human worker or downstream robot may be tasked with picking the misplaced item from the slot with too many items and placing it in an empty slot.

[0135] In various embodiments, the techniques disclosed herein are used to provide a robotic singulation system that can operate in a fully autonomous mode in most cases.

[0136] Although the above-described embodiments have been described in some detail for ease of understanding, the present invention is not limited to the details provided. There are many alternative ways to implement the present invention. The disclosed embodiments are illustrative and are not intended to be limiting. The present disclosure may be realized in the following forms: [Form 1] 1. A system comprising: a communication interface; a processor connected to the communication interface; Equipped with The processor: receiving sensor data via the communication interface, the sensor data including image data related to a workspace; generating a three-dimensional view of at least a portion of the workspace using the sensor data; the three-dimensional view includes boundaries of a plurality of items present in the workspace; determining a corresponding grasping strategy for each of at least a portion of the items, and for each grasping strategy, determining a corresponding probability of grasping success; The grasping strategy and the corresponding probability of successful grasping are used to business The system is configured to determine and implement a plan for autonomously operating a robotic structure to pick one or more items from space and place each item one-by-one at a corresponding location on a singulation transport structure. [Form 2] 10. The system according to claim 1, The system, wherein the robotic structure comprises a robotic arm. [Form 3] The system according to aspect 2, The system, wherein the robotic arm comprises an end effector configured to be used to grasp the one or more items. [Form 4] The system according to aspect 3, The end effector is a suction base ofThe system includes an end effector. [Form 5] The system according to aspect 4, the suction-based end effector comprises two or more independently actuated suction cup sets, each set comprising one or more suction cups, and the processor is configured to grasp a given item using one or more of the sets. [Form 6] The system according to aspect 5, the processor is configured to use two or more of the separately actuated suction cup sets to simultaneously grasp corresponding ones of the items from the workspace, move the items together to a destination position of a first one of the items, and sequentially place the items one by one into corresponding positions on the singulation transport structure. [Form 7] 10. The system according to claim 1, The singulation transport structure includes a partitioned conveyor. [Form 8] 10. The system according to claim 1, The processor 、 Receives sensor data; generating a three-dimensional view of at least a portion of the workspace using the sensor data; determining a corresponding grasp strategy for each of at least a portion of items present in the workspace; For each grasping strategy, determine the corresponding probability of grasp success; The grasping strategy and the corresponding probability of successful grasping are used to business Determining and implementing a plan for autonomously operating the robotic structure to pick the next one or more items from the space and place each item one-by-one at a corresponding location on the singulation transport structure.、 Repeat the process The system is configured as follows: [Form 9] The system according to aspect 8, The system, wherein the processor is configured to stop repetitive execution of the process in response to determining that no more items remain in the workspace. [Form 10] The system according to aspect 8, The processor is configured to abort operation based at least in part on a determination that no additional items are available for autonomous grasping despite one or more items remaining in the workspace. [Form 11] 11. The system of claim 10, The processor is configured to request human intervention based at least in part on a determination that no additional items are available for autonomous grasping despite one or more items remaining in the workspace. [Form 12] 10. The system according to claim 1, The processor is configured to determine the probability of successful grasping based at least in part on one or more of an item attribute and a degree of overlap of the item by one or more other items. [Form 13] 10. The system according to claim 1, The processor determines one or more candidate grasping strategies for the item and performs a grasping operation for each candidate grasping strategy. Mochinari Calculating the probability of success for each of said calculated outcomes Mochinari The system is configured to use a success probability to select a best grasping strategy selected for the item. [Form 14] 10. The system according to claim 1, The system, wherein the workspace comprises a chute or other container, the robotic structure comprises one of a plurality of robotic structures associated with the workspace, and the processor is configured to coordinate operations of the robotic structures to pick and place items from the workspace. [Form 15] 10. The system according to claim 1, The workspace includes a first workspace included in a plurality of workspaces associated with the singulation transport structure, each workspace having one or more robot structures associated therewith, and the processor controls the singulation transport structure to perform the singulation transport structure as described above. business a system configured to operate the respective robotic structures associated with each workspace to pick items from the spaces and place the items one-by-one on the singulation transport structure. [Form 16] 16. The system according to claim 15, The system, wherein the processor is configured to coordinate the operation of each of the robot structures associated with each workspace to maximize overall throughput in placing items one at a time onto the singulation transport structure. [Form 17] 17. The system according to claim 16, The system, wherein the processor is further configured to control operation of the singulation transport structure to maximize overall throughput in placing items one at a time onto the singulation transport structure. [Form 18] 16. The system according to claim 15, The system, wherein the processor is further configured to detect errors in the placement of items onto the singulation transport structure and perform responsive action. [Form 19] 19. The system according to claim 18, The response action includes assigning a robotic structure associated with a downstream workspace to correct the error. [Form 20] 1. A method comprising: General receiving sensor data via a communication interface, the sensor data including image data related to a workspace; generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present in the workspace; determining, for each of at least a portion of the items, a corresponding grasping strategy and, for each grasping strategy, a corresponding probability of grasping success; The grasping strategy and the corresponding probability of successful grasping are used to business determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from the space and place each item one-by-one at a corresponding location on a singulation transport structure; A method comprising: [Form 21] A computer program product embodied in a non-transitory computer-readable medium, General computer instructions for receiving sensor data via a communication interface, the sensor data including image data related to a workspace; computer instructions for generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present in the workspace; computer instructions for determining, for each of at least a portion of the items, a corresponding grasping strategy and, for each grasping strategy, a corresponding probability of grasping success; The grasping strategy and the corresponding probability of successful grasping are used to businesscomputer instructions for determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from space and place each item, one at a time, at a corresponding location on a singulation transport structure; A computer program product comprising:

Claims

1. 1. A system comprising: a communication interface; a processor connected to the communication interface; Equipped with The processor: receiving sensor data via the communication interface, the sensor data including image data related to a workspace; generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present within the workspace; determining a grasping strategy for each of at least a portion of the items; determining a probability of grasp success for each grasp strategy, the probability of grasp success being calculated based at least in part on learned data corresponding to past attempts to grasp similar items, the learned data including information regarding (i) a position of a portion of the item relative to one or more other items of the plurality of items, (ii) a degree of overlap of the items, and (iii) an orientation of the items; determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item one-by-one at a corresponding location on a singulation transport structure; A system configured to disturb at least a portion of the plurality of items present in the workspace.

2. 10. The system of claim 1, The processor: determining whether the workspace includes an item whose corresponding grasp success probability meets a predetermined minimum threshold; and perturbing at least a portion of the plurality of items present in the workspace in response to determining that the workspace does not include any items whose corresponding grasp success probabilities meet the predetermined minimum threshold. The system is configured as follows:

3. 3. The system of claim 2, The system, wherein disturbing at least a portion of the plurality of items present in the workspace includes the robotic structure disturbing the workspace or the at least a portion of the plurality of items present in the workspace.

4. 4. The system of claim 3, wherein the robotic structure disturbing the workspace or at least a portion of the plurality of items present in the workspace comprises using the robotic structure to poke, pull, or push one or more items to different locations in the workspace.

5. 10. The system of claim 1, The system, wherein the processor is configured to update the three-dimensional view, the grasping strategy, and the corresponding grasping success probability in response to an item being picked from the workspace and placed at the corresponding position on the singulation transport structure.

6. 10. The system of claim 1, The system, wherein the processor is configured to update the plan for autonomously operating a robotic structure to pick one or more items from the workspace in response to an item being picked from the workspace and placed at the corresponding position on the singulation transport structure.

7. 10. The system of claim 1, the robotic structure comprises a robotic arm, the robotic arm comprising a suction-based end effector configured to be used to grasp the one or more items, the suction-based end effector comprising two or more independently actuated suction cup sets.

8. 8. The system of claim 7, each suction cup set comprises one or more suction cups, and the processor is configured to simultaneously grasp a corresponding one of the items from the workspace using two or more of the separately actuated suction cup sets.

9. 9. The system of claim 8, wherein said two or more of said independently actuated suction cup sets move said items together to a destination location of a first one of said items.

10. 9. The system of claim 8, One of the independently actuated suction cup sets releases the first item at a destination of the first item.

11. 9. The system of claim 8, at least one other of said separately actuated suction cup sets sequentially places the remaining items one by one into corresponding locations on said singulation transport structure.

12. 10. The system of claim 1, The singulation transport structure includes a partitioned conveyor.

13. 10. The system of claim 1, The processor: Receives sensor data; generating a three-dimensional view of at least a portion of the workspace using the sensor data; For each of at least a portion of items present in the workspace, determining a corresponding grasping strategy, and for each grasping strategy, determining a corresponding probability of grasping success; determining and implementing a plan for autonomously operating a robotic structure to pick a next one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item one-by-one at a corresponding location on a singulation transport structure; and a system configured to repeatedly perform the steps of:

14. 14. The system of claim 13, The system, wherein the processor is configured to stop repetitive execution of the process in response to determining that no more items remain in the workspace.

15. 14. The system of claim 13, the processor is configured to abort operation based at least in part on a determination that no additional items are available for autonomous grasping despite one or more items remaining in the workspace.

16. 10. The system of claim 1, The processor is configured to determine the probability of successful grasping based at least in part on one or more of an item attribute of the item and a degree of overlap of the item by one or more other items.

17. 10. The system of claim 1, The processor is configured to determine one or more candidate grasping strategies for an item, calculate a probability of grasp success for each candidate grasping strategy, and use the calculated respective probability of grasp success to select a best grasping strategy for the item.

18. 10. The system of claim 1, The system, wherein the workspace comprises a chute or other container, the robotic structure comprises one of a plurality of robotic structures associated with the workspace, and the processor is configured to coordinate operation of the robotic structures to pick and place items from the workspace.

19. 10. The system of claim 1, The system, wherein the workspace includes a first workspace included in a plurality of workspaces associated with the singulation transport structure, each workspace having one or more robot structures associated therewith, and the processor is configured to operate each of the robot structures associated with each workspace to pick items from the workspace and place the items one by one onto the singulation transport structure.

20. 20. The system of claim 19, The system, wherein the processor is configured to coordinate the operation of each of the robot structures associated with each workspace to maximize overall throughput in placing items one at a time onto the singulation transport structure.

21. 1. A method comprising: receiving sensor data via a communications interface, the sensor data including image data related to a workspace; generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present in the workspace; determining a grasp strategy for each item of at least a portion of the items; determining a probability of grasp success for each grasp strategy, the corresponding probability of grasp success being calculated based at least in part on learned data corresponding to past attempts to grasp similar items, the learned data including information regarding (i) a position of a portion of the item relative to one or more other items of the plurality of items, (ii) a degree of overlap of items, and (iii) an orientation of the items; determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item one-by-one at a corresponding location on a singulation transport structure; disturbing at least a portion of the plurality of items present in the workspace; A method comprising:

22. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for receiving sensor data via a communications interface, the sensor data including image data related to a workspace; computer instructions for generating a three-dimensional view of at least a portion of the workspace using the sensor data, the three-dimensional view including boundaries of a plurality of items present in the workspace; computer instructions for determining a grasping strategy for each item of at least a portion of the items and for each grasping strategy determining a corresponding probability of grasping success, the corresponding probability of grasping success being calculated at least in part based on learned data corresponding to previous attempts to grasp similar items, the learned data including information regarding (i) a position of the portion of the item relative to one or more other items of the plurality of items, (ii) a degree of overlap of the items, and (iii) an orientation of the items; computer instructions for determining and implementing a plan for autonomously operating a robotic structure to pick one or more items from the workspace using the grasping strategies and corresponding grasping success probabilities and place each item one-by-one at a corresponding location on a singulation transport structure; computer instructions for disturbing at least a portion of the plurality of items present in the workspace; A computer program product comprising:

Images