Robotic system and method for controlling a robot to move items in a workspace

Abstract

This invention discloses a robotic single-dispatch system. In various embodiments, sensor data comprising image data associated with a plurality of items present in a work area is received. This sensor data is used to determine and implement a plan for autonomously operating a robotic structure to pick up one or more items from the work area and individually place each item in a corresponding position within a single-dispatch conveyor structure. This plan includes performing an active measure to modify or adapt a detected state or condition associated with one or more items in the work area.

Description

Technical Field

[0001] This application is generally concerned with the separate division of any mixed articles. Prior Technology

[0002] Parcel and other distribution centers may receive a random mix of items of various sizes, dimensions, shapes, weights, stiffness, and / or other attributes, often in a disorderly manner. Each item may have machine-readable information (such as text and / or optically or otherwise encoded information) that can be used to route the item via, for example, an automated sorting / routing system and / or processing. To read the information of a given item, in a typical method, the items are separated from each other via a process known as "single sorting".

[0003] Typically, sorting is performed manually by human workers. A batch of items arrives at a workstation via a chute or other conveyor, and a group of one or more human workers manually separate the items and place them onto a conveyor belt or similar in a defined space for a single item. For each item, its destination (or at least the next stage of transport) is determined by machine-readable information about the item, and the item is routed to a destination associated with the next stage (such as a bag, storage box, container, or other container associated with the next stage and / or a delivery vehicle or assembly area).

[0004] Manual sorting is labor-intensive and can be inefficient. For example, a downstream human worker may have almost no space to place sorted items, as upstream workers fill many individual item positions. Collective processing can be suboptimal.

[0005] Using robots to perform single-separation is challenging due to the chaotic mix of items arriving at a workstation, the dynamic flow of items at each station and overall, and the difficulty in using a robotic arm and end effector to automatically identify, grasp, and separate (single-separate) items. Summary of the Invention

[0006] This invention can be embodied in many ways, including as a program; a device; a system; a composition of matter; a computer program product embodied on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored in and / or provided by a memory coupled to and / or provided by the memory. In this specification, such embodiments or any other form of the invention may be referred to as technology. Generally, within the scope of this invention, the order of steps of the disclosed program may be changed. Unless otherwise stated, a component (such as a processor or a memory) configured to perform a task may be described as a general component temporarily configured to perform the task at a given time, or a specific component manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data (such as computer program instructions).

[0007] A detailed description of one or more embodiments of the invention is provided below, together with accompanying drawings illustrating the principles of the invention. The invention is described in conjunction with these embodiments, but is not limited to any particular embodiment. The scope of the invention is limited only by the scope of the invention claims, and the invention encompasses numerous alternatives, modifications, and equivalents. Many specific details are set forth in the following description to provide a thorough understanding of the invention. These details are provided for illustrative purposes, and the invention may be practiced according to the scope of the invention claims without some or all of these specific details. For clarity, known technical materials in the art related to the invention have not been described in detail so as not to unnecessarily obscure the invention.

[0008] A robotic system for performing sorting is disclosed. As used herein, sorting an item comprises picking up an item from a source stack / flow and placing the item on a conveyor structure (e.g., a segmented conveyor or similar conveyor). Sorting may, as needed, comprise classifying various items on the conveyor structure (e.g., by individually placing items from the source stack / flow into a slot or tray on the conveyor). In various embodiments, sorting and / or classification are disclosed. In various embodiments, sorting and / or classification are performed at least in part based on: detecting a state or condition associated with one or more items in a work area; and combining picking up an item from a source stack / flow (e.g., a work area) and placing the item on a segmented conveyor or similar conveyor for classification and routing to a downstream (e.g., final address / entity) destination with an active measure to adapt to the state or condition. In some embodiments, the robotic system determines a plan for sorting an item individually (e.g., picking up the item from a work area and placing it on a single-sorting conveyor structure), and takes proactive action in response to a detected state or condition initially determined after the plan is determined. In some embodiments, multiple robots are coordinated to maximize collective throughput. For example, the robotic system includes a plurality of robotic arms in the same work area, which operate to pick up a plurality of items from a source stack / flow and place such items on a single-sorting conveyor structure. The plurality of robotic arms can operate autonomously and independently.

[0009] A robotic system includes a robotic arm and an end effector for picking up items from a source stack / flow and placing them onto a segmented conveyor or similar transport for sorting and routing to a downstream (e.g., final location / physical) destination. In some embodiments, multiple robots are coordinated to perform a collective processing load. In various embodiments, one or more robots may be employed at a single station. A system may include multiple stations. As an example, each station may correspond to a distinct work area. Human workers may be employed at one or more stations. In various embodiments, the robotic system may be configured to invoke (request) the assistance of a human worker (e.g., via teleoperation of a robotic arm, manual task completion, etc.), for example, to handle an item that the robot cannot handle by fully automated processing and / or an item dropped by the robot. In some embodiments, multiple robotic arms operating in the same work area work independently to sort multiple items. One or more of the plurality of robotic arms may, in response to the detection of a collision or the possibility of a collision between two robotic arms, perform one of the active measures to avoid a collision between the two robotic arms.

[0010] According to various embodiments, a robotic sorting system performs an active measure to improve the sorting of an item (e.g., successfully picking up an item from a source pile / flow and placing it on a conveyor). For example, in response to determining that a detected state or condition hinders the implementation of a current plan to autonomously operate the robotic structure to pick up one or more items from a work area and place each item individually in a corresponding position in a sorting conveyor structure, the robotic structure performs one or more active measures to improve the likelihood of successful sorting (e.g., the robotic structure may determine, at least in part, the likelihood of successful sorting based on a detected state or condition). In some embodiments, the active measure includes using a robotic arm, an end effector of the robotic arm, a chute or other element in the work area to move or a blower to reassemble one or more items or debris on the source pile / flow or the reassembled work area. Active measures can be implemented to improve the scanning of a mark or identifier on a single item, improve the likelihood of picking up an item, improve the gripping of a single item, improve the release of an item from a robotic arm, or improve the operation of two robotic arms that independently separate items from the same work area (e.g., the same source stack / flow).

[0011] Parcel carriers, postal services, delivery services, large retailers or distributors, and other businesses and government entities that handle, transport and deliver items to and from different locations typically receive large quantities of items from different source locations, and each item will be delivered to one of a variety of destination locations.

[0012] Machines exist for handling, sorting, and routing items; however, for the use of machine readers and sorting equipment, items may need to be spaced apart and / or in a specific orientation so that a mark or tag can be read by a machine. This spacing and orientation may need to be achieved during the process of "introducing" items into a sorting / routing facility and may be performed in conjunction with a "sorting" or "sorting" procedure, such as a procedure that sorts items delivered to different locations according to a general destination (e.g., region, state, city, postal code, street, house number, etc.).

[0013] Machine readers (such as radio frequency (RF) tag readers, optical code readers, etc.) may require items to be spaced apart from each other (sometimes referred to as a "single" procedure) in order to reliably read a tag or code and enable the system to associate the obtained information with a specific item (such as an item in a specific location on a conveyor or other structure or tool).

[0014] In a typical parcel sorting operation, individual parcels may be picked from a bulk stack and placed onto a moving conveyor or tilting pallet sorting system. For most facilities, this type of sorting is entirely manual.

[0015] A typical, manual package sorting / classification procedure may include one or more of the following: • A chute containing unsorted packages slowly moves to a sorting table adjacent to a conveyor-based sorting system. • A worker's job is to "single" items onto a conveyor or a pallet-based sorting system. Workers ensure that each package being sorted onto the sorting machine is oriented so that a shipping barcode (or other optical code, electronic tag, etc.) can be read for sorting purposes (this orientation is usually determined by the facility's scanning infrastructure). Wait for an empty pallet or slot to pass through, and ensure that only one package is placed on each slot or pallet.

[0016] In a typical manual sorting / classification process, packages of various shapes and sizes arrive in bulk via manually (machine-fed) chutes, oriented in various directions. The packages may have different sizes, shapes, rigidities, and packaging. Typically, human workers remove packages from a chute fed to their respective workstations and place them one by one onto an open section or otherwise defined segment of a conveyor. Finally, at each workstation, numerous workers individually fill the packages at locations on one or more conveyors to facilitate downstream machine processing, such as reading codes or tags and performing automated sorting actions, such as routing the packages to a location within a facility associated with a destination to which they will be delivered. This location may involve further sorting (e.g., a more destination-specific location within the facility) and / or packing / loading the packages for further shipment (e.g., by truck or plane to a further destination where further sorting and delivery will occur, loading them onto a truck for local delivery, etc.). Simple Explanation of the Diagram

[0017] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0018] Figure 1 is a flowchart illustrating a procedure for receiving, classifying, and transporting goods for logistics distribution based on relevant technologies.

[0019] Figure 2A illustrates one of the robot single-system diagrams according to one of the various embodiments.

[0020] Figure 2B illustrates one of the robot single-system diagrams according to one of the various embodiments.

[0021] Figure 3A is a diagram of a procedure for picking up and placing items for sorting, according to various embodiments.

[0022] Figure 3B is a diagram of a procedure for picking up and placing items for sorting, according to various embodiments.

[0023] Figure 3C is a diagram of a procedure for picking up and placing items for sorting, according to various embodiments.

[0024] Figure 4 is a diagram of a procedure for using an active measure to separate one or more items according to various embodiments.

[0025] Figure 5A is a diagram of a procedure for using an active measure to separate one or more items according to various embodiments.

[0026] Figure 5B illustrates a robot separation system that uses an active measure to separate one or more items according to various embodiments.

[0027] Figure 5C is a diagram illustrating the use of an active measure to separate one or more items from a chute of a robot single-splitting system according to various embodiments.

[0028] Figure 5D is a diagram illustrating a procedure for separating one or more items by means of an active measure that interrupts an item on a chute of a robot separation system according to various embodiments.

[0029] Figure 5E is a diagram illustrating a procedure for separating one or more items by using an active measure of blowing air to reposition an item or debris on a chute of a robot separation system, according to various embodiments.

[0030] Figure 5F is a diagram illustrating a procedure for separating one or more items by means of an arm attached to a chute of a robot separation system to reposition an item or debris on the chute, according to various embodiments.

[0031] Figure 6A is a diagram illustrating a robot single-distribution system according to various embodiments, which uses one or more sensors to detect the state or condition of an item in the work area of ​​a robot single-distribution system.

[0032] Figure 6B is a diagram of a procedure for sorting an item using sensor data according to various embodiments.

[0033] Figure 7A illustrates a diagram of a robot separation system for separating one or more items, based at least in part on controlling one or more end effectors to release one or more items, according to various embodiments.

[0034] Figure 7B is a diagram of a procedure for separating one item based at least in part on controlling one or more end effectors to release one or more items, according to various embodiments.

[0035] Figure 8A is a diagram illustrating a robot subsystem performing a diagnostic to detect deviations from normal operation according to various embodiments.

[0036] Figure 8B is a diagram of a procedure for performing a diagnostic to detect deviations from the normal operation of a robot single subsystem, according to various embodiments.

[0037] Figure 9 is a diagram of a hierarchical scheduling system based on various embodiments of a robot single-system.

[0038] Figure 10A is a diagram illustrating a robotic separation system comprising one of a plurality of robotic arms operating within the same work area to separate items within that work area, according to various embodiments.

[0039] Figure 10B is a diagram illustrating a procedure for separating one of a plurality of items in the same work area using a robotic separation system comprising one of a plurality of robotic arms operating within a work area, according to various embodiments.

[0040] Figure 10C is a diagram illustrating a procedure for separating one of a plurality of items in the same work area using a robotic separation system comprising one of a plurality of robotic arms operating within a work area, according to various embodiments. Implementation

[0041] Figure 1 is a flowchart illustrating a procedure for receiving, classifying, and transporting goods for logistics distribution based on relevant technologies.

[0042] In the illustrated example, program 100 begins with a summarizing program 102, which provides items to one or more workstations for sorting via a sorting program 104. In various embodiments, the sorting program 104 is at least partially automated by a robotic sorting system as disclosed herein. The sorting program 104 receives piles or streams of different items via the summarizing program 102 and provides a stream of sorted items to a sorting / routing program 106. For example, the sorting program 104 may place items one by one onto a segmented conveyor or other structure that feeds items one by one into a sorting / routing machine. In some embodiments, the items are placed in an orientation such that a tag or label can be read by a downstream reader configured to read routing (e.g., destination address) information and use that routing information to sort the items to a corresponding destination (such as a pile, storage bin, or other groups of items destined for the same next intermediate and / or final destination). Once sorted, the group of items is processed by a transportation process 108 to a common next / final destination. For example, items may be placed in containers, loaded onto delivery or transport trucks or other vehicles for delivery to the next / final destination.

[0043] This discloses a robotic system that combines proactive measures—including picking up an item from a source stack / flow (e.g., a work area) and placing it on a segmented conveyor or similar means for sorting and routing to a downstream (e.g., final address / entity) destination—with or without human intervention. In various embodiments, a state or condition associated with one or more items in the work area is detected, and in response, an proactive measure is executed to facilitate the single-sorting of items. According to various embodiments, the single-sorting of items from a source stack / flow is improved by using a dynamic single-sorting method or system that implements a proactive measure in response to a detected state or condition (e.g., a state or condition that is absent and / or undetected when the robotic system determines that an initial plan for single-sorting an item in the work area is not used).

[0044] Figure 2A illustrates one of the robot single-system diagrams according to one of the various embodiments.

[0045] In the illustrated example, system 200 includes a robotic arm 202 equipped with a suction-type end effector 204. While the end effector 204 is a suction-type end effector in the illustrated example, in various embodiments, one or more other types of end effectors may be used in a single subsystem as disclosed herein, including (but not limited to) a pinch-type end effector or other types of actuated grippers. In some embodiments, end effector 204 includes one or more suction-type ends (e.g., one or more suction cups). In various embodiments, the end effector may be actuated by suction, air pressure, pneumatics, hydraulics, or other actuation. The robotic arm 202 and end effector 204 are configured to pick up packages or other items arriving via a chute or storage bin 206 and place each item in a corresponding position on a segmented conveyor 208. In this example, items are fed into the chute 206 from an inlet end 210. For example, one or more human and / or robotic workers may feed items directly or via a conveyor or other electromechanical structure configured to feed items into one of the chute 206 into the inlet end 210 of the chute 206.

[0046] In the examples shown, one or more of the robotic arm 202, end effector 204, and conveyor 208 are coordinated and operated by a control computer 212. In various embodiments, a robotic unit as disclosed herein may include one or more sensors from an environment of its modeled workspace. In the example shown in Figure 2A, system 200 includes image sensors, in this example including 3D cameras 214 and 216. In various embodiments, other types of sensors may be used (individually or in combination) in a unit system as disclosed herein, including a camera, an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, a weight sensor, and the like. In various embodiments, the control computer 212 includes a work area environment status system (such as a vision system) for identifying individual items, clutter in the work area, and the orientation of each item based on sensor data, such as image data provided by image sensors (in this example, including 3D cameras 214 and 216). In some embodiments, the work area environment status system includes sensors in a robotic arm for detecting the weight of an item (e.g., an item that has been grasped) or detecting information from which an estimated weight is determined. For example, information relating to a quantity of current, voltage, and / or power used by one or more motors driving the movement of the robotic arm can be used to determine the weight of the item (or an estimated weight). As another example, the chute includes a weight sensor, and the weight of the item is determined based on the difference between the weight on the chute measured by the weight sensor before and after the item is picked up. As another example, information relating to the output of one or more sensor arrays can be used to determine one position of an item in the work area, one position of an item when the item is grasped and / or moved by a robotic arm, and / or one position of the robotic arm (e.g., based on the output of one subset of sensors from one or more sensor arrays compared to another subset of sensors from one or more sensor arrays).

[0047] The work area environment status system generates an output used by the robot system to determine and implement a plan for autonomously operating a robotic structure to pick up one or more items from the work area and place each item in a corresponding available, defined location (such as a section of segmented conveyor 208) for machine identification and classification. In some embodiments, the work area environment status system generates an output used by the robot system to detect a state or condition associated with one or more items in the work area, and / or a state or condition associated with another element of the robot arm or work area (e.g., otherwise characterizing sensor data or information of the work area and the items within it). According to various embodiments, in response to detecting (e.g., determining) a state or condition associated with one or more items in the work area, the robot system implements one or more proactive actions in conjunction with sorting an item. The proactive actions may include updating the plan for autonomously operating a robotic structure to pick up one or more items from the work area and place each item individually in a corresponding location within a single-segment conveyor structure. In some embodiments, proactive measures or update plans may include manipulating the robot structure to alter or adapt to detected states or conditions (e.g., implementing changes to how an item is individually sorted, implementing reorganization of items within a state source pile / flow to make it easier to grasp a selected item, etc.).

[0048] In various embodiments, a robot system as disclosed herein (e.g.) includes and / or performs one or more of the following by operating a control computer (such as control computer 212): • By combining data from multiple sensors (including 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors), computer vision information is generated to produce a 3D view of a work area containing one or more sorting stations. The robotic system can determine the characteristics of one or more items and / or debris or other anomalies in this 3D view of the work area. A robotic system coordinates the operation of multiple robots to avoid collisions, interference, and contention for picking up the same item and / or placing an item at the same destination location as another robot (e.g., a segment of a conveyor). The system can coordinate the operations of multiple robots operating within the same work area to individually sort multiple items. For example, the system can coordinate the operations of multiple robots so that each robot can operate independently to pick up and place items. If a collision risk is detected, a response is taken to ensure that the multiple robots do not collide with each other during individual sorting. The robotic system coordinates the operations of multiple robots to ensure that all items are placed and that there is only one item per slot / location. For example, if robot A drops an item, the system delegates the task of picking it up to robot B; an item placed but incorrectly oriented is picked up by the same or another robot and adjusted or moved to another location; two or more items in a single destination slot cause a downstream robot station to pick up one of those two or more items from the conveyor and place it in a new location; etc. • Two or more robotic systems work together to pick up objects that are too heavy or too large for one robotic system to handle. The robot system continuously updates the motion plans of each robot and all robots together to achieve a desired collective throughput (e.g., maximizing collective throughput, reaching a predefined threshold for collective throughput, etc.). In response to the determination that two or more robots have collided or that a collision would occur according to their respective planned movements for individual items, the robot system implements an active measure to ensure that the two or more robots avoid collision or otherwise resets the independent operation of the two or more robots. If two robots are independently assigned the task of retrieving the same item, the system randomly selects one robot to retrieve the item while the other robot moves to the next item (e.g., identifying, selecting, determining the grasping strategy, picking up, moving and placing according to the plan). • The robot system can manage the independent operation of multiple robots to ensure that robots select items at different times to avoid the same item being selected by two different robots for single sorting. • Control the movement and / or speed of the conveyor as needed to avoid empty positions and achieve the desired robot productivity (processing capacity). In response to the determination that an item has been misplaced or dropped, the system assigns a robot or (if necessary) a human worker to pick up the item and return it to the source pile of the picking robot itself, or, if available or better, to place it on the next open slot on the conveyor. • Control the upstream robot to intentionally open some slots so that the downstream robot can place items on the conveyor. • A warning that a malfunction that the same or another robot cannot correct requires human (or other robot) intervention to resolve. • In response to the determination that a clamping strength (e.g., a pressure obtained by the end effector) is abnormal (e.g., less than expected during normal operation), a diagnostic procedure is performed that includes testing the clamping strength on a predefined surface and determining whether remedial action is required for the end effector. • In response to the determination that debris is in the work area and / or interferes with the single-item assembly of one or more items, take proactive measures to move / remove the debris or reposition the single-item assembly (e.g., increase the likelihood that the item is successfully picked up from the source stack / flow and placed on the conveyor structure). • In response to a detected state or condition, perform proactive measures such as moving / removing debris or reorganizing the single item to be picked up (e.g., improving the likelihood that the item is successfully picked up from the source pile / flow and placed on the conveyor structure). For example, operate (e.g., under robot control) a chute conveyor to bring the selected single item closer to the front of the chute for faster and easier pickup by a robotic arm. • Use sensor data from the work area environment status system (e.g., from one or more sensors within the work area) to model the chute flow (or model the work area environment), detect deviations from an expected chute flow (or an expected work area environment), use sensor data to detect blockages or anomalies in the chute flow or work area environment, and implement proactive measures to clear the blockages. • Using sensor data from the work area environmental status system to detect one or more characteristics of an item selected for individual use, determine the expected response of the item's gripping or release to an improved active measure, and implement the active measure to improve the gripping or release of the item. • Using sensor data to determine that the robotic arm has grasped a plurality of items by combining one of the items into a single part, and determining a plan for releasing the plurality of items so as to place each item individually in a corresponding position in a single part conveyor structure (e.g., selecting different positions in the single part conveyor structure where the corresponding items will be placed, and / or determining a strategy for operating the end effector to release a first subset of a plurality of items from a second subset of a plurality of items at different times). • Select a slot on the conveyor structure in which the selected item will be placed, based on the size of one of the selected items and / or one or more characteristics of one of the items in a slot on the conveyor structure. For example, select a slot to ensure that the selected item is not placed in a slot adjacent to one of the slots containing a tall or large item. • The movement and speed of a robotic arm that dispenses a single item are determined at least in part based on the speed of a conveyor belt. • The trajectory of an item to be sorted is determined based at least in part on one or more of the characteristics of the item, the characteristics of the work area environment, and / or the characteristics of the conveyor structure (e.g., the speed of the conveyor belt or the size of the item).

[0049] In various embodiments, the individual items to be mixed may include packages, parcels, and / or letters of various shapes and sizes. Some items may be standard packaging, and one or more of their attributes may be known, while others may be unknown. In various embodiments, sensor data (such as image data) is used to identify individual items (e.g., via image segmentation). The boundaries of partially occluded items may be estimated, for example, by identifying an item as a standard or known type and / or by extending the visible item boundaries to a logical estimation range (e.g., extrapolating to two edges meeting at an occluded corner). In some embodiments, an overlap degree (i.e., occlusion by other items) is estimated for each item, and this overlap degree is taken into account when selecting the next item to attempt to grasp. For example, for each item, a score may be calculated to estimate the probability of a successful grasp, and in some embodiments, this score is determined at least in part by the degree of overlap / occlusion by other items. Items with less occlusion may be more likely to be selected (e.g., other considerations are the same).

[0050] If a source stack / flow contains any items to be mixed, it typically includes items with different types of packaging (such as a cardboard box, a paper envelope, a plastic bag (e.g., a polyethylene bag)). The robotic system can determine the packaging of an item based on visual data obtained from self-sensors or based on pressure obtained between the end effector and the item when the robotic arm attempts to pick it up. Sensor data can be used to identify a packaging type corresponding to a specific item in the source stack / flow. In some embodiments, the robotic system determines a strategy for grasping the item based at least in part on the packaging type corresponding to the item. For example, a relatively heavy item packaged in a plastic bag will typically experience "tenting" between the end effector suction cups. Tenting can cause suboptimal suction from the end effector of the robotic arm, and therefore, the grasp of this item is suboptimal. According to various embodiments, in response to a determination that the item is relatively heavy (e.g., the weight exceeds a predefined threshold) and that the item is packaged in a plastic bag, or in response to a determination that a bulge is caused when the item is grasped, the robot structure performs an active measure to change or adapt to a determination of "bulge" or the item's packaging (e.g., a determination of a packaging type, a packaging material, etc.). As an example, the robot structure performs an active measure of partially lifting the package and dragging it from the chute to a corresponding slot in the conveyor structure.

[0051] In various embodiments, multiple 3D and / or other cameras can be used to generate image data. A 3D view of one of the scenes can be generated, and / or in some embodiments, a camera array is used to observe the scene from different angles, and the camera least obstructed relative to a work area and / or one or more specific items in that work area is selected and used in conjunction with the grasping and movement of one or more items. The image data can be used to detect debris on or within the chute, a blockage in the chute flow of items passing through the work area, the number of items grasped by the robot structure during a single segment of a selected item, characteristics of one or more items occupying slots on the conveyor structure, etc.

[0052] According to various embodiments, one or more cameras serve various purposes. One or more cameras can provide a richer, full 3D view of a scene (e.g., a work area). Alternatively or concurrently, one or more cameras can operate cohesively to minimize errors attributable to the brightness of the packaging when light reflected from a package and entering one camera can interfere with its operation; in this case, another camera positioned at a different location provides a backup. In some embodiments, one or more cameras can be selectively triggered by a predictive vision algorithm that determines which camera has the best viewing angle and / or lowest error rate for picking up a particular package. Thus, the robotic system can operate using information about an item obtained from one or more cameras (e.g., among multiple cameras in a work area) that are best suited for viewing an item. In some embodiments, one or more cameras are mounted on an actuated base, the position and orientation of which the system can change to provide a better perception (e.g., view) of a package.

[0053] In some embodiments, the robotic system may select the field of view of one or more cameras. The field of view of each camera may be selected (e.g., determined) to increase the quality of object segmentation by intentionally filtering out portions of the field of view and to increase segmentation speed by reducing computation on a larger field of view.

[0054] In various embodiments, another purpose supplied by the cameras is to detect any kind of unforeseen error or damage to the environment during robot operation. Cameras placed on the robot and in the environment have different error and accuracy profiles. Cameras on the robot can be more accurate because they are rigidly fixed to the robot, but are slower to use because their use requires the robot to slow down or stop. Cameras in the environment have a stable field of view and are actually faster because the robot can perform multiple tasks and do other things while a camera is taking a picture. However, if someone moves or shakes the camera mount, they can become out of sync with the robot and cause errors. In various embodiments, images from the robot and non-robot cameras are combined (e.g., occasionally or when a package is lost) to detect whether the robot is synchronized with the non-robot cameras. If the cameras are determined to be out of sync, the robot takes corrective action, such as performing a calibration or synchronization procedure, alerting a human operator, etc. In some embodiments, a camera may be non-rigidly mounted on a robotic arm, and in some of these embodiments, a gyroscope and / or accelerometer on the camera may be used to filter or compensate for the movement of the mounting base.

[0055] According to various embodiments, system 200 may include one or more sensors other than a plurality of cameras or other than a plurality of cameras, such as an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, and one or more of the like. Referring to FIG2A, in various embodiments, the robotic arm 202 is driven by one or more motors (e.g., one or more motors at various movable joints or mounting locations). In some embodiments, the work required to drive the robotic arm 202 (e.g., moving the robotic arm when it attempts to separate an item) indicates one or more characteristics of the item to be separated. For example, in some embodiments, the weight of an item may be calculated (or estimated) based on the work required to drive the robotic arm 202 when the item is in its grasp. In various embodiments, a current sensor, a voltage sensor, a power sensor, and / or the like, or a combination thereof, is used to measure the work required to drive the robotic arm 202. In response to determining the weight of an item during the sorting process, the robot system may perform an active measure adapted to the weight of the item. In some embodiments, in response to determining that the weight of the item is greater than a predefined threshold, the robot system 200 adjusts its plan to sort the item by partially picking it up and dragging it to a corresponding position on the conveyor structure (e.g., the opposite of fully picking up the item and moving the arm to place it on the conveyor structure). In some embodiments, in response to determining the weight of the item, the robot structure adjusts the speed at which it moves the robot arm (and the item). For example, the greater the weight of the item, the greater the shear force between the item and the end effector 204 as the robot arm 202 moves. Furthermore, the shear force may increase with the speed at which the robot arm operates (e.g., the speed at which the robot arm moves the item). Therefore, the robot system 200 may control the speed of the robot arm 202 at least in part based on the weight of the item to ensure that the item remains firmly gripped by the robot arm. Although the description herein is based on the use of a current sensor, a voltage sensor, a power sensor and / or the like to measure weight, a force sensor configured in the robot arm 202 or the end effector 204 may also be used to measure weight.

[0056] Referring further to Figure 2A, in the illustrated example, system 200 further includes an on-demand teleoperation device 218 available for use by a human worker 220 to operate one or more of the robotic arm 202, end effector 204, and conveyor 208 via teleoperation. In some embodiments, control computer 212 is configured to attempt to grasp and place items in a fully automated mode. However, if, after attempting to operate in a fully automated mode, control computer 212 determines that it does not have a (further) strategy for grasping one or more items, in various embodiments, control computer 212 sends an alert to obtain assistance from a human operator via teleoperation (e.g., by human operator 220 using teleoperation device 218). For example, in some embodiments, in response to detecting a state or condition affecting the flow of items through chute 206, control computer 212 may attempt to perform one or more actions to facilitate the movement of items. If a fully automated attempt to respond to a detected state or condition fails to resolve the state or condition, the control computer may prompt a human operator 220 (e.g., to use on-demand teleoperation device 218 to resolve the state or condition remotely. In various embodiments, the control computer 212 may display a user interface or other interface that identifies the state or condition and / or presents human-selectable options for controlling the robotic arm 202, end effector 204, and / or other elements and tools as disclosed herein (e.g., blowers, vibrators, chute conveyors, etc.) to change the state or condition.

[0057] In various embodiments, control computer 212 uses image data from cameras (such as cameras 214 and 216) to provide a visual display of the scene to human worker 220 to facilitate teleoperation. For example, control computer 212 may display a view of a pile of items in chute 206. In some embodiments, control computer 212 performs segmentation processing on image data generated by cameras 214 and 216 to identify item / object boundaries. Masking techniques may be used, for example, to highlight individual items using different colors. Operator 220 can use the visual display of the scene to identify the items(s) to be grasped and uses teleoperation device 218 to control robotic arm 202 and end effector 204 to pick up the items(s) from chute 206 and place each item in a corresponding position on conveyor 208. In various embodiments, once the items(s) that prompted human intervention have been placed on the conveyor, system 200 resumes fully automated operation. In various embodiments, in the presence of human intervention, the robotic system observes human workers (e.g., completing tasks manually or remotely using a robotic arm and end effector) and attempts to learn a strategy for completing tasks in an autonomous mode (preferably) in the future. For example, the system may learn a strategy for grasping an object, for instance, by observing the position of a human worker grasping the object and / or by remembering how the human worker grasped the object remotely using a robotic arm and end effector.

[0058] In some embodiments, system 200 invokes assistance from human operator 220 in response to a determination of an anomaly in the operation of system 200. One example of an anomaly is a lack of a threshold pressure obtained between end effector 204 and the item during the single-partitioning of an item. In response to detecting that the pressure obtained between end effector 204 and the item is less than a threshold pressure value, robot system 200 may perform a diagnostic procedure in conjunction with an assessment of whether robot system 200 is performing correctly. For example, system 200 may perform a diagnostic on the ability of end effector 204 to engage an item and obtain a predetermined threshold pressure value. In response to a determination that system 200 is not performing correctly (e.g., end effector 204 cannot engage an item and obtain a predetermined threshold pressure value), system 200 invokes assistance from human operator 220. In some embodiments, control computer 212 sends an alert to human operator 220. This alert may indicate the basis of the problem (e.g., an indication that the end effector cannot engage an item and obtain a predetermined threshold pressure value). For example, an alert can provide human operator 220 with a recommended or requested remedial action.

[0059] Figure 2B is a diagram illustrating one of the robot single-station systems according to various embodiments. In the example shown, the robot single-station system of Figure 2A has been expanded to include a plurality of single stations. Specifically, in addition to a robot arm 202 configured to pick up items from chute 206 and place each item on a corresponding available and / or assigned position on segmented conveyor 208, the system shown in Figure 2B includes three additional stations: robot arms 230, 232, and 234 positioned and configured to pick up / place items from chute 236, 238, and 240, respectively. In addition to cameras 214 and 216, additional cameras 224 and 226 are also included to provide a 3D view of the complete scene (including each of the four stations / chute 206, 236, 238, and 240 and conveyor 208).

[0060] In various embodiments, the control computer 212 coordinates the operation of four robotic arms 202, 230, 232 and 234 and associated end effectors together with the conveyor 208 to pick up / place items from chutes 206, 236, 238 and 240 onto the conveyor 208 in a manner that achieves a desired collective processing capacity of the system (e.g., a collective processing capacity that meets a processing capacity threshold).

[0061] Although each station in the example shown in Figure 2B has one robotic arm, in various embodiments, two or more robots may be deployed at a station to avoid interference between their operations and movements and to maximize their collective throughput (including by avoiding and / or managing contention for picking up and placing the same item) under the control of an associated control computer (such as control computer 212 in the example shown in Figure 2B). In some embodiments, multiple robotic arms operating in the same work area work independently to single-sort multiple items. One or more of the multiple robotic arms may respond to the detection of a collision or the possibility of a collision between two robotic arms and perform an active measure to avoid a collision between the two robotic arms. For example, control computer 212 may coordinate the operation of the multiple robots so that the multiple robots can operate independently while ensuring that the multiple robots and / or the items grasped by the multiple robots do not collide with each other during single-sorting. In some embodiments, control computer 212 implements / enforces a "force field" between two or more robots to prevent collisions between the two or more robots. As an example, a robot (or control computer 212) accesses information, determines its own position and the positions of one or more other robots from that information, and controls the robot to avoid an intersection between its own position and the positions of one or more other robots at a specific time. In some embodiments, a first robot reserves an airspace (e.g., a specific location) that will be used by the first robot during the single distribution of an item. In conjunction with a second robot scheduling the single distribution of an item, the second robot determines the single distribution plan based at least in part on the airspace reserved by the first robot. For example, in conjunction with scheduling the single distribution of items, the second robot determines that the plan cannot include movement through the airspace reserved by the first robot and that the second robot determines that no movement of the second robot or the item through the airspace reserved by the first robot is required during the time the airspace is reserved.

[0062] In various embodiments, a scheduler coordinates the operations of a plurality of robots (e.g., one or more robots working at various stations) to achieve the desired throughput without conflict between the robots, such as a robot placing an item in a location that the scheduler has assigned to another robot.

[0063] In some embodiments, at least a subset of a plurality of robots working in a work area independently picks up an item, and determines a corresponding plan for single-distributing the item. The at least subset of robots may pick up items in a predefined order such that no two robots simultaneously select or pick up an item. Each of the at least subset of robots may select or pick up an item based on items currently available at the time of selection. Therefore, a second robot, which picks up an item after a first robot, from at least two subsets of the plurality of robots, will select an item different from the one selected or picked up by the first robot for single-distribution.

[0064] In various embodiments, such as those disclosed herein, a robotic system coordinates the operation of multiple robots to pick up items one after another from a source storage bin or chute and place the items at an assigned location on a conveyor or other device to move the items to the next stage of machine identification and / or classification.

[0065] In some embodiments, multiple robots may pick up from the same chute or other source container. In the example shown in Figure 2B, for example, robot arm 202 may be configured to pick up from chute 206 or chute 236. Similarly, robot arm 230 may pick up from chute 236 or chute 238 and robot arm 232 may pick up from chute 238 or chute 240. In some embodiments, two or more robot arms configured to pick up from the same chute may have different end effectors. One of the robot sorting systems disclosed herein may select the robot arm most suitable for picking up and sorting a given item. For example, the system determines which robot arms can reach the item and selects one with the most suitable end effector and / or other properties that allow for successful grasping of the item.

[0066] Although a fixed robotic arm is shown in Figure 2B, in various embodiments, one or more robots may be mounted on a mobile delivery vehicle, such as a robotic arm mounted on a chassis configured to move along a rail, track, or other guide, or a robotic arm mounted on a mobile cart or chassis. In some embodiments, a robotic tool actuator other than a robotic arm may be used. For example, an end effector may be mounted on a rail and configured to move along the rail, and the rail may be configured to move on one or more axes perpendicular to the rail so that the end effector can move to pick up, translate, and place an item as disclosed herein.

[0067] According to various embodiments, system 200 manages a distributed data structure relating to the operation of a plurality of robots in system 200 and / or the state of a conveyor structure. For example, the distributed data structure may include one or more fields associated with slots in the conveyor structure. According to various embodiments, the distributed data structure operates at a speed significantly exceeding the speed of robot operation in system 200. For example, the distributed data structure operates (e.g., updates) at approximately 1 µs or 1 ms, while the time for robot physical operation / movement is approximately 100 ms. Because the robot operation speed is slower than the distributed data structure operation speed, the distributed data structure is updated relatively quickly to reflect changes in the state of the work area (e.g., the state of the conveyor structure), and by the time the robot combines a decision for a single item sorting plan / strategy (e.g., selecting / requesting a slot in the conveyor) to obtain and / or use information from the distributed data structure, the distributed data structure may already be updated with the latest state. In some embodiments, the relative speed of the distributed data structure reduces the likelihood of two robots simultaneously requesting a slot on the conveyor and causing a malfunction in the distributed data structure. Therefore, the distributed data structure can be updated based on the operation of a robot or on a single-unit plan associated with a robot. In various embodiments, single-unit robots, operating independently of a system associated with an output conveyor, update the distributed data structure using information related to a plan or information related to one or more characteristics of the work area (e.g., whether a slot in the conveyor is occupied or requested by a robot in the system for use as a planned destination for placing an item on the conveyor). In some embodiments, if a robot receives an error in conjunction with an attempt to write information to the distributed data structure (e.g., requesting a slot in the conveyor for its use), the robot waits for a predetermined time interval and re-attempts to write the information to the distributed data structure. If data cannot be written because another robot has already written data to that location (e.g., requested an associated slot in the output conveyor), the robot selects another slot determined to be available by reading another location in the data structure. According to various embodiments, in response to a robot updating the data structure, the data structure is automatically updated relative to one or more other robots within system 200. For example, in response to a decision to perform an update (e.g., a write or delete operation), that update is distributed to other robots within system 200. The distributed data structure can be a shared structure read by all robots, or a robot (e.g., individual robots) can store a local copy and propagate updates to other robots across the system. For example, a robot can synchronize modifications to the data structure (e.g., updates, such as schedules or reserved slots on a conveyor) to other robots within the system.

[0068] According to various embodiments, the distributed data structure includes a field associated with a slot in the transport structure, which is used to indicate whether the slot is occupied or reserved for the item in conjunction with a robot's single-splitting of an item. For example, a value in the field associated with a slot indicates whether the slot can be reserved or used by another robot to schedule an item. In some embodiments, a slot in the transport structure is reserved when a robot determines (or updates) a plan to single-splitting an item. The slot in the transport structure is reserved based at least in part on the distributed data structure relating to the state of the transport structure. For example, a slot associated with a field indicating that the slot is empty or unreserved may be reserved for single-splitting an item. Sometimes, a robotic arm may erroneously release an item in a slot other than the slot corresponding to the single-splitting plan, or in a manner in which the item crosses two slots (e.g., adjacent slots). A corresponding robot (or a downstream robot) can detect a slot containing an item that contradicts a corresponding field in the distributed data structure (e.g., indicating that the slot is empty or unreserved). In response to detecting an item in a slot that contradicts a corresponding field in the distributed data structure, the robot system updates the data structure to indicate whether the slot is occupied or reserved.

[0069] According to various embodiments, the distributed data structure includes information relating to a timestamp, a conveyor speed, and one or more characteristics of a slot in the conveyor (e.g., an indication of whether a slot is occupied or reserved). The robotic system can determine a plan for sorting an item from a source pile / flow into a slot in the conveyor based at least in part on the distributed data structure. For example, system 200 determines a set of slots from which an item picked up from the source pile / flow can be placed based on the timestamp and the conveyor speed. System 200 can select an empty or unreserved slot from this set of slots as the slot where the item will be sorted. The timestamp and conveyor speed are used because system 200 can determine, based on the operation of the corresponding robot, one or more slots that may cause the item to be sorted to intersect with it.

[0070] Figure 3A is a diagram of a procedure for picking up and placing items for sorting according to various embodiments. In some embodiments, procedure 300 is implemented by a robotic system (such as system 200 of Figures 2A and 2B) that operates to sort one or more items in a work area. The robotic system includes one or more processors (e.g., in control computer 212 in the examples shown in Figures 2A and 2B) that operate (including by executing procedure 300) to cause a robotic structure (e.g., a robotic arm) to pick up and place items for sorting.

[0071] At 310, sensor data relating to the work area is obtained. In some embodiments, a robotic system obtains sensor data relating to the work area from one or more sensors operated within the system. As an example, sensor data is obtained at least in part based on outputs from image sensors (e.g., 2D or 3D cameras), an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, and the like.

[0072] According to various embodiments, the sensor data obtained in relation to the work area includes information from which a model of the work area can be generated. For example, one or more characteristics associated with the work area are determined at least in part based on the sensor data. The sensor data may be used in conjunction with determining at least one characteristic of one or more items within the work area (such as items in the source pile / flow of a chute, or an item grasped by a robotic arm). In some embodiments, the sensor data may be used in conjunction with determining one or more characteristics of the conveyor structure (such as determining whether there are empty or unreserved slots on the conveyor, determining a speed of the conveyor, and / or determining the characteristics of at least one slot or at least one item already on the conveyor).

[0073] At 320, a plan or strategy for individually distributing one or more items in a work area is determined. In some embodiments, a robot system determines a plan or strategy for picking up at least one item from a source pile / flow in the work area and placing the at least one item individually in a slot on a conveyor. In various embodiments, the plan or strategy for individually distributing one or more items is determined on a per-robot basis, such that if the robot system comprises a plurality of robots, each robot operates independently of the other robots(s).

[0074] According to various embodiments, a plan or strategy for single-sorting one or more items in a work area is determined at least in part based on sensor data. For example, a plan or strategy for single-sorting one or more items includes selecting an item to be single-sorted within a source stack / stream. The selected item may be identified from other items or objects within the work area at least in part based on sensor data (e.g., the boundary between the item and other items or objects within the work area may be determined). As an example, one or more characteristics related to the selected item are determined at least in part based on sensor data. The one or more characteristics related to the selected item may include an item size, an item packaging, one or more identifiers or markings on the item (e.g., an item fragile indicator, a shipping mark on the item, etc.), an estimated weight of the item, and the like, or any combination thereof. As another example, a plan for single-sorting one or more items includes determining that a robotic structure (e.g., a robotic arm) on a conveyor structure (e.g., a slot on a conveyor) will place one item at a single location. The location of an item to be placed on a conveyor structure can be determined at least in part based on a timestamp, a speed of a conveyor, and one or more characteristics of a slot in the conveyor (e.g., an indication of whether the slot is occupied or reserved), and the like or any combination thereof. As another example, a plan or strategy for sorting one or more items includes determining a path or trajectory along which a robotic arm will move the item during sorting. The path or trajectory along which the item will be moved can be determined at least in part based on the location of one or more other objects within the work area (such as a frame of a chute, other items in a source stack / flow, items on a conveyor, other robots operating within the work area, a reserved space for the operation of other robots, sensors within the work area, etc.). For example, the path or trajectory of the item may be determined to move a portion of an item including an identifier (e.g., a shipping mark) to an area where a scanner can scan the identifier, or the path or trajectory of the item may be determined to maximize the likelihood that one or more scanners will read the identifier on the item along the path or trajectory.

[0075] At 330, items are individually sorted. In some embodiments, items are individually sorted in response to a plan or strategy for sorting items. For example, a robotic arm is operated to pick up one or more items from a work area and place each item individually in a corresponding location in a sorting conveyor structure. Item sorting includes picking up items from a work area (e.g., from a source stack / flow) and placing the items individually on the conveyor structure. The robotic system sorts items at least in part based on a plan or strategy for sorting items.

[0076] At 340, determine whether to further separate the items. If more items exist, perform one of steps 310, 320, and 330 to repeat further, and continue repeating until it is determined at 340 that there are no items to be picked up and placed in the chute (or other container or source).

[0077] Figure 3B is a diagram of a procedure for picking up and placing items for sorting, according to various embodiments. In various embodiments, the procedure of Figure 3B implements step 320 of procedure 300 of Figure 3A.

[0078] At 322, a plan or strategy for individually distributing one or more items in a work area is determined. In some embodiments, a robotic system determines a plan or strategy for picking up at least one item from a source stack / flow in the work area and placing the at least one item individually into a slot on a conveyor. The plan or strategy for individually distributing one or more items can be determined on a per-robot basis, such that if the robotic system comprises a plurality of robots, each robot operates independently of the other robots(s).

[0079] At 324, an attempt is made to implement a plan or strategy for sorting one or more items in a work area. In some embodiments, the robotic system operates to cause a robotic structure (e.g., a robotic arm) to pick up and place items according to the plan or strategy for sorting.

[0080] According to various embodiments, sensor data is received during the implementation of a plan or strategy for dividing one or more items in a work area. For example, the robot system continuously or at predefined time intervals acquires sensor data from one or more sensors. The acquired sensor data is processed continuously or at predefined time intervals. In some embodiments, the sensor data acquired during the implementation of the plan or strategy is used to update a model of the work area. The robot system may process the sensor data acquired during the implementation of the plan or strategy to determine whether a state of the work area environment has changed (e.g., due to the determination of an initial plan or strategy, or due to a recent update of the plan or strategy). For example, the sensor data acquired during the implementation of the plan or strategy is used to detect a state or condition associated with one or more items in the work area.

[0081] At 326, a determination is made as to whether to take an active action. In some embodiments, the robot system determines whether to take an active action based at least in part on sensor data acquired during the implementation of a plan or strategy. The sensor data acquired during the implementation of a plan or strategy can be used to update a model of the work area. For example, the robot system combines the determination of whether to take an active action with the state or conditions associated with one or more items in the pre-work area.

[0082] According to various embodiments, in response to a detected state or condition, the robot system determines whether to take an active action. In some embodiments, the state or condition is one that was absent and / or undetected when the robot system determined the initial plan for sorting one of the items in the work area or when the plan for sorting was last updated. The state or condition is detected at least in part based on sensor data obtained during the implementation of the plan or strategy. In some embodiments, not all states or conditions will cause the robot system to determine whether to take an active action. For example, the robot system determines to take an active action based at least in part on a determination of one type of state or condition, and / or on a determination of whether the detected state or condition will hinder the successful sorting of an item associated with the plan. The robot system may determine the probability or likelihood that a particular detected state or condition will hinder the successful sorting of an item associated with the plan, and if this probability or likelihood exceeds a predefined active action threshold, then determine to take an active action. Examples of detected states or conditions in which a robot system responds to and determines to perform an active action may include: an expected collision with another item or object while the item to be sorted is moving along a determined path; the location of an identifier or mark on the item to be sorted (e.g., related to one or more scanners configured to read the mark or identifier); a state of a conveyor, such as an item occupying a slot in which the item to be sorted will be placed, or an item being larger than a limit size of a slot adjacent to the slot in which the item to be sorted will be placed; the size or weight of the item to be sorted, etc.

[0083] In some embodiments, the robot system performs proactive actions in response to a detected state or condition that does not obstruct a single part of a currently selected item but may obstruct a single part of an item within the work area. Examples of states or conditions that may not obstruct a single part of a currently selected item but may obstruct a single part of an item within the work area include a blockage within the work area or fed to the work area (e.g., an item), debris on a chute, an item or debris blocking one or more scanners (e.g., which may prevent the one or more scanners from reading an identifier or mark on an item), a position of one or more robot arms within the work area, a reservation of space for another element in the robot system (e.g., another robot arm), etc.

[0084] In response to a determination at 326 to perform an active measure, procedure 320 continues to 322 and performs one of steps 322, 324, and 326 further repeated (e.g., in conjunction with a determination of a plan or strategy for sorting items using a detected state or condition). In some embodiments, an active measure is performed in conjunction with the execution of an updated or new plan or strategy for sorting items. For example, an updated plan or strategy for sorting items is determined at least in part based on the active measure to be performed (e.g., the plan or strategy is incorporated into at least a portion of the active measure). In response to a determination not to perform an active measure, procedure 320 continues to 328, where the current plan or strategy for sorting one or more items in the work area is implemented. For example, a robotic arm is operated to pick up one or more items from the work area and place each item individually in a corresponding position in a sorting conveyor structure. Sorting of items includes picking up items from the work area (e.g., from a source stack / flow) and placing the items individually on the conveyor structure. The robotic system sorts items at least in part based on a plan or strategy for sorting items individually. In some embodiments, 328 corresponds to 330 of procedure 300 in Figure 3A.

[0085] According to various embodiments, during the individual distribution of items (e.g., when implementing a plan or strategy), a determination is made to take an active measure. For example, current sensor data may be obtained during the implementation of a current plan or strategy for individual distribution of items, and the current sensor data is used in conjunction with a determination to take an active measure (e.g., whether to update the plan or strategy, or to determine a new plan or strategy).

[0086] Figure 3C is a diagram of a procedure for picking up and placing items for sorting, according to various embodiments. In various embodiments, the procedure of Figure 3C implements step 330 of procedure 300 of Figure 3A.

[0087] In 332, one or more items are identified for pickup from a work area (e.g., from a chute) according to a plan or strategy for a single correspondence. In some embodiments, a robotic system determines that one or more items are identified for pickup from the work area based at least in part on a plan or strategy and sensor data obtained from one or more sensors associated with the work area.

[0088] At 334, a robotic arm is used to pick up an item from a work area. In some embodiments, the robotic arm includes one or more end effectors. The robotic arm may be controlled by one or more processors of a robotic system. The robotic arm picks up the item from the work area based at least in part on a plan or strategy for distributing the item individually. For example, the manner in which one or more end effectors grasp the item (e.g., the type of end effector, the side of the item being grasped, etc.) is determined based at least in part on a plan or strategy for distributing the item individually.

[0089] At 336, a determination is made as to whether to update the plan or strategy. In some embodiments, the robot system determines whether to update the plan or strategy based at least in part on the time when the robot arm grasps an object (e.g., during the implementation of the plan or strategy). Sensor data acquired when the robot arm grasps the object can be used to update a model of the work area. In some embodiments, the robot system determines whether to update the plan or strategy based at least in part on an updated model of the work area. As an example, the robot system may use the updated model of the work area in conjunction with detecting a detected state or condition associated with the work area (e.g., associated with one or more objects in the work area). For example, the robot system may use the state or condition associated with one or more objects in the work area in conjunction with a determination to perform an active action.

[0090] In response to a determination at 336 to update the plan or strategy, procedure 330 continues to 332 and performs one of steps 332, 334, and 336 again (e.g., combining an updated plan or strategy for individually distributing items with the detected state or condition). According to various embodiments, an updated plan or strategy for individually distributing items is determined at least in part based on the active measures taken (e.g., the plan or strategy incorporates at least a portion of the active measures). In response to a determination not to update the plan or strategy, procedure 330 continues to 338, where items are placed on a conveyor structure (e.g., in a corresponding slot on a conveyor). For example, a robotic arm is operated to individually place items on the conveyor structure.

[0091] Figure 4 is a diagram illustrating a procedure for sorting one or more items using an active measure according to various embodiments. The procedure 400 of Figure 4 can be executed by the system 200 of Figures 2A and 2B. In some embodiments, the procedure 400 is implemented by a robotic system that operates to sort one or more items within a work area. This robotic system includes one or more processors that operate to cause a robotic structure (e.g., a robotic arm) to pick up and place items for sorting.

[0092] At 410, a current plan or strategy for separating one or more items is obtained, and a robotic structure is operated to implement the current plan or strategy. In some embodiments, the current plan or strategy for separating one or more items is determined using sensor data based on information obtained from one or more sensors associated with the work area.

[0093] Sensor data may include image data associated with a plurality of items present in the work area. According to various embodiments, the obtained sensor data related to the work area includes information from which a model of the work area can be generated. As an example, the model of the work area is used in conjunction with determining the current plan or strategy. Sensor data is used to determine one or more characteristics associated with the work area. In some embodiments, sensor data is used in conjunction with determining at least one characteristic of one or more items within the work area (such as items in the source pile / flow of a chute, or an item grasped by a robotic arm). In some embodiments, sensor data is used in conjunction with determining one or more characteristics of a conveyor structure (such as determining an empty or unreserved slot on the conveyor, determining a speed of the conveyor, and / or determining the characteristics of at least one slot or at least one item already on the conveyor).

[0094] According to various embodiments, sensor data is determined at least in part based on information output from one or more sensors. In some embodiments, the robot system includes one or more sensor arrays comprising at least a subset of one or more sensors. The one or more sensor arrays may include an infrared sensor array, a laser array, etc. Various other sensors may be configured in a sensor array. According to various embodiments, the position of an object (e.g., an object on a work area, an object grasped by a robot arm, an object on a conveyor, etc.), the presence of an obstruction or debris, the height of an object, etc., are detected using the output from a sensor array. For example, if a sensor array is configured such that a plurality of sensors are arranged vertically, the height of an object is determined based on determining a subset of sensors in the sensor array that are "tilted" and / or determining a subset of sensors in the sensor array that are not "tilted". As another example, if a sensor array is configured with one of a plurality of sensors arranged horizontally, the depth or height of an object is determined based on determining a subset of the sensors in the sensor array that are "tilted" and / or determining a subset of the sensors in the sensor array that are not "tilted".

[0095] At 420, current sensor data is obtained based on the output from one or more sensor arrays. According to various embodiments, current sensor data is obtained while the robot system is implementing a current plan or strategy. For example, when a robotic arm sorts an item from a work area to a conveyor structure, the robot system receives information from one or more sensor arrays.

[0096] According to various embodiments, one or more sensor arrays are positioned at various locations in or around the environment where the robot structure operates. As an example, one or more sensor arrays are positioned in or around a chute, a conveyor structure (e.g., a portion of the conveyor structure within the proximity or range of the robot arm), a work area in which the robot arm operates to separate one or more items, etc. The locations where one or more sensor arrays are positioned can be chosen to provide the robot system with information about the environment of the work area.

[0097] In some embodiments, a sensor array has a corresponding transmitter and receiver. The transmitter emits a signal (e.g., an infrared beam, a laser beam, etc.) and the receiver receives the beam. If the signal is interrupted because an object blocks its arrival at the receiver, the change in the signal reception state can be used to indicate the presence of an object. Mapping one or more sensor arrays to corresponding locations in or around the work area is used to indicate the location of one or more objects blocking the signals emitted from one or more transmitters.

[0098] At 430, a current detected state or condition is detected at least in part based on information associated with the output of one of the sensors from one or more sensor arrays. This current detected state or condition corresponds to a state or condition of the robot system, the work area, or other environment in which the robot system operates while implementing its current plan or strategy (e.g., when the robot system is sorting one or more items).

[0099] In some embodiments, in response to the output of one of a set of sensors from one or more sensor arrays, the robot system can determine current sensor data. For example, the robot system uses the output from that set of sensors to generate a current model corresponding to a work area or the environment in which the robot system (or robot arm) operates. This current model can be used to determine detected states or conditions.

[0100] In some embodiments, a sensor or sensor array is mapped to a location in a work area or an environment surrounding the work area. Alternatively, a particular sensor(s) may be individually mapped to a location in the work area or environment. Thus, a determination of a "tilted" set of sensors in a sensor array (e.g., compared to a subset of sensors that are not "tilted") can be used to determine that an object (e.g., an item, a robotic arm, debris, or another component) is located between (a subset of) corresponding transmitters and (a subset of) receivers of that set of sensors. If the set of sensors in the sensor array is a subset of all sensors in the sensor array, the robotic system can determine that an edge of the object is located between a last "tilted" sensor in the sensor array and a subsequent sensor that is not "tilted." This determination can be used to determine the position of the object and / or a size of the object (such as height, length, depth, etc.). When a robotic arm grasps an item and moves it from a work area to a conveyor, the robotic system can determine the current position of one of the items, at least in part, based on the determination of one or more sensors that "tumble" at that time.

[0101] In some embodiments, the robot system may determine one or more other properties associated with the work area based at least in part on a determination that one or more sensors are "tilting". For example, in response to a determination that a sensor is tilting at a location different from the current location of one of the items being sorted, the robot system may detect a state or condition of the robot system or the work area. A tilting of a sensor array located at a conveyor structure may be used in conjunction with a determination that an item on the conveyor exceeds one(s) predefined dimensions (e.g., the robot system may determine that a slot adjacent to one of the slots where an item is being sorted is occupied by an item exceeding a predefined size). A tilting of a sensor array located at a chute may be used in conjunction with a determination that there is or is a blockage in the work area (e.g., of an item) being fed into the work area, and / or a determination that an item or debris is blocking one or more scanners (e.g., a scanner for reading a mark or identifier on an item to be sorted, thereby preventing the reading of the mark or identifier).

[0102] At 440, it is determined whether to update the current plan or strategy. In some embodiments, the robot system determines whether to update the current plan or strategy based at least in part on the currently detected state or conditions.

[0103] In response to the decision at 440 to update the current plan or strategy, procedure 400 continues to 410 and performs one of steps 410, 420, 430, and 440 in a further repetition (e.g., combining the determination of the plan or strategy used to distribute items using the detected state or conditions). In the next repetition at 410, the plan or strategy is updated in conjunction with the plan or strategy obtained at that time. For example, the current plan or strategy is determined at least in part based on the current detected state or conditions.

[0104] In contrast, if the decision is not to update the current plan / strategy, procedure 400 continues to 450, where the robot system continues to implement the current plan or strategy. For example, the robot system continues to distribute one or more items according to the current plan or strategy.

[0105] According to various embodiments, the robot system analyzes the current plan or strategy based on the current detected state or conditions, and determines whether to update the current technology or strategy. In some embodiments, the robot system determines to update the current plan or strategy in response to a single determination that the current detected state or conditions are expected to hinder one or more items.

[0106] According to various embodiments, the robot system determines to update its current plan or strategy in response to a determination that the separation of a single item is expected to be hindered. In response to a determination that a detected state or condition hinders the implementation of a current plan to autonomously operate the robot structure to pick up one or more items from the work area and place each item individually in a corresponding position within a single-item conveying structure, the robot structure determines to update its current plan or strategy to one that improves the likelihood of successful separation (e.g., the robot structure may take proactive measures, at least in part, based on the detected state or condition, to determine the likelihood of successful separation).

[0107] In some embodiments, the robot system determines a plurality of plans or strategies for separating one or more items, and corresponding properties of the plurality of plans or strategies (such as a measure of the probability of successfully separating the one or more items, a measure of the efficiency of a plan or strategy, etc.). The measure of the probability of successfully separating one or more items may correspond to the expected probability that the robot structure will successfully separate one or more items (e.g., based on the current state or conditions of a work area). The measure of the efficiency of a plan or strategy may include the speed at which one or more items will be separated if the corresponding plan or strategy is implemented, the throughput of the robot system (or robot structure), etc. According to various embodiments, the robot system may determine a plan or strategy to be implemented based on the corresponding properties of the plurality of plans or strategies. For example, the robot system determines a composite score representing a particular plan or strategy, and the robot system selects a plan or strategy to be implemented based on the composite score corresponding to this plan or strategy. The robotic system can rank composite scores corresponding to at least a subset of a plurality of plans or strategies and select a plan or strategy to be implemented based at least in part on the ranking of the composite scores.

[0108] Figure 5A is a diagram illustrating a procedure for sorting one or more items using an active measure according to various embodiments. The procedure 500 of Figure 5A can be executed by the system 200 of Figures 2A and 2B. In some embodiments, the procedure 500 is implemented by a robotic system that operates to sort one or more items within a work area. This robotic system includes one or more processors that operate to cause a robotic structure (e.g., a robotic arm) to pick up and place items for sorting.

[0109] At 510, a detected state or condition associated with one or more items in the work area is detected. In some embodiments, the robot system uses sensor data from a work area environmental state system to detect the state or condition. The sensor data may be obtained at least in part based on information output from one or more sensors located in or around the work area, robot structure, and / or conveyor structure. In some embodiments, the robot system uses information output from one or more sensors to generate a model of the work area, and this model of the work area is used to detect the state or condition.

[0110] Examples of detected states or conditions include deviations from a planned chute flow (e.g., detecting a blockage or anomaly in the chute flow or work area environment), deviations from a planned gripping or releasing mechanism of an end effector of a robotic arm used for sorting an item, detection of debris in the work area, an object along the planned path of the item during sorting (or within a predefined proximity of the planned path), a possible collision event between the robotic arm or item and another object (e.g., another robotic arm operating in the work area) during sorting, a state of the conveyor structure (e.g., conveyor speed, an indication that a slot is occupied, a conveyor slot being occupied by an item exceeding a certain size limit, etc.), and the weight of an item currently gripped by the robotic arm. Various other states or conditions are possible.

[0111] According to various embodiments, the robot system detects blockages or anomalies in the chute flow or work area environment by modeling the (item) flow through a chute using sensor data and determining whether the modeled flow deviates from an expected flow (e.g., in normal operation or within a flow threshold defined according to normal operation). The robot system may use a vision system to model the flow. For example, a 3D model of the flow may be determined based at least in part on sensor data from the vision system.

[0112] In some embodiments, the robot system detects a blockage or anomaly in the chute flow or work area environment based on a determination that the front portion of a chute (e.g., a location of the conveyor structure closest to the work area and / or a chute of a robot structure) is not filled with an item or has been refilled. For example, the chute includes a conveyor that moves items from one inlet of the chute to the front portion of the chute. In response to a determination that the conveyor included in the chute is operating and that an item has not been refilled in the front portion of the chute, the system detects a blockage or anomaly in the chute flow or work area environment. The system may determine that the front portion of the chute does not contain an item and that an item has not been refilled based on an output from one or more sensors located at the front portion of the chute. A determination that one or more infrared sensors may be located at the front portion of the chute and that one or more sensors are not tilted may be considered an indication that no item is located at the front portion of the chute.

[0113] At 530, a single item is determined to be impeded by a detected state or condition. In some embodiments, the robot system determines that a single item is expected to be impeded by a detected state or condition.

[0114] A robotic system can determine, based on a comparison with a current plan or strategy for sorting at least one item, that a detected state or condition may impede the sorting of at least one item. As an example, if the detected state or condition corresponds to the position of an object within a work area (e.g., a chute and / or conveyor structure), the robotic structure determines that the position of that object intersects with a planned path or trajectory of the item to be sorted and accordingly determines that the detected state or condition may impede the sorting. As another example, if the detected state or condition corresponds to a determination that a slot in which at least one item will be sorted already contains another item (e.g., an item sorted by an upstream robotic structure), the robotic structure determines that the presence of the other item impedes the sorting of the item to be sorted by the robotic structure. As yet another example, if the detected state or condition corresponds to a detected weight of an item grasped by a robotic arm, the robotic system determines that the weight of the item will impede the sorting (e.g., if the weight of the item exceeds a weight that can be moved by the robotic arm or that an end effector can maintain a grasp on). As another example, if the detected state or condition corresponds to a pressure or other value indicating that the robotic arm has grasped one of the items to be sorted, the robotic system determines whether the pressure or grip strength is consistent with the normal operation of the robotic arm (e.g., whether an abnormal pressure or grip strength is measured) and / or whether the pressure or grip strength is sufficient to maintain the grip of the item to be sorted (e.g., based on the packaging type, size, and / or weight of the item to be sorted).

[0115] According to various embodiments, a model of the probability of successfully splitting an item is generated. The robotic system may determine the probability of successfully splitting an item based at least in part on one or more properties associated with the item to be split, one or more properties of the work area (or other objects within the work area), etc. The model of the probability of successful splitting may include a measure of the probability that the item to be split is successfully split by a corresponding robotic structure. According to various embodiments, in response to detecting a new state or condition (e.g., a state or condition that does not exist when determining the current plan or strategy for splitting, or a state or condition that was not considered when determining the current strategy), the model of the probability of successful splitting is updated to reflect the current probability that the splitting of the item is successful. A determination of whether to perform an active measure (e.g., changing the current plan for splitting the item or adapting it to a detected state or condition associated with one or more items in the work area) may be based at least in part on the updated model of the probability of successful splitting.

[0116] At 550, a single active measure for improving one or more items is determined. In some embodiments, the active measure is a measure that combines picking up an item from a source stack / flow (e.g., a work area) and placing the item on a segmented conveyor or similar conveyor for sorting and routing to a downstream (e.g., final address / entity) destination to adapt one or more items to a state or condition.

[0117] In some embodiments, proactive measures include using a robotic arm, an end effector of the robotic arm, a chute in the work area, or another element to move or blower to reassemble one or more items or debris within the reassembled source stack / flow or reassembled work area. Proactive measures can be performed to improve the scanning of a mark or identifier on an item to be sorted, improve the likelihood of picking up an item, improve the gripping of an item after sorting, improve the release of an item from the robotic arm, improve the operation of two robotic arms independently sorting items from the same work area (e.g., the same source stack / flow), and / or improve the path or trajectory of the item to be sorted (e.g., by changing the path or trajectory or by removing a detected obstacle along a planned path or trajectory). Various other proactive measures are possible.

[0118] At 570, an active measure is implemented in conjunction with the individual distribution of one or more items. In some embodiments, in response to a determination to perform an active measure, the plan or strategy for the individual distribution of one or more items is updated (e.g., to include the active measure), and the active measure is implemented.

[0119] Various active measures are described in conjunction with Figures 5B to 5F. In various embodiments, a robotic system as disclosed herein may implement one or more of the active measures illustrated in Figures 5B to 5E and / or one or more other measures.

[0120] Figure 5B is a diagram illustrating a robot separation system that uses an active measure to separate one or more items according to various embodiments.

[0121] In various embodiments, a robotic system including one or more robotic arms performs sorting / sorting, including performing one or more active actions in response to a detected state or condition associated with one or more items in the work area of ​​the robotic system. The robotic system may include one or more robotic arms, each having one or more end effectors. In the example shown in FIG5B, system 580 includes one or more of a plurality of robotic arms 590a, 590b, 590c, and 590d operating in corresponding work areas 585a, 585b, 585c, and 585d. The plurality of robotic arms 590a, 590b, 590c, and 590d operate to sort items within work areas 585a, 585b, 585c, and 585d to conveyor structure 593. The plurality of robotic arms 590a, 590b, 590c, and 590d include one or more end effectors (not shown). The end effector can be a suction end effector, a pinch end effector, or various other types of end effectors.

[0122] System 580 includes image sensors, in this example including 3D cameras 581a, 581b, 581c and / or 581d. In various embodiments, other types of sensors may be used (individually or in combination) in a single subsystem as disclosed herein, including a camera, an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, and the like. In various embodiments, control computer 599 includes a workspace environment status system (such as a vision system) for identifying individual items, clutter in the workspace, and the orientation of each item based on sensor data (such as image data provided by image sensors (in this example including 3D cameras 581a, 581b, 581c and / or 581d)). The work area environment status system may also include sensors in the robotic arm for detecting the weight of an object (e.g., an object that has been grasped) or for detecting information from which an estimated weight is determined. For example, information relating to a quantity of current, voltage, and / or power used by one or more motors driving the movement of the robotic arm can be used to determine the weight of the object (or an estimated weight). As another example, information relating to the output of one of one or more sensor arrays can be used to determine the position of the object in the work area, the position of the object when it is grasped and / or moved by the robotic arm, and / or the position of the robotic arm (e.g., based on the output of one subset of sensors from one or more sensor arrays compared to another subset of sensors from one or more sensor arrays).

[0123] The work area environment status system generates an output used by system 580 to determine and implement a plan for autonomously operating at least one of a plurality of robotic arms 590a, 590b, 590c, and 590d to pick up one or more items from a plurality of work areas 585a, 585b, 585c, and 585d and place each of the one or more items in a corresponding available, defined location (such as a section of a segmented conveyor 593) for machine identification and classification. In some embodiments, the work area environment status system generates an output used by system 580 to detect a state or condition associated with one or more items in the work area, and / or a state or condition associated with another element of the robotic arm or work area (e.g., sensor data or information of the work area and the items within the work area in other ways). According to various embodiments, in response to detecting (e.g., determining) a state or condition associated with one or more items in the work area, the robotic system performs one or more proactive actions in conjunction with a single item. Proactive measures may include updating a plan to autonomously operate a robotic structure to pick up one or more items from a work area and place each item individually in a corresponding location within a single-dispatch conveyor structure. In some embodiments, proactive measures or updated plans may include operating the robotic structure to alter or adapt to detected states or conditions (e.g., implementing changes to how an item is individually dispensed, implementing reorganization of items within a source stack / flow to make grasping a selected item easier, etc.).

[0124] The following will describe in more detail the different proactive measures implemented in work areas 585a, 585b, 585c and 585d.

[0125] At work area 585a, active measures include adjusting the chute. Examples of adjusting the chute include: adjusting the inclination of one of the chutes, causing the chute to vibrate (e.g., by activating one or more motors operatively connected to the chute), adjusting the height of one of the chutes (e.g., by raising or lowering the chute), etc. Various adjustments to the chute are possible. In some embodiments, adjusting the chute is used to interrupt the chute or one or more items within the work area.

[0126] Figure 5C is a diagram illustrating a procedure for sorting one or more items by adjusting an active measure of a chute in a robotic sorting system according to various embodiments. Procedure 550a of Figure 5C can be executed by system 580 of Figure 5B. In various embodiments, the procedure of Figure 5C implements steps 550 and / or 570 of procedure 500 of Figure 5A. In some embodiments, procedure 550a is implemented by a robotic system operating to sort one or more items within a work area (such as work area 585a of system 580 in Figure 5B). System 580 includes one or more processors that operate to cause a robotic arm 590a to pick up and place items for sorting.

[0127] At 551a, it is determined that the flow of items through one of the chutes is obstructed. According to various embodiments, system 580 determines that the flow of items through the chutes is obstructed based at least in part on sensor data obtained from information output from one or more sensors within system 580. For example, system 580 determines that the flow of items through the chutes is obstructed based at least in part on a detected state or condition associated with one or more items in the work area. In some embodiments, system 580 determines that the flow of items through the chutes is obstructed in response to determining that a model of a current flow of items through one of the chutes deviates from a model of a normal flow through the chutes. The model of the flow of items through the chutes may be generated at least in part based on sensor data relating to a work area and / or one or more items within that work area.

[0128] At 552a, a determination is made to adjust the chute. In some embodiments, the system 580 determines to adjust the chute to interrupt one or more items or debris within the chute or work area 585a. The determination to adjust the chute may be based at least in part on a determination that the flow of items through the chute obstructs the flow of one or more items.

[0129] According to various embodiments, in response to a determination that the flow of items through the chute is obstructed, system 580 determines whether adjusting one of the chute's components will alter or adapt the flow of items through the obstructed chute. System 580 may determine whether adjusting one of the chute's components will improve or normalize the flow of items through the chute. In response to a determination that adjusting one of the chute's components will improve or normalize the flow of items through the chute, system 580 determines to adjust the chute.

[0130] In 553a, a plan or strategy for adjusting the chute (e.g., interrupting one or more items or miscellaneous items) is implemented. In some embodiments, one or more processors cause system 580 to implement a plan or strategy for adjusting the chute.

[0131] According to various embodiments, system 580 determines, at least in part, how to adjust a chute based on a detected state or condition associated with one or more items in the work area. For example, system 580 determines a cause (or a possible cause) obstructing the flow of items through the chute, and system 580 determines that an adjustment to the chute is expected (or possible) to improve the flow of items through the chute. System 580 may determine an adjustment to the chute that is expected to return the flow of items through the chute to a normal state. Adjustments to the chute in work area 585a are indicated by dashed lines surrounding the chute.

[0132] Adjusting a chute to interrupt one or more items or debris within the chute or work area may include adjusting the inclination of one of the chute sections. The inclination of the chute can be adjusted by one or more motors operably connected to the chute to raise or lower a portion of the chute, or both. The inclination of the chute can also be achieved by various other mechanisms.

[0133] Adjusting a chute to interrupt one or more items or debris within the chute or work area may include vibrating the chute. Vibration of the chute can be achieved by operating a motor operably connected to the chute to vibrate at least a portion of the chute, lowering a portion of the chute, or both of these motors. Vibration of the chute can also be achieved by various other mechanisms.

[0134] Adjusting a chute to interrupt one or more items or debris within the chute or work area may include adjusting the height of the chute. The chute may be raised or lowered by one or more motors operably connected to it for raising or lowering. As an example, the chute may be configured with a rack and pinion mechanism, a screw jack mechanism, a pneumatic and / or hydraulic mechanism, or other mechanisms for raising / lowering the chute. In the case where the chute is configured with a rack and pinion mechanism, system 580 may operate a motor to drive (e.g., rotate) a pinion across a rack, and the chute will rise / lower accordingly. Adjustment of the chute height may be achieved by various other mechanisms.

[0135] At 554a, it is determined whether the flow of goods through the chute has been improved. In some embodiments, in response to implementing a plan or strategy for adjusting the chute, system 580 obtains current sensor data and uses this data to determine whether the adjustment to the chute has improved the flow of goods through the chute. In response to the determination that the flow through the chute has been improved, procedure 550a ends (e.g., the implementation of the active measure is completed). Conversely, in response to the determination that the flow of goods through the chute has not been improved, procedure 550a continues to 551a and performs one of steps 551a, 552a, 553a, and 554a for further repetition. In subsequent repetitions of procedure 550a, system 580 may determine that a different adjustment to the chute has been performed (e.g., because the current adjustment to the chute has not resolved the obstructed flow of goods through the chute).

[0136] Whether to improve the flow of goods through the chute can be determined at least in part based on a model of the flow of goods through the chute. The model of the flow of goods can be generated at least in part based on sensor data related to the work area (e.g., sensor data associated with one or more items in the work area). System 580 can determine whether to improve the flow of goods based on a measure of the amount of goods processed through the chute and / or a comparison between the current flow through the chute and an expected normal flow through the chute.

[0137] Figure 5D is a diagram illustrating a procedure for separating one or more items by means of an active action of a robot structure interrupting an item on a chute of a robot separation system, according to various embodiments. Procedure 550b of Figure 5D can be executed by system 580 of Figure 5B. In various embodiments, the procedure of Figure 5D implements steps 550 and / or 570 of procedure 500 of Figure 5A. In some embodiments, procedure 550b is implemented by a robot system operating to separate one or more items within a work area (such as work area 585b of system 580 in Figure 5B).

[0138] At 551b, it is determined that the flow of articles through one of the chutes is obstructed. According to various embodiments, 551b of procedure 550b of FIG. 5D may be similar to or the same as 551a of procedure 550a of FIG. 5C.

[0139] In 552b, a determination is made to use a robotic structure to interrupt one or more items or debris within a chute or work area. According to various embodiments, the robotic structure for interrupting one or more items or debris within a chute or work area corresponds to the robotic arm 590b of FIG. 5B. The robotic arm 590b can be used to push, grasp, and pull one or more items within a chute or work area 585b to clear the flow of items through the chute or otherwise reconfigure one or more items within the work area. The use of a robotic structure to interrupt one or more items or debris within a chute or work area 585b can be implemented to improve the individual distribution of one or more items within the work area 585b.

[0140] According to various embodiments, in response to a determination that the flow of items through the chute is obstructed, system 580 determines whether using a robotic structure to interrupt one or more items or miscellaneous items within the chute or work area 585b will alter the individual composition of one or more items or adapt it to the obstructed flow of items through the chute. System 580 may determine whether using a robotic structure to interrupt one or more items or miscellaneous items will improve or normalize the flow of items through the chute, or otherwise increase the likelihood that the individual composition of one or more items will be successful. In response to a determination that using a robotic structure to interrupt one or more items or miscellaneous items will improve or normalize the flow of items through the chute, system 580 determines whether to use a robotic structure to interrupt one or more items or miscellaneous items.

[0141] In 553b, a plan or strategy is implemented to use a robotic structure to interrupt one or more items or debris within the chute or work area 585b. In some embodiments, one or more processors cause system 580 to implement a plan or strategy to use a robotic structure to interrupt one or more items or debris within the chute or work area 585b.

[0142] According to various embodiments, system 580 determines, at least in part, the manner in which robotic arm 590b will interrupt one or more items or debris in the chute or work area 585b based on a detected state or condition associated with one or more items in the work area. For example, system 580 determines a cause (or a possible cause) obstructing the flow of items through the chute, and system 580 determines the use of a robotic arm that will improve the flow of items through the chute. System 580 may determine the use of a robotic arm that is intended to return the flow of items through the chute to a normal state. As illustrated in FIG5B, robotic arm 590b is used to engage item 596 in work area 585b. Robotic arm 590b may be used to push item 596, or grasp and pull item 596 (e.g., forcefully move item 596, or move item 596 to avoid obstructing the use of sensors placed in work area 585b).

[0143] According to various embodiments, the robotic arm 590b is operated to extend back into the chute (e.g., toward a portion of the chute through which a new item is introduced into the chute), and the robotic arm 590b is used to engage with one or more items or debris within the chute or work area 585b to interrupt the one or more items or debris.

[0144] At 554b, it is determined whether the flow of items through the chute has been improved. In some embodiments, in response to a plan or strategy to use a robotic arm 590b to interrupt one or more items or debris within the chute or work area 585b, the system 580 obtains current sensor data and uses that data to determine whether the flow of items through the chute has been improved. If the flow through the chute is determined to have been improved, procedure 550b ends (e.g., the implementation of the active measure is complete). Conversely, if the flow through the chute is determined not to have been improved, procedure 550b continues to 551b and performs one of steps 551b, 552b, 553b, and 554b in further repetition. In subsequent repetitions of procedure 550b, the system 580 may determine to use the robotic arm 590b to engage or interrupt one different item or debris within the chute or work area, or to use a different strategy to engage the same item or debris.

[0145] Figure 5E is a diagram illustrating a procedure for separating one or more items by using air blowing to actively reposition an item or debris on a chute of a robotic separation system, according to various embodiments. Procedure 550c of Figure 5E can be executed by system 580 of Figure 5B. In various embodiments, the procedure of Figure 5E implements steps 550 and / or 570 of procedure 500 of Figure 5A. In some embodiments, procedure 550c is implemented by a robotic system operating to separate one or more items within a work area (such as work area 585c of system 580 in Figure 5B).

[0146] At 551c, it is determined that the flow of articles through one of the chutes is obstructed. According to various embodiments, 551c of procedure 550c of FIG. 5E may be similar to or the same as 551a of procedure 550a of FIG. 5C.

[0147] At 552c, a determination is made to use one or more blowers to interrupt one or more items or debris within the chute or work area. According to various embodiments, one or more blowers are positioned within the work area. As illustrated in FIG5B, the work area 585c includes blowers 595a and 595b positioned on one or more sides of the chute or work area 585c. In some embodiments, the robotic arm 590c includes one or more blowers. For example, if the robotic arm 590c includes a suction-type end effector, the system 580 can use the suction-type end effector to blow air outward from the suction-type end effector to the surrounding environment (e.g., by reversing the airflow in a corresponding pneumatic system). The use of one or more blowers to interrupt one or more items or debris within the chute or work area 585c can be implemented to improve the individual separation of one or more items within the work area 585c.

[0148] One or more blowers may be pneumatic components that blow air to remove (e.g., blow out) debris or reposition one or more items within the repositioning work area 585c. The pressure from the pneumatic components can be adjusted. For example, if one or more items or debris to be interrupted are relatively large, the system 580 may configure the blowers to blow air more forcefully toward one or more items or debris. In some embodiments, one or more blowers disposed within the chute are "knife-type" blowers positioned along one side of the chute.

[0149] According to various embodiments, in response to a determination that the flow of items through the chute is obstructed, system 580 determines whether using one or more blowers to interrupt one or more items or debris within the chute or work area 585c will alter the individual composition of one or more items or adapt it to the obstructed flow of items through the chute. System 580 may determine whether using one or more blowers to interrupt one or more items or debris will improve or normalize the flow of items through the chute, or otherwise increase the likelihood that the individual composition of one or more items will be successful. In response to a determination that using one or more blowers to interrupt one or more items or debris will improve or normalize the flow of items through the chute, system 580 determines that using one or more blowers to interrupt one or more items or debris will alter the individual composition of one or more items or debris.

[0150] In 553c, a plan or strategy is implemented to interrupt one or more items or debris within the chute or work area 585c using one or more blowers. In some embodiments, one or more processors cause system 580 to implement a plan or strategy to interrupt one or more items or debris within the chute or work area 585c using one or more blowers 595a and / or 595b.

[0151] According to various embodiments, system 580 determines, at least in part, a manner to interrupt one or more items or debris in the chute or work area 585c using one or more detected states or conditions associated with one or more items in work area 585c. For example, system 580 determines a cause (or a possible cause) obstructing the flow of items through the chute, and system 580 determines that using blowers 595a and / or 595b would improve the flow of items through the chute. As another example, system 580 determines whether to use pre-configured blowers 595a and / or 595b, or whether to reverse the airflow in one suction-type end effector on robot arm 590c (or use another blower mounted on robot arm 590c) to interrupt one or more items or debris in the chute or work area 585c. In some embodiments, blowers 595a and / or 595b include vents, and system 580 can configure such vents to direct airflow in a particular direction. In some embodiments, the robotic arm 590c is operated to move to a position where a blower mounted on the robotic arm 590c can guide airflow in a desired direction.

[0152] At 554c, it is determined whether the flow of items through the chute has been improved. In some embodiments, in response to implementing a plan or strategy to interrupt one or more items or debris in the chute or work area 585c using one or more blowers, system 580 obtains current sensor data and uses the current sensor data to determine whether the flow of items through the chute has been improved. In response to determining that the flow through the chute has been improved, procedure 550c ends (e.g., the implementation of the active measure is completed). Conversely, in response to determining that the flow of items through the chute has not been improved, procedure 550c continues to 551c and performs one of steps 551c, 552c, 553c, and 554c for further repetition. In subsequent repetitions of procedure 550c, system 580 may determine to use a different blower or change a setting of a previously used blower (e.g., the direction of a vent, the amount of airflow, etc.) to engage or interrupt a different item or debris in the chute or work area, or to use a different strategy to engage the same item or debris.

[0153] Figure 5F is a diagram illustrating a procedure for separating one or more items by means of an arm attached to a chute of a robotic separation system to reposition an item or debris on the chute, according to various embodiments. Procedure 550d of Figure 5F can be executed by system 580 of Figure 5B. In various embodiments, the procedure of Figure 5F implements steps 550 and / or 570 of procedure 500 of Figure 5A. In some embodiments, procedure 550d is implemented by a robotic system operating to separate one or more items within a work area (such as work area 585d of system 580 in Figure 5B).

[0154] At 551d, it is determined that the flow of articles through one of the chutes is obstructed. According to various embodiments, 551d of procedure 550d in FIG5F may be similar to or the same as 551a of procedure 550a in FIG5C.

[0155] At 552d, a determination is made to interrupt one or more items or debris within the chute or work area using one of the arms or other components contained in the chute. According to various embodiments, the arm or other component contained in the chute for interrupting one or more items or debris within the chute or work area is different from the robotic arm 590d. The arm or other component contained in the chute may be integrated into or operatively connected to the chute. As illustrated in FIG5B, arm 594 is positioned on the inner side of one of the chutes in work area 585d. The arm may extend to engage with items or debris within the chute. In some embodiments, arm 594 is operated to extend and push one or more items within the chute or work area 585d to clear the flow of items through the chute or otherwise reposition one or more items within the work area. It is possible to use an arm or other component contained in the chute to interrupt one or more items or debris in the chute or work area 585d in order to improve the individual separation of one or more items in the work area 585d.

[0156] In some embodiments, the arm or other element included in the chute includes a chute conveyor. The chute conveyor can operate in a "jittering" mode to move back and forth in continuous pulses to de-stack or shake items off each other or otherwise reposition items to be picked up from the work area. In some embodiments, a chute conveyor as disclosed herein includes a modular, movable hardware having dimensions suitable for placement on a portion of the chute closest to one end of the robotic arm. The chute conveyor may have a low profile (height) that facilitates, for example, the flow of items moving through the chute by gravity to a top surface of the chute conveyor. The chute conveyor may be located in a recess such that its top surface is flush with (i.e., at the same level) a gravity feed portion of the chute, which is positioned ahead of the chute conveyor in the direction of the flow of items through the chute.

[0157] According to various embodiments, in response to a determination that the flow of items through the chute is obstructed, system 580 determines whether using an arm or other element included in the chute to interrupt one or more items or debris within the chute or work area 585d will alter the individual composition of one or more items or adapt it to the obstructed flow of items through the chute. System 580 may determine whether using an arm or other element included in the chute to interrupt one or more items or debris will improve or normalize the flow of items through the chute, or otherwise increase the likelihood that the individual composition of one or more items will be successful. In response to a determination that using an arm or other element included in the chute to interrupt one or more items or debris will improve or normalize the flow of items through the chute, system 580 determines whether using an arm or other element included in the chute to interrupt one or more items or debris.

[0158] At 553d, a plan or strategy is implemented to interrupt one or more items or debris within the chute or work area 585d using an arm or other component contained in the chute. In some embodiments, one or more processors cause system 580 to implement a plan or strategy to interrupt one or more items or debris within the chute or work area 585d using a robotic structure.

[0159] According to various embodiments, system 580 determines, at least in part, the manner in which robotic arm 590d will interrupt one or more items or debris in the chute or work area 585d based on a detected state or condition associated with one or more items in the work area. For example, system 580 determines a cause (or a possible cause) obstructing the flow of items through the chute, and system 580 determines the use of a robotic arm that will improve the flow of items through the chute. System 580 may determine the use of a robotic arm that is intended to return the flow of items through the chute to a normal state. As illustrated in FIG5B, arm 594 is used to engage one item in work area 585d. Arm 594 may be used to push an item (e.g., forcefully move item 596, or move item 596 to avoid obstructing the use of sensors placed in work area 585d, reposition an item, flip an item, etc.).

[0160] According to various embodiments, the operating arm 594 is extended back into the chute (e.g., toward a portion of the chute through which a new item is introduced into the chute), and the arm 594 is configured to engage with one or more items or debris within the chute or work area 585d to interrupt the one or more items or debris.

[0161] At 554d, it is determined whether the flow of items through the chute has been improved. In some embodiments, in response to a plan or strategy to interrupt one or more items or debris in the chute or work area 585d using arm 594, system 580 obtains current sensor data and uses this data to determine whether the flow of items through the chute has been improved. In response to the determination that the flow through the chute has been improved, procedure 550d ends (e.g., the implementation of the active measure is complete). Conversely, in response to the determination that the flow of items through the chute has not been improved, procedure 550d continues to 551d and performs one of steps 551d, 552d, 553d, and 554d for further repetition. In some embodiments, in response to the determination that the flow of items has not been improved, system 580 determines to use a different type of active measure to improve the flow or separation of items. In subsequent iterations of procedure 550d, system 580 may determine whether to use arm 594 to engage or disengage one of the different items or miscellaneous items in the chute or work area 585d, or to use a different strategy to engage the same item or miscellaneous item.

[0162] In some embodiments, proactive measures include requesting human intervention from human operator 597 via manual intervention by human operator 597 or via remote intervention using an on-demand teleoperation device, which human operator 597 can use to control one or more of robotic arms 590a, 590b, 590c and 590d, or can use control computer 599 to configure one of the settings of system 580.

[0163] Although the embodiments described in conjunction with Figures 5A to 5F include the use of one or more active measures in response to a determination that the flow of items is obstructed in the chute, an active measure (such as the active measures described in Figures 5A to 5F) may be performed in response to other detected states or conditions (such as states or conditions associated with one or more items in the work area). Active measures may be performed to reconfigure or rearrange items or debris in a work area to improve the individual distribution of one or more items. For example, an item may be reconfigured to improve the scanning of an identifier or mark on that item. As another example, an item may be reconfigured to improve the gripping of the item. In the case where an item is configured such that its longest side is perpendicular to the surface of the chute, an active measure may be implemented to reconfigure the item so that its longest side is parallel to the surface of the chute (e.g., knocking the item over). Various active measures may be implemented in response to detected states or conditions in various contexts.

[0164] Figure 6A is a diagram illustrating a robot single-distribution system according to various embodiments, which uses one or more sensors to detect the state or condition of an item in the work area of ​​a robot single-distribution system.

[0165] In various embodiments, a robotic system including one or more robotic arms performs single-point / inductive reasoning, including performing one or more active actions in response to a detected state or condition associated with one or more items in the work area of ​​the robotic system. The robotic system may include one or more robotic arms, each having one or more end effectors.

[0166] In the example shown, system 600 includes a robotic arm 603a operated to individually transfer one or more items in work area 605a to one of conveyor structures 607, and a robotic arm 603b operated to individually transfer one or more items in work area 605b to conveyor structures 607.

[0167] In various embodiments, a system (such as system 600) includes one or more sensors from one of its modeled workspace environments. In the example shown in FIG6A, system 600 includes image sensors, in this example including 3D cameras 601a, 601b, 601c, and 601d. In various embodiments, other types of sensors may be used (individually or in combination) in a single subsystem as disclosed herein, including a camera, an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, and the like. As illustrated in FIG6A, system 600 includes one or more sensor arrays disposed within or around a chute. For example, workspace 605b includes sensor arrays 609a and 609b. According to various embodiments, one or more of sensor arrays 609a and 609b include a vertical sensor array. System 600 may include one or more sensors or sensor arrays disposed around a working area. As illustrated, system 600 includes sensors 611a, 611b, and 611c disposed around working area 605a, and sensors 611d, 611e, and 611f disposed around working area 605b. According to various embodiments, system 600 includes one or more sensor arrays disposed at transport structure 607. For example, a sensor array may be disposed at the transport structure in a manner that transmits a signal (e.g., an IR beam) across the transport structure.

[0168] In various embodiments, a control computer (not shown) includes a workspace environment status system (such as a vision system) for identifying individual items, clutter in the workspace, and the orientation of each item based on sensor data, such as image data provided by image sensors (in this example, including 3D cameras 601a, 601b, 601c, and 601d). This workspace environment status system may also include sensors in a robotic arm for detecting the weight of an item (e.g., an item grasped) or detecting information from which an estimated weight is determined. For example, information relating to a quantity of current, voltage, and / or power used by one or more motors driving the movement of the robotic arm can be used to determine the weight of the item (or an estimated weight). As another example, information relating to the output of one of one or more sensor arrays (e.g., sensor arrays 609a and 609b) can be used to determine one position of an item in the work area, one position of an item when the item is grasped and / or moved by a robotic arm, and / or one position of the robotic arm (e.g., based on a determination from the output of one subset of sensors of one or more sensor arrays relative to another subset of sensors of one or more sensor arrays).

[0169] According to various embodiments, information output from sensor arrays 609a and / or 609b is used to determine the position and / or characteristics of one or more items within work area 605b. The height of an item within a work area (such as work area 605b) can be determined at least in part based on a determination by one of the sensors in sensor arrays 609a and / or 609b that indicates an item has been tipped over. In some embodiments, system 600 determines that an item within work area 605b is configured such that its longest side is perpendicular to the surface of the chute, and that the item's position would be improved if it were gripped from a different side. Therefore, system 600 determines and implements an active measure to reconfigure the item so that its longest side is parallel to the surface of the chute (e.g., knocking the item over using a robotic arm 603b, a blower mechanism, etc.). System 600 can be configured to implement an active measure to reconfigure the item if its height within work area 605b exceeds a predefined height threshold.

[0170] According to various embodiments, the work area environment status system generates an output for system 600 to determine and implement a plan for autonomously operating robotic arms 603a and / or 603b to pick up one or more items from work area 605a and / or work area 605b and place each item in a corresponding usable, defined location (such as a section of segmented conveyor 607) for machine identification and classification. In some embodiments, the work area environment status system generates an output for the robot system to detect a state or condition associated with one or more items in the work area, and / or a state or condition associated with another element of the robotic arm or work area (e.g., sensor data or information of the work area and the items within the work area, in other ways). According to various embodiments, in response to detecting (e.g., determining) a state or condition associated with one or more items in the work area, the robot system implements one or more proactive actions in conjunction with individual items. Proactive measures may include updating a plan to autonomously operate a robotic structure to pick up one or more items from a work area and place each item individually in a corresponding location within a single-dispatch conveyor structure. In some embodiments, the proactive measures or update plan include operating the robotic structure to alter or adapt to detected states or conditions (e.g., implementing a change relative to one method of dispatch an item, implementing measures to reorganize items within a source stack / flow to make gripping a selected item easier, manipulating to reposition a robotic arm or end effector to increase the ability to grip items, etc.).

[0171] The output used by system 600 may correspond to sensor data containing image data associated with a plurality of items present in a work area. System 600 may use the sensor data to identify items that cannot be separated within work areas 605a and / or 605b. For example, system 600 may identify predefined items mapped to a type of item that cannot be separated. As another example, system 600 may identify items within work areas 605a and / or 605b for which a plan or strategy for separation from work areas 605a and / or 605b to conveyor structure 607 cannot be determined. In response to the determination of an item that cannot be separated, system 600 may operate to implement an active measure. The active measure implemented in response to the determination of an item that cannot be separated may include removing the item from the corresponding work area, invoking human intervention (e.g., alerting a human operator), etc. In some embodiments, in response to the determination that an item cannot be separated within work area 605a and / or work area 605b, system 600 operates robotic arms 603a and / or 603b to move the item to a designated location. This designated location may be a rejected item storage bin or other area for storing items to be disposed of outside the separation process. Examples of items identifiable as non-separable include items with a specific shape (e.g., a poster tube), items for which system 600 cannot locate a mark or other predefined identifier to be scanned, and items with a predefined identifier on one side (e.g., a mark indicating the item includes a fragile object, a mark indicating the item is not stacked, a mark indicating the item includes a hazardous substance, etc.).

[0172] In some embodiments, the chute in the corresponding work area (e.g., work area 605a or work area 605b) includes a weight sensor. System 600 can determine the weight of an item to be sorted based on a difference between a weight measured by the weight sensor in the chute before the robotic arm grasps the item and a weight measured by the weight sensor after the item is grasped (and completely moved away from the chute). In response to determining the weight of the item, system 600 can combine a determination of whether to implement an active measure related to sorting the item to use the item, and if so, implement the active measure. For example, in response to determining the weight, system 600 can update the path or trajectory of the item according to a corresponding plan or strategy for sorting the item. If the weight of the item is greater than a predefined weight threshold, system 600 can update the plan for sorting the item based on the weight of the item. In some cases, if the weight of the item exceeds a weight threshold, the robotic arm is operated to lift the item from the chute section and to drag the item to the conveyor structure 607.

[0173] Figure 6B is a diagram of a procedure for sorting one item using sensor data according to various embodiments. The procedure 650 of Figure 6B can be executed by the system 600 of Figure 6A. In some embodiments, the procedure 650 is implemented by a robotic system that operates to sort one or more items within a work area (such as work area 605a or 605b of system 600 in Figure 6A). System 600 includes one or more processors that operate to cause robotic arms 603a or 603b to pick up and place items for sorting.

[0174] At 651, sensor data is received. According to various embodiments, system 600 receives sensor data at least in part based on the outputs of image sensors (in this example, including 3D cameras 601a, 601b, 601c, and 601d) and / or sensor arrays (e.g., sensor arrays 609a and 609b) disposed within the work area. System 600 may obtain sensor data from a work area environmental status system, using this sensor data to identify corresponding items and objects within the work area. The work area environmental status system may further use the outputs from sensors disposed around or within proximity to the work area (e.g., sensors 611a, 611b, 611c) and / or sensors (not shown) disposed at the transport structure 607 to generate sensor data.

[0175] At 653, the position of an item or the path of an item grasped or a robotic structure deviates from a planned path. As illustrated in Figure 6A, a robotic arm 603a is operable to single-separate an item 613 from a work area 605a to a conveyor structure 607. According to various embodiments, system 600 determines that the robotic arm 603a will proceed according to its single-separation plan or strategy for item 613. In response to received sensor data, system 600 determines whether the position of item 613 or the current path of item 613 deviates from a planned path (e.g., the path specified in the corresponding plan or strategy for single-separation) during single-separation by the robotic arm 603a.

[0176] In some embodiments, system 600 uses sensor data to determine the current position or path of item 613 during a single movement. For example, as the item moves through the space above the chute, a sensor array disposed within the chute may tilt. As the sensor array tilts, corresponding outputs from the sensor array are used to generate sensor data associated with the work area. Thus, if the sensor array within the chute is tilted by the item, the sensor data reflects a position of the item. As another example, as item 613 moves from work area 605a to conveyor structure 607, sensors 611a, 611b, and 611c disposed around work area 605a detect the movement of the item. Sensors 611a, 611b, and 611c may be infrared sensors or cameras.

[0177] System 600 uses sensor data to compare the current position or current path of one of the items 613 with the expected position or expected path of one of the items 613 based on a plan or strategy for distributing the item 613 individually. Based on the comparison between the current position or current path and the plan or strategy for distributing the item 613 individually, system 600 can determine whether the position or path of one of the grasped items (or robot structures) has deviated from the expected path.

[0178] At 655, the plan or strategy for individual item distribution is updated. According to various embodiments, in response to a determination that the position or path of an item (or robotic structure) has deviated from the expected path, system 600 determines to execute an active measure to change or adapt to that deviation. System 600 updates the plan or strategy for individual item distribution 613 based on the determination that the item's position or path deviates from the expected position or path.

[0179] In some cases, if the weight of an item exceeds what the robot system anticipates when determining its plan or strategy for sorting items, the item's position or path may deviate from the expected position or path. The weight of the item can affect the speed at which the robotic arm can move during sorting. In some cases, the weight of the item may exceed the weight that the robotic arm (or end effector) can support. In some embodiments, one active measure in response to an item's weight exceeding its expected weight is to operate the robotic arm to lift the item from the chute section and drag it across the work area to a conveyor structure on which the item will be placed individually. In conjunction with a decision to update the plan or strategy, the robot system can model a 3D view of the top surface of the chute or stack of items within the work area and determine a path along which the robotic structure drags the item.

[0180] In some cases, if an object wobbles (e.g., vibrates) more than expected during the movement of the robotic arm, its position or path may deviate from the intended position or path. In response to the determination that the object wobbles more than expected during the separation period, the robotic system can implement an active measure to suppress the wobbling. For example, in response to the determination that the object wobbles during the separation period, the robotic system can update the plan or strategy used for separating the object to include operating a torque controller to maintain sufficient suction across the suction-type end effector during oscillation. The level of suction applied during the separation period can be determined at least in part based on the weight of the object, a detected oscillation (e.g., a measure of the degree of wobbling during movement), etc.

[0181] In other cases, if the robotic arm grasps more items than expected, the position or path of the items deviates from the expected position or path. In some embodiments, the robotic system determines that the robotic arm has picked up multiple items based on the detection of items grasped by the robotic arm using a vision system (e.g., sensor data from a camera). In some embodiments, the robotic system determines that the robotic arm has picked up multiple items based on sensor data from one or more sensors in the robotic arm. For example, a pressure difference measured across a suction-type end effector (e.g., a pressure difference across different suction cups) can indicate that the robotic arm is grasping different items. As another example, the difference between the weight of the items(s) grasped by the robotic arm and the expected weight of one of the items to be sorted can indicate that the robotic arm has grasped more items than expected according to a plan or strategy for sorting (the items). In some embodiments, a combination of information from various sensors within the robotic system (e.g., cameras, weight systems, pressure sensors, etc.) is used to determine that the robotic arm has grasped more items than expected.

[0182] In response to the determination that the robotic arm has grasped more items than expected, the robotic system may determine one of the active measures for sorting multiple items. In some embodiments, the active measure for sorting multiple items includes operating the robotic arm to identify a corresponding slot on a conveyor structure where multiple items will be placed, moving the multiple items to the conveyor structure, and individually placing the multiple items in the corresponding slot. In conjunction with individually placing the multiple items in the corresponding slot, the robotic arm is operated to release the multiple items in an alternating manner. For example, in the case where the robotic arm has a suction-type end effector including multiple suction cups, the robotic arm is operated to release the suction of at least a subset of the multiple suction cups at different times to release the grasp of the multiple items at different times. The weight difference grasped by the robotic arm between before releasing the set of suction cups and after releasing the at least subset of the multiple suction cups can be used to determine the weight of the items placed on the conveyor structure. In some embodiments, the weight of the items placed by the robotic arm in response to the release of the suction of the set of suction cups can be used to identify items individually placed on the conveyor structure. In some embodiments, the robot system identifies an item placed alone in the conveyor structure based on an identifier associated with the detected item placed on the conveyor structure. In response to the identification of an item placed alone in the conveyor structure, the robot system may update a distributed data structure related to the state of the conveyor structure to associate the item with a slot in the conveyor structure where the item is placed.

[0183] In some embodiments, active measures for sorting multiple items include operating a robotic arm to return one or more items to the work area and operating the robotic arm to continue sorting the intended items according to a plan or strategy. The robotic arm may return one or more items by staggered release of items, such as using the staggered release of suction as discussed above.

[0184] At 657, an updated plan or strategy for individually distributing items is implemented. In some embodiments, in response to updating the plan or strategy for individually distributing items, the robotic system implements the updated plan or strategy. For example, the robotic system operates a robotic arm to place items individually into a corresponding slot on a conveyor structure.

[0185] Figure 7A illustrates a diagram of a robot separation system for separating one or more items, based at least in part on controlling one or more end effectors to release one or more items, according to various embodiments.

[0186] In various embodiments, a robotic system including one or more robotic arms performs sorting / sorting, including performing one or more active actions in response to a detected state or condition associated with one or more items in the work area of ​​the robotic system. The robotic system may include one or more robotic arms, each having one or more end effectors. In the example shown in FIG7A, system 700 includes a robotic arm 703 operated to sort one or more items within work area 710 to a conveying structure 713.

[0187] System 700 may include one or more sensors from one of its modeled workspace environments. In the example shown in Figure 7A, system 700 includes image sensors, in this example including 3D cameras 701 and 702. In various embodiments, other types of sensors may be used (individually or in combination) in a single subsystem as disclosed herein, including a camera, an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, and the like. Robot system 700 operates a robotic arm 703 to pick up an item from chute 711 and place the item individually on conveyor structure 713.

[0188] As illustrated in Figure 7A, the robotic arm 703 includes an end effector 704 by which the robotic arm 703 grasps an object. The end effector 704 may include a plurality of end effectors. In this example, the end effector 704 includes a suction end effector 705a and a suction end effector 705b. Although in the illustrated example, end effectors 705a and 705b are suction end effectors, in various embodiments, one or more other types of end effectors may be used in a single-system as disclosed herein, including (but not limited to) a pinch end effector or other types of actuated grippers.

[0189] According to various embodiments, a robotic arm 703 is operated to pick up one or more items using a plurality of end effectors 705a and 705b. For example, the robotic arm 703 may be operated to pick up a plurality of items 706a and 706b simultaneously. In some embodiments, the robotic arm 703 is operated to pick up items 706a and 706b individually according to a determined plan or strategy for separating one of the items 706a and 706b. In some embodiments, the robotic arm 703 is operated to pick up one of the items 706a and 706b individually, and the robotic arm unintentionally picks up both items 706a and 706b. In response to a determination that the robotic arm 703 has picked up both items 706a and 706b, the system 700 may determine how to place items 706a and 706b separately in the conveyor structure 713.

[0190] If the robotic arm 703 unintentionally picks up items 706a and 706b simultaneously (e.g., if such picking up of multiple items is not included in the determined plan for single sorting), then according to various embodiments, the system 700 determines to execute one of the active measures for single sorting items 706a and 706b. This active measure may include operating the robotic arm 703 to release items 706a and 706b in an alternating manner, such as by staggered release of suction from end effectors 706a and 706b. For example, in the case where the robotic arm 703 has a suction-type end effector comprising multiple suction cups, operating the robotic arm to release the suction of at least a subset of the multiple suction cups at different times to release the grip on the multiple items at different times.

[0191] In some embodiments, the robotic arm 703 releases items 706a and 706b alternately, placing them individually into different slots on the conveyor structure 713. As an example, the robot system 700 may update a distributed data structure associated with the conveyor structure to associate items 706a and 706b with their respective slots. As another example, the robot system 700 may send a fault message indicating that items 706a and 706b have been placed in their respective slots on the conveyor structure 713, but the association between the respective identifiers of items 706a and 706b and their respective slots has not been stored, to a downstream robot system or robot arm of the robotic arm 703. In response to receiving the fault, the downstream robot system or robot arm can determine one or more identifiers associated with items 706a and 706b and store such identifiers in their respective slots on the conveying structure 713 (e.g., the downstream robot system can use this association to update the distributed data structure).

[0192] In some embodiments, the robotic arm releases one or more of a plurality of items it has grasped back to a chute. For example, an active measure determined in response to picking up a plurality of items could be to return at least a subset of the plurality of items to a source stack / flow in the chute, and to operate the robotic arm to individually separate any remaining items (if any) grasped by the robotic arm. As an example, in the case of FIG7A, in response to the determination that the robotic arm 703 has unintentionally grasped both items 706a and 706b, the robot system 700 determines to return one of items 706a and 706b to chute 711, and individually separates items that have been grasped by the robotic arm 703 into a corresponding slot in the transport structure. An association between an identifier of an individually separated item and its corresponding slot is stored. For example, the robot system 700 updates the distributed data structure and stores the identifier of an item individually separated into the corresponding slot of the transport structure 713. As another example, robot system 700 notifies a downstream robot structure or robot arm of a fault. In response to the fault, the downstream robot structure or robot arm determines an identifier corresponding to a single-divided item and associates the identifier with a slot in which the item is single-divided by robot arm 703.

[0193] The staggered release of grips from a plurality of end effectors on a robotic arm can be achieved by releasing the grips at predetermined time intervals or in response to the fulfillment of one or more conditions. As an example, if the robotic arm 703 has gripped items 706a and 706b, the suction force from the end effector 705a is released a predetermined time after the suction force from the end effector 705b is released. As another example, the suction force from the end effector 705a is released in response to a determination that the robotic arm 703 has been operated to move item 706a above a position where item 706a will be released (and at a time when the release of item 706a will be released into a specific slot of a conveying structure). Similarly, the suction force from the end effector 705b is released in response to a determination that the robotic arm 703 has been operated to move item 706b above a position where item 706b will be released (and at a time when the release of item 706b will be released into another specific slot of a conveying structure 713).

[0194] In some embodiments, proactive measures include requesting human intervention from human operator 719 via manual intervention or remote intervention using an on-demand teleoperation device 717, which may be used to control one or more robotic arms 703, or to configure one of the settings of system 700 using a control computer 715. Human operator 719 may intervene to ensure that items 706a and 706b are correctly separated on the conveyor structure 713, or to return one or more items 706a and 706b to the chute 711.

[0195] Figure 7B is a diagram illustrating a procedure for sorting one item, at least in part, based on controlling one or more end effectors to release one or more items, according to various embodiments. The procedure 750 of Figure 7B can be executed by the system 700 of Figure 7A. In some embodiments, the procedure 750 is implemented by a robotic system operating to sort one or more items within a work area (such as work area 710 of system 700 in Figure 7A). System 700 includes one or more processors that operate to cause a robotic arm 703 to pick up and place items for sorting.

[0196] At 751, sensor data is received. System 700 receives sensor data associated with work area 710 (e.g., sensor data related to robot arm 703 and conveying structure 713). System 700 uses the sensor data in conjunction with a determination that robot arm 703 has picked up a plurality of items.

[0197] In some embodiments, system 700 uses a vision system (e.g., sensor data from a camera) to detect items grasped by robotic arm 703 and determine that robotic arm 703 deviates from a plan or strategy for sorting items individually. In some embodiments, robotic system 700 determines that robotic arm 703 has picked up multiple items based on sensor data from one or more sensors in robotic arm 703. For example, a pressure difference measured across suction end effectors 705a and 706b can indicate that the robotic arm is grasping different items. As another example, the difference between the weight of the items(s) grasped by the robotic arm and the expected weight of one of the items to be sorted can indicate that the robotic arm has grasped more items than expected according to a plan or strategy for sorting items(s). In the case of Figure 7A, before picking up items 706a and 706b, if the plan or strategy for separating item 706a includes picking up 706a separately and placing item 706a on the conveyor structure 713, then system 700 can obtain an expected weight of item 706a and compare this expected weight with information from (a plurality of) weight sensors by combining it with a determination that the robotic arm 703 will also pick up item 706b. The expected weight of item 706a can be obtained based on the size of item 706a and / or an identifier on item 706a. As an example, the expected weight of item 706a can be obtained by obtaining a mapping from weight or weight range to the size of the item (e.g., dimensions, volume, etc.), and the size of item 706a is used as a lookup to determine the expected weight.

[0198] In some embodiments, a combination of information from various sensors within the robotic system (e.g., cameras, weight systems, pressure sensors, etc.) is used to determine that the robotic arm has grasped more items than expected.

[0199] At 752, a plan or strategy for releasing one or more items is determined, at least in part, based on sensor data. According to various embodiments, in response to a determination that the robotic arm 703 (unintentionally) picks up a plurality of items, the system 700 determines a plan or strategy for releasing one or more of the plurality of items grasped by the robotic arm 703. The plan or strategy for releasing one or more items includes an indication of a location for releasing each item (e.g., returning to the source stack / flow at chute 711, or placing it individually in a slot on the transport structure 713), and a release strategy for operating end effectors 705a and 705b. In some embodiments, the release strategy for operating end effectors 705a and 705b includes an indication of a timing for releasing suction from end effectors 705a and 705b and / or one or more conditions that must be met for releasing suction from end effectors 705a and 705b. In some embodiments, the release strategy for operating the end effectors 705a and 705b includes a strategy for staggered release of suction to release one of a plurality of items at different times and / or at different locations.

[0200] At 753, the plan or strategy for single-item separation is updated to include a plan or strategy for releasing one or more items. According to various embodiments, in response to determining a plan or strategy for releasing one or more items, system 700 updates the plan or strategy for single-item separation at least in part based on the plan or strategy for releasing one or more items. For example, if the initial plan for single-item separation of item 706a includes a plan or strategy for picking up item 706a individually, then the robotic arm 703 is operated to move item 706a to the conveyor structure 713 and place item 706a individually on the conveyor structure 713. If the robotic arm 703 unintentionally picks up item 706b while picking up item 706a, then the update to the plan or strategy for single-item separation of item 706a includes a plan or strategy for releasing item 706b (e.g., releasing item 706b back into the chute 711 or placing item 706b individually in a corresponding slot on the conveyor structure 713).

[0201] At 754, a set of suction cups or end effectors are operated in conjunction with the single-dispensing of one or more items. In some embodiments, system 700 implements an updated plan or strategy for single-dispensing one or more items, including operating robotic arm 703 to move one or more items to a planned location and releasing one or more items according to the plan or strategy. System 700 controls robotic arm 703 based on the updated plan or strategy for single-dispensing one or more items to release one or more items at different times (e.g., staggered releases) or simultaneously.

[0202] Figure 8A is a diagram illustrating a robot subsystem performing a diagnostic to detect deviations from normal operation according to various embodiments.

[0203] In various embodiments, a robotic system comprising one or more robotic arms performs a sorting / induction process, including executing one or more active measures in response to a detected state or condition associated with one or more items in the robotic system's work area. The detected state or condition may correspond to a deviation from normal operation of a robotic arm or a portion thereof. For example, the robotic system may use sensor data to determine that an end effector on the robotic arm is malfunctioning, and in response to this determination, the robotic system may execute a diagnostic procedure to determine whether the end effector should be replaced or repaired.

[0204] The robotic system may include one or more robotic arms, each having one or more end effectors. In the example shown in Figure 8A, system 800 includes a robotic arm 803 operated to individually separate one or more items within a work area 810 to a conveyor structure 813. System 800 may include one or more sensors from the environment of its modeled work area. In the example shown in Figure 8A, system 800 includes image sensors, in this example including 3D cameras 801 and 802. In various embodiments, other types of sensors may be used (individually or in combination) in an individual separation system as disclosed herein, including a camera, an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, a weight sensor, and the like.

[0205] In various embodiments, the control computer 817 includes a work area environment status system (such as a vision system) for identifying individual items, clutter in the work area, and the orientation of each item based on sensor data (such as image data provided by image sensors (in this example, including 3D cameras 801 and 802)). This work area environment status system may include sensors in a robotic arm for detecting the weight of an item (e.g., an item being grasped) or detecting information from which an estimated weight is determined. For example, information relating to a quantity of current, voltage, and / or power used by one or more motors driving the movement of the robotic arm can be used to determine the weight of the item (or an estimated weight). As another example, information relating to the outputs of one or more sensor arrays can be used to determine a position of the item in the work area, a position of the item when it is grasped and / or moved by the robotic arm, and / or a position of the robotic arm (e.g., based on a determination from the output of one subset of sensors of one or more sensor arrays compared to another subset of sensors of one or more sensor arrays). The work area environment status system may include sensors for detecting the amount of pressure generated between an object (e.g., a gripped object) and a suction-type end effector, the force of gripping the object, or other information related to the quality of the grip of the robotic arm 803 on the object it grips.

[0206] According to various embodiments, the work area environment status system generates an output used by system 800 to determine and implement a plan for autonomously operating the robotic arm 803 to pick up one or more items from work area 810 (e.g., from chute 811) and place each item in a corresponding usable, defined location (such as a section of segmented conveyor 813) for machine identification and classification. In some embodiments, the work area environment status system generates an output used by the robot system to detect a state or condition associated with one or more items in the work area, and / or a state or condition associated with another element of the robotic arm or work area (e.g., otherwise characterizing sensor data or information of the work area and the items within it). According to various embodiments, in response to detecting (e.g., determining) a state or condition associated with one or more items in the work area, the robot system implements one or more proactive actions in conjunction with individual items. The proactive actions may include updating a plan for autonomously operating the robot system 800 to perform a diagnostic on the robotic arm 803 or a portion thereof (such as end effector 804). In some embodiments, proactive measures or update plans include operating the robot structure to change or adapt to detected states or conditions (e.g., implementing a procedure to diagnose one of the detected deviations from normal operation).

[0207] According to various embodiments, during the process of separating one or more items from the chute 811 to the conveying structure 813, the system 800 uses sensor data related to the work area 810 (including sensor data related to the operation of the robotic arm 803) to determine whether the system 800 is operating according to normal operation. Normal operation of the system 800 can be defined by various states or predetermined ranges of measurement values. As an example, one mapping of normal operation of the end effector 804 and / or the suction-type end effectors 805a and 805b disposed at the distal end of the end effector 804 includes a range of pressure values ​​or one or more pressure thresholds corresponding to normal operation (or satisfactory operation) of the corresponding end effector when the end effector engages with the surface of an item or object. In some embodiments, the system 800 compares the obtained sensor data related to the operation of the system 800 (including the operation of the robotic arm 803 and / or the end effectors disposed thereon) with the values ​​obtained regarding normal operation (e.g., predetermined ranges or thresholds). If the sensor data related to the operation of system 800 falls outside the values ​​indicating normal operation of system 800, system 800 may determine that system 800 has deviated from normal operation or perform a diagnostic procedure to assess whether to implement a remedial proactive measure.

[0208] An example of a system 800 determining that it is deviating from normal operation or performing a diagnostic procedure in conjunction with an assessment of whether to implement a remedial proactive measure includes the system 800 obtaining sensor data relating to one or more pressure values ​​corresponding to the pressure obtained by end effectors 805a and / or 805b when the robotic arm has picked up item 806, and the system 800 determining whether a comparison between the sensor data and values ​​relating to normal operation (e.g., a predetermined range or threshold value) indicates that the current operation of the system 800 is deviating from normal operation. Values ​​relating to normal operation may include pressure values ​​that are expected to be obtained by end effectors 805a and / or 805b in normal operation when the robotic arm has picked up item 806. In response to determining that the sensor data corresponding to the current operation of the system 800 falls outside (or is otherwise inconsistent with) the predetermined range or threshold value corresponding to the normal operation of the system 800, the system 800 determines that it is deviating from normal operation or performs a diagnostic procedure in conjunction with an assessment of whether to implement a remedial proactive measure. If the pressure obtained by end effector 805a and / or end effector 805b is less than a threshold pressure value corresponding to normal operation, system 800 determines that the end effector with unsatisfactory obtained pressure is deviating from normal operation. System 800 may consider the end effector with unsatisfactory obtained pressure as damaged or requiring maintenance (e.g., through human intervention by human operator 820).

[0209] According to various embodiments, in response to a determination that the current operation of system 800 deviates from the expected normal operation of system 800, system 800 determines to perform a diagnostic on system 800. System 800 may perform a diagnostic on a portion of system 800 based at least in part on sensor data (e.g., current sensor data) indicating a deviation from normal operation or within a threshold range or percentage of deviation from normal operation. In the case of pressure measurements obtained by end effectors 805a and / or 805b described above, system 800 may determine which particular end effector(s) will be subject to a diagnostic procedure. In some embodiments, in response to the result of performing a diagnostic procedure and determining that system 800 (or one of its components) deviates from normal operation or otherwise falls within a threshold range or percentage of deviation from normal operation, system 800 implements one or more proactive measures. Examples of proactive measures include replacing components (e.g., switching an end effector), determining that the system 800 operates in a manner that avoids using components deviating from normal operation or does not apply strain exceeding a predefined strain threshold on components, and / or invoking human intervention (e.g., notifying a human operator 820 of the deviation). Various other proactive measures may be implemented.

[0210] Continuing with the example described above in conjunction with Figure 8A, system 800 at least partially responds to system 800's determination that system 800 has deviated from normal operation or will combine an assessment of whether to implement a remedial proactive measure to perform a diagnostic procedure, thereby determining to perform a diagnostic procedure. In response to determining that the pressure obtained by end effector 805a and / or end effector 805b deviates from the expected normal operation of one of the end effectors 805a and / or end effector 805b when the robotic arm has picked up item 806, system 800 determines to perform a diagnosis relative to the operation of (a number of) end effectors 805a and / or end effector 805b that have been determined to deviate from normal operation. According to various embodiments, a diagnostic procedure for performing a diagnosis relative to an end effector includes manipulating robotic arm 803 to move to a predetermined position and engaging the end effector with a predetermined surface (in this example, such as surface 815). Surface 815 may be located within work area 810. In some embodiments, surface 815 is partially or operatively connected to chute 811 or conveying structure 813. When the end effector engages a predetermined surface, system 800 controls the end effector to grip the predetermined surface. In response to controlling the end effector to grip the predetermined surface, system 800 obtains sensor data relating to a measurement of the gripping strength of the end effector on the predetermined surface. For example, in response to engaging the end effector with surface 815, system 800 controls the end effector to apply a suction force to surface 815. System 800 obtains sensor data including one or more pressure values ​​obtained between the end effector and surface 815. System 800 then determines whether the pressure obtained by the end effector when gripping surface 815 deviates from an expected normal operation of gripping surface 815. In some embodiments, system 800 compares the sensor data including one or more pressure values ​​obtained between the end effector and surface 815 with one or more predetermined ranges or threshold values ​​mapped to normal operation of the end effector. If the pressure obtained between the end effector and surface 815 is inconsistent with the normal operation of the end effector (e.g., if one or more pressure values ​​obtained between the end effector and surface 815 fall outside one or more predetermined ranges or thresholds mapped to the normal operation of the end effector), then system 800 determines that the end effector is not working correctly (e.g., the end effector deviates from normal operation).

[0211] According to various embodiments, in response to system 800 determining that the end effector is not functioning correctly, system 800 updates its plan, at least in part, to implement one or more proactive measures based on this determination. This proactive measure includes alerting human operator 820 that the end effector is not working. For example, system 800 sends a notification of deviation from normal operation to human operator 820. In some embodiments, system 800 provides human operator 820 with one or more recommended proactive measures. In response to receiving the notification of deviation from normal operation, human operator 820 may implement human intervention to replace or repair the applicable end effector. In some embodiments, human operator 820 uses on-demand teleoperation device 819 to control system 800 to implement a proactive measure, such as controlling robotic arm 803 to replace the applicable end effector. In some embodiments, human operator 820 may select at least one of one or more recommended proactive measures, and in response to this selection, system 800 controls robotic arm 803 to implement the selected proactive measure (e.g., without additional human intervention).

[0212] Figure 8B is a diagram illustrating a procedure for performing a diagnostic to detect deviations from normal operation of a robotic sorting system, according to various embodiments. The procedure 850 of Figure 8B can be executed by the system 800 of Figure 8A. In some embodiments, the procedure 850 is implemented by a robotic system that operates to sort one or more items within a work area (such as work area 810 of system 800 in Figure 8A). System 800 includes one or more processors that operate to cause a robotic arm 803 to pick up and place items for sorting.

[0213] In 851, a condition that causes an obstructive ability to grasp an object is detected. In some embodiments, the condition corresponding to the obstructive ability to grasp an object is an indication that an end effector has a low suction value.

[0214] At 853, a diagnostic test is performed. In some embodiments, a robotic arm is manipulated to move to a predetermined position and a diagnostic test is performed. In some embodiments, the diagnostic test corresponds to a suction test in which a value indicates the suction force obtained by an end effector (e.g., when engaged with a surface such as a predetermined surface). The value indicating the suction force obtained by the end effector may be determined at least in part based on the output of one or more pressure sensors. In some embodiments, in the case of a pinch-type end effector, the diagnostic test is a clamping test (e.g., a test measuring a clamping strength). Various other diagnostic tests may be performed based on the various types of end effectors implemented.

[0215] At 855, the results from the diagnostic test are analyzed to determine whether the diagnostic test indicates that the robot system has deviated from normal operation. In some embodiments, the robot system determines whether the diagnostic test indicates that the robot system has deviated from normal operation based on a comparison of one or more values ​​corresponding to the results from the diagnostic test with one or more values ​​or ranges corresponding to normal operation. If the results of the diagnostic test are outside or inconsistent with one or more values ​​or ranges corresponding to normal operation, the robot system determines that the diagnostic test indicates that the robot system has deviated from normal operation.

[0216] Returning to the example discussed above in conjunction with Figure 8B, system 800 determines whether end effector 805a or 805b is operating correctly, at least in part, based on diagnostic tests performed relative to surface 815. If the pressure obtained by end effector 805a and / or end effector 805b is less than a threshold pressure value corresponding to normal operation (or outside a predetermined pressure range), system 800 determines that the end effector is deviating from normal operation. System 800 may consider the end effector to be damaged or require repair (e.g., through human intervention by human operator 820). Conversely, if the pressure obtained by end effector 805a and / or end effector 805b is consistent with a threshold pressure value corresponding to normal operation (or within a predetermined pressure range), system 800 determines that the end effector is operating correctly.

[0217] If a diagnostic test indicates that the end effector is operating normally (or within one of the ranges corresponding to normal operation), then procedure 850 continues to operation 859, in which autonomous operation is resumed. In some embodiments, in response to the determination that the end effector is operating normally, the robot system operates a robot structure (e.g., robot arm 803) to implement a plan or strategy for separating one or more items.

[0218] In contrast, in response to a diagnostic test indicating that the end effector is not operating correctly (e.g., operating outside a range corresponding to normal operation), procedure 850 invokes an active measure at 857. As an example, an active measure could be a remedial measure for abnormal operation (e.g., replacing the malfunctioning end effector). As another example, an active measure could be using other end effectors and avoiding gripping objects with end effectors (e.g., using some end effectors to grip an object and avoiding the use of invalid end effectors). As yet another example, an active measure could be requesting assistance or invoking human intervention to remedy a deviation from normal operation by sending an alert to a human operator (e.g., human operator 820 could manually remove and replace an invalid end effector, or human operator 820 could use on-demand teleoperation 819 to control the robotic arm 803 to replace the invalid end effector).

[0219] Figure 9 is a diagram of a hierarchical scheduling system according to various embodiments of a robot single-station system. In various embodiments, the hierarchical scheduling system 900 of Figure 9 is at least partially implemented on a computer (such as control computer 212 of Figures 2A and 2B, control computer 599 of Figure 5B, control computer 715 of Figure 7A, and control computer 817 of Figure 8A). In the illustrated example, the hierarchical scheduling system 900 includes a global scheduler 922 configured to optimize throughput by coordinating the operation of a plurality of robot single-stations and a segmented conveyor (or similar structure) configured to place items on the robot single-stations. According to various embodiments, the global scheduler 922 is configured to coordinate the operation of a plurality of robot arms within a single work area to ensure that a desired throughput is achieved while ensuring that the plurality of robots do not collide. The global scheduler 922 can be configured to take proactive action in response to the determination that multiple robotic arms (e.g., operating within a single work area) have collided or are expected to collide (e.g., if the multiple robots continue to implement their respective plans or strategies for sorting items individually). The global scheduler 922 can be implemented as a processing module or other software entity running on a computer. The global scheduler monitors and coordinates the work within a single robotic station at least in part by monitoring and, as needed, controlling and / or otherwise providing input to multiple robot single-station schedulers 924, 926, 928, and 930.

[0220] In some embodiments, a robotic single station includes a single robotic arm controlled to single-dispense one item within a work area, and a work area may include a plurality of robotic single stations. In some embodiments, a robotic single station includes a plurality of robotic arms coordinated to single-dispense a plurality of items within a work area. In some cases, if a robotic single station includes a plurality of robotic arms, a particular work area may include a single robotic single station.

[0221] Each of the robot single-station schedulers 924, 926, 928, and 930 is associated with a corresponding robot single-station and controls and coordinates the operation of one or more robot arms and associated end effectors to pick up items from a corresponding chute or other item container and place them individually on a segmented conveyor or similar structure. Each of the robot single-station schedulers 924, 926, 928, and 930 is associated with a set of one or more corresponding station sensors 932, 934, 936, and 938, and each uses sensor data generated by the sensors at its station to perform automated single-station dispatching at its robot single-station. In some embodiments, each implements and executes procedures 300, 320 and 330 of Figures 3A to 3C, procedure 400 of Figure 4, procedures 500 and 550a to 550d of Figures 5A and 5C to 5F, procedure 650 of Figure 6B, procedure 750 of Figure 7B, procedure 850 of Figure 8B, and procedures 1050 and 1075 of Figures 10B and 10C.

[0222] In various embodiments, each of the robot single-station schedulers 924, 926, 928, and 930 reports one or more of the following to the global scheduler 922: image and / or other station sensor data; object recognition, grasping strategy, and success probability data; pick / placement plan information; and expected item single-station processing volume information. The global scheduler 922 is configured to use information received from the robot single-station schedulers 924, 926, 928, and 930, as well as sensor data received from other sensors 940 (such as cameras pointing to other parts of the work area not covered or well or completely covered by the station sensors), to coordinate the operation of each robot single station under the control of its station-specific scheduler 924, 926, 928, and 930, and to control the operation (e.g., speed) of the segmented conveyor via the conveyor controller 942, in order to optimize (e.g., maximize) the collective single-station processing volume of the system.

[0223] In various embodiments, the global scheduler 922 employs one or more techniques to implement collision avoidance among the multiple robots, including the robot splitting system, when performing splitting using multiple robots, (e.g.) to maximize overall throughput or achieve a desired overall throughput level. For example, a particular work area may contain multiple robots operating to split items within that particular work area (e.g., the multiple robots pick up items from the same source stack / flow in a chute and place each item individually into a corresponding slot in a conveyor structure). Since the work area contains multiple robot arms splitting items therein, the robot arms may have overlapping ranges of motion. In some embodiments, the global scheduler 922 detects a anticipated collision event at least in part based on information received from robot single-station schedulers 924, 926, 928, and 930 and sensor data received from other sensors 940 (such as cameras pointing to other parts of the work area not covered or well- or completely covered by the station sensors) to coordinate the operation of the respective robot single-stations. The global scheduler 922 may use information received from the respective robot single-station schedulers 924, 926, 928, and 930 to determine the positions of a plurality of robots and the probability that two of the plurality of robots will collide with each other or that one robot will collide with another object within the work area where the robot is operating. For example, the global scheduler detects a anticipated collision event at least in part based on the position of at least one robot and the path or strategy (for single-item sorting) for at least one robot. In some embodiments, a planned collision event is detected based on a determination that the position or path of a first robot (or an object grasped by the first robot) is planned to intersect with the position or path of a second robot (or an object grasped by the second robot) at a specific time.

[0224] In some embodiments, each robot operating in a work area where multiple robots pick up items from a common source (such as the same chute) operates largely independently to identify and grasp the item to be picked up by that robot from that source. Collisions can be minimized or avoided by configuring each robot to pick up from potentially overlapping assigned areas of the chute or other sources. For example, each robot may (primarily) pick up from the side of the chute closest to itself. Alternatively, in some embodiments, one robot may be configured to (primarily) pick up from the rear of the chute (farthest from the destination conveyor) while another robot picks up from the front of the chute (closest to the destination conveyor). In some embodiments, in the event of a detected collision (risk), each robot implements a random waiting time interval and resumes operation. If different / random waiting results in no collision risk, operation resumes and continues. In some embodiments, a heuristic or logic may be applied to resolve / avoid collisions, such as by allowing a robot to target an item further back in the chute to reach and grasp first, followed by a robot targeting an item closer to the front (i.e., closer to the destination conveyor). In some embodiments, a robot that picks up an item from a position closer to the end of the chute may move first to place its item, followed by a robot that grasps an item from a position further back.

[0225] In some embodiments, multiple robots operating in a single work area operate to perform their respective programs in parallel to individually dispense items, and each of the multiple robots simultaneously operates to perform a specific step or set of steps in a sequence of steps of its respective program to individually dispense items before any of the multiple robots moves to the next step or set of steps in that sequence. For example, each of the multiple robots operates to pick up an item from a source stack / flow and each of the multiple robots waits until (several) other robots accordingly pick up an item to individually dispense items from the work area before proceeding to the next step or sequence of steps in the program. As another example, each robot operates to move an item from a chute area (from where it grips an item) to a specific location on a conveyor structure (where the item will be placed), and each of the multiple robots waits until (several) other robots accordingly move the items they are gripping to the corresponding location on the conveyor structure to individually dispense items (e.g., place the item on the conveyor structure) before proceeding to the next step or sequence of steps in the program. In the aforementioned manner, multiple robots operate independently to determine plans and strategies for sorting items within a single work area; however, the multiple robots are coordinated as the sorting process progresses. According to various embodiments, a collision event is detected based on a determination that a position or path of a first robot (or an item grasped by the first robot) intersects with the position or path of a second robot (or an item grasped by the second robot) at a specific step in a sorting sequence. In this way, the position or path of a second robot is used only by a single robot during a specific step or set of steps in the sorting process.

[0226] According to various embodiments, robot single-station schedulers 924, 926, 928, and 930 register plans or strategies for operating the corresponding robots to single-distribute items with the global scheduler 922, or otherwise store such plans or strategies in a storage location accessible to the global scheduler 922. Robot single-station schedulers 924, 926, 928, and 930 can independently determine the plans or strategies for operating the corresponding robots to single-distribute items. In some embodiments, although robot single-station schedulers 924, 926, 928, and 930 operate independently to determine their respective plans or strategies, they may determine their respective plans or strategies at different times (e.g., causing the same item not to be selected for single-distribution by two robots). In some embodiments, robot single-station schedulers 924, 926, 928, and 930 operate independently to determine their respective plans or strategies, and register their respective plans or strategies with global scheduler 922 at different times. If, during the registration of their plan or strategy with global scheduler 922, this plan or strategy conflicts with an existing registered plan or strategy, global scheduler 922 may send a fault to a robot single-station scheduler. Examples of plans or strategies considered to be in conflict include multiple plans or strategies related to a single item from the work area, multiple plans or strategies using the same path or trajectory, multiple plans or strategies using intersecting paths or trajectories, multiple plans or strategies involving moving a portion of a robot arm or item to the same location, multiple plans or strategies selecting the same slot at a conveyor structure where the corresponding item will be placed, etc. Various other conflicts are possible between plans or strategies.

[0227] Although the robot single-station schedulers 924, 926, 928, and 930 operate independently to determine the plans or strategies for operating the corresponding robots to single-distribute items, the global scheduler 922 can provide oversight of the entire robot system to detect a anticipated collision event and implement proactive measures to prevent it. In some embodiments, implementing proactive measures includes sending a fault to a robot single-station scheduler (e.g., corresponding to at least one of the robots identified as being associated with the anticipated collision event). For example, in response to sending the fault to the robot single-station scheduler, the global scheduler 922 may require the robot single-station scheduler to update its current plan or strategy (e.g., to avoid the anticipated collision event). The fault sent to the robot single-station scheduler may include an indication of a cause of the fault (e.g., an indication of an anticipated collision event, and / or information related to the anticipated collision event, such as an indication of how the plan or strategy of one robot single-station scheduler conflicts with the plan or strategy of another robot single-station scheduler).

[0228] In various embodiments, the global scheduler 922 employs one or more techniques to optimize the use of multiple robots, including the robot splitting system, to perform splitting, for example, optimizing the overall throughput or achieving a desired overall throughput level. For example, if four robots are present sequentially, a leading (or other upstream) robot can be controlled to place packages in a manner that leaves open slots, so that a downstream robot does not wait for an empty slot. This approach has an impact because downstream robots may wait for an unknown / random amount of time due to package flow, etc. Therefore, a primary strategy (such as the leading robot placing packages into every four empty slots) may not optimize the collective throughput. Sometimes, if its packages are not flowing, it may be better for the leading robot to place two or three packages sequentially into consecutive slots, but overall, the system makes such decisions based on the status and flow at each station. In some embodiments, the optimal strategy for leaving open slots for downstream robots is based on the downstream robot's anticipated request for an open slot (e.g., varying according to its package flow). In some embodiments, information from the local station scheduler is used to anticipate the maximum throughput of each station and control conveyor speed and how many empty slots are left by the upstream robot to ensure that the downstream robot can access slots proportional to its (current) pick-up / place speed. In some embodiments, when the segmented conveyor is full due to some bottleneck in the downstream sorting process, a robotic sorting system as disclosed herein can pre-sort one or more packages, for example, within its corresponding chute or in a nearby aggregation area, while tracking the posture of each pre-sorted package. Once some empty space becomes available from the segmented conveyor, the system / station moves the pre-sorted packages individually and rapidly onto the segmented conveyor without additional vision processing time.

[0229] In some embodiments, the presence of a human working alongside the robot influences placement and multi-robot coordination strategies because the robot or associated computer vision or other sensing systems must now also observe human behavior and adjust robot placement in real time. For example, if a human takes over a conveyor belt chute scheduled for use by a robot, the system must adjust its global and local schedulers / plans accordingly. In another instance, if a human damages a robot's picked-up package and causes it to register as unpicked, the system adjusts to correct the error. Alternatively, if a human corrects a robot's picking error (instructing the robot to place a package in chute A but accidentally placing it across chute A and adjacent chute B; and the human places it in chute B even though the system memory indicates the package was in chute A), the system must observe the human's actions and adjust downstream robot actions.

[0230] In various embodiments, global scheduler 922 can cause a station to operate slower than its maximum possible throughput at a given time. For example, global scheduler 922 can explicitly instruct local station schedulers (e.g., 924, 926, 928, and 930) to slow down and / or make fewer slots available to the local station, for example, explicitly by assigning fewer slots to the station or indirectly, such as by allowing upstream stations to fill more slots.

[0231] In various embodiments, global scheduler 922 may transmit a fault to one or more local station schedulers (e.g., 924, 926, 928, and 930) in response to detecting a state or condition associated with one or more items in the work area. The fault transmission may be combined with an active measure that causes one or more local station schedulers to perform a change or adaptation to the detected state or condition. For example, if global scheduler 922 determines that a robot has placed two items into a single slot of a conveyor (e.g., based on sensor data related to the conveyor, or in response to receiving an instruction from a robot structure corresponding to the robot), global scheduler 922 sends a fault to one or more local station schedulers downstream of the robot that placed the two items into the single slot or downstream of the slot containing the two items. In response to receiving the fault, a local station scheduler can determine a plan or strategy for an active measure to pick up one of two items from the slot and place the item separately in the other slot on the conveyor (and update the distributed data structure using an association between the item(s) and the slot(s)).

[0232] In some embodiments, global scheduler 922 responds to a determination by which an item placed on a conveyor crosses one of two different slots by transmitting a fault to one or more local station schedulers (e.g., 924, 926, 928, and 930). Global scheduler 922 transmits the fault to cause a local station scheduler downstream of the item to determine a plan or strategy for repositioning the item into a single slot on the conveyor, and causes a corresponding robot to implement that plan or strategy for repositioning the item. Following repositioning, the distributed data structure can be updated accordingly using an association between the item and the slot it occupies.

[0233] Figure 10A is a diagram illustrating a robotic separation system comprising one of a plurality of robotic arms operating within the same work area to separate items within that work area, according to various embodiments.

[0234] As illustrated in Figure 10A, system 1000 includes three additional stations: robotic arms 1003, 1005, 1009, 1013, and 1017, which are positioned and configured to pick up / place items from chutes 1007, 1011, 1015, and 1019. According to various embodiments, one or more robotic arms can be operated to individually dispense items from the same chute. For example, robotic arms 1003 and 1005 operate to pick up / place items from chute 1007. System 1000 includes one or more sensors that provide information from sensor data determined to be related to a work area. System 1000 includes cameras 1001a, 1001b, 1001c, and 1001d that can be configured or configured to provide a 3D view of one of the four stations / chutes 1007, 1011, 1015, 1019, and conveyor 1021.

[0235] In various embodiments, the control computer 1023 coordinates the operation of the robotic arms 1003, 1005, 1009, 1013 and 1017 and their associated end effectors, together with the conveyor 1021, to pick up / place items from the chutes 1007, 1011, 1015, 1019 onto the conveyor 1021 in a manner that achieves a desired collective processing capacity of the system (e.g., a collective processing capacity that meets a processing capacity threshold). In various embodiments, the control computer 1023 implements the global scheduler 922 and robot single-station schedulers 924, 926, 928 and 930 of the hierarchical scheduling system 900 of FIG9 to coordinate the operation of robot arms 1003, 1005, 1009, 1013 and 1017 to single-separate items from chutes 1007, 1011, 1015 and 1019 to conveyor 1021 in a manner that ensures that collision events are avoided during the operation of robot arms 1003, 1005, 1009, 1013 and 1017.

[0236] In various embodiments, two or more robots may be deployed at a single station to avoid interference with each other's operation and movement and to maximize their collective throughput (including by avoiding and / or managing contention for picking up and placing the same item) under the control of an associated control computer (such as control computer 1023 in the example shown in Figure 10A). In some embodiments, multiple robotic arms operating in the same work area work independently to single-sort multiple items. One or more of the multiple robotic arms may respond to the detection of a collision or the possibility of a collision between two robotic arms and perform an active measure to avoid a collision between the two robotic arms. For example, control computer 1023 may coordinate the operation of the multiple robots so that the multiple robots can operate independently while ensuring that the multiple robots and / or the items grasped by the multiple robots do not collide with each other during single-sorting. In some embodiments, control computer 1023 implements / enforces a "force field" between two or more robots to prevent collisions between the two or more robots. As an example, a robot (or control computer 1023) accesses information, determines its own position and the positions of one or more other robots from that information, and controls the robot to avoid an intersection between its own position or path and the positions or paths of one or more other robots at a specific time or during a specific segmentation phase. In some embodiments, a first robot reserves an airspace (e.g., a specific location) that will be used by the first robot during the segmentation of an item. In conjunction with a second robot scheduling the segmentation of an item, the second robot determines the segmentation plan based at least in part on the airspace reserved by the first robot. For example, in conjunction with scheduling the segmentation of an item, the second robot determines that the plan cannot include movement through the airspace reserved by the first robot and that the second robot determines that during the time the airspace is reserved, neither the second robot nor the item needs to move through the airspace reserved by the first robot.

[0237] In various embodiments, a scheduler coordinates the operation of a plurality of robots (e.g., one or more robots working at various stations) to achieve the desired throughput without conflicts between robots (e.g., one robot placing an item in a location that the scheduler has assigned to another robot).

[0238] In some embodiments, at least a subset of a plurality of robots working in a work area independently picks up an item, and determines a corresponding plan for single-distributing the item. The at least subset of robots may pick up items in a predefined order such that no two robots simultaneously select or pick up an item. Each of the at least subset of robots may select or pick up an item based on items currently available at the time of selection. Therefore, a second robot, which picks up an item after a first robot, from at least two subsets of the plurality of robots, will select an item different from the one selected or picked up by the first robot for single-distribution.

[0239] In various embodiments, such as those disclosed herein, a robotic system coordinates the operation of multiple robots to pick up items one after another from a source storage bin or chute and place the items at an assigned location on a conveyor or other device to move the items to the next stage of machine identification and / or classification.

[0240] In some embodiments, multiple robots may pick up from the same chute or other source container. In the example shown in Figure 10A, for example, robot arm 1005 may be configured to pick up from chute 1007 or chute 1011. Similarly, robot arm 1013 may pick up from chute 1015 or chute 1019, etc. In some embodiments, two or more robot arms configured to pick up from the same chute may have different end effectors. One of the robot sorting systems disclosed herein may select the robot arm most suitable for picking up and sorting a given item. For example, the system determines which robot arms can reach the item and selects one with the most suitable end effector and / or other properties that allow for successful grasping of the item.

[0241] Although a fixed robotic arm is shown in Figure 10A, in various embodiments, one or more robots may be mounted on a mobile delivery vehicle, such as a robotic arm mounted on a chassis configured to move along a rail, track, or other guide, or a robotic arm mounted on a mobile cart or chassis. In some embodiments, a robotic tool actuator other than a robotic arm may be used. For example, an end effector may be mounted on a rail and configured to move along the rail, and the rail may be configured to move on one or more axes perpendicular to the rail so that the end effector can move to pick up, translate, and place an item as disclosed herein.

[0242] Figure 10B is a diagram illustrating a procedure for sorting one of a plurality of items in the same work area using a robotic sorting system comprising one of a plurality of robotic arms operating within a work area, according to various embodiments. The procedure 1050 of Figure 10B can be executed by the system 1000 of Figure 10A. In some embodiments, procedure 1050 is implemented by a robotic system operating to sort one or more items within one or more work areas (such as the work areas corresponding to chutes 1007, 1011, 1015, and / or 1019 of system 1000 of Figure 10A). System 1000 includes one or more processors that operate to cause robotic arms 1003, 1005, 1009, 1013, and / or 1017 to pick up and place items for sorting.

[0243] In steps 1051a, 1051b, and 1051c, a plan or strategy for distributing items individually is determined. According to various embodiments, the plan or strategy for distributing items individually is determined independently of each other. For example, each of steps 1051a, 1051b, and 1051c may be implemented at least partially by different schedulers (such as robot single-station schedulers 924, 926, 928, and 930 of the hierarchical scheduling system 900 in Figure 9). The plan or strategy determined in steps 1051a, 1051b, and 1051c is associated with the operation of different robots within a robotic system.

[0244] In 1053, attempt to implement a plan or strategy for single-item distribution by independently operating a plurality of corresponding robots. The plan or strategy is implemented by controlling a plurality of robots through a single-station scheduler for each robot.

[0245] In some embodiments, implementing a plan or strategy for distributing items includes providing information related to the plan or strategy to a global scheduler (such as the global scheduler 922 of the hierarchical scheduling system 900 in Figure 9). This global scheduler is configured to coordinate the operation of a plurality of robot substations (e.g., a plurality of robots) and a segmented conveyor (or similar structure) configured to place items on the robot substations.

[0246] At 1055, it is determined whether a anticipated collision event has been detected. In some embodiments, the system 1000 determines whether a anticipated collision event has been detected based at least in part on sensor data relating to the work area (e.g., robots, chutes, etc.) and / or the plan or strategy for sorting items. In response to an attempt to implement the plan or strategy by independently operating a plurality of robots, the system 1000 determines whether a anticipated collision event has been detected.

[0247] According to various embodiments, the universal scheduler determines whether a anticipated collision event has been detected. For example, the universal scheduler responds to information provided by the robot single-station scheduler to the universal scheduler, which is associated with the respective plans or strategies for distributing items by the corresponding robot single station, to determine whether a anticipated collision event has been detected. The robot single-station scheduler may register the determined plan or strategy with the universal scheduler.

[0248] In response to the detection of an anticipated collision event at 1055, procedure 1050 returns to 1051a, 1051b, and 1051c, and one or more of the robot single-station schedulers determine a new plan or strategy for one of the single-item sorting tasks. In response to the detection of an anticipated collision event, the general scheduler sends a fault to one or more of the robot single-station schedulers to cause the robot single-station schedulers to implement a new plan or strategy for one of the single-item sorting tasks.

[0249] In contrast, in response to the determination that no expected collision event was detected at 1055, procedure 1050 continues to 1057, at which point the current plan or strategy is implemented (e.g., operating the corresponding robot to distribute items individually).

[0250] Figure 10C is a diagram illustrating a procedure for sorting one of a plurality of items in the same work area using a robotic sorting system comprising one of a plurality of robotic arms operating within a work area, according to various embodiments. Procedure 1075 of Figure 10C can be executed by system 1000 of Figure 10A. In some embodiments, procedure 1075 is implemented by a robotic system operating to sort one or more items within one or more work areas (such as the work areas corresponding to chutes 1007, 1011, 1015, and / or 1019 of system 1000 of Figure 10A). System 1000 includes one or more processors operating to cause robotic arms 1003, 1005, 1009, 1013, and / or 1017 to pick up and place items for sorting.

[0251] At 1077, multiple plans or strategies for individually distributing multiple items using multiple robotic arms are determined. According to various embodiments, the plans or strategies for individually distributing items are determined independently of each other. For example, each of 1051a, 1051b, and 1051c in Figure 10B can be implemented at least partially by different schedulers (such as the robot single-station schedulers 924, 926, 928, and 930 of the hierarchical scheduling system 900 in Figure 9). The plans or strategies determined at 1051a, 1051b, and 1051c are associated with the operation of different robots within a robotic system.

[0252] In 1079, an attempt was made to implement multiple plans or strategies for distributing multiple items individually. Multiple robots were controlled by a corresponding robot single-station scheduler to implement the plans or strategies.

[0253] At 1081, a collision criterion is determined. In some embodiments, system 1000 determines whether one or more collision criteria are detected based at least in part on sensor data relating to the work area (e.g., robots, chutes, etc.) and / or a plan or strategy for sorting items. According to various embodiments, a collision criterion corresponds to a criterion that, if satisfied, indicates that a collision between two objects (e.g., two robots, two items, a robot and an item, a robot and a portion of the work area (such as a chute), etc.) is expected (or considered to have occurred). Collision criteria may include an intersection between two objects along their expected paths, an intersection of the positions of two objects during sorting, or two objects that will be within a certain proximity to each other during sorting (e.g., if sorting is performed according to a corresponding plan or strategy). Various other criteria may be implemented for determining whether a collision event is expected.

[0254] In response to an attempt to implement a plan or strategy using the independent operation of multiple robots, system 1000 determines whether a collision criterion has been detected (or is met). According to various embodiments, the general scheduler determines whether a collision criterion has been detected. For example, the general scheduler determines whether a collision criterion has been detected in response to information provided by the robot single-station scheduler to the general scheduler, which is associated with the respective plans or strategies for distributing items by the corresponding robots. The robot single-station scheduler may register the determined plan or strategy with the general scheduler.

[0255] In response to the collision criterion detected at 1081, procedure 1075 continues to 1083, where assistance is obtained or a plan or strategy is updated to avoid collisions. For example, in response to the collision criterion detection, the general scheduler may cause an active measure to be implemented to change the sorting of items or adapt the sorting of items to the probability of a collision occurring (e.g., detecting an expected collision event). In some embodiments, the general scheduler causes one or more robot sorting station schedulers to determine a new plan or strategy for sorting items or for avoiding collisions. For example, in response to the collision criterion detection, the general scheduler sends a fault to one or more robot sorting station schedulers to cause the robot sorting station schedulers to implement a new plan or strategy for sorting items (e.g., in a way that sorting will avoid expected collisions).

[0256] In response to the determination that no collision was detected at 1081, the procedure continued from 1075 to 1085. At 1085, the autonomous operation of the multiple robots distributing multiple items was restored.

[0257] Although the foregoing embodiments have been described in considerable detail for the purposes of clarity, the invention is not limited to the details provided. Many alternative ways of carrying out the invention exist. The disclosed embodiments are illustrative and not restrictive.

[0258] 100: Program 102: Inductive Procedure 104: Single-part procedure 106: Classification / Route Program 108: Transportation Procedures 200: System / Robot System 202: Robotic Arm 204: Suction-type end effector / End effector 206: Chute or storage tank / station 208: Segmented Conveyor / Conveyor 210: Entry point 212: Controlling the computer 214: 3D Camera / Camera 216: 3D Camera / Camera 218: On-demand remote operation device / remote operation device 220: Human worker / operator / human operator 224: Camera 226: Camera 230: Robotic Arm 232: Robotic Arm 234: Robotic Arm 236: Sluice / Station 238: Sluice / Station 240: Sluice / Station 300: Program 310: Steps 320: Steps / Procedures 322: Steps 324: Steps 326: Steps 328: Steps 330: Steps / Procedures 332: Steps 334: Steps 336: Steps 338: Steps 340: Steps 400: Program 410: Steps 420: Steps 430: Steps 440: Steps 450: Steps 500: Program 510: Steps 530: Steps 550: Steps 550a: Program 550b: Program 550c: Program 550d: Program 551a: Steps 551b: Steps 551c: Steps 551d: Steps 552a: Steps 552b: Steps 552c: Steps 552d: Steps 553a: Steps 553b: Steps 553c: Steps 553d: Steps 554a: Steps 554b: Steps 554c: Steps 554d: Steps 570: Steps 580: System 581a: 3D Camera 581b: 3D Camera 581c: 3D Camera 581d: 3D Camera 585a: Work Area 585b: Work Area 585c: Workspace 585d: Workspace 590a: Robotic Arm 590b: Robotic Arm 590c: Robotic Arm 590d: Robotic Arm 593: Conveying Structure / Segmented Conveyor 594: Arm 595a: Blower 595b: Blower 596: Items 597: Human Operator 599: Controlling the computer 600: System 601a: 3D Camera 601b: 3D Camera 601c: 3D Camera 601d: 3D Camera 603a: Robotic Arm 603b: Robotic Arm 605a: Work Area 605b: Work Area 607: Conveying Structure / Segmented Conveyor 609a: Sensor Array 609b: Sensor Array 611a: Sensor 611b: Sensor 611c: Sensor 611d: Sensor 611e: Sensor 611f: Sensor 613: Items 650: Program 651: Steps 653: Steps 655: Steps 657: Steps 700: System / Robot System 701:3D camera 702:3D Camera 703: Robotic Arm 704: End effector 705a: Suction-type end effector / End effector 705b: Suction-type end effector / End effector 706a: Items 706b: Items 710: Work Area 711: Slide 713: Conveying Structure 715: Controlling the computer 717: On-demand remote operation device 719: Human Operator 750: Program 751: Steps 752: Steps 753: Steps 754: Steps 800: System / Robot System 801:3D camera 802:3D camera 803: Robotic Arm 804: End effector 805a: Suction-type end effector 805b: Suction-type end effector 806: Items 810: Work Area 811: Slide 813: Conveying Structure / Segmented Conveyor 815: Surface 817: Controlling the computer 819: On-demand remote operation device 820: Human Operator 850: Program 851: Steps 853: Steps 855: Steps 857: Steps 859: Steps 900: Hierarchical Scheduling System 922: Global Scheduler 924: Robot Single-Station Scheduler / Station-Specific Scheduler 926: Robot Single-Station Scheduler / Station-Specific Scheduler 928: Robot Single-Station Scheduler / Station-Specific Scheduler 930: Robot Single-Station Scheduler / Station-Specific Scheduler 932: Station Sensor 934: Station Sensor 936: Station Sensor 938: Station Sensor 940: Sensor 942: Conveyor Controller 1000: System 1001a: Camera 1001b: Camera 1001c: Camera 1001d: Camera 1003: Robotic Arm 1005: Robotic Arm 1007: Slide 1009: Robotic Arm 1011: Slide 1013: Robotic Arm 1015: Slide 1017: Robotic Arm 1019: Slide 1021: Conveyor 1023: Controlling the computer 1050: Program 1051a: Steps 1051b: Steps 1051c: Steps 1053: Steps 1055: Steps 1057: Steps 1075: Program 1077: Steps 1079: Steps 1081: Steps 1083: Steps 1085: Steps

Claims

1. A robotic system comprising: A communication interface; A first robotic arm includes a suction-type end effector; and a processor coupled to and configured to: receive sensor data via the communication interface, the sensor data including image data associated with a plurality of items present in a work area; and use the sensor data to determine and implement a plan for autonomously operating the first robotic arm to pick up one or more items from the work area and place each item in a corresponding location, wherein determining the plan includes: selecting a destination location for placing a first item; and implementing the plan includes: determining to implement an active measure based at least in part on an active measure associated with a detected state or condition of the one or more items in the work area, including: detecting, at least in part on the sensor data, that a location adjacent to the destination location contains a second item having a characteristic satisfying one of a predefined criteria; and in response to detecting that the adjacent location contains the second item having the characteristic satisfying the predefined criteria, determining to execute the active measure, wherein executing the active measure includes: Select a new destination location for the first item; and update the plan to operate the first robot to place the first item at the new destination location.

2. The robot system of claim 1, wherein updating the plan to operate the first robot to place the first item at the new destination location comprises: updating a distributed data structure to associate the new destination location with the first item, wherein the distributed data structure is used to manage the retention of items placed in a placement area by a plurality of robot arms.

3. The robot system of claim 1, wherein the characteristic of the second item is the size of the item, and the predefined criterion is that the size of the item is greater than a threshold size.

4. The robot system of request item 1, wherein the destination location is a single-transport structure.

5. The robot system of claim 4, wherein the processor is further configured to: detect the detected state or condition, including by using sensor data to detect that an observed flow of items through the work area has deviated from a predicted flow based on a model of the flow of items through the work area; determine that the detected state or condition hinders the implementation of a current plan to autonomously operate the first robotic arm to pick up one or more items from the work area and place each item individually in a corresponding position in the single-transfer structure; and in response to the determination, operate to implement the proactive measures to improve the flow of the items through the work area.

6. The robot system of claim 4, wherein the sensor data includes data associated with an infrared beam interruption detector and the detected state or condition is detected at least in part based on a determination by which one of the infrared beams of the infrared beam interruption detector has not been interrupted at an expected time.

7. The robot system of claim 4, wherein the first robotic arm is used to push or pull at least one item in the work area.

8. The robot system of claim 4, wherein the first robot arm includes a robot-controlled blower, and the active measure includes using the blower to clear a blockage by blowing air onto the one or more items in the work area.

9. The robotic system of claim 4, wherein the active measure includes using the first robotic arm to clear one of the obstructions of the flow of items through the work area.

10. The robotic system of claim 4, wherein the active action comprises: operating the first robotic arm to remove debris obstructing the flow through the work area.

11. The robot system of claim 1, wherein the detected state or condition includes a determination that an item exceeds a predetermined limit weight and the active action includes a determination to pull, drag, or push the item through at least a portion of the work area to a corresponding destination location.

12. The robotic system of request item 9, wherein the item is pulled, dragged, or pushed to an abnormal handling area.

13. The robotic system of claim 1, wherein performing the active action comprises dragging an object along the top surface of one of the other objects present in the work area.

14. The robot system of claim 1, wherein the processor is configured to determine and implement the plan at least in part by: determining, at least in part, to change the position and orientation of at least one of the one or more items based on three-dimensional image data associated with the work area; and operating the first robot arm according to the plan.

15. The robot system as described in request item 14, wherein: The first robotic arm includes an end effector; the end effector includes one or more suction cups; and the active measure includes changing one or both of the position and orientation of the at least one of the one or more articles by blowing air out of at least one of the one or more suction cups.

16. The robot system as described in Request 1, wherein: The suction-type end effector includes one or more suction cups; and the processor is configured to perform a diagnostic test, including by: manipulating the first robotic arm to move the end effector to a predetermined surface; causing the first robotic arm to grasp the predetermined surface; measuring a pressure affected by the one or more suction cups when engaging the predetermined surface; and comparing the pressure affected by the one or more suction cups when engaging the predetermined surface with a preset threshold pressure value.

17. The robotic system of claim 1, wherein the determination that the first robotic arm has picked up more items than the expected number of items is further based on one or more of image sensor data, weight data, pressure data and tactile sensor data.

18. The robotic system of claim 1, wherein the suction-type end effector includes a plurality of independently actuated suction cups, each group including one or more suction cups, and the active action includes operating the respective independently actuated suction cup groups in an interleaved manner to release each object held in the grip to a corresponding destination location.

19. The robot system as described in Request 1, wherein: The system includes a second robotic arm, and the first and second robotic arms operate autonomously and independently of each other, each picking up and placing items from the work area and placing them individually in a corresponding single destination location in a manner that achieves a combined processing capacity while avoiding collisions or other interference between the first and second robotic arms.

20. The robotic system of claim 19, wherein the detected condition or state includes a potential collision between the first robotic arm and the second robotic arm and the active measure includes delaying the next operation of one of the first robotic arms until after one of the second robotic arms has moved.

21. The robotic system of claim 19, wherein the destination location is a single-unit conveyor structure, and the active action is based at least in part on a determination that the next operation of the second robotic arm is to grasp a second item further away from a first item in the single-unit conveyor structure than the next item to be grasped by the first robotic arm.

22. A method for controlling a robot to move objects in a work area, comprising: Sensor data, which includes image data associated with a plurality of objects present in a work area, is received by one or more processors. The method uses sensor data to determine and implement a plan to autonomously operate a first robotic arm to pick up one or more items from a work area and place each item in a corresponding location. Determining the plan includes: selecting a destination location for placing a first item; and implementing the plan includes: determining to implement an active measure based at least in part on an active measure associated with a detected state or condition of the one or more items in the work area, including: detecting, at least in part on the sensor data, that a location adjacent to the destination location contains a second item having a characteristic satisfying one of a predefined criteria; and in response to detecting that the adjacent location contains the second item having the characteristic satisfying the predefined criteria, determining to execute the active measure, wherein executing the active measure includes: selecting a new destination location for placing the first item; and updating the plan to operate the first robot to place the first item at the new destination location.

Images