Robotic system for identifying items

Abstract

A method and system for obtaining an identifier from an item is disclosed.SOLUTION: And autonomously operating the robotic structure to move the item along a predetermined path from the source location to the destination location, and autonomously operating the robotic structure to place the item at the destination location based at least in part on the plan. The item comprises one or more identifiers, and responsive to a determination that at least one of the one or more identifiers was not acquired by the one or more sensors, an active measure is performed to cause the one or more sensors to acquire the at least one identifier that was not acquired. The predetermined path corresponds to a path along which the item is moved from the movement source position to the movement destination position. The predetermined path is planned such that the item is moved within a threshold range of the one or more sensors while the item is moved along the predetermined path.SELECTED DRAWING: Figure 2C

Description

[Background technology]

[0001] Robots have been used in manufacturing and other fields to perform tasks, such as tasks in environments that may be hazardous or otherwise dangerous to humans, tasks that require the application of greater forces than humans can exert, and tasks that require a high degree of precision and consistency over extended periods of time.

[0002] An autonomous robot performs at least some tasks automatically, without the need for human control or direction. For example, autonomous robots have been utilized to perform repetitive and / or otherwise predetermined tasks and sequences of tasks, typically in controlled environments (such as factories). More recently, self-driving cars, delivery drones, and other autonomous vehicles are under development.

[0003] In the field of robotics, teleoperation refers to the remote control of a robot by an operator. For example, robots have been used to perform surgery, defuse bombs, and perform other tasks under the control of a skilled human operator.

[0004] Processes associated with kitting and singulation have traditionally been highly labor-intensive processes that have been difficult to employ robotics for because of movement constraints and the difficulty of providing and programming robots to perform tasks such as reaching into bins or shelves and picking up items of any size, fragility, or consistency, or to sort any mixed items. As a result, large-scale kitting and / or singulation operations continue to be human labor-intensive.

[0005] Another difficulty in utilizing robots for processes related to kitting and singulation is the design and selection of end effectors. The end effector of a robotic arm is a module that the robotic arm can engage with items in the source pile / stream. Different types of end effectors may be better optimized for specific sizes, package types, weights, shapes, etc. Additionally, the size of the end effector or the robotic arm (e.g., wrist) to which it is attached may hinder the robotic arm's ability to reach bins / shelves, source piles / streams, etc. [Brief explanation of the drawings]

[0006] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0007] [Figure 1] FIG. 1 illustrates a singulation system according to various embodiments.

[0008] [Figure 2A] 1 illustrates a kitting system according to various embodiments.

[0009] [Figure 2B] 1 illustrates a kitting system according to various embodiments.

[0010] [Figure 2C] 1 illustrates a kitting system according to various embodiments.

[0011] [Figure 2D] 1 illustrates a kitting system according to various embodiments.

[0012] [Figure 3A] 1 is a flowchart illustrating a method for picking and placing an item according to various embodiments.

[0013] [Figure 3B]1 is a flowchart illustrating a method for picking and placing an item according to various embodiments.

[0014] [Figure 3C] 1 is a flowchart illustrating a method for scanning an item according to various embodiments.

[0015] [Figure 3D] 1 is a flowchart illustrating a method for determining whether to take proactive measures in connection with scanning an item, according to various embodiments.

[0016] [Figure 3E] 1 is a flowchart illustrating a method for determining whether to take proactive measures in connection with scanning an item, according to various embodiments.

[0017] [Figure 4] 1 illustrates a kitting system according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0018] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0019] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0020] Kitting, as used herein, includes picking one or more items / objects from corresponding locations and placing one or more items in predetermined locations such that a set of one or more items corresponds to a kit.

[0021] As used herein, singulation includes picking one or more items / objects from a source pile or stream and placing the one or more items one-by-one into corresponding predetermined locations (e.g., on a partitioned conveyor (e.g., in a tray on a conveyor) or similar transportation means) to be sorted and routed for transportation to a downstream (e.g., final destination / physical) destination.

[0022] As used herein, an identifier includes a label, a bar code, a symbol, an image, an alphanumeric string, a code, etc. The identifier may be printed on a label attached to the item, provided on the side of the item, embedded in a radio frequency identification (RFID) tag attached to the item, etc. In some embodiments, the identifier comprises machine-readable information (such as text and / or optically or otherwise encoded information), which may be read by a machine and utilized in connection with kitting or singulating objects and / or items, for example, via an automated kitting system and / or process and / or an automated singulation system and / or process.

[0023] As used herein, a sensor includes a machine reader (such as a radio frequency (RF) tag reader, an optical code reader, etc.) that may obtain (e.g., read) machine-readable information corresponding to an item (e.g., an identifier on or otherwise embedded in the item, etc.).

[0024] "Kitting machines" or "kitting systems" and their incorporation into highly automated kitting operations are disclosed. In various embodiments, the kitting machines disclosed herein comprise at least a partially robotically controlled unit that delivers and places items to facilitate locating, picking, and / or placing items in and / or for packaging and / or shipping as part of a kitting operation. In various embodiments, the kitting machines disclosed herein comprise one or more kitting system modules, each module comprising modular components. The kitting system modules disclosed herein may comprise one or more shelves, bins, or other containers. In some embodiments, the shelves, bins, or other containers may be robotically positioned to place items for pickup. The kitting system modules disclosed herein may be integrated with one or more other kitting system modules, one or more robotic arms, and / or other components to form an at least partially automated kitting system capable of locating, selecting, and packing a predetermined quantity of each of one or more any individual items, such as items contained in an order, invoice, or similar data.

[0025] A kitting system configured to perform kitting is disclosed. In some embodiments, the kitting system includes a kitting shelving system for use in connection with kitting. The kitting shelving system may include one or more shelves housing one or more items for use in the kitting process to build one or more kits. The kits may be built at least in part based on corresponding orders (e.g., based on packing slips associated with the orders). Various embodiments include one or more robotic systems. The robotic systems may include one or more robotic arms, each configured to operate autonomously to pick items / objects from a first location (e.g., a source location) and place them at a second location (e.g., a destination location). The robotic arms included in the kitting system may be controlled to operate to pick and place items / objects (e.g., autonomously) according to a plan to build the kits.

[0026] Each item or object on the kitting shelf (e.g., in a container on the kitting shelf) may have machine-readable information (e.g., text and / or optically or otherwise encoded information), which may be machine-readable and utilized in connection with kitting the object and / or item, e.g., via an automated kitting system and / or process. As one example, to read the information of a given item (or object within an item), one or more sensors may acquire information about the item while the item is in the kitting shelf system (e.g., on a shelf of the kitting shelf system, such as on a presentation surface of the shelf). As another example, to read the information of a given item (or object within an item), one or more sensors may acquire information about the item while the item is being moved (e.g., by a robotic arm) from the kitting shelf system to a corresponding container (e.g., information about the item is scanned along the item's path / trajectory from the kitting shelf system to the container). Items or objects on the kitting shelf may be placed in containers on the kitting shelf, which may be trays, boxes (e.g., cardboard boxes), totes, etc. In some examples, the containers are placed in a large container disposed on a kitting shelf. In some embodiments, a workspace corresponding to the kitting shelf includes one or more sensors, and the one or more sensors may acquire information about items / objects in the workspace. For example, information related to the items / objects may be acquired while a robotic arm approaches the items / objects to pick them up. As one example, a sensor (e.g., a camera) disposed above a chute (inclined platform) may capture information related to the items / objects (e.g., barcodes or other identifiers) so that the system has information before the robotic arm grasps the items / objects. As another example, a camera attached to the robotic arm may capture information related to the items / objects (e.g., barcodes or other identifiers) as the robotic arm moves to grasp / pick up the items / objects.As another example, as the robotic arm moves to grasp / pick up an item / object, a sensor attached to the robotic arm may capture information related to the item / object (e.g., information from an RFID tag on the item).

[0027] A robotic system for performing singulation is disclosed. In various embodiments, the singulation is performed based on data related to a workspace or items within the workspace. A plan (e.g., a plan for singulating items) is determined based at least in part on attributes of items within the workspace. The attributes of the items may be determined at least in part on sensor data acquired about the workspace. As used herein, a workspace (e.g., a workspace related to at least a singulation system and / or singulation process) may include a chute or other transport structure and / or receptacle in which a source pile / stream of items is located, a destination transport structure into which the items from the chute are placed one by one, and a robotic structure including a robotic arm that picks one or more items from the chute (or other source) and places the one or more items, one by one, in corresponding positions on the destination transport structure. The workspace may further include a control computer that acquires sensor data related to the workspace and / or an on-demand remote control device that a human operator can use to control elements within the workspace (e.g., the robotic arm and / or transport structure). As used herein, the terms slot or tray may be used interchangeably in reference to describing a particular location on a conveyor.

[0028] A robotic system comprises a robotic arm and end effectors used to pick items from a source pile / stream (e.g., a source location) and place the items on a partitioned conveyor or similar transport vehicle to be sorted and routed for transportation to a downstream (e.g., final destination / physical) destination (e.g., a destination location). As used herein, singulation of items includes picking items from a source pile / stream and placing the items one-by-one in or on a destination (e.g., a location on a destination transport structure (e.g., a partitioned conveyor or similar transport vehicle)). In some embodiments, multiple robots are coordinated to achieve a desired overall throughput. In various embodiments, one or more robots may be used at a singulation station. A robotic system may comprise multiple stations. As an example, each station may correspond to a different workspace (e.g., a different chute with a source pile / stream). Human workers may be deployed at one or more stations. In various embodiments, the robotic system may be configured to request (require) assistance from a human worker, e.g., by remotely operating a robotic arm, manually completing a task, etc., to handle items that the robot cannot handle through fully automated processing and / or items dropped by the robot. In some embodiments, multiple robotic arms operating in the same workspace work independently to singulate multiple items. In connection with singulating an item, a plan or strategy may be determined for singulating the item from the source pile / stream in which the item is located to a corresponding location on a conveyor. The corresponding location on the conveyor may be a particular slot or tray on the conveyor. In some embodiments, the slot or tray on the conveyor is associated with an identifier (e.g., a unique identifier for the conveyor within the robotic system).

[0029] In some embodiments, multiple robots (e.g., robotic arms) may operate to singulate items in a particular chute. For square or rectangular items, identifiers may be placed on any one or more of the item's six sides. The multiple robots may coordinate their respective positions and movements to ensure that no robot arm not grasping a particular item is between the item and at least one of the one or more scanners, or to ensure that a robot arm does not otherwise block / obstruct the view or ability of at least one sensor to acquire information about the item. In some embodiments, multiple robots work together to both grasp and move items to the conveyor, and to move items within the range of sensors (e.g., a predetermined range of the sensor). For example, multiple robots may be controlled to pick up items that are relatively flat or relatively large (e.g., large enough that movement of the item by a single robot may not meet a threshold for stability in grasping the item).

[0030] In some embodiments, a plan / strategy related to singulating an item may include presenting one or more sides of the item to one or more scanners. For example, a robotic arm may be controlled to move the item such that multiple sides of the item are presented to the scanners. Presenting multiple sides of the item to the scanners may include rotating or reorienting the item as it is moved (e.g., from a chute to a conveyor where the item is singulated).

[0031] According to various embodiments, the plan or strategy includes instructions for items to be singulated (e.g., from a source pile / stream), locations on a conveyor where the items will be placed one by one, and a path or trajectory along which the items will be moved from the source pile / stream to their locations on the conveyor. The plan or strategy includes information regarding the locations on the conveyor where the items will be placed one by one (e.g., identifiers of slots or trays on the conveyor where the items will be placed). In some embodiments, the plan or strategy includes instructions that a robotic structure that singulates the items utilizes to singulate the items. As an example, the instructions provide instructions that the robotic structure follows to control corresponding robotic arms to pick items from chutes, move the items along paths or trajectories, and place the items at determined locations on the conveyor.

[0032] A path or trajectory along which an item is singulated is determined based at least in part on sensor data, according to some embodiments. The robotic system may acquire multiple sensors that output information about the workspace, including items or objects within the workspace. The sensor data is acquired based on information output from one or more sensors and used in connection with determining the path or trajectory. In some embodiments, the path or trajectory is determined based at least in part on one or more attributes of the item to be singulated. Examples of item attributes include weight, size (e.g., one or more dimensions), type of packaging, an identifier on the item, the location of an identifier or label on the item, the location of the item relative to the chute and / or conveyor, information obtained from the identifier or label on the item, etc. Various other attributes may also be used in connection with determining the path or trajectory. Determining the path or trajectory of the item may further be based at least in part on the location on the conveyor where the item is to be placed, attributes of items already on the conveyor, attributes of items in the workspace (e.g., items in the source pile / stream), the grip force with which the robotic arm grasped the item, the speed at which the robotic arm moves the item, etc. In some embodiments, the path or trajectory of an item is based at least in part on the location of one or more sensors within the workspace. In some embodiments, the path or trajectory is determined based at least in part on the likelihood that one or more sensors will acquire information associated with the item (e.g., an identifier on the item). For example, historical information may be used to determine one or more locations or ranges in the workspace where sensors have been successful in acquiring information from the item. The robotic system may utilize the historical information in connection with modeling the workspace to determine locations or ranges associated with relatively high success for sensors in acquiring sensor-related information.For example, the historical information may suggest that a location with coordinates x1, y1, z1 is associated with a 99% effective scan success rate (e.g., of a scan obtaining information from an item at such location), and that a location with coordinates x2, y2, z2 is associated with a 98% effective scan success rate. In some embodiments, the system may use the historical information to determine a path or trajectory to optimize the likelihood of obtaining information from the item. In some embodiments, the system determines a path or trajectory based at least in part on a threshold related to the likelihood of obtaining information from the item (e.g., such that the selected path or trajectory meets a predetermined minimum likelihood of obtaining information from the item). A value related to the likelihood of obtaining information from an item may be included in a cost function associated with determining a plan for moving the item (e.g., determining a source location, a destination location, a path or trajectory, etc.). The system may store a mapping of areas / locations in the workspace to values ​​related to the likelihood of obtaining information associated with an item in that area / location. For example, the mapping may include a mapping of coordinates to a probability of successful scanning. The mapping may store information on a per-sensor basis, such that the probability of obtaining information from an item is determined for one or more particular sensors, and / or the mapping may store information on a workspace basis, such that the likelihood of obtaining information from an item for a set of sensors is aggregated or calculated as a collective value. The mapping or modeling of the likelihood of obtaining information related to an item may be updated over time (e.g., at predetermined intervals, or in response to an update request, or continuously (e.g., with each movement of an item)).

[0033] Various embodiments include a robotic system including a robotic arm configured to move an item along a predetermined path from a source location to a destination location. The predetermined path may be determined based on sensor data from one or more sensors in a workspace (e.g., sensor data related to items on a shelving system in a kitting system, sensor data related to a source pile / stream and / or conveyor in a singulation system, etc.). The predetermined path may be configured (e.g., determined) such that when an item is moved along the path, the item is within a threshold range of one or more sensors while the item is being moved along the predetermined path. The threshold range of one or more sensors may include a line of sight or area (e.g., in the case of radio frequency identification (RFID) sensors, a range or proximity within which information can be obtained from an RFID tag, etc.) within which the one or more sensors can obtain information (e.g., information provided on or otherwise associated with the item). As the item is moved through the threshold range, the one or more sensors may obtain a formation associated with the item. For example, the one or more sensors may obtain one or more identifiers on the item (e.g., capture an image of the identifier).

[0034] According to various embodiments, the robotic system determines whether one or more sensors have acquired sufficient information from one or more identifiers as the item is moved along a predetermined path. For example, in response to the item being moved through a threshold range of one or more sensors, the robotic system determines whether one or more sensors have acquired information corresponding to one or more identifiers. The robotic system may determine whether information about at least one of the one or more identifiers was not captured (e.g., it may determine that one or more sensors failed to acquire information from at least one identifier). In some embodiments, information indicating multiple identifiers on an item is stored in advance (e.g., before the item is kitted or singulated). For example, the system may store a mapping of items or item types to multiple identifiers. The number of identifiers mapped to an item or item type may correspond to an expected number of identifiers. If the robotic system determines that the number of identifiers acquired (or information acquired) by one or more sensors is less than the expected number of identifiers for the item, the robotic system may determine that the information acquired by the one or more sensors is insufficient (e.g., information about at least one of the one or more identifiers was not captured). In some embodiments, the system determines that more than enough information has been acquired by one or more sensors. For example, one or more sensors may acquire information from multiple items (e.g., when attempting to acquire information for only a portion of the multiple items). Examples of when more than enough information may be captured include when a robotic system is moving an item within range of a sensor and another item intrudes into the frame or otherwise occupies part of the frame and is captured by one or more sensors. An item may intrude into the frame of a sensor based on the flow or instability of items in the workspace, by a robotic arm when grasping an item, or the like.In some embodiments, in response to determining that additional information has been acquired (e.g., information in excess of that expected for the item (e.g., information about another item in the frame)), the robotic system may determine that some of the captured information corresponds to the item being moved (e.g., is information intended to be acquired). In some embodiments, in response to determining that additional information has been acquired, the robotic system controls a robotic arm or another element in the workspace (e.g., another robotic arm, a compressed air blower, an air knife, etc.) to remove another item from the frame or line of sight of the sensor.

[0035] According to various embodiments, a robotic system associates position coordinates with information about an item (e.g., a label (e.g., a barcode on an item)) or with the item's position coordinates based on the sensor's detection of the information or item. For example, a robotic arm (or robotic system) may be calibrated with one or more sensors in the workspace. A mapping of the sensors to the workspace (e.g., a mapping of the positions of various sensors to the workspace) may be used to determine a plan for moving the item, such as determining a path or trajectory along which the item will be moved. The mapping of the sensors to the workspace may also be used to determine a portion of the information acquired by the sensors in the workspace that corresponds to a particular item. For example, if sensors in the workspace capture information about two or more items, the robotic system may use the time the information was captured, the location of the items when the information was captured, and the path or trajectory along which the items were moved to determine the portion of the captured information related to the items. If two or more objects are within the sensor frame and information for two items is acquired, the robotic system may determine the portion of such information that corresponds to a particular item based on when the information was captured and where the item was located when the information was captured (e.g., based on the path or trajectory the item was moved).

[0036] According to various embodiments, the mapping of sensors to the workspace may be predefined. For example, a robotic system may be calibrated to determine various predetermined ranges or frames of one or more sensors (e.g., sensors within the workspace). Calibrating the robotic system may include moving a robotic arm through a series of fixed points while grasping a predetermined item. The robotic system may analyze information acquired by the one or more sensors while the robotic arm is moved through the series of fixed points. The analysis of the information acquired by the one or more sensors may be used to determine the range (e.g., 3D area) within which one or more scanners acquire information about the items (e.g., to calibrate the locations at which the sensors can acquire / scan information from the items and / or the orientation of the items at which the sensors can scan / acquire information from the items at particular locations). Calibrating the robotic system may further include simulating the movement of one or more items through the workspace to ensure that expected information is acquired by one or more sensors within the workspace, determining expected information acquired for the items during the simulation, and then executing the movement of the one or more items.

[0037] According to various embodiments, one or more sensors are dynamically controlled to acquire information based at least in part on the location of the item (or the expected location of the item). The robotic system may use the size of the item and the plan or trajectory along which the item will be moved in connection with determining when to activate the sensors to acquire information. For example, the robotic system may determine when the item is expected to be within a predetermined range of the sensor (or within a frame of the sensor). When the item is within the predetermined range of the sensor (or is expected to be within the predetermined range), the robotic system may activate the sensor to acquire information about the item (e.g., scan a barcode on the item). The robotic system may deactivate the sensor or otherwise stop capturing information after it is determined that the item is outside the predetermined range of the sensor (or when the item is expected to be outside the predetermined range). Dynamic control of sensors based on the location of the item (or the expected location of the item) may reduce memory and computational load requirements by reducing the amount of images or information captured by the sensors.

[0038] Various embodiments include performing an active countermeasure in response to determining that one or more sensors did not acquire information about at least one identifier. For example, in response to a robotic system determining that the number of identifiers acquired by one or more sensors (or identifiers for which information was acquired) is less than the expected number of identifiers for an item, the robotic system may perform an active countermeasure. The active countermeasure may include performing one or more additional attempts to scan the at least one identifier. For example, the active countermeasure may include moving the item through a threshold range (e.g., within the line of sight of at least one sensor). The active countermeasure may include changing the orientation of the item (e.g., one or more lateral orientations relative to one or more sensors). The robotic system may change the orientation before performing one or more additional attempts to have information corresponding to the at least one identifier acquired by one or more sensors. The orientation of the item may be changed while the robotic arm is moving. For example, the robotic arm may be controlled to rotate a wrist of the robotic arm. In some embodiments, the orientation of the item is changed while the robotic arm is moving through at least a portion of a threshold range of one or more sensors (e.g., while the item is within the line of sight of at least one sensor). Proactive measures may include repeatedly attempting to acquire at least one identifier (e.g., an identifier that was not acquired) until (i) all of the item's identifiers have been acquired (or a threshold number of identifiers have been acquired) and / or (ii) a threshold number of retries to acquire at least one identifier have been performed. As an example, retries to acquire at least one identifier may be performed repeatedly until either (i) or (ii) is satisfied.

[0039] In various embodiments, the integrated kitting and / or singulation systems disclosed herein operate automatically unless / until the system becomes stuck and no strategies are available for continuing automatic operation. In some embodiments, in response to such a condition, the system requests human intervention, e.g., via manual assistance, remote operation, etc.

[0040] FIG. 1 is a diagram illustrating a singulation system according to various embodiments.

[0041] In the depicted example, system 100 includes a robotic arm 102 with a suction-based end effector 104. In the depicted example, end effector 104 is a suction-based end effector; however, in various embodiments, one or more other types of end effectors, including, but not limited to, pinch-based end effectors or other types of actuated grippers, may be used in the singulation systems disclosed herein. In some embodiments, end effector 104 includes one or more suction-based ends (e.g., one or more suction cups). In various embodiments, the end effector may be actuated by one or more of suction, pneumatic, air, hydraulic, or other actuation. Robotic arm 102 and end effector 104 are configured to pick up parcels or other items arriving via chute (or bin) 106 and place each item in a corresponding location (e.g., a destination location) on partitioned conveyor 108. In this example, items are fed into chute 106 from intake end 110. For example, one or more human and / or robotic workers may supply items to the intake end 110 of the chute 106 either directly or via a conveyor or other electromechanical structure configured to supply items to the chute 106.

[0042] In the illustrated example, one or more of the robotic arm 102, end effector 104, and conveyor 108 are coordinately operated by a control computer 112. In some implementations, the control computer 112 is configured to control multiple robotic arms operating at one or more work stations. In various embodiments, the robotic singulation systems disclosed herein may include one or more sensors used to model the environment of the workspace. In the example shown in FIG. 1 , the system 100 includes image sensors (such as 3D cameras 114 and 116 in this example). In various embodiments, other types of sensors may be used (individually or in combination) in the singulation systems disclosed herein, such as cameras, infrared sensor arrays, laser arrays, scales, gyroscopes, current sensors, voltage sensors, power sensors, force sensors, pressure sensors, weight sensors, etc. In various embodiments, control computer 112 includes a workspace environmental condition system (e.g., a vision system) that is used to determine individual items, debris on the workspace, and the orientation of each item based on sensor data (e.g., image data) provided by image sensors (e.g., 3D cameras 114 and 116 in this example). The workspace environmental condition system in some embodiments includes a sensor on the robotic arm to detect the weight of an item (e.g., a grasped item) or information for determining an estimated weight. For example, information regarding the amount of current, voltage, and / or power utilized by one or more motors driving the movement of the robotic arm may be used to determine the weight (or estimated weight) of the item. As another example, the chute includes a weight sensor, and the weight of the item is determined based on the difference between the weight on the chute measured by the weight sensor before and after the item is picked up.As another example, information about the output from one or more sensor arrays may be used to determine the location of an item within the workspace, the location of an item while the item is being grasped and / or moved by a robotic arm, and / or the location of the robotic arm (e.g., based on determining the output from some of the sensors of one or more sensor arrays compared to other portions of the sensors of one or more sensor arrays). As another example, information about the output from one or more sensor arrays may be used to determine the dimensions or size of an item to be singulated and / or another item or object within the workspace. The information about the output from one of the sensor arrays may include information indicative of one or more aspects of the item with an identifier (e.g., a label, etc.).

[0043] The workspace environmental state system generates outputs used by the robotic system to determine and execute plans for autonomously operating the robotic structure to pick one or more items from the workspace and place each item in a corresponding defined location (such as a partitioned portion of the partitioned conveyor 108) available for machine identification and sorting. In some embodiments, the workspace environmental state system generates outputs (e.g., sensor data or information otherwise characterizing the workspace and / or items within the workspace) used by the robotic system to detect states, conditions, and / or attributes associated with one or more items within the workspace and / or states or conditions associated with the robotic arm or other elements of the workspace. According to various embodiments, in response to detecting (e.g., determining) states, conditions, and / or attributes associated with one or more items within the workspace, the robotic system executes one or more proactive measures in connection with singulating the items. As an example, a proactive measure may include updating a plan for autonomously operating the robotic structure to pick one or more items from the workspace and place each item one-by-one at a corresponding location on the singulation transport structure. As an example, a proactive measure may include updating the plan to include an updated path or trajectory of the items so that the items are moved within a threshold range of one or more sensors in the workspace (e.g., within line of sight of one or more sensors to enable the one or more sensors to acquire information from an identifier on the item). In some embodiments, proactive measures, i.e., updating the plan, may include operating the robotic structure to change or adapt to detected states, conditions, and / or attributes (e.g., changing how the items are singulated, changing the path or trajectory along which the items are singulated, changing how the items are grasped, changing the grasped position on the items when the items are grasped, etc.).

[0044] In various embodiments, the robotic systems disclosed herein comprise and / or perform, for example, by operation of a control computer (such as control computer 112), one or more of the following: Computer vision information is generated by merging data from multiple sensors, including one or more of 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors, to generate a three-dimensional view of the workspace including one or more sorting stations. The robotic system determines characteristics of items and / or debris or other anomalies in the three-dimensional view of the workspace. The robotic system coordinates the motion of multiple robots to avoid collisions, obstructions, and competition with another robot to pick up the same item and / or place the item in the same destination location (e.g., a partitioned section of a conveyor). The robotic system coordinates the motion of multiple robots operating within the same workspace to singulate multiple items. For example, in various embodiments, multiple robots operate independently to pick and place items. If a risk of collision is detected, responsive action is taken to prevent the multiple robots from colliding with each other during singulation. The robotics system coordinates the actions of multiple robots to ensure that every item is placed once per slot / position. For example, if robot A drops an item, the system tasks robot B with picking it up; an item that is placed but improperly oriented can be picked up and adjusted or moved to another position by the same or another robot; if more than one item falls into a single destination slot, a robot at a downstream station can pick one of the two or more items from the conveyor and place it in the new position. The robotic system continually updates motion plans for each robot and for all robots to achieve a desired overall throughput (e.g., to maximize overall throughput, to reach a predetermined threshold of overall throughput, etc.). In response to determining that two or more robots have collided or will collide when moving according to their respective plans for singulation of an item, the robotic system implements proactive measures to ensure that the two or more robots avoid the collision or otherwise reset the independent operation of the two or more robots. Upon determining that two robots are tasked with independently retrieving the same item, the system randomly selects one robot to take the item, while the other robot transitions to the next item (e.g., identifies, selects, decides on a grasping strategy, picks, moves according to a plan, and places). The robotic system can manage the independent operation of multiple robots to ensure that the robots select items at different times to avoid the same item being selected for singulation by two different robots. Conveyor movement and / or speed controlled as needed to eliminate vacant positions and achieve desired robot productivity (throughput). In response to an item being determined to be misplaced or dropped, the system assigns the item to a robot, or a human worker if necessary, to pick it up and either return it to the pile from which the retrieval robot itself came, or, if available or more optimal, place it in the next available slot on the conveyor. An upstream robot that is controlled to intentionally leave some slots empty for the downstream robot to place items on the conveyor. A downstream robot that is controlled to correct errors in the placement of an item upstream onto the conveyor (e.g., to correct the placement of an item in more than one slot / tray, to update a data structure with the relationship between the item's identifier and the slot into which the upstream robot placed the item, etc.). A downstream sensor may be controlled to obtain information about one or more identifiers on items being conveyed on the conveyor (or information may be obtained from such a sensor). Failures or errors that cannot be corrected by the same or another robot result in an alert being issued to obtain human (or other robot) intervention to resolve. Move / remove debris within the workspace or reconfigure items to be singulated (e.g., to improve the likelihood that an item will be successfully picked from the source pile / stream and placed onto the transport structure). Controlling a chute conveyor to reconfigure items in the workspace (e.g., to move an item selected for singulation closer to the front of the chute for faster and easier access by the robotic arm, to reposition one or more items to improve the ability of the robotic arm to grasp the item, etc.). Using sensor data from the workspace environmental state system (e.g., from one or more sensors in the workspace) to model flow in the chute (or model the workspace environment), detect deviations from predicted chute flow (or from the predicted workspace environment), and use the sensor data to detect blockages or anomalies in the chute flow or workspace environment and take proactive measures to clear the blockage. Using sensor data from the workspace environmental state system, detect one or more characteristics (e.g., attributes) of an item selected for singulation, determine that grasping or release of the item is expected to improve upon implementation of proactive measures, and implement proactive measures to improve grasping or release of the item. Using the sensor data, determine that the robotic arm has grasped a plurality of items in association with a singulation within the items, and determine a plan for releasing the plurality of items to place each item one-by-one at a corresponding location on the singulation transport structure (selecting different locations on the singulation transport structure at which the corresponding items are placed, and / or determining a strategy for operating the end effector to release a first subset of the plurality of items at a different time than a second subset of the plurality of items). Selecting a slot on the transport structure in which to place the selected item based on the size of the selected item and / or one or more characteristics of the item in the slot on the transport structure, e.g., the slot is selected to ensure that the selected item is not placed in a slot adjacent to a slot containing a tall or large item. Selecting a path for singulating an item onto a transport structure based on attributes of the item (e.g., size of the selected item, weight of the item, etc.) and / or one or more attributes (e.g., characteristics) of the item in a slot on the transport structure. For example, a path may be determined to place the item above one placed in a slot adjacent to a slot containing a tall or large item. As another example, a path for singulating an item may be determined based on the location of an identifier on the item (e.g., the location of the identifier relative to a sensor or scanner in the workspace). · Determining the movement and speed of the robotic arm that singulates the items based at least in part on the speed of the conveyor belt. Determining the trajectory of the item to be singulated based at least in part on one or more of the following: characteristics of the item (e.g., the location of an identifier on the item), characteristics of the workspace environment, the location of a sensor or scanner, and / or characteristics of the transport structure (e.g., the speed of a conveyor belt). Determine the singulation success probabilities corresponding to one or more paths / trajectories of the item to be singulated, and select the path / trajectory along which the item will be singulated based on the corresponding success probabilities. Determine positioning of the robotic arm and / or the robotic arm's end effector for a successful grasp (e.g., as determined based on the probability of successful grasp, the item's packaging type, the item's dimensions, the expected grasp force relative to a threshold, etc.). Positioning the end effector may include controlling movement of the robotic arm or the robotic arm's wrist so that the end effector is perpendicular to the item's surface. Update the robotic system's ability to detect empty slots or trays. For example, the definition of an empty slot / tray used by the robotic system to identify an empty slot / tray is updated over time.

[0045] In various embodiments, any mixed items to be singulated may include luggage, packages, and / or letters of various shapes and sizes. Some items may be standard packages, one or more attributes of which may be known, while others may be unknown. Sensor data (e.g., image data) is used to identify individual items (e.g., by image segmentation) in various embodiments. The boundaries of partially occluded items may be evaluated, for example, by recognizing the item as a standard or known type and / or by extending the visible item boundaries to a logical evaluation range (e.g., extrapolating so that two edges meet at an occluded corner). In some embodiments, the degree of overlap (i.e., occlusion by other items) is evaluated for each item, and the degree of overlap is taken into account when selecting the next item to attempt grasping. For example, a score may be calculated for each item to assess the probability of successful grasping, and in some embodiments, the score depends at least in part on the degree of overlap / occlusion by other items. An item that is less occluded may, for example, be more likely to be selected, other considerations being equal.

[0046] When a source pile / stream has any mixed items to be singulated, the source pile / stream typically includes items with different types of packaging, such as cardboard box packaging, paper wrapping packaging, and polybag packaging (e.g., polyethylene bags). The robotic system can determine the packaging of an item based on visual data obtained from sensors or based on the pressure generated between the end effector and the item when the robotic arm attempts to pick up the item. The sensor data can be utilized to determine the type of packaging corresponding to a particular item in the source pile / stream. In some embodiments, the robotic system determines a strategy for grasping an item based at least in part on the type of packaging corresponding to the item. For example, relatively heavy items packaged in polybags typically experience "tenting" between the end effector suction cups. Tenting can cause suboptimal adhesion from the end effector of the robotic arm, resulting in suboptimal grasping of such items. According to various embodiments, in response to determining that an item is relatively heavy (e.g., weight exceeds a predetermined threshold), that the item is packaged in a polybag, or in response to determining that tenting is occurring while gripping the item, the robotic structure performs active measures to change or adapt to the determination of "tenting" or the item's packaging. As one example, the robotic structure performs active measures such as partially lifting the package and pulling it from a chute to a corresponding slot in the transport structure.

[0047] The robotic system may determine a path or trajectory (or the trajectory of the robotic arm / end effector when approaching an item for grasping) based on the type of packaging of the item to avoid tenting or otherwise improve grasping of the item. As one example, the robotic arm (e.g., wrist) and / or end effector may be controlled to be perpendicular to the surface of the item to be grasped when grasping the item. As another example, the path or trajectory of the robotic arm and / or end effector may be determined to flip or otherwise reposition the item before grasping it.

[0048] In various embodiments, multiple 3D and / or other cameras may be used to generate image data. A 3D view of the scene may be generated, and / or in some embodiments, a combination of cameras may be used to view the scene from different angles, e.g., for the workspace and / or one or more particular items within the workspace, the least occluded camera is selected and utilized in connection with grasping and moving one or more items. The image data may be utilized to detect debris on the chute or within the workspace, blockages in the chute flow of items through the workspace, the number of items grasped by the robotic structure during singulation of selected items, characteristics of one or more items occupying slots on the transport structure, etc. In some embodiments, the image data is utilized to determine characteristics (e.g., attributes) of one or more items within the workspace. As one example, the image data may be utilized in connection with determining (e.g., estimating) the height or dimensions of an item. As another example, the image data may be utilized to obtain information regarding an identifier (e.g., a label, etc.) on an item. The image data may also be utilized to determine the side of the item on which the label is included.

[0049] Multiple cameras serve many purposes in various embodiments. First, they provide a richer, full 3D view of the scene. Second, they work together to minimize errors due to package shine when light reflecting off the package interferes with the camera's operation; in this case, another camera in a different position provides backup. In some embodiments, they can be selectively triggered by a predictive vision algorithm that determines which camera has the best viewing angle and / or lowest error rate for picking a particular package, so that each package is viewed by the optimal camera. In some embodiments, one or more cameras are mounted on an operating base, and the system can change the position and orientation of the base to provide a more optimal perception (e.g., view) of the package. In embodiments, one or more cameras are mounted on the robot structure (e.g., on the end effector of a robot arm, etc.).

[0050] Another purpose served by the camera, in various embodiments, is to detect any kind of unforeseen error in the robot's operation or any disturbance to the environment. Cameras placed on the robot and in the environment have different error and accuracy profiles. Cameras on the robot may be more accurate because they are firmly fixed to the robot, but may be slower to use because the robot must slow down or stop to use such cameras. Cameras in the environment are substantially faster because they have a stable view and allow the robot to multitask and do other things while the camera takes pictures. However, if someone moves or shakes the camera stand, the camera may become out of sync with the robot, causing errors. In various embodiments, images from the robot camera and the non-robotic camera are combined (occasionally or in the event of a package mismatch) to detect whether the robot is out of sync with the non-robotic camera. If the camera is determined to be out of sync, the robot takes corrective action (performing a calibration or synchronization process, changing the human operator, etc.). In some embodiments, the camera may not be firmly mounted on the robot arm, and in some embodiments, a gyro and / or accelerometer on the camera may be used to filter or compensate for movement of the mounting base.

[0051] According to various embodiments, system 100 may include one or more sensors (e.g., one or more of an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, etc.) in addition to or in addition to the multiple cameras. Information received from various other sensors is used in determining one or more attributes of the item to be singulated and / or attributes of other items or objects in the workspace, etc.

[0052] Referring to FIG. 1 , in various embodiments, the robotic arm 102 is driven by one or more motors (e.g., one or more motors at each movable joint or mounting location). In some embodiments, the work required to drive the robotic arm 102 (e.g., to move the robotic arm as it attempts to singulate an item) is indicative of one or more characteristics of the item to be singulated. For example, in some embodiments, the weight of the item may be calculated (or estimated) based on the work required to drive the robotic arm 102 while the item is grasped by the robotic arm 102. In various embodiments, the work required to drive the robotic arm 102 is measured using current sensors, voltage sensors, power sensors, and / or the like, or some combination thereof. In response to determining the weight of the item during singulation, the robotic system determines a path / trajectory for the item to be singulated based at least in part on the weight of the item. The robotic system may take proactive measures to adapt to the weight of the item (e.g., updating the path or trajectory in response to determining the weight of the item, etc.). In some embodiments, in response to determining that the weight of the item is greater than a predetermined threshold, the robotic system 100 adjusts its plan to singulate the item by partially picking up the item and pulling it to a corresponding location on the transport structure (e.g., as opposed to fully picking up the item and moving the arm to place the item on the transport structure). In some embodiments, in response to determining the weight of the item, the robotic system adjusts the speed at which the robot arm (and item) is moved. For example, the heavier the item, the greater the shear force between the item and the end effector 104 when the robotic arm 102 is moved. Furthermore, the shear force may increase with the speed at which the robotic arm is operated (e.g., the speed at which the robotic arm moves the item).Thus, the robotic system 100 can control the velocity of the robotic arm 102 based at least in part on the weight of the item to ensure that the item remains firmly grasped by the robotic arm. Although the description herein describes weight as being measured based on using current sensors, voltage sensors, power sensors, and / or the like, weight may also be measured using force sensors configured within the robotic arm 102 or end effector 104. However, because force sensors are relatively expensive, low-level hardware information (such as measurements of motor torque or work used by a motor) is an effective method for determining (e.g., estimating) the weight of an item.

[0053] Information regarding the output from one or more sensor arrays may be used to determine the location of an item within the workspace, the location of the item while it is being grasped and / or moved by the robotic arm, and / or the position of the robotic arm (e.g., based on determining the output from some of the sensors of one or more sensor arrays compared to other portions of the sensors of one or more sensor arrays). As another example, information regarding the output from one or more sensor arrays may be used to determine the dimensions or size of the item to be singulated and / or another item or object within the workspace. Information received from one or more sensor arrays may be used in connection with determining the height of the item to be singulated and / or another item or object within the workspace. In some embodiments, the robotic system determines a path or trajectory (or updates a path or trajectory) based at least in part on the height (or other dimension) of the item to be singulated and / or another item or object within the workspace. For example, the robotic system determines the location on a conveyor where an item is to be placed based at least in part on the height (or other dimension) of one or more other items on the conveyor. Planning to place an item in a slot / tray adjacent to another slot / tray containing a relatively large (e.g., tall, wide, etc.) item can increase the likelihood of a collision during singulation. Furthermore, relatively large items on a conveyor can hinder the robotic system's ability to obtain information about adjacent items. Because the vision system's line of sight may be blocked by the relatively large item, the sensor data may not contain accurate information about the adjacent item (or other items in the large item's vicinity). As another example, if an item has an identifier or label on the side facing the relatively large item or on a surface close to the large item, the vision system may not be able to locate or read the identifier or label.In some embodiments, in response to determining that the robotic system's ability to acquire information about an adjacent item is impeded, the robotic system may implement one or more proactive measures. The one or more proactive measures may include performing one or more actions to acquire information about the item. Examples of proactive measures may include capturing information using one or more downstream sensors (e.g., overhead sensors and / or side sensors, etc.), updating the path / trajectory of the item to include a portion of the path that brings the item within range of one or more sensors (e.g., modifying the path to bring the item within line of sight of sensors in the workspace, based on modeling of the workspace and the likelihood of a successful scan for a location within the workspace, etc.), etc. Various other proactive measures may also be implemented.

[0054] 1 , in the depicted example, the system 100 further includes an on-demand teleoperator 118 that can be used by a human worker 120 to remotely operate one or more of the robotic arm 102, the end effector 104, and the conveyor 108. In some embodiments, the control computer 112 is configured to attempt to move items from a source pile (e.g., a source location) to the conveyor 108 (e.g., a destination location) in a fully automated mode. As an example, the control computer 112 is configured to pick up the items from the source pile and move the items (e.g., along such a path / trajectory) such that one or more identifiers on the items or information about the one or more identifiers on the items are acquired (e.g., scanned) by one or more sensors in the workspace. However, if, after attempting to operate in fully automatic mode, control computer 112 determines that no (further) strategies are available to have at least one of the identifiers (or information thereabout) acquired by one or more sensors, in various embodiments, control computer 112 sends an alert to obtain assistance remotely from a human operator, for example, by human operator 120 using remote operator 118. Remote operator 118 displays one or more images on a user interface, the one or more images corresponding to images of the items or workspace captured by a vision system (e.g., camera 114, camera 116, one or more other sensors, etc.). The user interface may be configured to allow human operator 120 to manually input information about the identifiers on the images (e.g., information seen in one or more images) and / or information about the items.

[0055] In some embodiments, control computer 112 is configured to attempt to grasp and place items in a fully automated mode. However, if, after attempting to operate in a fully automated mode, control computer 112 determines that no (further) strategies are available for grasping one or more items, in various embodiments, control computer 112 sends an alert to obtain assistance from a human operator remotely, e.g., by human operator 120 using teleoperated device 118. For example, in some embodiments, in response to detecting a condition or condition affecting the flow of items through chute 106, control computer 112 may attempt to perform one or more actions to facilitate singulation. If it is determined that a fully automated attempt to address the detected condition or condition did not resolve the condition or condition, the control computer may instruct human operator 120 to address the condition or condition, e.g., via teleoperation using on-demand teleoperated device 118. In various embodiments, the control computer 112 may display a user interface or other interface that presents human-selectable options for controlling the robotic arm 102, the end effector 104, and / or other elements and devices disclosed herein (e.g., blowers, vibrators, chute conveyors, etc.) to identify a state or condition and / or modify a state or condition.

[0056] In various embodiments, control computer 112 uses image data from cameras (such as cameras 114 and 116) to provide a visual representation of the scene to human operator 120 to facilitate remote operation. For example, control computer 112 may display a view of the pile of items in chute 106. In some embodiments, a segmentation process is performed by control computer 112 on the image data generated by cameras 114 and 116 to determine item / object boundaries. Masking techniques may be used to highlight individual items, for example, using different colors. Operator 120 may use the visual representation of the scene to identify items to be grasped and, using teleoperation device 118, control robotic arm 102 and end effector 104 to pick items from chute 106 and place each item in a corresponding position on conveyor 108. In various embodiments, once the items prompting human intervention are placed on the conveyor, system 100 resumes fully automated operation. In various embodiments, during human intervention, the robotic system observes the human worker (e.g., completing a manual task, completing a task remotely using a robotic arm and end effector) and attempts to learn strategies for (better) completing future tasks in autonomous mode. For example, the system may learn a strategy for grasping an item by observing the location of the grasp on the item as the human worker grasps it and / or by memorizing how the human worker utilized the robotic arm and end effector to grasp the item remotely.

[0057] In some embodiments, the system 100 seeks assistance from a human operator 120 in response to determining that an anomaly in the operation of the system 100 exists. One example of an anomaly is a failure to achieve a threshold pressure between the end effector 104 and an item during singulation of the item. In response to detecting that the pressure achieved between the end effector 104 and the item is less than the threshold pressure value, the robotic system 100 can perform a diagnostic process related to evaluating whether the robotic system 100 is performing normally. For example, the system 100 can perform a diagnostic of the end effector 104's ability to engage an item and achieve a predetermined threshold pressure value. In response to determining that the system 100 is not performing normally (e.g., the end effector 104 is not able to engage an item and achieve a predetermined threshold pressure value), the system 100 seeks assistance from a human operator 120. In some embodiments, the control computer 112 sends an alert to the human operator 120. The alert may indicate the cause of the problem (e.g., indicating that the end effector is not able to engage an item and achieve a predetermined threshold pressure value). For example, the alert may provide a human operator 120 with recommended or required corrective action.

[0058] According to various embodiments, in response to determining that the current operation of the system 100 deviates from the expected normal operation of the system 100, the system 100 determines to perform a diagnostic on the system 100. The system 100 can perform the diagnostic on portions of the system 100 that the system 100 determines, based at least in part on sensor data (e.g., current sensor data), deviates from normal operation or is within a threshold range or threshold percentage of deviation from normal operation. In the case of measurements of pressure achieved by the end effector 104 described above, the system 100 can determine the particular end effector on which the diagnostic process is performed. In some embodiments, in response to performing a diagnostic process and determining that the results of the diagnostic process indicate that the system 100 (or a component thereof) has deviated from normal operation or is otherwise within a threshold range or threshold percentage of deviation from normal operation, the system 100 implements one or more proactive measures. Examples of proactive measures include replacing a component (e.g., replacing an end effector), deciding to operate the system 100 not to utilize a component that is deviating from normal operation or not to strain a component that exceeds a predetermined strain threshold, and / or seeking human intervention (e.g., notifying a human operator 120 of the deviation). Various other proactive measures may also be implemented.

[0059] Continuing with the example described above in connection with FIG. 1 , system 100 determines to perform a diagnostic process at least in part in response to system 100 determining that system 100 is deviating from normal operation, or that a diagnostic process be performed in connection with assessing whether corrective proactive measures are implemented. In response to determining that the pressure achieved by end effector 104 upon robotic arm picking up an item deviates from expected normal operation of end effector 104, system 100 determines to perform a diagnostic regarding the operation of the end effector (e.g., one or more suction cups on the end effector) determined to be deviating from normal operation. According to various embodiments, the diagnostic process for performing a diagnostic regarding the end effector includes operating robotic arm 102 to move to a predetermined location and engage the end effector with a predetermined surface (such as surface 115 in this example). Surface 115 may be located within workspace 110. In some embodiments, surface 115 is part of chute 106 or operatively connected to chute 106 or conveyor 108. When the end effector engages the predetermined surface, the system 100 controls the end effector to grip the predetermined surface. In response to controlling the end effector to grip the predetermined surface, the system 100 acquires sensor data related to a measurement of the gripping force with which the end effector grips the predetermined surface. For example, in response to engaging the end effector with the surface 115, the system 100 controls the end effector to apply a suction force to the surface 115. The system 100 acquires sensor data including one or more values ​​of pressure achieved between the end effector and the surface 115. The system 100 determines whether the pressure achieved by the end effector when gripping the surface 115 deviates from normal operation expected when gripping the surface 115. In some embodiments, the system 100 compares the sensor data including one or more values ​​of pressure achieved between the end effector and the surface 115 to one or more predetermined ranges or thresholds mapped to normal operation of the end effector.If the pressure achieved between the end effector and the surface 115 is inconsistent with the normal operation of the end effector (e.g., if one or more values ​​of the pressure achieved between the end effector and the surface 115 fall outside one or more predetermined ranges or thresholds mapped to the normal operation of the end effector), the system 110 determines that the end effector is not operating properly (e.g., the end effector is deviating from normal operation).

[0060] According to various embodiments, in response to the system 100 determining that the end effector is not operating properly, the system 100 updates the plan to implement one or more proactive measures based at least in part on the determination that the end effector is not operating properly. Such proactive measures may include alerting the human operator 120 that the end effector is not operating properly. For example, the system 100 may send a notification of a deviation from normal operation to the human operator 120. In some embodiments, the system 100 provides one or more recommended proactive measures to the human operator 120. In response to receiving the notification of a deviation from normal operation, the human operator 120 may intervene to replace or repair the end effector. In some embodiments, the human operator 120 controls the system 100 using an on-demand remote device 118 to implement the proactive measures, such as controlling the robotic arm 102 to replace the end effector. In some embodiments, the human operator 120 can select at least one of the one or more recommended active measures, and in response to such selection, the system 100 controls the robotic arm 102 to implement the selected active measure (e.g., without further human intervention).

[0061] In various embodiments, the control computer 112 operates the robotic arm 102 (or a system associated therewith) to actuate suction cups on the end effector 104. The end effector 104 may include multiple suction cups, and the multiple suction cups may be actuated independently (e.g., independently of one another). For example, the control computer 112 may select one or more suction cups (from the multiple suction cups on the end effector) for actuation and send a signal to the end effector 104 (or a robotic arm or system associated therewith) to actuate the selected one or more suction cups. In some embodiments, the multiple suction cups include multiple sets of one or more suction cups. One or more sets of suction cups may be actuated independently from another set of one or more suction cups. In some embodiments, each set of one or more suction cups may be actuated independently from the other set(s) of one or more suction cups. The suction cups (or sets of one or more suction cups) may be actuated according to a grasping strategy to grasp an item. For example, the control computer 112 may select an item to be grasped, and the control computer 112 may determine a plan for grasping the item, such as in connection with picking up the item and placing the item in another location (e.g., a container for a kitting operation, or a tray or partitioned conveyor for a singulation operation).

[0062] According to various embodiments, one or more suction cups on the end effector 104 are actuated based at least in part on a gripping strategy. The gripping strategy may be determined by the control computer 112, which may send one or more signals (e.g., control signals) to the actuation mechanism. In some embodiments, the actuation mechanism controls the actuation of at least one of the one or more suction cups on the end effector 104. For example, the actuation mechanism can selectively actuate a first set of one or more suction cups from among multiple suction cups on the end effector. The actuation mechanism may be a module executing on the control computer 112 or a module operably connected to the robotic arm 102 and / or the end effector 104.

[0063] Controlling the suction cups included on the end effector 104 may include moving the suction cups between an inactive position and an active position. In the inactive position, the suction cups may be retracted relative to the bottom of the end effector (e.g., relative to the base plate of the end effector and / or relative to the position of the suction cups in the active position). In some embodiments, when a suction cup is in the inactive position, it does not engage an item or other surface when other suction cups included on the end effector are positioned in their corresponding active positions. Conversely, in the active position, the suction cups may be protruded relative to the bottom of the end effector 104 and / or relative to the position of the suction cups in the inactive position. According to various embodiments, a set of suction cups on the end effector 104 may be selectively controlled to engage an item, such as in connection with gripping the item. Selective control of the set of suction cups may include controlling a first set of suction cups to move to the active position and / or controlling a second set of suction cups to move to the inactive position. The first set of suction cups may be actuated to create pressure between at least one of the first set of suction cups and the item being gripped.

[0064] Although examples are described relating to singulation operations (e.g., using the robotic arm 102 to singulate items from the chute 106 to a location on the conveyor 108), various embodiments include a robotic arm performing kitting operations. For example, a robotic arm with an end effector 104 (which may, for example, include one or more suction cups) may be used to pick one or more items from corresponding determined locations (e.g., from a tray on a shelf) and arrange or assemble the items to form a kit (e.g., placing the items in a box, tray, or other container). The kit may be formed based at least in part on an order (e.g., an order received via an e-commerce website).

[0065] According to various embodiments, a set of paths or trajectories for singulating items is determined, and a path or trajectory along which the items will be singulated is selected from the set of paths or trajectories. The paths or trajectories may be selected based on various characteristics associated with corresponding paths or trajectories within the set. Examples of characteristics associated with corresponding paths or trajectories that may be utilized in connection with selecting a path or trajectory include the speed at which the items will be singulated, the position on the conveyor at which the items will be singulated, the success probability of the items being singulated along a particular path or trajectory, an indication of whether another item or object in the workspace will intersect with the particular path or trajectory, etc. In some embodiments, the success probability of the items being singulated along a particular path or trajectory is determined for at least a portion of the set of paths or trajectories, and the path or trajectory is selected based on the corresponding success probability compared to the success probabilities corresponding to other paths or trajectories. As one example, the path or trajectory is selected in response to determining that the success probability corresponding to the path or trajectory exceeds a threshold probability or exceeds a threshold number or percentage of other paths or trajectories within the subset.

[0066] According to various embodiments, the robotic singulation system implements proactive measures to improve the singulation of items (e.g., to successfully pick items from the source pile / stream and place them on the conveyor). The robotic system dynamically updates the path or trajectory of the items during singulation based on workspace conditions (e.g., the state or condition of the items, characteristics of the items, other items in the workspace, etc.). For example, in response to determining that a detected state or condition prevents the execution of a current plan for autonomously operating the robotic structure to pick one or more items from the workspace and place each item one by one in a corresponding position on the singulation transport structure, the robotic structure implements one or more proactive measures to increase the likelihood of successful singulation (e.g., the robotic structure can determine proactive measures expected to increase the likelihood of successful singulation based at least in part on the detected state or condition). In some embodiments, active measures include moving a robotic arm, a robotic arm end effector, a chute, or other element of the workspace, or utilizing an air blower, to reconfigure the source pile / stream or to reconfigure one or more items or debris on the workspace. Active measures may be implemented to improve scanning of labels or identifiers on items to be singulated, to increase the likelihood that an item can be picked up, to improve gripping of an item being singulated, to improve release of an item from a robotic arm, or to improve the operation of two robotic arms independently singulating items from the same workspace (e.g., the same source pile / stream).

[0067] FIG. 2A is a diagram illustrating a kitting system according to various embodiments.

[0068] In the illustrated example, kitting system 200 includes kitting shelf system 202 and robotic arm 212. In some embodiments, the kitting system includes multiple kitting shelf systems and / or multiple robotic arms. The robotic arm may operate autonomously to pick items (or objects from within items) from the kitting shelf system and place the items (or objects) at predetermined locations. In some embodiments, the robotic arm picks and places one or more items at predetermined locations based at least in part on a plan, such as a plan for kitting one or more items (e.g., to build a kit based on an order, etc.).

[0069] In some embodiments, the robotic arm 212 is movable relative to the kitting shelving system 202 and / or relative to the conveyor or other location where the containers are located. In the example shown in FIG. 2A , the robotic arm 212 is mounted on a carriage 214 that is configured to run along a rail or other linear guide 216 that is positioned substantially parallel to the conveyor 220. As an example, the robotic arm 212 may be mounted on the opposite side of the kitting shelving system 202. As an example, the robotic arm 212 may be mounted on the same side of the kitting shelving system 202. In some embodiments, one or more robotic arms are mounted on the same side of the conveyor 220 as the kitting shelving system, and one or more robotic arms are mounted on the opposite side of the conveyor 220 from the kitting shelving system. In various embodiments, a motor, belt, chain, or other motive force source is applied via a controller (not shown in FIG. 2A ) to move carriage 214 and attached robotic arm 212 along rails or guides 216 to facilitate automated retrieval of items from one or more kitting shelf systems and placement of items into containers (e.g., boxes, trays, etc. (such as container 218)) as they move along conveyor 220. Control of the robotic arm may be adjusted based at least in part on one or more items to be picked and placed into container 218, the location of container 218 (e.g., the predetermined location where the items are to be kitted), and / or the path of container 218 (e.g., based on the determined movement of conveyor 218).

[0070] In some embodiments, kitting system 200 includes control computer 230 and / or driving remote control 232. In the illustrated example, the operation of kitting shelf system 202, conveyor 220, and robotic arm 212 and / or carriage 214 are coordinated under the control of control computer 230. In the illustrated example, control computer 230 communicates (e.g., wirelessly) with controllers (not shown in FIG. 2A ), each configured to control the operation of a corresponding element of kitting system 200 (e.g., kitting shelf system 202, robotic arm 212, conveyor 220, carriage 214, and / or container source (not shown)). While wireless connections are shown in FIG. 2A , in various embodiments, wired connections or a combination of wired and wireless connections may be used.

[0071] 2A , the robot arm 212 has an end effector corresponding to a two-fingered gripper. In various embodiments, the robot arm 212 may be equipped with one or more other and / or different types of end effectors / acquisition tools, including, but not limited to, grippers with more than two fingers, grippers with different finger attributes than those shown (e.g., cushioned fingers, small fingers, large fingers, etc.), and / or acquisition tools that are not grippers (e.g., configured to pick up items using suction, friction, electrostatic force, magnetic force, etc.). In some embodiments, the gripper of the robot arm 212 may be interchangeable with one or more different end effectors depending on one or more attributes (e.g., weight, fragility, compressibility, stiffness, size, shape, etc.) of the item to be acquired. In some embodiments, the gripper of the robot arm 212 may be used to acquire different end effectors (e.g., gripper-held tools) for use in picking and placing the item, for example, depending on one or more attributes of the item to be acquired. One or more attributes of the item may be determined based at least in part on information obtained from one or more sensors (eg, camera 210).

[0072] In various embodiments, control computer 230 is configured, for example, by software executing on control computer 230, to receive data related to an invoice, order, parts list, pickup list, or other list of items to be retrieved and packed together, determine a strategy / plan for accomplishing the retrieval and packing of the required items, and operate the elements of kitting system 200 (e.g., kitting shelf system 202, conveyor 220, and robotic arm 212 and / or carriage 214) in coordination to fulfill the request. In some embodiments, kitting system 200 includes multiple kitting shelf systems and / or multiple robotic arms, and one or more control computers are controlled to coordinate / operate the elements of kitting system 200.

[0073] In some embodiments, control computer 230 is configured to receive a list of items to be packed. Control computer 230 may determine which items are associated with which kitting shelving system (or which items are associated with a particular shelf in a kitting shelving system (e.g., kitting shelving system 202)) and may create a plan to retrieve and pack the items. In some embodiments, control computer 230 controls a box assembly machine (not shown) or a container supply module to place containers on conveyor 220 and control conveyor 220 to advance the containers to a position where one or more first items will be placed. Control computer 230 controls carriage 220 and / or robotic arm 212 to position robotic arm 212 to retrieve one or more first items from an associated kitting shelving system. Control computer 230 may, for example, control kitting shelving system 202 to ensure that the required items in the required quantities are present in a pickup zone (e.g., presentation surface) at the end of kitting shelving system 202 (or a shelf of kitting shelving system 202) closest to conveyor 218 and robotic arm 212. Control computer 230 controls robotic arm 212 to retrieve the items from the corresponding pickup zone, place the items in a container (e.g., 218), and then move to perform coordinated retrieval and packing of any additional items needed to be included in that particular kit. In response to determining that all items have been retrieved and packed (e.g., according to a plan for kitting one or more items), control computer 230 controls conveyor 220 to advance the container (e.g., container 218) to the next stage of fulfillment (e.g., a station where boxes are sealed, labeled, and sent for shipping), not shown in FIG. 2A .

[0074] According to various embodiments, kitting system 200 includes one or more sensors within the workspace. For example, kitting system 200 may include one or more cameras, barcode scanners, RFID scanners, infrared scanners, 3D scanners, and / or the like. As shown in FIG. 2A , kitting system 200 may include one or more cameras (such as camera 210). Some of the one or more cameras may have line of sight to kitting shelving system 202 and / or one or more items on the shelves of kitting shelving system 202. In some embodiments, kitting system 200 includes sensor 222, sensor 224, sensor 226, and / or sensor 228. Sensor 222, sensor 224, sensor 226, and / or sensor 228 may be arranged in a sensor array. Sensor 222, sensor 224, sensor 226, and / or sensor 228 may be configured to capture information (e.g., machine-readable information from the items).

[0075] Sensors in the workspace (e.g., camera 210) may capture information about one or more items on at least one shelf of kitting shelving system 202. For example, information (e.g., machine-readable information) about identifiers on at least one side of the items may be obtained by the sensors. The one or more identifiers on the items may be used by kitting system 200 in connection with determining that the items have been kitted (e.g., retrieved from kitting shelving system 202 and placed in receptacle 216), in connection with confirming at least partial fulfillment of a corresponding order (e.g., an order in which the items are components), and / or in connection with determining whether to replenish kitting shelving system 202 with items (e.g., items that are of a type corresponding to the one or more identifiers of the items removed from kitting shelving system by robotic arm 212). Information about one or more identifiers on an item may be acquired while the item is on the kitting shelving system 202 (e.g., on a presentation surface of a shelf) and / or while the item is being moved by the robotic arm from the kitting shelving system 202 to the bin 216. For example, the robotic arm 212 may move the item along a path / trajectory determined at least in part based on the item (e.g., the size of the item, the type of item, the location or expected location of one or more identifiers on the item) and / or the location of one or more sensors (e.g., barcode scanner, RFID scanner, etc.) in the workspace. The path / trajectory may be predetermined, defined, or otherwise specified in a plan (e.g., a plan for kitting one or more items) (e.g., to build a kit based on an order, etc.). In some embodiments, the path / trajectory is determined at least in part based on a condition in which the item is moved through a threshold area or threshold range of one or more sensors. For example, the path / trajectory is determined such that at least one side of the item is moved within the line of sight of at least one sensor.

[0076] According to various embodiments, as robotic arm 212 moves an item from kitting shelf system 202 to a bin (e.g., bin 218 on conveyor 220), one or more of sensors 222, 224, 226, and / or 228 capture information about the item when the item is brought within range / view of the sensor. For example, sensors 222, 224, 226, and / or 228 may capture information about one or more identifiers on the item. The information about the one or more identifiers may be machine-readable information that a computer (e.g., control computer 230) can process and determine information about the item (e.g., item type, manufacturer, serial number, model number, brand, lot number, etc.). Kitting system 200 may include a sensor array including sensors arranged to capture information from different angles / lines of sight. For example, sensor 226 and / or sensor 228 may be positioned to have an upward line of sight (e.g., substantially perpendicular to the ground), and sensor 226 and / or sensor 228 may obtain information corresponding to an identifier on the bottom surface of the item. As another example, sensors 222 and 224 may be positioned at a different angle than sensor 226 and / or sensor 228. Sensor 222 and / or sensor 224 may be positioned at a 45-degree angle with respect to the ground. Sensor 222 and / or sensor 224 may be positioned to view at least one side of the item (e.g., a side of the item other than the bottom surface of the item scanned by sensor 226 and / or sensor 228).In some embodiments, kitting system 200 includes camera 210 configured to acquire information corresponding to one or more identifiers on a top surface of an item while the item is on a shelf of kitting shelf system 202, sensors (e.g., sensor 226 and / or sensor 228) configured to acquire information corresponding to one or more identifiers on a bottom surface of the item while the item is being moved from kitting shelf system 202 to a container (within the range / field of view of the sensor), and sensors (sensor 222 and / or sensor 224) configured to acquire information corresponding to one or more identifiers on one or more sides of the item (e.g., a surface substantially perpendicular to the ground) while the item is being moved from kitting shelf system 202 to a container (within the range / field of view of the sensor).

[0077] According to various embodiments, the plan for moving items from kitting shelving system 202 to bins (e.g., the plan for retrieving and packing items) includes a path or trajectory along which robotic arm 212 moves the items. The path along which robotic arm 212 moves the items may be determined based at least in part on the locations of one or more sensors (e.g., sensors 222, 224, 226, and 228) (or a sensor array) within the workspace. For example, the path may be determined to include moving the items within a threshold range or area of ​​one or more sensors. The threshold range or area may be a defined location or area within which at least one sensor can acquire information from an identifier on an item. As one example, for an RFID scanner, the threshold range or area may be based at least in part on the proximity within which the RFID scanner can acquire information from an RFID tag on an item. As another example, for a barcode scanner, the threshold range or area may correspond to the distance within which the barcode scanner can acquire information from a barcode on an item. 2A , as the item is being moved through threshold range or threshold area 236, information about the item (e.g., an identifier on the item) may be obtained by sensor 222, sensor 224, sensor 226, and / or sensor 228. Accordingly, a plan may be determined that includes a path for the item along which the item will be moved through at least a portion of threshold range or threshold area 236. In some embodiments, kitting system 200 stores a mapping of threshold range or threshold area 236 to the workspace (control computer 230 stores a definition of the boundaries of threshold range or threshold area 236). In some embodiments, threshold range or threshold area 236 is mapped on a sensor-by-sensor basis (e.g., each sensor may be mapped to a threshold range or threshold area).

[0078] In some embodiments, determining a plan for kitting (or singulating) an item may include determining a pick-and-place method for the item that optimizes the likelihood that the identifier will be read by a sensor in the workspace. Based on the location of one or more sensors in the workspace, the robotic system can determine the side of the item that includes the label, determine which side to grasp when grasping the item, and determine a path / trajectory along which the item will be singulated to expose the label on the item to one or more sensors (e.g., a vision system or other sensor array, such as a barcode reader). In some embodiments, the path is determined based at least in part on the likelihood that at least one of the one or more identifiers will be scanned (or information about such identifiers will be obtained) by one or more sensors (e.g., sensors 222, 224, 226, and / or 228). Determining a path to optimize the likelihood that one or more labels or identifiers will be detected by one or more sensors includes determining a set of paths along which the item will be moved and corresponding probabilities that the label or identifier on the item will be scanned if the item is moved along the path, and selecting the path with the highest corresponding probability. In some embodiments, a path is determined that exceeds a threshold probability or falls within a predetermined percentage or number of paths in the set of paths that have the highest corresponding probability. In some embodiments, a path and a corresponding probability that a label or identifier on an item will be scanned if the item is moved along such a path are determined, and the probability is compared to a predetermined threshold for a minimum desired probability. In response to determining that the probability corresponding to a particular path does not exceed the predetermined threshold for the minimum desired probability, a new path and corresponding probability are determined. The robotic system may continue to iterate determining paths and corresponding probabilities until the robotic system determines that the corresponding probability exceeds the predetermined threshold for the minimum desired probability. Determining a path may include weighting different factors and maximizing a composite score for the weighted values ​​of the different elements.The different factors may include the likelihood of obtaining one or more identifiers, the likelihood of successfully picking and placing the item, the expected time to pick and place the item if the item is moved along the corresponding path, the presence / location of other objects in the workspace (e.g., another robotic arm, a container, another item, etc.), etc.

[0079] According to various embodiments, in response to obtaining information regarding the identifiers on the items, the control computer 230 updates a data structure that stores mappings for information regarding the items. The mappings may correspond to mappings of identifiers or items to orders, mappings of identifiers or items to containers, or mappings of identifiers or items to the kitting shelf system (or specific shelves in the kitting shelf system). Information regarding the identifiers on the items (obtained by one or more sensors) may be utilized in connection with determining the completion of order fulfillment and / or determining to replenish the kitting shelf system 202. Information regarding the identifiers on the items (e.g., labels, barcodes, QR codes, etc.) may be scanned by sensors. The item identifiers may be used to look up item attributes (e.g., item size, item weight, indication that the item is fragile, type of packaging, etc.). The item attributes may be used in connection with determining / updating a packing or gripping strategy and / or a packing strategy (e.g., the arrangement within the container to follow when packing the items), etc.

[0080] While a "kitting" operation is shown in FIG. 2A and described herein with reference to FIG. 2A and other figures, in various embodiments, the kitting system (and kitting shelving system) and integrated system disclosed herein may be used to perform the reverse operation, e.g., by placing items removed from a box that was initially full or partially full of items onto a shelf, bin, and / or kitting machine. For example, in the example shown in FIG. 2A , bin 218 may include multiple items associated with kitting shelving system 202, and robotic arm 212 may be used to remove the items from bin 218 and place the items on kitting shelving system 202, e.g., from the back or feed end as shown. In some embodiments, the robotic arm provides (e.g., stocks) items onto a feeder portion of a shelf or a feeder portion of the kitting shelving system.

[0081] In some embodiments, items on a kitting shelving system (e.g., kitting shelving system 202) or on a shelf accessed by or included in a kitting system disclosed herein may be bins or trays containing objects to be “kited.” In some embodiments, a system (e.g., kitting system 200 of FIG. 2A ) is configured to detect that a bin is empty, for example, based on computer vision or other sensors and / or techniques, and remove the bin from the kitting system or shelf by having a robotic arm pick up the bin and place it in a corresponding empty bin location (e.g., a nearby stack of empty bins). In some embodiments, automatic bin removal makes way for the next non-empty bin to move to a location on the kitting machine or shelf, allowing the robotic arm to access the bin and pick up an item from it. In some embodiments, in response to determining that a bin / tray is empty, the robotic arm (e.g., robotic arm 212) removes the bin / tray from the corresponding shelf of kitting shelving system 202 and uses the empty bin / tray as the next container for kitting another item / object. For example, the robotic arm 212 can pick up an empty bin / tray from a shelf and place the empty bin / tray on the conveyor 220 for use as a receptacle (eg, receptacle 218).

[0082] In various embodiments, the robotic systems disclosed herein comprise and / or perform, for example, by operation of a control computer (such as control computer 230), one or more of the following: The computer vision information is generated by merging data from multiple sensors, including one or more of 2D cameras, 3D (e.g., RGBD) cameras, infrared, and other sensors, to generate a three-dimensional view of the workspace, which may include one or more kitting shelving systems (and, accordingly, one or more shelves of the kitting shelving systems). The robotic system determines characteristics of items and / or debris or other anomalies in the three-dimensional view of the workspace. The robotic system coordinates the motion of multiple robots to avoid collisions, obstructions, and competition with another robot to pick up the same item and / or place it in the same destination location (e.g., a bin on a conveyor). The robotic system coordinates the motion of multiple robots operating in the same workspace to perform kitting on multiple items / objects (e.g., to kitte items in different bins or in the same bin). As an example, in various embodiments, multiple robots operate independently to pick and place items. As another example, multiple robots operate independently to pick and place items for different orders (e.g., to place different sets of items in different bins). If a risk of collision is detected, responsive action is taken to prevent the multiple robots from colliding with each other during singulation. The robotics system coordinates the actions of multiple robots to ensure that all items are placed in their corresponding bins. For example, if Robot A drops an item, the system tasks Robot B with picking it up; an item that is placed but improperly oriented can be picked up and adjusted or moved to another position by the same or another robot; when two or more items of different orders are placed in a single bin, a downstream robot can pick one of the two or more items from the bin and place it in the new bin. The robotic system continually updates motion plans for each robot and for all robots to achieve a desired overall throughput (e.g., to maximize overall throughput, to reach a predetermined threshold of overall throughput, etc.). In response to determining that two or more robots have collided or will collide when moving according to their respective plans for kitting items, the robotic system implements proactive measures to ensure that the two or more robots avoid the collision or otherwise reset the independent operation of the two or more robots. Upon determining that two robots are tasked with independently retrieving the same item, the system randomly selects one robot to take the item, while the other robot transitions to the next item (e.g., identifies, selects, decides on a grasping strategy, picks, moves according to a plan, and places). The robotic system can manage the independent operation of multiple robots to ensure that the robots select items at different times to avoid the same item being selected for singulation by two different robots. Conveyor movement and / or speed controlled as needed to achieve desired robot productivity (throughput) and allow sufficient time for the robot to place objects in the desired bins. In response to determining that an item has been misplaced or dropped, the system assigns the misplaced item to a robot or, if necessary, a human worker to pick up the misplaced item and return it to the appropriate kitting shelving system (e.g., on a shelf, such as via a feeder section) or, if available or more optimal, place it in a bin on a conveyor. An upstream robot that is controlled to intentionally leave some bins empty in order for a downstream robot to place items on the conveyor (e.g., into the corresponding bins). A downstream robot that is controlled to correct errors in the placement of an item upstream into a bin on a conveyor (e.g., to correct placement of an item in more than one bin (e.g., tray) or an item on the conveyor instead of the appropriate bin, to update a data structure with the relationship between the item or kit / order identifier and the bin into which the upstream robot placed the item, etc.). As a result of a failure that cannot be corrected by the same or another robot, an alert is communicated to obtain human (or other robot) intervention to resolve. In response to determining that the gripping force (e.g., pressure achieved by the end effector) is abnormal (e.g., lower than expected during normal operation), perform diagnostic processing including testing the gripping force on a predetermined surface to determine whether corrective action is required with respect to the end effector. Move / remove debris within the workspace or reconfigure items being kitted (e.g., to improve the likelihood that an item will be successfully picked from a shelf (or shelf presentation) and placed on a conveyor (e.g., into a bin on a conveyor)). Using sensor data from the workspace environmental state system, detect one or more characteristics (e.g., attributes) of an item selected for kitting, determine that grasping or release of the item is expected to improve in response to implementation of proactive measures, and implement proactive measures to improve grasping or release of the item. Using the sensor data, determine that the robotic arm has grasped multiple items in association with kitting one of the items, and determine a plan for releasing the multiple items to place each item one-by-one in a corresponding position in one or more bins on the conveyor or to return one of the items to a shelf / presentation surface of the kitting shelving system (e.g., determine a strategy for operating the end effector to release a first subset of the multiple items at a different time than a second subset of the multiple items). Selecting a path for kitting items into a container on a conveyor based on attributes of the items (e.g., size of the selected item, weight of the item, etc.) and / or one or more attributes (e.g., characteristics) of the items in the container on the conveyor. Selecting a path for kitting items into bins on a conveyor based on an identifier on the item (e.g., identifier type, identifier location, etc.) and / or one or more sensors in the workspace (e.g., sensor location, sensor range, sensor type, etc.). · determining the movement and speed of a robotic arm to pick items from a kitting shelving system and place the items in the appropriate bins based at least in part on the speed of a conveyor belt; Determining the trajectory of the items to be kitted based at least in part on one or more of the following characteristics: characteristics of the items, characteristics of the workspace environment, and / or characteristics of the conveyor (e.g., speed of the conveyor belt). Determine the kitting success probabilities corresponding to one or more paths / trajectories of the items to be kitted, and select the path / trajectory along which the items will be kitted based on the corresponding success probabilities. Determine positioning of the robotic arm and / or the robotic arm's end effector for a successful grasp (e.g., as determined based on the probability of successful grasp, the item's packaging type, the item's dimensions, the expected grasp force relative to a threshold, etc.). Positioning the end effector may include controlling movement of the robotic arm or the robotic arm's wrist so that the end effector is perpendicular to the item's surface. Update the robotic system's ability to detect empty containers. For example, the definition of an empty container used by the robotic system to identify empty containers is updated over time. Controlling the robotic arm to orient the item while moving it from the kitting shelving system to the bin (e.g., rotating the wrist of the robotic arm to rotate the item while it is being moved within a sensor threshold range). Controlling the robotic arm to repeatedly move the item within the sensor threshold range to ensure that all identifiers on the item are acquired (or the required set of identifiers is acquired).

[0083] According to various embodiments, kitting shelving system 202 includes one or more shelves (e.g., shelves 204, 206, and 208). Each of the one or more shelves may be an integrated unit including a side where items are loaded onto the shelf and a side where items can be picked up by robotic arm 212. In some embodiments, at least one of the shelves includes a gating structure that controls the flow of items to an area where the items are picked up by robotic arm 212. For example, a shelf may include a presentation surface. The presentation surface corresponds to an area or surface where items are placed on the kitting shelving system. In connection with kitting one or more items, the robotic arm picks items (or objects from within the items) from at least one presentation surface. Kitting shelving system 202 (or a shelf included in kitting shelving system 202) may include one or more feeder sections. In some embodiments, kitting shelving system 202 includes a gating structure configured to control the transport of items from the feeder section to a corresponding presentation surface. The gating structure may be coupled to or integrated with the presentation surface. In some embodiments, the kitting shelf system 202 comprises one or more shelves whose presentation surfaces are integrated with corresponding feeder sections (e.g., the presentation surfaces may be fixed and non-movable relative to the feeder sections).

[0084] According to various embodiments, the gate structure allows or prevents items from flowing from the feeder section to the presentation surface. Controlling the flow of items from the feeder section to the presentation surface may prevent items from becoming cluttered on the presentation surface and may help provide sufficient space and order for a robotic arm to pick and place items / objects from the presentation surface. The gate structure may be configured to prevent / restrain more than a threshold number of items from being placed on the presentation surface at any particular time. In some embodiments, the gate structure is switched (e.g., moved) between an open position and a closed position. As one example, the open position may correspond to an orientation of the gate structure when no items are present on the presentation surface. As another example, the open position may correspond to an orientation of the gate structure when the quantity of items on the presentation of items is less than a threshold number of items. As another example, the open position may correspond to an orientation of the gate structure when the weight on (or force acting on) the presentation surface is less than a threshold weight (or threshold force). When the gate structure is oriented in the open position, the flow or transfer of items from the feeder portion to the presentation surface may be permitted (e.g., not blocked), and conversely, when the gate structure is oriented in the closed position, the flow or transfer of items from the feeder portion to the presentation surface may be prevented (e.g., the flow of items is blocked).

[0085] In some embodiments, the presentation surface of the shelf is movable. For example, the presentation surface switches between an empty position and an occupied position. The presentation surface may be oriented in any one of multiple positions / orientations based at least in part on the context of the workspace (e.g., items presented on the presentation surface, the position of a robotic arm picking the items, the position of a sensor or camera, etc.).

[0086] In various embodiments, the vacant position corresponds to a position to which the presentation surface is oriented to facilitate the flow / transport of one or more items from the feeder section to the presentation surface. For example, the vacant position may correspond to a position to which the presentation surface is oriented when the quantity of items on the presentation surface is less than a threshold number. In another example, the vacant position corresponds to a position to which the presentation surface is oriented when the presentation surface is empty (e.g., no items are placed on it). As another example, the vacant position corresponds to a position to which the presentation surface is oriented when the weight on (or force acting on) the presentation is less than a threshold weight (or threshold force). In some embodiments, each presentation surface in a particular kitting shelf system (e.g., kitting shelf system 202) may have the same azimuth angle (e.g., angle relative to a perpendicular to the ground) when oriented to the corresponding vacant position. In some embodiments, two or more presentation surfaces in a particular kitting shelf system (e.g., kitting shelf system 202) have different azimuth angles (e.g., angle relative to a perpendicular to the ground) when oriented to the corresponding vacant positions. Configuring shelves (e.g., presentation surfaces) at different angles may allow for better line of sight of sensors (e.g., camera 212) within the workspace, and such configurations may improve information about the workspace (e.g., identifiers on items may be obtained more easily or accurately, etc.).

[0087] In various embodiments, an occupied position corresponds to a position at which a presentation surface is oriented to facilitate picking of one or more items from the presentation surface (e.g., by a robotic arm). A presentation surface may be oriented in an occupied position when the transfer / flow of one or more items from a feeder section to the presentation surface should be prevented. In some embodiments, each presentation surface in a particular kitting shelf system (e.g., kitting shelf system 202) may have the same azimuth angle (e.g., angle relative to a normal to the ground) when oriented toward the corresponding vacant position. In some embodiments, two or more presentation surfaces in a particular kitting shelf system have different azimuth angles (e.g., angle relative to a normal to the ground) when oriented toward the corresponding occupied positions. As one example, a first set of one or more presentation surfaces has an azimuth angle that is different from the azimuth angle of a second set of one or more presentation surfaces. As another example, each presentation surface in a particular kitting shelf system has a different azimuth angle (e.g., angle relative to a normal to the ground) when oriented toward the corresponding occupied position. Configuring shelves at different angles may allow for better line of sight of sensors (such as camera 212) within the workspace, and such a configuration may improve information about the workspace (e.g., identifiers on items may be obtained more easily or accurately, etc.).

[0088] In some embodiments, the orientation of the presentation surface may be tilted downward more sharply the lower the corresponding shelf is relative to the ground. Such an orientation of the presentation surface may enhance one or more sensors (e.g., camera 210212) for acquiring information about one or more items / objects on the presentation shelf or presentation surface. Furthermore, such an orientation may enhance the ability of the robotic arm to engage items with its end effector. The robotic arm may have limitations regarding its wrist extension capability / configuration and / or its wrist flexion capability / configuration. As an example, the orientation of the presentation surface (e.g., at least in the occupied position) may be configured based at least in part on the degree of wrist extension required by the robotic arm to pick an item / object from the presentation surface. The shelf / presentation surface may be configured based at least in part on the range of motion (e.g., range of motion for wrist extension / flexion) of the wrist of the robotic arm in the kitting shelving system. The end effector or wrist component of the robotic arm may have size limitations that inhibit the ability of the robotic arm to engage items placed on the presentation surface (e.g., at certain angles and heights / positions). Thus, the orientation of the presentation surface of a shelf (e.g., at least in an occupied position) may be configured to improve the possibility / ability of the robotic arm to configure its position to engage an item / object on the presentation surface with the robotic arm's end effector perpendicular to the item / object. The orientation of the presentation surface when in an occupied position may correspond to an orientation in which the tray / item placed thereon is optimally tilted (at each level / shelf) for better viewing from an onboard camera (e.g., a camera located in the workspace and / or on the robotic arm or its chassis). In some embodiments, the orientation of the presentation surface in the vacant and / or occupied positions is based at least in part on the configuration of the corresponding gate structure. For example, if the gate structure is a hinge, the orientation of the presentation surface in the vacant and / or occupied positions is based at least in part on the range of motion of the hinge.

[0089] According to various embodiments, kitting shelf system 202 includes one or more feeder sections. In some embodiments, kitting shelf system 202 may have a single feeder section that transports one or more items to multiple presentation surfaces. In other embodiments, kitting shelf system 202 has a single feeder section for each presentation surface (e.g., a one-to-one correspondence between feeder sections and presentation surfaces). A feeder section may be configured to transport items to a presentation surface. As one example, item transport may be passive, such as by gravity acting on items placed on the feeder section (e.g., when the feeder section is configured to slope toward the presentation surface). As another example, item transport may be at least partially active, such as when the feeder section is configured with a conveyor that carries items from an input location of the feeder section to the presentation surface. In various embodiments, the feeder section is configured to sequentially accept items on a receiving end (e.g., input to the feeder section) and transport the items to a destination end (e.g., operatively connected / coupled to a presentation surface or otherwise exiting the feeder section to an appropriate presentation surface). A series of items may be manually loaded into the feeder section or kitting shelving system (e.g., by a human operator 228), or a series of items may be automatically loaded into the feeder section (e.g., by a robotic arm / component or based at least in part on the feeder section being coupled to a chute that transports items from a source stream / pile).

[0090] FIG. 2B is a diagram illustrating a kitting system according to various embodiments.

[0091] According to various embodiments, the robotic arm 212 moves items from the kitting shelf system 202 to the bin 218 in a manner that improves the likelihood that at least one identifier on the item will be acquired / scanned by one or more sensors in the workspace (e.g., sensor 222, sensor 224, sensor 226, and / or sensor 228). For example, as shown in FIG. 2B at 240, the robotic arm may rotate the item. Rotating the item may increase the likelihood that an identifier on a side of the item (e.g., a surface adjacent to the bottom of the item) will be scanned by a sensor. The item may be rotated when the item is within the threshold range / threshold area 236. In some cases, the item may be rotated 360 degrees (or may be rotated continuously when the item is within the threshold range / threshold area 236). In other cases, the item may be rotated based at least in part on the range of motion of the robotic arm 212 (e.g., the wrist of the robotic arm) and / or an attribute of the item.

[0092] In some embodiments, the robotic arm 212 moves an item along a predetermined path. The path includes at least one path portion in which the item is moved within at least a portion of the threshold range / threshold area 236. The robot system (e.g., control computer 230) may determine the position of the robotic arm 212 (or an item being moved by the robotic arm 212) within the workspace and / or relative to one or more sensors (such as sensor 222, sensor 224, sensor 226, and / or sensor 228). The robotic arm may rotate the item (e.g., an item being grasped by it) based at least in part on the position of the robotic arm 212 and / or the item. For example, in response to determining that the robotic arm 212 and / or the item are within the threshold range / threshold area 236 (or within a predetermined distance of the threshold range / threshold area 236), the robotic arm 212 may rotate the item. The robotic arm may rotate the item by rotating the wrist of the robotic arm 212. In some embodiments, the axis of rotation of the item is perpendicular to the ground (or substantially perpendicular to the ground).

[0093] Rotating the item may improve the ability of a sensor (e.g., sensor 222 or sensor 224 in FIG. 2B ) to scan / acquire identifiers on at least one side of the item. As one example, the robotic arm 212 may continue to rotate the item while the item is within the threshold range / threshold area 236. As another example, the robotic arm may continue to rotate the item until all identifiers (e.g., all identifiers on the sides of the item) have been acquired, or until a threshold number of identifiers have been acquired by one or more sensors. The threshold number of identifiers may correspond to an expected number of identifiers (e.g., the number of identifiers mapped to the type of item to which the item corresponds).

[0094] According to various embodiments, the rotation of an item may be controlled based at least in part on one or more attributes of the item. For example, the speed or degree of rotation may be controlled based at least in part on attributes of the item. Attributes of the item may include weight, length, size, etc. The rotation of the item may be controlled to prevent shear forces on the item and / or the end effector of the robotic arm 212 from setting the item down (e.g., dropping the item onto the robotic arm 212). In some embodiments, in the case of a suction-based end effector, the rotation (e.g., the speed or degree of rotation) may be controlled based on pressure detected between the end effector and the item. The robotic system may also, or alternatively, increase the suction force applied to the item as it is being rotated. The strategy or plan determined in connection with rotating (or reorienting the item) may be determined based on a determination of the item's final orientation following packaging or placement (e.g., into a container).

[0095] 2B and 2C are diagrams illustrating kitting systems according to various embodiments.

[0096] The robotic system (control computer 230) may execute one or more proactive measures related to scanning / capturing one or more identifiers on an item, such as in connection with ensuring that all identifiers (e.g., required identifiers) are captured. According to various embodiments, the robotic system executes at least one proactive measure upon determining that at least one identifier was not scanned or captured. The at least one proactive measure may increase the likelihood that the one or more identifiers will be captured. By way of example, the at least one proactive measure selected for execution may be selected based at least in part on a corresponding likelihood that the at least one identifier will be captured in response to the execution of the proactive measure. Examples of active measures include shaking the item in one or more ways, dropping off the item and picking it up in a different way (e.g., to reorient the item or to grasp the item from a more pickable position), dropping off the item, adjusting the item's orientation, and picking it up in a different way, adjusting the packaging of items with an end effector such as by applying pressure to the item / package (e.g., pushing the bag, spreading the bag, etc.), and adjusting the items using a robotic arm, end effector, or workspace peripheral (e.g., using a compressed air blower, air knife, static discharger, etc.).

[0097] Determining that at least one identifier was not scanned or acquired may be based at least in part on the number of identifiers acquired by one or more scanners (e.g., camera 210 and / or sensors 222, 224, 226, and / or 228) and the expected number corresponding to the item. According to various embodiments, the number of identifiers for an item (or type of item) is known and stored in advance. For example, a mapping of item types to identifier numbers on an item may be pre-stored. Control computer 230 may obtain the expected number of identifiers corresponding to an item based at least in part on performing a lookup in the mapping of item types to identifier numbers. In some embodiments, control computer 230 compares the number of identifiers acquired from an item by one or more sensors to the expected number of identifiers. In response to determining that the number of identifiers acquired is less than the expected number of identifiers, control computer 230 may determine that at least one identifier was not scanned or acquired. In some implementations, the control computer 230 may determine the number of identifiers that were not scanned or acquired by one or more sensors (e.g., based on a comparison of the number of acquired identifiers to an expected number of identifiers). The comparison may be performed after the items are moved through the threshold range / threshold area 236 (e.g., before the items are placed in the receptacle 218). In some embodiments, the control computer determines the aspects of the items for which identifiers were not acquired based on the determination of the identifiers (or types of identifiers) that were not scanned (e.g., based on a comparison of the expected identifiers for the items to the acquired identifiers).

[0098] In response to determining that at least one identifier was not scanned or acquired, one or more proactive measures may be implemented (e.g., to attempt to acquire at least one identifier). As shown in FIG. 2C , in some embodiments, the proactive measure includes attempting to rescan the item. For example, the proactive measure may include moving the item through a threshold range / area one or more additional times. In some embodiments, the proactive measure is implemented until all of the one or more identifiers on the item have been scanned (e.g., based on an updated comparison of the number of acquired identifiers to the expected number of identifiers), a threshold number of attempts have been made to scan one or more identifiers, or other conditions. As one example, the proactive measure may be implemented until all of the one or more identifiers on the item have been acquired, or a threshold number of attempts have been made to scan one or more identifiers. As another example, the proactive measure may be implemented until a threshold number of identifiers on the item have been acquired. As another example, the proactive measure may be implemented until a particular type of identifier (e.g., serial number, model number, product number, etc.) on the item is acquired. The control computer 230 may store an indication of one or more particular types of identifiers that need to be obtained (e.g., in a profile associated with the item type that corresponds to the item). As shown in FIG. 2C , movement 250 may correspond to an initial attempt or movement through a threshold range / threshold area 240 (not shown in FIG. 2C ). In response to the robotic arm 212 determining to perform an active countermeasure, the item may be moved according to movement 252 and / or movement 254. In some embodiments, at least one movement 254 is through the threshold range / threshold area. For example, movement 254 may be a remaining movement that positions the item so that the robotic arm 212 can again move the item through the threshold range / threshold area according to movement 254.

[0099] According to various embodiments, the plan for kitting (or singulating) the item may be updated in response to a determination that proactive measures will be implemented and / or in response to a determination that at least one identifier on the item was not acquired. Control computer 230 may update the plan to include a new path / trajectory. For example, the plan for kitting the item may be updated to further include movements 252 and 254.

[0100] In some embodiments, the proactive measures further include moving the item within a different portion of the threshold range / threshold area. As one example, one or more subsequent attempts to acquire the at least one identifier may include moving the item along a path closer to one or more sensors. As another example, each further attempt to acquire the at least one identifier may be performed in a different portion of the threshold range / threshold area or at a different distance from the at least one sensor. Moving the item within a different portion of the threshold range / threshold area may move the item in an area where one or more sensors have a better line of sight or a better ability to acquire the identifier from the item. If the identifier is relatively small, the identifier may be better captured if the item is moved closer to one or more sensors.

[0101] In some embodiments, the proactive countermeasure includes orienting the item such that the ability of at least one sensor to acquire an identifier on at least one surface of the item is improved. For example, the robot arm 212 may be controlled to orient the item such that a side of the item is oriented toward the sensor. In the example shown in FIG. 2C , during an attempt to rescan the item, the robot arm 212 may be controlled to tilt the item such that a side of the item is oriented toward the sensor 222 and / or the sensor 224. The degree of tilt may be selected based on the positioning / configuration of the sensors. For example, the robot arm 212 may be controlled to tilt the item such that a side of the item is perpendicular to the line of sight of the sensor. In addition to or instead of the robot arm 212 being controlled to tilt the item, the robot arm 212 may be controlled to at least partially rotate the item, such as by rotating the wrist of the robot arm 212 (e.g., while the item is being moved through a threshold range / area in connection with the proactive countermeasure). In some embodiments, the robotic arm 212 may be controlled to vary the tilt and / or rotation of the item according to a random or predetermined sequence of movements, at least while the item is being moved through a threshold range / area in connection with an active countermeasure.

[0102] According to various embodiments, the sensors may be mounted on the robot arm, the robot chassis, etc. Active measures may include reorienting the item relative to the sensors mounted on the robot arm and / or attempting to rescan the item with the mounted sensors.

[0103] After attempting to perform proactive measures a threshold number of times (e.g., after attempting to rescan the item a threshold number of times), kitting system 200 may determine that an error has occurred. In response to determining that an error has occurred, kitting system 200 may alert a human operator 234 via remote control device 232. Alerting the human operator may include displaying a warning on a user interface. The warning may include an indication of the type of error (e.g., that the item could not be properly scanned, that an identifier could not be found on the item, etc.).

[0104] In some cases, the kitting shelf system 202 may be improperly loaded with items such that the items on the shelf do not match the expected item type (e.g., the wrong model of a certain type of item may be loaded onto the shelf). Incorrect loading of the kitting shelf system may cause errors during attempts to obtain identifiers from items. For example, if the control computer 230 expects a first type of item to be loaded onto the shelf where the robotic arm is instructed to pick an item, but a second type of item is actually loaded, the expected number of items to be scanned (e.g., based on the mapping of item types to identifier numbers) may be inaccurate. Control computer 230 may determine an expected number of identifiers for a first type of item, and if a second type of item has a different number of identifiers (e.g., the first type of item has five expected identifiers and the second type of item has four expected identifiers), then confirmation that all expected identifiers were scanned during kitting of the item will be uncertain / inaccurate (e.g., the kitting system may attempt to take proactive measures to rescan the item, but regardless of the number of attempts to rescan the identifiers, the kitting system may not be able to confirm that all expected identifiers were scanned). Thus, kitting system 200 may determine that an error has occurred. Similarly, in response to obtaining identifiers from the items, kitting system 200 may verify that the items to be picked and placed match the plan (e.g., order, packing slip, etc.). If the kitting shelf system is improperly stocked or if robotic arm 212 picks an item from the wrong shelf or bin, the identifiers on the items to be picked and placed may differ from the item corresponding to the plan. Thus, kitting system 200 may determine that an error has occurred. In response to determining that an error has occurred, the kitting system 200 may attempt proactive measures, such as returning the item to the kitting shelving system or return bin and picking up the correct item.Alternatively, in response to determining that an error has occurred, kitting system 200 may alert a human operator 234 via remote control 232. Alerting the human operator may include displaying a warning on a user interface. The warning may include an indication of the type of error (e.g., that the kitting shelf system has been improperly stocked, etc.).

[0105] According to various embodiments, sensors may be located in or around the workspace, on the robotic arm (e.g., on the end effector of the robotic arm), on the kitting shelving system, and / or on a carriage carrying the robotic arm. As shown in FIGS. 2B and 2C , threshold range / area 236 may exist in or around the sensors that capture information about items moved therethrough. For example, threshold range / area 236 may be an area corresponding to sensors 222, 224, 226, and / or 228. In some embodiments, one or more sensors are located on carriage 214.

[0106] FIG. 2D is a diagram illustrating a kitting system according to various embodiments.

[0107] 2D , sensors 224 and 228 are disposed on carriage 214. In some embodiments, sensors 224 and 228 may be in a module fixed to carriage 214. In some embodiments, sensors 224 and / or 228 are integrated with carriage 214.

[0108] According to various embodiments, when sensor 224 and / or sensor 228 are disposed on / in carriage 214, the relative position of sensor 224 and / or sensor 228 with respect to robot arm 212 (e.g., at least the base of robot arm 212) is fixed. Additionally, threshold range 236 corresponding to sensor 224 and / or sensor 228 may be fixed with respect to the position of robot arm 212 (e.g., at least the base of robot arm 212). The fixed relative position of threshold range / threshold area 236 may improve the efficiency with which the robot arm may determine a plan or strategy for kitting or singulating items. For example, kitting system 200 (or singulation system) may store a predetermined routine for moving or otherwise attempting to scan an item through threshold range / threshold area 236. In some embodiments, a plan or strategy for moving an item to a destination location may include moving the item to a predetermined location (e.g., a location fixed relative to the robotic arm 212 or its base) and then executing a routine to move the item through a threshold range / threshold area 236 or otherwise attempt to scan the item with sensors 224 and / or 228. After the item is scanned, the robotic arm may move the item to a destination location (e.g., to a bin or a predetermined location on a conveyor). The predetermined routine may be based on a determined path or trajectory or set of paths or trajectories that correspond to an optimal path or trajectory, a path or trajectory that meets a threshold regarding the likelihood that information related to the item will be acquired by one or more sensors when the item passes through the threshold range / threshold area 236.

[0109] In some embodiments, the routine or manner in which an item is moved or redirected while moving through the threshold range / threshold area 236 may be determined based on how the item is grasped, the size of the item, the location of information on the object being acquired relative to the item, the location of the sensor relative to the robotic arm, etc.

[0110] The path or trajectory and / or the manner in which an item is moved or redirected while being moved to a destination location may be determined offline (e.g., by another computer system such as a web server) and / or may be determined online by the computer system controlling the robotic arm.

[0111] FIG. 3A is a flowchart illustrating a method for picking and placing an item according to various embodiments.

[0112] According to various embodiments, the process 300 of FIG. 3A is performed by the singulation system 100 of FIG. 1 and / or the kitting system 200 of FIGS. 2A-2C.

[0113] At step 310, sensor data about the workspace is acquired. In some embodiments, the robotic system acquires sensor data about the workspace from one or more sensors operating within the system. By way of example, the sensor data may be acquired based at least in part on output from an image sensor (e.g., a 2D or 3D camera), an infrared sensor array, a laser array, a scale, a gyroscope, a current sensor, a voltage sensor, a power sensor, a force sensor, a pressure sensor, or the like.

[0114] According to various embodiments, sensor data acquired about the workspace includes information from which a model of the workspace may be generated. For example, one or more characteristics associated with the workspace may be determined based at least in part on the sensor data. The sensor data may be utilized in connection with determining at least one characteristic (e.g., attribute) of one or more items in the workspace (e.g., an item at a source location, an item being grasped by a robotic arm, an item already placed at a destination location, a container identifier, etc.). In some embodiments, the sensor data is used in connection with determining one or more characteristics of a transport structure, such as determining empty or unscheduled containers or slots on a conveyor, determining a speed of the conveyor, and / or determining a characteristic of at least one container or slot or a characteristic of at least one item already on the conveyor.

[0115] At step 330, a plan or strategy is determined for picking and placing one or more items in the workspace. The plan or strategy for singulating or kitting the items in the workspace is determined based at least in part on the sensor data. In some embodiments, the robotic arm determines a plan or strategy for picking at least one item from a source location (e.g., a source pile or stream, a shelf on a kitting system, etc.) in the workspace and placing the at least one item at a destination location (e.g., a bin or slot on a conveyor, etc.). In various embodiments, the plan or strategy for picking and placing one or more items is determined for each robot when the robotic system includes multiple robots, such that each robot operates independently of the other robots.

[0116] According to various embodiments, a plan or strategy for singulating or kitting one or more items in the workspace is determined based at least in part on sensor data (e.g., sensor data including data related to items present in the workspace (e.g., items to be singulated and / or items already placed on a conveyor) or other objects included in the workspace). As one example, when determining a plan for kitting items, the plan or strategy may be determined based on a stored mapping of items (or types of items) to orders (e.g., packing slips) and / or source locations (e.g., shelves or bins on shelves in a kitting shelving system). As another example, when determining a plan for singulating items, the plan or strategy for singulating one or more items includes selecting items in the source pile / stream to be singulated.

[0117] The selected item may be identified from among other items or objects in the workspace (or other bins or trays of items on a kitting shelving system) based at least in part on the sensor data. As an example, one or more characteristics (or attributes) of the selected item may be determined based at least in part on the sensor data. The one or more characteristics of the selected item may include dimensions of the item, packaging of the item, one or more identifiers or labels on the item (e.g., an indication that the item is fragile, a shipping label on the item, etc.), height of the item, length of the item, estimated weight of the item, etc., or any combination thereof. As another example, planning for picking and placing one or more items includes determining a destination location on a transport structure (e.g., a bin on a conveyor, a slot on a conveyor, etc.) where a robotic structure (e.g., a robotic arm) will place the item. The destination location on the transport structure where the item is placed may be determined based at least in part on a timestamp, the speed of the conveyor, and one or more characteristics of a slot or receptacle on the conveyor (e.g., an indication of whether the slot or receptacle is occupied or reserved), or any combination thereof. As another example, a plan or strategy for singulating one or more items includes determining a path or trajectory of the item as a robotic arm moves the item during singulation. The path or trajectory of the item as it is moved may be determined at least in part based on the positions of one or more other objects in the workspace, the item on the conveyor, other robots operating in the workspace, airspace reserved for operation of other robots, sensors in the workspace, etc. For example, the path or trajectory of the item may be determined to move a portion of the item that includes an identifier (e.g., a shipping label) to an area where a scanner can scan the identifier, or the path or trajectory of the item may be determined to maximize the likelihood that the identifier on the item will be read by one or more scanners along the path or trajectory.

[0118] In some embodiments, determining a plan for picking and placing an item includes determining how an appropriate robotic arm will grasp the item. The plan for singulating the item may indicate a particular end effector to be used to pick up the item from a destination location and one or more settings associated with the end effector related to grasping the item (e.g., a grasping force, pressure to apply for a suction-based end effector, etc.). The plan for singulating the item may indicate an orientation of one or more of the robotic arm, robotic arm wrist, and / or end effector. In some embodiments, the end effector is positioned perpendicular to the item when grasping the item. To so position the end effector relative to the item, the robotic structure may be controlled to manipulate the orientation of one or more of the robotic arm, robotic arm wrist, and / or end effector.

[0119] At step 350, an item is picked from a source location and placed at a destination location. According to various embodiments, a robotic system (e.g., a control computer) controls a robotic arm to move the item from the source location to the destination location based at least in part on a plan or strategy. The robotic arm may move the item along a path or trajectory defined in the corresponding plan or strategy for picking and placing the item.

[0120] A determination is made at step 370 whether more items are to be picked and placed. If more items are present, further iterations of steps 310, 330, and 350 are performed, and so on, until step 370 determines that there are no more items to be picked and placed (e.g., in a singulation system chute, kitting system shelf, or other container or source).

[0121] FIG. 3B is a flowchart illustrating a method for picking and placing an item according to various embodiments.

[0122] According to various embodiments, process 350 of Figure 3B may correspond to step 350 of Figure 3A. Process 350 of Figure 3B may be performed by singulation system 100 of Figure 1 and / or kitting system 200 of Figures 2A-2C.

[0123] In step 351, an item is picked up by a robotic arm. In some embodiments, an end effector of the robotic arm is used in connection with gripping the item. The item may be picked up from a source location (such as a tray on a shelf of a kitting shelving system or a source stream / pile). The robotic arm may pick up the item based at least in part on a plan or strategy for picking and placing the item (e.g., based at least in part on a gripping strategy).

[0124] At step 356, the item is moved according to the current plan. The current plan may include a path along which the item will travel toward the destination location. According to various embodiments, the current plan corresponds to an initial plan (e.g., a plan determined when the item was determined to be picked and placed) or an updated plan (e.g., an updated version of the initial plan in connection with the implementation of one or more proactive measures, etc.). In some embodiments, the path is determined to enable one or more scanners to scan one or more identifiers on the item. As an example, moving the item along the path includes moving the item within a threshold range or area corresponding to one or more sensors. The threshold range / area (e.g., threshold range / area 236 of kitting system 200 in FIG. 2A ) may be predefined based at least in part on one or more sensors in the workspace. The one or more sensors acquire one or more identifiers from the item as the item is moved along the path. For example, an RFID sensor may acquire information from an RFID tag on the item as the item is moved along the path. As another example, a barcode or QR code on the surface of an item is scanned by a barcode reader or QR code reader as the item is moved along a path.

[0125] In step 361, a decision is made regarding whether to implement proactive measures. The decision to implement proactive measures may be made while the item is being moved along the path to the destination location. For example, the decision to implement proactive measures may be made based on an interruption being caused during the movement of the item.

[0126] According to various embodiments, the robotic system determines to perform proactive measures in response to determining that at least one identifier on an item was not acquired. Determining that at least one identifier was not scanned or acquired may be based at least in part on the number of identifiers acquired by one or more scanners and the expected number corresponding to the item. The number of identifiers for an item (or type of item) may be known and stored in advance. For example, a mapping of item types to numbers of identifiers on an item may be pre-stored. The robotic system may obtain the expected number of identifiers corresponding to the item based at least in part on performing a lookup in the mapping of item types to identifier numbers. In some embodiments, the robotic system compares the number of identifiers acquired from the item by one or more sensors to the expected number of identifiers. In response to determining that the number of identifiers acquired is less than the expected number of identifiers, the robotic system may determine that at least one identifier was not scanned or acquired. In some implementations, the robotic system may determine the number of identifiers not scanned or acquired by one or more sensors (e.g., based on comparing the number of acquired identifiers to the expected number of identifiers). The comparison may be performed after the item is moved through the threshold range / area (e.g., before the item is placed in the receptacle). In some embodiments, the robotic system determines the aspects of the item for which an identifier was not obtained based on a determination of the identifier (or type of identifier) ​​that was not scanned (e.g., based on a comparison of the expected identifier of the item with the obtained identifier).

[0127] In some embodiments, the robotic system determines to take proactive measures in response to determining that an identifier was not properly acquired (e.g., the identifier was partially scanned, etc.). If an item is moved too quickly across the line of sight or within a threshold range of one or more sensors, the one or more sensors may not properly capture the identifier on the image. In response to determining that at least one identifier was not properly captured (e.g., only part of a barcode or QR code was captured, or an RFID tag was recognized but complete information was obtained therefrom), the robotic system may determine to take proactive measures.

[0128] In some embodiments, the robotic system may determine that the grasped item is inappropriate. For example, the item may be a different type of item than the type of item specified in the plan (or the order from which the plan was generated / determined). In response to obtaining at least one identifier on the item, the robotic system may determine that the at least one identifier does not match an expected identifier or type of identifier based, at least in part, on the plan (or the order from which the plan was generated / determined). For example, in response to scanning the identifier on the item, the robotic system may determine that the model type of the item is incorrect. In response, the robotic system may determine that proactive measures are to be taken. The proactive measures may include returning the item to the kitting shelving system, returning the item to a location designated as a return area or a discard area, and / or alerting a human operator of the error.

[0129] The robotic system may determine that an active countermeasure be taken based, at least in part, on an area in which at least a portion of the robot arm is located or an area in which an item is located. The position of at least a portion of the robot arm or an item (e.g., an item being grasped by the robot arm) may be determined based on information obtained by one or more sensors or sensor arrays in the workspace. In some embodiments, the robotic system determines to take an active countermeasure in response to determining that the item or a portion of the robot arm (e.g., an end effector) is within a threshold range / threshold area or within a predetermined distance from the threshold range / threshold area. As an example, the active countermeasure may include rotating or otherwise reorienting the item (e.g., rotating or tilting the item while the item is being moved through the threshold range / threshold area).

[0130] In response to a determination that proactive measures are to be implemented, process 350 returns to step 356 where the plan is updated and items are moved based at least in part on the current plan (e.g., the updated plan). Alternatively, in response to a determination that proactive measures are not to be implemented, process 350 may end.

[0131] FIG. 3C is a flowchart illustrating a method for scanning an item according to various embodiments.

[0132] According to various embodiments, process 356a of Figure 3C may correspond to step 356a of Figure 3B. Process 356a of Figure 3C may be performed by singulation system 100 of Figure 1 and / or kitting system 200 of Figures 2A-2C.

[0133] In step 357a, a current plan is obtained. The plan may be determined for picking an item from a source location to a destination location. In some embodiments, in connection with an initial attempt to acquire one or more identifiers from the item, the current plan may include moving the item within a proximity of one or more sensors (e.g., within a threshold range / threshold area). According to various embodiments, in response to determining that an active measure will be implemented, according to various embodiments, the robotic system updates a plan (e.g., a plan for picking and placing the item). The plan may be updated based at least in part on the active measure (e.g., the type of active measure determined). After determining that an active measure will be implemented, the current plan may correspond to the updated plan.

[0134] For example, the plan may be updated to include retrying to obtain at least one identifier from the item. Retrying to obtain at least one identifier from the item may include moving the item within the vicinity of one or more scanners (e.g., moving the item through the threshold range / area one or more additional times). Accordingly, the plan may be updated to include instructions for so moving the item.

[0135] In step 358a, the item is moved based at least in part on the current plan. The robotic arm may be controlled to autonomously move the item according to the current plan. For example, in connection with an initial attempt to obtain one or more identifiers from the item, the item is moved along a path from a source location to a destination location, the path including moving the item near one or more scanners.

[0136] At step 359a, an item rotation process is initiated while the item is being moved through the threshold area. According to various embodiments, the item is rotated while being moved. For example, the item may be rotated along substantially the entire path from the source location to the destination location. As another example, the item may be rotated in response to a determination that the item is within the vicinity of one or more sensors (e.g., in response to a determination that the item is within a threshold range / threshold area).

[0137] At step 360a, the item rotation process is stopped and the item is further moved to the destination location. In some embodiments, the robotic system stops rotating the item after it is determined that the item is no longer in proximity to one or more sensors. For example, the robotic arm may be controlled to stop rotating the item when the item is within a threshold distance of the destination location. As another example, the robotic arm may be controlled to stop rotating the item in response to determining that the item has left a threshold range / area.

[0138] FIG. 3D is a flowchart illustrating a method for determining whether to implement proactive measures in connection with scanning an item, according to various embodiments.

[0139] According to various embodiments, process 356b of Figure 3D may correspond to step 356 of Figure 3B. Process 356b of Figure 3D may be performed by singulation system 100 of Figure 1 and / or kitting system 200 of Figures 2A-2C.

[0140] In step 357b, a current plan is obtained. The plan may be determined for picking the item from the source location to the destination location. In some embodiments, in connection with an initial attempt to acquire one or more identifiers from the item, the current plan may include moving the item within a proximity of one or more sensors (e.g., within a threshold range / threshold area). According to various embodiments, in response to determining that an aggressive measure will be implemented, according to various embodiments, the robotic system updates the plan (e.g., a plan for picking and placing the item). The plan may be updated based at least in part on the aggressive measure (e.g., the type of aggressive measure determined). After determining that an aggressive measure will be implemented, the current plan may correspond to the updated plan.

[0141] For example, the plan may be updated to include retrying to obtain at least one identifier from the item. Retrying to obtain at least one identifier from the item may include moving the item within the vicinity of one or more scanners (e.g., moving the item through the threshold range / area one or more additional times). Accordingly, the plan may be updated to include instructions for so moving the item.

[0142] The current plan may be obtained based at least in part on determining the likelihood that one or more identifiers on the item will be acquired if the item is moved along a corresponding path. For example, the current plan may be determined based at least in part on determining a pick-and-place method for the item that optimizes the likelihood that the identifiers will be read by sensors in the workspace. Determining the likelihood that one or more identifiers on the item will be acquired if the item is moved along a corresponding path may be based at least in part on determining a path that optimizes the likelihood that one or more labels or identifiers will be detected by one or more sensors. Determining a path that optimizes the likelihood that one or more labels or identifiers will be detected by one or more sensors may include determining a set of paths along which the item will be moved and corresponding probabilities that the labels or identifiers on the item will be scanned if the item is moved along the path (e.g., the probabilities may be based on the position of the item relative to the sensor or the position of a side of the item relative to the sensor, the distance of the item to the sensor, etc.), and selecting the path with the highest corresponding probability. In some embodiments, a path that exceeds a threshold probability or that falls within a predetermined percentage or number of paths with the highest corresponding probability among the set of paths is determined. In some embodiments, a path and a corresponding probability that a label or identifier on an item will be scanned if the item is moved along such a path are determined, and the probability is compared to a predetermined threshold for a minimum desired probability. In response to determining that the probability corresponding to a particular path does not exceed the predetermined threshold for the minimum desired probability, a new path and corresponding probability are determined. The robotic system may continue to iterate determining paths and corresponding probabilities until the robotic system determines that the corresponding probability exceeds the predetermined threshold for the minimum desired probability. Determining a path may include weighting different factors and maximizing a composite score for the weighted values ​​of the different elements.The different factors may include the likelihood of obtaining one or more identifiers, the likelihood of successfully picking and placing the item, the expected time to pick and place the item if the item is moved along the corresponding path, the presence / location of other objects in the workspace (e.g., another robotic arm, a container, another item, etc.), etc.

[0143] At step 358b, the item is moved based at least in part on the current plan. The robotic arm may be controlled to autonomously move the item according to the current plan. For example, in connection with an initial attempt to acquire one or more identifiers from the item, the item is moved along a path from a source location to a destination location, the path including moving the item within the vicinity of one or more scanners. As another example, in connection with performing proactive countermeasures, the item is moved within the vicinity of one or more scanners one or more additional times (e.g., the robotic arm moves the item back and forth through a threshold range / area).

[0144] FIG. 3E is a flowchart illustrating a method for determining whether to implement proactive measures in connection with scanning an item, according to various embodiments.

[0145] According to various embodiments, process 361 of Figure 3E may correspond to step 361 of Figure 3B. Process 361 of Figure 3B may be performed by singulation system 100 of Figure 1 and / or kitting system 200 of Figures 2A-2C.

[0146] In step 362, one or more identifiers scanned on the item are acquired. In some embodiments, the robotic system determines one or more identifiers acquired by one or more sensors (e.g., in the workspace). The acquired one or more identifiers may be stored in a data structure in which the acquired identifiers are mapped to the items. In some embodiments, acquiring one or more identifiers scanned on the item includes determining the number of identifiers (e.g., different identifiers) acquired by one or more sensors in association with scanning the item. Acquiring one or more identifiers scanned on the item may include acquiring one or more attributes associated with the one or more identifiers (e.g., type of identifier, such as model number, serial number, lot number, manufacturer identifier, etc.).

[0147] An expected number of identifiers to be scanned on the items is obtained in step 363. The robotic system may obtain the expected number of identifiers to be scanned based at least in part on the plan (e.g., based on the items to be picked and placed and / or the types of items that correspond to the items to be picked and placed).

[0148] In some embodiments, information indicating multiple identifiers on an item is stored in advance (e.g., before the item is kitted or singulated). For example, the robotic system may store a mapping of items or item types to multiple identifiers. The number of identifiers mapped to an item or item type may correspond to an expected number of identifiers. If the robotic system determines that the number of identifiers acquired (or information acquired) by one or more sensors is less than the expected number of identifiers for the item, the robotic system may determine that the information acquired by the one or more sensors is insufficient (e.g., information about at least one of the one or more identifiers was not captured).

[0149] In step 364, the system determines whether to take proactive measures based at least in part on the identifiers scanned on the items and the expected number of identifiers to be scanned on the items.

[0150] Various embodiments include performing an active countermeasure in response to determining that one or more sensors did not acquire information about at least one identifier. For example, in response to a robotic system determining that the number of identifiers acquired by one or more sensors (or identifiers for which information was acquired) is less than the expected number of identifiers for an item, the robotic system may perform an active countermeasure. The active countermeasure may include performing one or more additional attempts to scan the at least one identifier. For example, the active countermeasure may include moving the item through a threshold range (e.g., within the line of sight of at least one sensor). The active countermeasure may include changing the orientation of the item (e.g., one or more lateral orientations relative to one or more sensors). The robotic system may change the orientation before performing one or more additional attempts to have information corresponding to the at least one identifier acquired by one or more sensors. The orientation of the item may be changed while the robotic arm is moving. For example, the robotic arm may be controlled to rotate a wrist of the robotic arm. In some embodiments, the orientation of the item is changed while the robotic arm is moving through at least a portion of a threshold range of one or more sensors (e.g., while the item is within the line of sight of at least one sensor). Proactive measures may include repeatedly attempting to acquire at least one identifier (e.g., an identifier that was not acquired) until (i) all of the item's identifiers have been acquired (or a threshold number of identifiers have been acquired) and / or (ii) a threshold number of retries to acquire at least one identifier have been performed. As an example, retries to acquire at least one identifier may be performed repeatedly until either (i) or (ii) is satisfied.

[0151] FIG. 4 is a diagram illustrating a kitting system according to various embodiments.

[0152] 4, kitting system 400 includes a transport system that transports (e.g., distributes) totes to various kitting shelf machines. Totes (e.g., containers) may be input to the transport system at input area 405. Individual totes within the totes are distributed to areas corresponding to kitting shelf machines (e.g., kitting shelf machine 410). Multiple kitting shelf machines may be interconnected via the transport system.

[0153] In some embodiments, the transport structure terminates at such a kitting shelf machine. If the transport structure terminates at a kitting shelf machine, the totes filled with one or more items may be removed (or otherwise manually removed) from the kitting shelf machine, such as by a human operator, or removed from a different system.

[0154] In some embodiments, multiple kitting shelf machines are interconnected via a transport structure. For example, multiple kitting shelf machines may share a common input area 405 where totes (e.g., empty totes) enter the kitting system. As another example, multiple kitting shelf machines may share a common output area 455 (where totes are sent for packaging, labeling, and / or other processing, etc.) where totes (e.g., totes filled with one or more items) exit the kitting system. As another example, multiple kitting shelf machines may share the common output area 455 and the common input area 405. In some embodiments, a kitting shelf system may include multiple kitting shelf machines (e.g., kitting shelf machines positioned on either side of a conveyor). In some embodiments, each kitting shelf system may include one or more control computers that control multiple kitting shelf machines and corresponding robotic arms. In other embodiments, each kitting shelf machine may include its own control computer that controls the kitting shelf machine and corresponding robot. In other embodiments, kitting system 400 includes a control computer that controls multiple kitting shelf systems and / or multiple kitting shelf machines.

[0155] In some embodiments, the kitting system 400 may store and / or manage data structures relating to relationships between totes (e.g., trays) and orders or items. For example, a kitting shelf machine (or its associated controller) may reserve an empty tote for the kitting shelf machine to place an item that a corresponding robotic arm picks from the kitting shelf. In another example, in response to an item being placed in a tote, the data structure may be updated to include a mapping of a tote identifier to the order into which the item is being kitted and / or a mapping of a tote identifier to one or more items contained in the tote.

[0156] Kitting system 400 may include multiple robotic arms (e.g., robotic arm 415, robotic arm 425, and / or robotic arm 435). One or more of the multiple robotic arms may be connected to a base having a fixed position relative to kitting shelf machine 410. As another example, one or more of the robotic arms may be fixed to one or more carriages that each move on one or more rails to move at least one robotic arm relative to kitting shelf machine 410.

[0157] Kitting system 400 may include one or more sensors and / or one or more sensor arrays located along the transport structure or in or around multiple robotic arms (e.g., robotic arm 415, robotic arm 425, and robotic arm 435). For example, kitting system 400 may include one or more sensors in input area 405. As another example, kitting system 900 may include one or more sensors in output area 455. As another example, kitting system 400 may include sensors in an area along the transport structure where kitting machines are located. Sensors may be used in connection with associating totes with orders and / or items to be kitted (or kitted) within the totes. Sensors may be used in connection with determining one or more attributes associated with the totes and / or items within the totes (e.g., obtaining an identifier on an identifier and / or an identifier on one or more items, etc.).

[0158] The one or more sensors may be located within the workspace of the robotic arm (e.g., one or more sensors 410 located within a vicinity of the robotic arm 415, or one or more sensors 430 located within a vicinity of the robotic arm 425). If the one or more sensors are located in fixed locations relative to the kitting shelf machine 410, the computer system may determine a plan or strategy for moving items through threshold areas or threshold ranges corresponding to a set of one or more sensors, where the threshold areas or threshold ranges have fixed locations defined relative to the kitting shelf machine.

[0159] According to various embodiments, kitting system 400 includes one or more sensors located on or integrated into the base of the robotic arm or the carriage to which the robotic arm is attached. For example, sensor 445 and / or sensor 450 are located on or integrated into the base of robotic arm 435. When sensor 445 and / or sensor 450 are located on / in the base of robotic arm 435, the relative position of sensor 445 and / or sensor 450 to robot arm 435 (e.g., at least the base of robot arm 435) is fixed. Furthermore, the threshold ranges corresponding to sensor 445 and / or sensor 450 may be fixed with respect to the position of robotic arm 435 (e.g., at least the base of robot arm 435). The fixed relative positions of the threshold ranges / threshold areas may improve the efficiency with which the robotic arm may determine a plan or strategy for kitting or singulating items. For example, kitting system 400 (or singulation system) may store predetermined routines for moving or otherwise attempting to scan an item through threshold range / threshold area 436. In some embodiments, a plan or strategy for moving an item to a destination location may include moving the item to a predetermined position (e.g., a position that is fixed relative to robotic arm 435 or its base) and then executing a routine for moving the item through a threshold range / threshold area or otherwise attempting to scan the item with sensor 445 and / or sensor 450. After the item is scanned, the robotic arm may move the item to a destination location (e.g., to a receptacle or predetermined location on a conveyor).

[0160] Although the above-described embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. A system comprising: a communication interface; one or more processors connected to the communication interface; Equipped with the one or more processors: autonomously operating the robotic structure to move the item from a source location to a destination location along a predetermined path; the item comprises one or more identifiers; In response to determining that at least one of the one or more identifiers was not acquired by one or more sensors, proactive measures are taken to cause the one or more sensors to acquire the at least one identifier that was not acquired; the predetermined path corresponds to a path along which the item will be moved from the source location to the destination location; the predetermined path is planned such that the item moves within a threshold range of the one or more sensors while the item is moved along the predetermined path; a system configured to autonomously operate the robotic structure to place the item at the destination location based at least in part on the plan;

2. 10. The system of claim 1, The one or more processors further obtaining the plan for operating the robotic structure to move and place the item from the source location to the destination location; configured to autonomously operate the robotic structure to pick the item from the source location based at least in part on the plan; autonomously operating the robotic structure to move the item from the source location to the destination location along the predetermined path, determining whether the one or more sensors acquire the one or more identifiers of the item as the item is moved along the predetermined path.

3. 3. The system of claim 2, wherein the proactive measures include: autonomously operating the robotic structure to repeatedly move the item within the threshold range of the one or more sensors until either (i) all of the one or more identifiers of the item are acquired by the one or more sensors, or (ii) attempting to move the item within the range of the one or more sensors for a threshold number of iterations.

4. 3. The system of claim 2, wherein the proactive measures include: autonomously operating the robotic structure to again move the item within the thresholds of the one or more sensors; the robotic structure changes the orientation of the item relative to the one or more scanners; wherein the orientation of the item relative to the one or more scanners when the item is again moved within the threshold range of the one or more sensors is different from the orientation of the item when the item was moved within the threshold range of the one or more sensors during an initial attempt.

5. 3. The system of claim 2, wherein the proactive measures include: determining an aspect of the item that includes the at least one identifier that was not obtained; determining a reconfigured orientation of the item corresponding to an increased likelihood that the one or more sensors acquire the at least one identifier when the item is within the threshold range; autonomously operating the robotic structure to again move the item within the thresholds of the one or more sensors; The item is moved within the threshold range based at least in part on the reconstructed orientation.

6. 3. The system of claim 2, wherein the proactive measures include: determining an update plan for moving the item and placing the item at the destination location; the update plan includes an update path along which the item will be moved; and autonomously operating the robotic structure to move and place the items based at least in part on the update plan.

7. 7. The system of claim 6, wherein the update plan is determined based at least in part on the at least one identifier that was not acquired when the item was moved within the threshold range of the one or more sensors.

8. 7. The system of claim 6, wherein the update plan is determined based at least in part on aspects of the item that include the at least one identifier that was not acquired and on locations of the one or more sensors.

9. 10. The system of claim 1, the one or more sensors include a barcode reader; the item comprises one or more identifiers; wherein the one or more of the identifiers comprises at least one barcode.

10. 10. The system of claim 1, the one or more sensors include a radio frequency identification (RFID) sensor; the item comprises one or more identifiers; The system, wherein the one or more of the identifiers include an RFID tag.

11. 2. The system of claim 1, wherein autonomously operating the robotic structure to move the item from the source location to the destination location along a predetermined path comprises: and autonomously operating the robotic structure to rotate the item at least when the item is being moved within a threshold range of one or more sensors.

12. 10. The system of claim 1, wherein the threshold range of the one or more sensors corresponds to a field of view of the one or more sensors.

13. The system of claim 1 , wherein at least one of the one or more sensors is located in a fixed position relative to the robotic structure.

14. 14. The system of claim 13, wherein the at least one of the one or more sensors is located in a fixed position relative to a base of the robotic structure.

15. 14. The system of claim 13, wherein the at least one of the one or more sensors is located in a fixed position relative to a carriage on which the robotic structure is mounted.

16. 10. The system of claim 1, further comprising: obtaining the plan for operating the robotic structure to move and place the item from the source location to the destination location; The obtaining includes: determining the plan based at least in part on a position of at least one of the one or more sensors.

17. 10. The system of claim 1, further comprising: obtaining the plan for operating the robotic structure to move and place the item from the source location to the destination location; At least one of the one or more sensors is located in a fixed position relative to the robot structure; The system, wherein the plan includes executing a predetermined routine related to moving the item through the at least one threshold area or threshold range of the one or more sensors.

18. 1. A method comprising: executing, by one or more processors, a plan for moving and placing items from a source location to a destination location; Implementing the plan comprises: autonomously operating a robotic structure to move the item from the source location to the destination location along a predetermined path; the item comprises one or more identifiers; responsive to determining that at least one of the one or more identifiers was not acquired by one or more sensors, performing proactive measures to cause the one or more sensors to acquire the at least one identifier that was not acquired; the predetermined path corresponds to a path along which the item will be moved from the source location to the destination location; the predetermined path is configured such that the item is moved within a threshold range of the one or more sensors while the item is moved along the predetermined path; autonomously operating the robotic structure to place the item at the destination location based at least in part on the plan; A method comprising:

19. 20. The method of claim 18, further comprising: obtaining the plan for operating the robotic structure to move and place the item from the source location to the destination location; autonomously operating the robotic structure to pick the item from the source location based at least in part on the plan; Equipped with the item comprises one or more identifiers; autonomously operating the robotic structure to move the item from the source location to the destination location along the predetermined path; determining whether the one or more sensors acquire the one or more identifiers of the item as the item is moved along the predetermined path.

20. 20. The method of claim 19, wherein performing the proactive measures comprises: autonomously operating the robotic structure to again move the item within the thresholds of the one or more sensors; the robotic structure changes the orientation of the item relative to the one or more scanners; wherein the orientation of the item relative to the one or more scanners when the item is again moved within the threshold range of the one or more sensors is different from the orientation of the item when the item was moved within the threshold range of the one or more sensors during an initial attempt.

21. 20. The method of claim 19, wherein performing the proactive measures comprises: determining an aspect of the item that includes the at least one identifier that was not obtained; determining a reconfigured orientation of the item corresponding to an increased likelihood that the one or more sensors acquire the at least one identifier when the item is within the threshold range; autonomously operating the robotic structure to again move the item within the threshold of the one or more sensors; Equipped with The item is moved within the threshold range based at least in part on the reconstructed orientation.

22. 20. The method of claim 19, wherein performing the proactive measures comprises: determining an update plan for moving the item and placing the item at the destination location, the update plan including an update path along which the item will be moved; autonomously operating the robotic structure to move and place the items based at least in part on the update plan; A method comprising:

23. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for executing, by one or more processors, a plan for moving and placing items from a source location to a destination location; Implementing the plan comprises: autonomously operating a robotic structure to move the item from the source location to the destination location along a predetermined path; the item comprises one or more identifiers; responsive to determining that at least one of the one or more identifiers was not acquired by one or more sensors, performing proactive measures to cause the one or more sensors to acquire the at least one identifier that was not acquired; the predetermined path corresponds to a path along which the item will be moved from the source location to the destination location; the predetermined path is configured such that the item moves within a threshold range of the one or more sensors while the item is moved along the predetermined path; and autonomously operating the robotic structure to place the item at the destination location based at least in part on the plan.

24. 24. The computer program product of claim 23, further comprising: obtaining the plan for operating the robotic structure to move and place the item from the source location to the destination location; autonomously operating the robotic structure to pick the item from the source location based at least in part on the plan; the item comprises one or more identifiers; autonomously operating the robotic structure to move the item from the source location to the destination location along the predetermined path, determining whether the one or more sensors acquire the one or more identifiers of the item as the item is moved along the predetermined path.

25. 25. The computer program product of claim 24, wherein performing the proactive measures comprises: autonomously operating the robotic structure to again move the item within the thresholds of the one or more sensors; the robotic structure changes the orientation of the item relative to the one or more scanners; an orientation of the item relative to the one or more scanners when the item is again moved within the threshold range of the one or more sensors that is different from the orientation of the item when the item was moved within the threshold range of the one or more sensors during a first attempt.

26. 25. The computer program product of claim 24, wherein performing the proactive measures comprises: determining an aspect of the item that includes the at least one identifier that was not obtained; determining a reconfigured orientation of the item corresponding to an increased likelihood that the one or more sensors acquire the at least one identifier when the item is within the threshold range; autonomously operating the robotic structure to again move the item within the thresholds of the one or more sensors; The item is moved within the threshold range based at least in part on the reconfigured orientation.

27. 25. The computer program product of claim 24, wherein performing the proactive measures comprises: determining an update plan for moving the item and placing the item at the destination location; the update plan includes an update path along which the item will be moved; and autonomously operating the robotic structure to move and place the items based at least in part on the update plan.

Images