Robotic handling of soft products in non-rigid packaging

Abstract

Techniques are disclosed for performing robotic handling of soft goods products in non-rigid packaging. In various embodiments, sensor data associated with a workspace is received. An action to be performed in the workspace using one or more robotic elements is determined, the action including moving an end effector of one of the robotic elements relatively quickly to a position proximate to an article to be grasped; actuating a grasping mechanism of the end effector to grasp the article using an amount of force and structure associated with a minimized risk of damage to the article and one or both of its packaging; and using sensor data generated after the article has been grasped to ensure that the article has been securely grasped. Control communications are sent to the robotic elements via a communications interface to cause the robotic elements to perform the action.

Description

[0001] Cross-references to other applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 809,398, filed February 22, 2019, entitled “ROBOTIC HANDLING OF SOFTPRODUCTS IN NON-RIGID PACKAGING,” which is incorporated herein by reference for all purposes. Technical Field

[0003] This application relates to the robotic handling of soft products in non-rigid packaging. Background Art

[0004] Shipping and distribution centers, warehouses, shipping docks, air cargo terminals, big box stores, and other operations that ship and receive non-homogeneous collections of items use strategies such as packing and unpacking dissimilar items in boxes, crates, containers, conveyor belts, pallets, etc. Packing dissimilar items in boxes, crates, pallets, etc. enables the resulting collection of items to be handled by heavy lifting equipment such as forklifts, cranes, etc., and enables more efficient packaging of items for storage (e.g., in a warehouse) and / or shipping (e.g., in a truck, cargo hold, etc.).

[0005] In some scenarios, items may be so dissimilar in size, weight, density, volume, stiffness, packaging strength, etc. that any given item or collection of items may or may not have attributes that would enable those items to support the size, weight, weight distribution, etc. of given other items that may need to be packaged (e.g., in boxes, containers, pallets, etc.) When assembling a pallet or other collection of dissimilar items, the items must be carefully selected and stacked to ensure that the palletized stack does not collapse, tilt, or otherwise become unstable (e.g., so that it cannot be handled by equipment such as a forklift) and to avoid damage to the items.

[0006] Some products may be subject to being crushed or otherwise damaged and / or may be in flexible packaging. For example, bread and bread-like products may be packaged in plastic bags or similar packaging. A robotic gripper can easily crush the bag and / or the product inside. Even a slightly crushed product or severely crumpled packaging can render the item less than desirable to the consumer.

[0007] Currently, pallets are typically stacked and / or unpacked by hand. Human workers select items for stacking based on, for example, shipping invoices or manifests, and use human judgment and intuition, such as selecting larger and heavier items for placement on the bottom. However, in some cases, items are simply delivered via conveyors or other mechanisms and / or selected from bins in the order listed, resulting in unstable pallets or otherwise packaged collections.

[0008] In the case of bread and other products sold from retail locations, it may be necessary to distribute to stores in odd batches, for example, with a specific mix and quantity of each item ordered by each respective store. Bread products are typically hand-packed in stackable pallets, loaded into trucks, and delivered to stores by bringing the pallets into the stores and loading the products onto shelves.

[0009] In many environments, the use of robots is made more challenging by the variety of items, such as the varying order, quantity, and mix of items to be packed on a given pallet, and the various types and locations of totes and / or feed mechanisms from which items must be picked for placement on pallets or other totes. In the case of bread and similar products, robotic handling is further challenged by the need to pick and place arbitrary collections of fragile product items in flexible packaging, such as plastic bags. Summary of the Invention

[0010] In a first aspect of the present invention, a robotic system is provided, comprising: a communication interface; and a processor coupled to the communication interface and configured to: receive sensor data associated with a workspace via the communication interface, in which there are one or more robotic elements at least partially controlled by the robotic system, wherein the sensor data is received from a plurality of different types of sensors, the plurality of different types of sensors including at least a torque sensor for detecting whether an item has been grasped close to the center of mass and a pressure sensor for detecting air leaks that impair the grasp; determine, based at least in part on the sensor data, an action to be performed in the workspace using the one or more robotic elements, the action comprising relatively quickly moving an end effector of one of the robotic elements to a grasping position. actuating a gripping mechanism of the end effector to grasp the item using an amount of force and structure associated with minimizing risk of damage to one or both of the item and its packaging; and while the item is being moved after being successfully grasped using the end effector, evaluating sensor data from each sensor type to determine whether the data is consistent with a continued secure grasp of the item, wherein if a threshold number of sensor types provide data indicating that the grasp is no longer secure, the item is moved to its original source location if the robotic system determines that the grasp is secure enough and the original source location is close enough to do so, otherwise the item is moved to a buffered location; and sending control communications to the robotic element via the communication interface to cause the robotic element to perform the actions.

[0011] According to an example, the processor is further configured to move the grasped item to a destination location using the robotic element.

[0012] According to an example, the end effector includes a suction-based gripping mechanism.

[0013] According to an example, the end effector comprises one or more suction cups.

[0014] According to an example, the one or more suction cups comprise a flexible material that gives way when in contact with an object.

[0015] According to an example, the sensor data includes image data generated by one or more cameras.

[0016] According to an example, the processor is further configured to move the grasped item to a destination location using the robotic element and place the item at the destination location.

[0017] According to an example, the processor is configured to place the item at least in part by: in a first operational phase, moving the item to a position proximate to a destination location, and using sensor data to adjust the position of the item to be more closely adjacent to a structure or a second item adjacent to the destination location.

[0018] According to an example, the processor is configured to determine the action based at least in part on the plan.

[0019] According to an example, the processor is configured to generate the plan based on a high-level goal.

[0020] According to an example, the high-level goal includes identification of a plurality of items and a corresponding quantity for each item to be included in a set of items associated with a final destination.

[0021] According to an example, the items include different bread products, and the high-level goal includes an order and the quantity of each item to be shipped to a retail store.

[0022] According to an example, the processor is further configured to move the grasped item to a destination location using the robotic element and place the item at the destination location, including verifying the placement of the item before releasing the item from the grasping mechanism of the end effector by using subsequently received sensor information.

[0023] According to an example, the processor uses sensor information to verify placement of the item prior to releasing the item from a gripping mechanism of an end effector at least in part by using sensor data to determine a height of the item above a height reference and comparing the determined value to an expected value associated with a successful placement.

[0024] According to an example, the end effector includes a plurality of independently actuated sets of one or more suction cups, and the processor is further configured to control the robotic element to simultaneously grasp two or more items, each item using one or more suction cups from the set of independently actuated one or more suction cups.

[0025] According to an example, the processor is further configured to determine an optimal plan for picking and placing multiple items, including by determining possible combinations of two or more items that can be grasped simultaneously, and determining an optimal plan that includes a mix of single-grasp and multi-grasp pick and place actions.

[0026] According to an example, the processor is further configured to determine a strategy for grasping the item using the robotic element.

[0027] In a second aspect of the present invention, a method for controlling a robotic system is provided, comprising: receiving sensor data associated with a workspace via a communication interface, in which workspace there are one or more robotic elements at least partially controlled by the robotic system, wherein the sensor data is received from a plurality of different types of sensors, the plurality of different types of sensors including at least a torque sensor for detecting whether an item has been grasped close to the center of mass and a pressure sensor for detecting air leaks that weaken the grasp; determining, at least in part based on the sensor data, an action to be performed in the workspace using the one or more robotic elements, the action comprising relatively quickly moving an end effector of one of the robotic elements to a position close to the item to be grasped. actuating a gripping mechanism of the end effector to grasp the article using an amount of force and structure associated with minimizing risk of damage to one or both of the article and its packaging; and while the article is being moved after being successfully grasped using the end effector, evaluating sensor data from each sensor type to determine whether the data is consistent with a continued secure grasp of the article, wherein if a threshold number of sensor types provide data indicating that the grasp is no longer secure, the article is moved to its original source location if the robotic system determines that the grasp is secure enough and the original source location is close enough to do so, otherwise the article is moved to a buffered location; and sending control communications to the robotic element via the communication interface to cause the robotic element to perform the actions.

[0028] In a third aspect of the application, there is provided a computer program product, the computer program product being embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: receiving, via a communication interface, sensor data associated with a workspace in which there is one or more robotic elements controlled at least in part by a robotic system, wherein the sensor data is received from a plurality of different types of sensors, the plurality of different types of sensors including at least a torque sensor for checking whether an item has been grasped proximate to a center of mass and a pressure sensor for checking for air leaks that weaken a grasp; determining, based at least in part on the sensor data, an action to be performed in the workspace using the one or more robotic elements, the action including moving an end effector of one of the robotic elements relatively quickly to a position proximate to an item to be grasped; actuating a grasping mechanism of the end effector to grasp the item using an amount of force and structure associated with a minimized risk of damage to the item and one or both of its packaging; and while the item is being moved after being successfully grasped using the end effector, evaluating sensor data of each sensor type to determine whether the data is consistent with a continued secure grasp of the item, wherein if a threshold number of sensor types provide data indicating that the grasp is no longer secure, the item is moved to its original source location if the robotic system determines that the grasp is secure enough and the original source location is close enough to do so, otherwise the item is moved to a buffer location; and sending, via the communication interface, control communications to the robotic elements to cause the robotic elements to perform the action. BRIEF DESCRIPTION OF DRAWINGS

[0029] Various embodiments of the application are disclosed in the following detailed description and the accompanying drawings.

[0030] Figure 1 is a block diagram illustrating an embodiment of a robotic system that handles soft goods products in non-rigid packaging.

[0031] Figure 2 is a block diagram illustrating an embodiment of a system that controls a robotic system.

[0032] Figure 3 is a flowchart illustrating an embodiment of a process that controls a robotic system.

[0033] Figure 4 is a flowchart illustrating an embodiment of a process that determines a plan for performing work using a robot.

[0034] Figure 5 is a flowchart illustrating an embodiment of a process that uses a robot to pick / place items according to a plan.

[0035] Figure 6A is a schematic diagram illustrating an embodiment of an end effector that includes a plurality of suction cups.

[0036] Figure 6B FIG. 6A is a schematic diagram illustrating an embodiment of an end effector including a plurality of suction cups.

[0037] FIG. 6C is a schematic diagram illustrating an embodiment of an end effector including a plurality of suction cups.

[0038] Figure 7 FIG. 7 is a flowchart illustrating an embodiment of a process of grasping an item with a robotic arm and an end effector.

[0039] Figure 8 FIG. 8 is a flowchart illustrating an embodiment of a process of safely moving an item that has been grasped by a robotic arm and / or an end effector to a destination.

[0040] Figure 9 FIG. 9 is a block diagram illustrating an embodiment of a suction-based end effector.

[0041] Figure 10 FIG. 10 is a flowchart illustrating an embodiment of a process of picking / placing items simultaneously as needed / if needed to best achieve goals assigned to a robotic system.

[0042] Figure 11 FIG. 11 is a flowchart illustrating an embodiment of a process of adjusting a final placement of an item.

[0043] Figure 12 FIG. 12 is a flowchart illustrating an embodiment of a process of selecting a destination location for an item.

[0044] Figure 13 FIG. 13 is a flowchart illustrating an embodiment of a process of detecting misplacement of an item.

[0045] Figure 14 FIG. 14 is a schematic diagram illustrating an embodiment of a robotic system that carries soft products in non-rigid packaging. DETAILED DESCRIPTION

[0046] The application can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product on a computer readable storage medium; and / or a processor, such as a processor configured to fetch and execute instructions stored on a memory coupled to the processor and / or provided by the memory. In this specification, these implementations, or any other form that the application can take, can be referred to as techniques. In general, the order of the steps of disclosed processes can be altered, unless otherwise specified. Unless otherwise stated, components that are described as being configured to perform a task can be implemented as o general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term 'processor' refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0047] A detailed description of one or more embodiments of the application is provided below along with accompanying figures that illustrate the principles of the application. The application is described in connection with such embodiments, but the application is not limited to any embodiment. The scope of the application is limited only by the claims and the application encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the application. These details are provided for the purpose of example and the application can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the application has not been described in detail so that the application is not unnecessarily obscured.

[0048] Techniques are disclosed for programmatically using a robotic system that includes one or more robots (e.g., robotic arms with suction and / or grippers at an operating end) to palletize / de-palletize and / or to otherwise package and / or unpack arbitrary collections of non-homogeneous items (e.g., items that differ in size, shape, weight, weight distribution, hardness, fragility, etc.), including items such as bread and similar products that are sold in plastic bags or similar packaging.

[0049] In various embodiments, 3D cameras, force sensors, and other sensors are used to detect and determine properties of items to be picked up and / or placed. Items whose type is determined (e.g., with sufficient confidence, e.g., as indicated by a confidence score determined by a program) can be grasped and placed using strategies derived from models specific to the item type. Items that cannot be identified are picked up and placed using strategies that are not specific to a given item type. For example, models that make use of size, shape, and weight information can be used.

[0050] In various embodiments, a library of item types, corresponding attributes, and gripping strategies is used to determine and implement a strategy for picking and placing each item. In some embodiments, the library is dynamic. For example, the library can be expanded to add newly encountered items and / or additional attributes learned about items, such as gripping strategies that work or don't work in a given scenario. In some embodiments, human intervention can be invoked if the robotic system gets stuck. The robotic system can be configured to observe (e.g., using sensors) and learn (e.g., updating the library and / or model) based on how a human teleoperator intervenes, thereby using the robotic arm to pick and place items, for example, via teleoperation.

[0051] In some embodiments, a robotic system as disclosed herein can engage in a process of collecting and storing (e.g., adding to a library) attributes and / or strategies for identifying and picking / placing unknown or newly discovered types of items. For example, the system can hold items at various angles and / or positions to enable a 3D camera and / or other sensor to generate sensor data to augment and / or create library entries characterizing the item type and store a model for how to identify and pick / place items of that type.

[0052] In various embodiments, a high-level plan is created for picking and placing items on a pallet or other container or location. For example, a strategy is applied based on sensor information to implement the plan by picking and placing individual items according to the plan. Deviations from the plan and / or re-planning may be triggered, for example, if the robotic system becomes stuck, if the stacking of items on the pallet is detected to have become unstable (e.g., via computer vision, force sensors, etc.), etc. A partial plan can be formed based on known information (e.g., the next N items visible on the conveyor, the next N items on the invoice or manifest, etc.) and adjusted as additional information is received (e.g., the next item visible, etc.). In some cases, the system is configured to stage (e.g., set aside, within reach) items that are determined to be unlikely to fit in the layer or container currently being built by the system (such as a lower (or upper) layer in a pallet or container, etc.). Once more information about the next items to be picked / placed is known, a strategy is generated and implemented that takes into account the staged (buffered) items and the next items.

[0053] In the case of bread and similar products in plastic bags or similar packaging, in various embodiments, the robotic system disclosed herein utilizes a robotic arm with a suction-type gripper at the operating end. For example, in some embodiments, a gripper including two large suction cups is used. In other embodiments, a single suction cup or more than two suction cups may be included. In various embodiments, bread is picked from a conveyor, bin, shelf, etc. and placed in a large, stackable plastic tray for distribution. Invoices for each destination can be used to pick and place items.

[0054] In various embodiments, algorithms, heuristics, and other programming techniques are used to generate a plan for packaging bread items. For example, items of the same type, size, shape, etc., can be placed together for efficient packaging. In some embodiments, the planner module can attempt to add items to a tray in a manner that minimizes gaps, perimeter (e.g., the sum of edges not adjacent to other products and / or a side of the tray), etc.

[0055] In various embodiments, a robotic system for picking and placing bread and similar items as disclosed herein may be used, at least in part, in Figures 1 to 14 Graphic and / or combined Figures 1 to 14 Some or all of the techniques described may be implemented.

[0056] Figure 1 is a block diagram illustrating an embodiment of a robotic system for handling soft products in non-rigid packaging. In the illustrated example, the system 100 includes a robotic arm 102 rotatably mounted on a carriage 104, which is configured to translate along a track 106, for example, under computer control. In this example, the robotic arm 102 is movably mounted on the carriage 104 and the track 106, but in various other embodiments, the robotic arm 102 can be stationary, or can be fully or partially mobile rather than via translation along a track, such as being mounted on a turntable, fully mobile on a motorized chassis, etc.

[0057] In the example shown, the robotic arm 102 has an end effector 108 at its distal operative end (farthest from the carriage 104). The end effector 108 includes a flexible vacuum (or "suction") cup 110. In various embodiments, the suction cup 110 comprises silicone or another natural or synthetic material that is durable but also flexible enough to at least slightly "give way" when they come into (first and / or gently) contact with an item (such as bread or other soft and / or fragile item in a non-rigid packaging such as a plastic bag or outer packaging) that the robotic system 100 is attempting to grasp (e.g., via suction) using the robotic arm 102 and the end effector 108.

[0058] In this example, the end effector 108 has a camera 112 mounted on the side of the end effector 108. In other embodiments, the camera 112 may be more centrally located, such as on a downwardly facing surface of the body of the end effector 108 (on the Figure 1). Additional cameras may be mounted elsewhere on the robotic arm 102 and / or end effector 108, for example, on an arm segment that comprises the robotic arm 102. Additionally, cameras 114 and 116, which are mounted on the wall in this example, provide additional image data that can be used to construct a 3D view of the scene in which the system 100 is positioned and configured to operate.

[0059] In various embodiments, the robotic arm 102 is used to position the suction cup 110 of the end effector 108 over the item to be picked (as shown), and the vacuum source provides suction to grasp the item, lift it from its source location, and place it at the destination location.

[0060] exist Figure 1 In the example shown in , the robotic arm 102 is configured to pick arbitrary, and in this example, disparate items, such as bread items picked from a pallet of items received from a bakery, from a source pallet 120 and place them on a destination pallet 118. In the example shown, the destination pallet 118 and the source pallet 120 comprise stackable pallets that are configured to be stacked on a base having wheels at the four corners. In some embodiments, the pallets 118 and / or 120 can be pushed into position by a human worker. In some embodiments, the pallets 118 and / or 120 can be stacked on a motorized and / or robotically controlled base that is configured to move bin stacks to a location from which they are to be picked up and / or placed, and / or to move completed pallet stacks to be loaded onto a delivery vehicle for transport to a staging and / or shipping area, such as a retail location. In some embodiments, Figure 1 Other robots not shown in the figures may be used to push the pallets 118 and / or 120 into position for loading / unloading and / or into a truck or other destination for transport, etc.

[0061] In various embodiments, 3D or other image data generated by one or more of cameras 112, 114, and 116 can be used to generate a 3D view of the work area of ​​system 100 and items within the work area. The 3D image data can be used to identify items to be picked up / placed, such as by color, shape, or other attributes. In various embodiments, one or more of cameras 112, 114, and 116 can be used to read text, logos, photos, drawings, images, markings, barcodes, QR codes, or other coded and / or graphical information or content visible on the work area of ​​system 100 and / or including items within the work area of ​​system 100.

[0062] Further references Figure 1In the example shown, the system 100 includes a control computer 122 that is configured to communicate with the robotic arm 102, the carriage 104, the actuator 108, and sensors (such as cameras 112, 114, and 116 and / or weight, force, and / or Figure 1 In various embodiments, the control computer 122 is configured to use information from sensors such as cameras 112, 114, and 116 and / or weight, force, and / or Figure 1 The control computer 122 utilizes input from other sensors (not shown) to view, identify, and determine one or more attributes of items to be loaded into the pallet 120 and / or unloaded from the pallet 120 to the pallet 118. In various embodiments, the control computer 122 uses item model data stored on the control computer 122 and / or in a library accessible by the control computer 122 to identify the item and / or its attributes, for example, based on images and / or other sensor data. The control computer 122 uses the model corresponding to the item to determine and implement a plan for stacking the item along with other items in / on the destination, such as the pallet 118. In various embodiments, the item attributes and / or model are used to determine a strategy for grasping, moving, and placing the item in a destination location (e.g., a determined location where the item is determined to be placed as part of the planning / replanning process for stacking the item in / on the pallet 118).

[0063] In the example shown, the control computer 122 is connected to an “on-demand” teleoperation device 124. In some embodiments, if the control computer 122 is unable to proceed in a fully automated mode, e.g., is unable to determine a strategy for picking, moving, and placing an item and / or fails in a manner such that the control computer 122 does not have a strategy for completing the pick and place of an item in a fully automated mode, the control computer 122 prompts the human user 126 to intervene, e.g., by using the teleoperation device 124 to operate one or more of the robotic arm 102, carriage 104, and / or end effector 108 to pick, move, and place the item.

[0064] Figure 2 is a block diagram illustrating an embodiment of a system for controlling a robotic system. In various embodiments, Figure 1 The control computer 122 is as follows Figure 2 In the example shown, the control computer 122 includes a communication interface 202 configured to send and receive one or more of wireless, network, and other communications between the control computer 122 and other elements comprising the robotic system, such as Figure 1one or more of the robot arms 102, carriages 104, end effectors 108, cameras 112, 114, and 116, and / or robot bases 118 and / or 120 in the system 100.

[0065] In the illustrated example, the control computer 122 includes a planning module 204. The planning module 204 receives invoices, inventory, space / location information (e.g., locations and status of robotic elements, sensors, inventory, empty totes / cabinets, etc.). The planning module 204 uses this information to determine and / or recalculate plans for picking / placing items from source locations to destination locations, e.g., to fulfill one or more high-level requirements. For example, in the case of a robotic system to fulfill orders to deliver bread or other baked goods to retail stores— each according to an invoice or manifest indicating the bread items to include for each order / retail store— in some embodiments, the planning module 204 determines sequences of operations to pick bread items from source totes to fill destination totes according to each order, and cause the totes associated with each order or set of orders to be loaded onto one or more delivery vehicles for transport to the corresponding retail stores. All or some of the information received by the planning module 206 and / or the plans generated by the planning module 206 can be stored in an order / inventory database 206.

[0066] In some embodiments, the planning module 204 can be configured to develop and / or re-evaluate plans on a rolling and / or periodic basis. For example, images or other sensor data can be received and processed to generate updated three-dimensional views of the workspace scenario and / or to update information about which inventory items are located in which source locations within the workspace. In the example of bread, as new totes or stacks of totes arrive at the workspace, e.g., from a bakery, storage room, etc., the items are identified, and plans are generated, updated, and / or confirmed to make progress toward achieving the high-level goal (e.g., efficiently fulfilling and transporting all orders), such as by picking available inventory from current source locations, and moving / placing the items in filled totes to fulfill respective individual orders.

[0067] Further reference is made to Figure 2 In the illustrated example, the control computer 122 further includes a robot control module 208. In various embodiments, the robot control module 208 receives and processes sensor data from a sensor module 210 (e.g., image data from cameras, force and / or torque sensor signals, vacuum / pressure readings, etc.). The sensor module 210 receives input from one or more sensors, buffers the data, and provides the input to the robot control module 208 in a form usable by the robot control module 208 to control the controlled robotic system (e.g., the robot arm 102, carriage 104, end effector 108, etc.). Figure 1operable elements of system 100) to perform tasks associated with implementation of the plan received from planning module 204. In various embodiments, the operable elements include: one or more of each sensor and / or actuator, including robot arms and / or end effectors, such as Figure 1 robot arm 102, carriage 104, and end effectors 108 (e.g., motors for moving arm segments, valves or other elements for applying vacuum to suction cups, etc.) of system 100; other sensors in the workspace, such as Figure 1 cameras 114 and 116 of system 100; and other robots and / or elements that are manipulated or moved by robots, such as Figure 1 pallet stacks 118 and 120 of system 100.

[0068] Robot control module 208 uses robot model(s) 210 of the robot(s), which are configured to control movement and other actuation of each robot. For example, Figure 1 models of robot arm 102 and carriage 104 in system 100 can be used to determine ways to efficiently and relatively smoothly actuate joint motors of arm 102 to rotate arm 102 on carriage 104, and / or move carriage 104 along track 106, to pick up and place items from a source location to a destination location, e.g., according to a plan received from planning module 204, without colliding with humans, other robot elements, or other obstacles present in the workspace.

[0069] In various embodiments, robot control module 208 maintains and updates a collection of item models 212 and / or grasping strategies 214. Item models 212 in various embodiments describe attributes of items and / or categories or types of items to be picked / placed, such as size, shape, dimensions, weight, hardness, type of packaging, markings or text on packaging, etc. In some embodiments, item models 212 can be used to identify items within the workspace, e.g., based on images, weight, optical code scanning, and / or other information.

[0070] In various embodiments, robot control module 208 uses grasping strategies 214 (e.g., based on determined items or item types, size, shape, weight, etc., and situational information, such as the item’s location and orientation, what it is adjacent to, etc.) to determine strategies for grasping items to be moved (e.g., from a source pallet to a destination pallet).

[0071] In various embodiments, the robot control module 208 can update one or both of the item model(s) 212 and the grasp strategies 214 over time. For example, in some embodiments, machine learning or other techniques can be used to learn new and / or better grasp strategies over time. For example, grasp strategies that frequently fail can be de-rated and / or removed, and new strategies developed and learned to replace them. In some embodiments, new strategies can be learned by observing how human operators perform grasps under teleoperation. The grasp strategies 214 can be specific to items, collections of items, and / or item type(s). Some grasp strategies 214 can be associated with the size, shape, dimensions, weight, etc. of unidentified items.

[0072] The term“grasp” is used in this disclosure to refer to any mechanism by which an end effector obtains sufficient control of an item to be able to move the item, such as by grasping the item in the case of an end effector of the gripper type, or by the vacuum / suction of an end effector such as the end effector 108 of Figure 1 The techniques disclosed herein can be used with any type of end effector.

[0073] In various embodiments, the robot control module 208 and / or the sensor module 210 receive and process image data to generate a three-dimensional view of a workspace associated with the robotic system controlled by the control computer 122 and the items present in the workspace. In various embodiments, the robot control module 208 and / or the sensor module 210 can be configured to control sensors, such as to control the position and orientation of cameras present in the workspace, to generate a more complete view of the workspace and the items in the workspace.

[0074] In various embodiments, the control computer 122 can be configured to control more than one robotic system in more than one physical location and / or workspace, simultaneously, through time sharing, etc. In some embodiments, Figure 2 Elements shown in FIG. 1 as including a single physical computer 122 can be dispersed across multiple physical or virtual machines, cabinets, etc.

[0075] In various embodiments, one or more of the planning module 204, the robot control module 208, and the sensor module 210 can include dedicated hardware (e.g., ASICs, programmed FPGAs, etc.) and / or one or both of functional modules provided by execution of software instructions on a hardware processor included in the control computer 122.

[0076] Figure 3 is a flowchart illustrating an embodiment of a process of controlling a robotic system. In various embodiments, Figure 3The process 300 may be performed by a computer, such as Figure 1 122. In the example shown, at 302, input data indicating a high-level goal, such as items to be included in each of one or more individual orders, is received. At 304, a plan for fulfilling the one or more orders is determined and / or updated. At 306, a strategy for grabbing, moving, and placing items according to the plan is determined. At 308, human intervention via teleoperation is invoked if and as long as necessary, for example, to perform or complete tasks that cannot be completed in a fully automated mode. Processing continues until all tasks have been completed and the high-level goal(s) have been achieved (310), at which point process 300 ends.

[0077] Figure 4 is a flow chart illustrating an embodiment of a process for determining a plan for performing work using a robot. In various embodiments, Figure 3 Step 304 is at least partially performed by Figure 4 In the illustrated example, at 402, initialization routines and / or operations are performed to determine the position (e.g., in three-dimensional space) and pose / state of each robotic element, along with items, tools, containers, obstacles, etc., within the workspace. In some embodiments, 402 may be performed at the beginning of one or more sets of operations, when the environment has been altered by human or other robotic intervention, or when data indicating that reinitialization may be necessary (e.g., a failed action, an unexpected state, etc.) is received. At 404, image and / or other sensor data is received and processed to generate a three-dimensional view of the workspace or other scene or environment. At 406, the location(s) and contents(s) of a source container (such as a bread tray in the bakery example) are determined. For example, one or more of image data, optical code scans, identifying markings or text on packaging, etc., may be used to determine which items are located. At 408, the location and state of a destination container are determined. For example, a stack of empty bins, one or more partially filled bins, etc., may be located using image data, RF or other electronic tags, optical codes, or other sensor data. The determined item and container location information, item identification, and other data (e.g., which items are to be sent to which manifest(s) of retail outlet(s) or other point-of-sale locations, etc.) is used at 410 to calculate (or recalculate) a plan for picking / placing items in the workspace to fulfill the order. For example, if a tray full of wheat bread and a tray full of white bread are identified and located in the workspace, and the order requires X loaves of wheat bread and Y loaves of white bread, then in some embodiments, at 410, a plan can be determined to pick and place loaves of wheat bread and white bread as needed to construct one or more trays having the required number of loaves of each type of bread already placed therein.

[0078] In various embodiments, the plan determined at 410 includes steps required to retrieve one or more empty totes from staging or buffer areas, and place / stack them in the loading area as needed to enable items to be placed therein. Similarly, source totes, once emptied, can be moved by the robotic system to staging or buffer areas, e.g., for future use to fulfill subsequent processed orders.

[0079] In various embodiments, the plan determined at 410 includes determining, for each item to be placed, a location on the destination tote that is suitable for the item (e.g., large enough and having the correct dimensions to accommodate the item). In some embodiments, the plan determined at 410 takes into account a goal of ensuring efficient use of available tote space, e.g., by packing items on each tote in a manner that uses an optimal amount of surface space, given the items included in the order (or items left to be picked / placed to fulfill the order). In various embodiments, the plan can be updated at 410 on a continuous and / or periodic basis. Updated state information can be used to update the plan. For example, the state information can indicate, for a given tote, which items have already been placed in the tote, the location and orientation of each item, the remaining space available in the tote, and the next (or remaining) N items to be placed to fulfill the order.

[0080] Figure 4 The process 304 continues until the high-level operations have been completed and there are no additional tasks remaining to be planned (412).

[0081] Figure 5 is a flowchart illustrating an embodiment of a process of using the robots to pick / place items according to the plan. In various embodiments, Figure 3 The step 306 of the process 300 is performed, at least in part, by Figure 5implementation. In the illustrated example, at 502, a pick / place task associated with a plan is received. For example, a robot control module can assign, place in a queue, pull from a queue, etc. a particular pick / place task to be performed to make progress toward completing one or more higher-level sets of operations. At 504, one or more of sensor data, item model information, and grasp strategy repository information are used to determine a strategy for grasping an item to be moved. At 506, an attempt is made to grasp the item according to the strategy. If the grasp is successful (508), e.g., image, weight, and / or other sensor data all indicate that the item has been successfully grasped, then at 510, the item is moved to a destination location where it is to be placed, e.g., according to a previously determined plan. If the grasp attempt is not successful (508), then at 512, it is determined whether to retry the grasp, e.g., using the same or possibly a different strategy. In some embodiments, a retry can be attempted for a configured or prescribed number of times, or until it is determined that no further strategies are available for grasping the item under full automation control, whichever occurs first. If a retry is to be attempted (512), then the process returns to step 504, where one or more of sensor, item model, and grasp strategy information are used to determine a (next) strategy for grasping the item at 506. If no further retries are to be attempted (512), then human intervention is invoked (514), e.g., by remote control operation, manual task performance, etc., and a human worker / operator performs the task to completion (510). Performance Figure 5 of subsequent iterations of the process until all items have been picked / placed (516), at which point the process ends.

[0082] Figure 6A is a schematic diagram illustrating an embodiment of an end effector including multiple suction cups. In the illustrated example, end effector 602 is movably connected to robot arm segment 604. In various embodiments, arm segment 604 includes internal wiring to provide power and a vacuum source to end effector 602. In the illustrated example, end effector 602 includes two suction cups, including leftmost suction cup 606. In various embodiments, end effector 602 includes internal actuator elements (not shown) to move the suction cups, e.g., to open and close the suction cups, to move the suction cups relative to each other, etc. Figure 6AThe end effector 602 includes two suction cups (not shown) that are configured to independently apply suction to the left and / or right suction cup. For example, only one suction cup can be used to grasp a smaller item. Or, two items can be grasped simultaneously, each using an independently (or jointly) controlled suction cup. In some embodiments, the end effector 602 includes sensors such as force or other sensors attached to the suction cup mounts that, in various embodiments, are used to detect initial contact with an item to be grasped, and / or to determine (e.g., by weight) what the item is and / or whether the item has been successfully grasped. In some embodiments, the end effector 602 includes one or more torque sensors, e.g., attached at the "wrist" point where the end effector 602 connects to the arm segment 604. In some embodiments, the torque sensor(s) can be used, e.g., to determine whether an item has been grasped at or near its center of gravity. In some embodiments, the end effector 602 includes pressure sensors to determine whether a suction cup-based item grasp was successful, e.g., by detecting a good or bad seal between the suction cup and the item and / or its packaging.

[0083] Figure 6B is a schematic diagram illustrating an embodiment of an end effector including multiple suction cups. In the example shown, the end effector 622 is movably attached to a robotic arm segment 624 and includes six suction cups arranged in a 2x3 grid. In various embodiments, each of the six suction cups and / or adjacent subgroups of two or three of them can be independently actuated under the control of a robotic control system as disclosed herein.

[0084] Figure 6C is a schematic diagram illustrating an embodiment of an end effector including multiple suction cups. In the example shown, the end effector 642 is movably attached to a robotic arm segment 644 and includes eighteen suction cups arranged in a 3x6 grid. In various embodiments, each of the suction cups and / or adjacent subgroups of the suction cups can be independently actuated under the control of a robotic control system as disclosed herein.

[0085] While Figure 6A , Figure 6B The examples shown in Figures 6A, 6B, and 6C include the number and arrangement of suction cups shown, but in various embodiments, more or fewer suction cups are included. In some embodiments, grasping structures other than suction cups are included.

[0086] Figure 7 is a flowchart illustrating an embodiment of a process of grasping an item with a robotic arm and end effector. In various embodiments, Figure 7 The process 700 can be performed by a computer such as the control computer 122 that is configured to operate a robotic arm and end effector such as the robotic arm 102 and end effector 104 of the robotic system 100. Figure 1robotic arm 102 and end effector 108. In some embodiments, process 700 is performed by Figure 2 image and / or other sensor data is used to move the end effector near the item to be grasped. For example, based on image data, the location in three-dimensional space of the surface of the package to be engaged by a suction-type end effector is determined, and a trajectory is calculated to move the item-engaging end of the end effector's suction cup to within a specified distance of the surface. As the end effector is moved near the package (or item) surface, image data continues to be monitored. In various embodiments, movement is stopped once the end effector has moved to the end position and orientation of the calculated trajectory, and / or if image or other sensor data indicates that the end effector has reached the vicinity of the item before reaching the end of the trajectory; for example, if a force sensor in the end effector detects that the suction cup has contacted the package / item, movement can be stopped. Once the end effector has been moved to a position near the item to be grasped (702, 704), at 706, the large-scale movement of the end effector and arm (e.g., of a relatively large distance and / or of a relatively high velocity) is stopped, and an attempt is made to grasp the item by one or more of fine movement of the end effector and actuation of the end effector's suction or other item-engaging mechanism.

[0087] In various embodiments, one or more of the following can be performed, presented, and / or implemented:

[0088] • Grasping using suction minimizes the necessary free space around the item, requiring only the top of the item to be accessible.

[0089] • Flexible suction cups attenuate contact forces to protect the product (i.e., the item being grasped).

[0090] • Pressure sensors to detect when the cup is in close contact with the product surface. In some embodiments and / or grasps, the outer package can be securely grasped even without contacting the contained product.

[0091] • Force sensors to detect when force is applied to the product.

[0092] • Any other combination of sensors that detect two grasp states, (1) that the end effector has made sufficient contact with the product to grasp, and (2) that the end effector cannot move closer without risking damage to the product. In some embodiments, these sensors can include proximity sensors and / or limit switches on the suction cup mount.

[0093] • A closed control loop that moves the robotic end effector toward the product until it encounters either state (1) or (2) from above.

[0094] At 708, it is determined whether the item has been successfully grasped. For example, a pressure sensor can be used to determine whether a suction-based grasp has been successfully completed. For example, if a required vacuum has been achieved, it can be determined that the item has been successfully grasped. In some embodiments, it can be determined that the item has been successfully grasped based on properties of the item, such as weight, and whether a vacuum has been achieved relative to at least a minimum number of suction cups of the end effector, such as a minimum number of suction cups associated with securely grasping an item of that weight.

[0095] If / once the grasp has been determined to be successful (708), at 710, the item is moved to the associated destination location. For example, the robotic arm and end effector are moved along a trajectory calculated by the control computer and / or module to move the item to the destination and release the item from the grasp of the end effector. If the grasp was (not) successful (708), at 712, the image and / or other sensor data is evaluated to determine a (new / revised) grasping strategy. In some embodiments, step 712 includes an evaluation of whether to attempt a (further) retry; if not, human intervention is initiated. If a new grasping strategy is determined at 712, the grasping strategy is attempted at 706, and if successful (708), the item is moved to its destination at 710. Once the item has been moved to its destination (e.g., at 710 and / or through human intervention if automated grasping / moving has failed), at 714, it is determined whether there are additional items to be grasped and moved. If so, at 716, the process proceeds to the next item to be moved, and further iterations of process 700 are performed relative to that item. If there are no additional items to be grasped and moved (714), the process ends.

[0096] Figure 8 is a flowchart illustrating an embodiment of a process of safely moving an item that has been grasped by a robotic arm and / or end effector to a destination. In various embodiments, Figure 8 Process 800 of can be performed by a computer, such as control computer 122, that is configured to operate a robotic arm and end effector, such as Figure 1 robotic arm 102 and end effector 108 of In some embodiments, process 800 is performed by Figure 2 robotic control module 208 of

[0097] In various embodiments, the robotic system, for example, via Figure 7 Process 700 of grasps an item, and then employs one or more sensor-based checks to ensure that the grasp is correct and safe and remains correct and safe, for example, the robotic system can:

[0098] • Query a database of SKU-specific or other item information, e.g., to determine item weight and / or other attributes.

[0099] • Check air leak inspection pressure sensor readings for compromised grasp.

[0100] • Check force sensor readings to ensure the correct item type and quantity has been grasped.

[0101] • Camera detects position of grasped item relative to end effector to corroborate readings from pressure and force sensors. Confirms that the size of the grasped item matches the size in the SKU or other item database.

[0102] • Torque readings check that the item has been grasped near the center of mass.

[0103] • Once there is any (or greater than a threshold) discrepancy between sensor readings or any check fails, the robot returns the grasped object to its original location or to some “safe” zone nearby for further attempts / checks. It can then continue picking (e.g., other) products.

[0104] Referring to Figure 8 , in the illustrated example, while an item is being moved (802) (e.g., after being successfully grasped using a robotic arm and end effector), sensor data from multiple sensors and / or sensor types (e.g., image, force, pressure, etc.) is monitored at 804. For each sensor / sensor type, the sensor data is evaluated at 806 to determine whether that data is consistent with a (continuing) secure grasp of the item. If all (or a threshold proportion of) the sensors indicate that the grasp remains secure (808), then at 810 the item is moved (and / or continues to be moved) to its destination. If any (or a threshold number of) the sensors and / or sensor types provide data indicating that the grasp is not secure or is no longer secure (808), then at 812 the item is moved to a buffer location (or its original source location, if the robotic system determines that the grasp is sufficiently secure and / or that the source location is sufficiently close to do so), and once in the safe location, the robotic system determines a grasping strategy and attempts to re-grasp the item. The monitoring of multiple sensors (804) and use of sensor data to evaluate whether the grasp is secure / whether it remains secure (806, 808) continues until the item has been successfully moved to its destination (814), after which the process 800 ends.

[0105] Figure 9 is a block diagram illustrating an embodiment of a suction-based end effector. In various embodiments, Figure 9 the robotic arm end effector 900 of Figure 1 may be used to implement the end effector 108 of

[0106] In the example shown, the end effector 900 includes a body or housing 902 attached to a robotic arm 904 via a rotatable coupling. In some embodiments, the connection between the housing 902 and the robotic arm 904 may include a controller (e.g., a controller) configured to control the robot. Figure 1 The end effector 900 further includes a suction or other pneumatic line 906 that extends through the robotic arm 904 into the housing 902 to supply a vacuum source to a suction control module 908. In various embodiments, the control module 908 is connected to a control computer external to the end effector 900, such as a control computer 118, via wireless and / or wired communication, for example, via a communication interface 914. Figure 1 The control computer 118 of the embodiment of the present invention is a control module 908. The control module 908 includes electronic and / or electromechanical components that are operable to supply suction to the suction cups 910, 912 comprising the end effector 900, for example, to attach the end effector to an item to be picked up, moved, and placed using the end effector 900 by suction.

[0107] In various embodiments, the suction control module 908 is configured to independently apply suction to the trays 910 and 912 so that only one or the other can be used to grab a given item. In some embodiments, two adjacent items (such as two loaves of bread) can be grabbed and moved simultaneously, with each item being grabbed by a corresponding one of the trays 910 and 912.

[0108] In the example shown, a camera 916 mounted on the side of the housing 902 provides image data of the field of view below the end effector 900. A plurality of force sensors 918, 920, 922, 924, 926, and 928 measure the forces applied to the suction cups 910 and 912, respectively. In various embodiments, the force measurements are transmitted to an external and / or remote control computer via the communication interface 914. The sensor readings are used in various embodiments to enable the robotic arm 904 and end effector 900 to be used to hold an item against other items and / or a sidewall or other structure, and / or to detect instability (e.g., when an item is still under suction but is depressed in a position where it is expected to be placed and stabilized, with insufficient pushback). In various embodiments, horizontally mounted force sensor pairs (e.g., 918 and 922, 924 and 928) are positioned at right angles in the xy plane to enable force determination in all horizontal directions.

[0109] In some embodiments, force sensors can be used to detect initial contact between the suction cups 910, 912 and the item to be grasped. Upon detecting initial contact, the robotic arm 904 stops to avoid damage to the item, such as crushing a loaf of bread or wrinkling its packaging. Suction is then applied to grasp the item using suction.

[0110] Figure 10 is a flowchart illustrating an embodiment of a process of picking / placing items simultaneously as needed / if needed to best achieve goals assigned to the robotic system. In various embodiments, Figure 10 Process 1000 of FIG. 10 can be performed by a computer, such as control computer 122, configured to operate a robotic arm and end effector, such as Figure 1 robotic arm 102 and end effector 108 of FIG. 1. In some embodiments, process 1000 is performed by Figure 2 robotic control module 208 of FIG. 2.

[0111] In various embodiments, a robotic system as disclosed herein can:

[0112] • pick multiple items in one run to increase the throughput of the picking system.

[0113] • generate different picking (grasping) poses for different combinations of items.

[0114] • detect combinations of pickable items that fit the current end effector configuration for a secure tight grasp. For example, a gripper (end effector) with two large suction cups can pick two items of the same or similar size at the same time.

[0115] • decide among all pickable items, according to the current quantity of customer demand (e.g., current order), whether to perform multiple grasps.

[0116] • adapt to different system conditions to perform multiple grasps in a safe manner. For example, if an item is located at the edge of the work area, adjust the gripper pose to accommodate possible collisions. Identify and filter out potential dangerous situations.

[0117] • adapt to different gripper configurations: evaluate the feasibility of multiple grasps and generate different grasping poses based on the given gripper configuration (such as the diameter between gripper entry points, etc.).

[0118] • perform a cost evaluation between possible single and multiple grasps.

[0119] • evaluate the grasping cost of all possible multiple grasps to achieve the lowest risk in a stable multiple grasp.

[0120] Reference is made to Figure 10In the example shown at 1002, item data, such as item attribute data, quantity of each item needed to fulfill a current / next order, source(s) / origin location of items to be moved, status of source and destination locations (e.g., pallets), etc., is evaluated. At 1004, individual items and supported multi-item item combinations that can be picked and placed as part of a set of actions to complete an order are evaluated. For example, if two or more of the same item are needed to fulfill an order, and adjacent items exist in a source pallet, and space is available (or, in a scenario evaluated in the plan, will or can become available), moving two (or more) items in a single task can be considered to be included in a plan to fulfill the order. At 1006, a lowest (overall / total) cost (e.g., time, energy, etc.) and / or lowest risk (e.g., risk of dropping an item can be considered to be greater than a measurable or predictable amount than moving items one at a time) mix of single item and / or multi-item pick / place tasks to fulfill the order is determined. In some embodiments, each of the cost and risk can have an associated weight in a cost function, for example, and a plan that minimizes the cost function can be determined. At 1008, items are picked / placed according to the plan determined at 1006. Processing continues until completion (1010).

[0121] In various embodiments, one or more steps of the process 1000 can be performed on a continuous and / or periodic basis, e.g., to refine the plan determined at 1006 based on information that becomes available while the plan is being executed, such as new source pallets or other item source containers being moved into the workspace, human or other unexpected intervention or events that change the status of any items and / or containers in the workspace, failed pick / place tasks that result in a pallet (e.g., of a destination) having an actual status different than expected, etc.

[0122] Figure 11 is a flowchart illustrating an embodiment of a process to adjust a final placement of an item. In various embodiments, Figure 11 The process of can be performed by a computer, such as the control computer 122, that is configured to operate a robotic arm and end effector, such as Figure 1 the robotic arm 102 and end effector 108 of In some embodiments, Figure 11 The process of is performed by Figure 2The robot control module 208 executes. In the example shown, while an item is being moved (1102), the system detects that the item has been moved to a position in close proximity to an intended destination location (1104). For example, image sensors, force sensors, and / or other sensor data can be used to detect proximity to the final location. In some embodiments, a vertically oriented force sensor can be used to detect that the item has made contact with the bottom of a pallet or other container surface on which the item is to be placed. At 1106, in response to detecting proximity to the final destination location, a final placement operation is performed to ensure a snug fit before the item is released. For example, horizontal or other force sensors, torque sensors, contact sensors, image sensors, or some combination thereof can be used to assess contact or near contact of the item with adjacent items and / or structures (such as the sidewalls of a container). The robot arm and / or end effector can be manipulated to increase proximity, such as by ensuring that the placed item makes contact with adjacent items along any edges and / or surfaces against which it should be placed in close proximity according to the pick / place plan. For example, if the force sensor readings of sensors associated with the same horizontal entry and / or contact edge or surface of an item are not (substantially) the same, the end effector can be rotated and / or the arm moved to reposition the item to a position where the force sensors sense substantially the same force.

[0123] Figure 12 is a flow chart illustrating an embodiment of a process for selecting a destination location for an item. In various embodiments, Figure 12 The process may be performed by a computer (such as control computer 122) configured to operate a robotic arm and end effector, such as Figure 1 The robotic arm 102 and the end effector 108. In some embodiments, Figure 12 The process of Figure 2 The robot control module 208 executes the following steps. In the example shown, at 1202, the state of a destination pallet to which one or more items are to be moved is evaluated. For example, image data and data from previously executed pick / place tasks can be used to determine available slots, spaces, and / or locations that can be used to receive items to be placed on the destination pallet or other container. At 1204, the best slot is selected as available from those determined at 1202.

[0124] In some embodiments, the optimal slot can be selected based on an algorithm and / or other criteria. For example, in some embodiments, the slot that results in the lowest outer perimeter for the group of items in the destination tray can be selected. In some embodiments, if information about N items is available, the slots for the next N items to be placed can be determined simultaneously to ensure that the set of placement locations is optimal. In some embodiments, if a prior placement decision impacts the availability and / or optimality of a subsequent placement decision, the previously placed item can be moved to a programmatically determined position and / or orientation to make a (more optimal) slot available for the next (or multiple) items to be placed. For example, the decision to place the first item can be made before the system knows which item(s) to place next and / or before the system has complete information about such item(s). In such a situation, the system may have selected a placement for the previously placed item that results in an overall less-than-optimal situation given the subsequently placed items. In some embodiments, the system is configured to detect such a situation and move and / or change the placement / or orientation of the previously placed item if doing so opens up more / better possibilities for the next or later placed item(s).

[0125] In some embodiments, if it is determined at 1202 and / or 1204 that no slots are available to receive an item, and in some embodiments, if no smaller item(s) that could potentially fit in the pallet are available to be picked up / placed into a space available and sufficient to receive such an item, the system determines that the pallet is full and retrieves an empty pallet, for example, from a staging area. If the pallet stack is not too high and the already filled pallet and the next new pallet to be filled are going to the same location, for example, the same delivery vehicle / route, the new pallet can be stacked on the pallet that has been determined to be full.

[0126] At 1206, once the appropriate / optimal slot has been selected to place the item(s), the item(s) are placed in that location, e.g., via Figure 7 and 11 One or more processes.

[0127] Figure 13 is a flow chart illustrating an embodiment of a process for detecting misplaced items. In various embodiments, Figure 13 The process may be performed by a computer (such as control computer 122) configured to operate a robotic arm and end effector, such as Figure 1 The robotic arm 102 and the end effector 108. In some embodiments, Figure 13 The process of Figure 2Executed by the robot control module 208. In the illustrated example, as an item is placed (1302) (e.g., moved and snugged into a final position but not yet released from the end effector's grip), image and / or other sensor data is used to assess the item's placement (1304). For example, the three-dimensional image data may indicate that the item is higher than expected for a successful placement. For example, the height of the item's top surface may be expected to be equal to the height of the tray bottom plus the expected height of the item. If the top surface is at a higher height, such information may indicate that the item has been placed in whole or in part on top of another item, or that the item may have been crushed or otherwise deformed during the pick / place operation, or that the wrong item may have been placed, etc. If no error is detected (1306), the item is released from the grip (1308), and the arm and end effector are moved away to perform the next pick / place task. If an error is detected (1306), the item is returned to the starting position, or in some embodiments, to another staging or buffer area, at 1310, and the pick / place operation is attempted again. In some embodiments, repeated attempts to pick / place an item resulting in detection of an erroneous pick / place item may trigger an alarm and / or other action to initiate human intervention.

[0128] Figure 14 is a schematic diagram illustrating an embodiment of a robotic system for handling soft products in non-rigid packaging. In the example shown, system 1400 includes robotic arms 1408 and 1410 mounted to move along rails 1404 and 1406, respectively, under computer control. Robotic arms 1408 and 1410 terminate in suction-type end effectors 1412 and 1414, respectively. In various embodiments, robotic arms 1408 and 1410 and end effectors 1412 and 1414 are controlled by a robotic control system that includes a control computer, such as Figure 1 and 2 The control computer 122.

[0129] exist Figure 14In the example shown in FIG. 14, robotic arms 1408 and 1410 and end effectors 1412 and 1414 are used to move items (such as loaves of bread) from a source tray on wheeled base 1402 to destination trays on wheeled bases 1416 and 1418. In various embodiments, a human and / or a robot can position the source tray and wheeled base 1402 into a starting position between tracks 1404 and 1406, as shown. Wheeled base 1402 can advance through the channel formed by tracks 1404 and 1406, for example, in the direction of the arrow starting at the distal end of base 1402, as shown. In various embodiments, base 1402 can be advanced as follows: using one or both of robotic arms 1408 and 1410; by being manually pushed by one or more human operators, one or more stationary or non-stationary robots, etc.; along a conveyor belt or chain-type mechanism that stretches along and / or between tracks 1404 and 1406; a robotically-controlled propulsion and / or conveyance mechanism incorporated into base 1402, such as computer-controlled motorized wheels and brakes; etc.

[0130] As wheeled base 1402 advances along / between tracks 1404 and 1406, and / or as base 1402 is temporarily in a stationary position between tracks 1404 and 1406, in various embodiments, the robotic system uses robotic arms 1408 and 1410, and end effectors 1412 and 1414, to move items such as loaves of bread from the source tray on wheeled base 1402 to the destination trays on wheeled bases 1416 and 1418 according to a plan. For example, the plan can be computed based on inventory and / or sensor data indicating which type(s) of bread are available in the source tray on base 1402 and / or other bases in the work area, and further based on a manifest or other data indicating which mix of breads or other items are to be placed in the destination trays for shipment to which respective final shipping locations (such as retail stores).

[0131] In the example shown, once the destination trays on bases 1416 and 1418 have been filled, bases 1416 and 1418 are moved outward away from tracks 1404 and 1406, respectively, for example to staging and loading areas to be loaded onto shipping vehicles.

[0132] While in the example shown in FIG. 14, the robotic arms 1408 and 1410 and end effectors 1412 and 1414 are used to move items (such as loaves of bread) from a source tray on wheeled base 1402 to destination trays on wheeled bases 1416 and 1418, in various embodiments, the robotic arms 1408 and 1410 and end effectors 1412 and 1414 can be used to move items in other ways, such as to move items from a source tray on wheeled base 1402 to a destination tray on wheeled base 1402, to move items from a source tray on wheeled base 1402 to a destination tray on a different wheeled base, etc. Figure 14A single set of source trays on a single base 1402 is shown, but in various embodiments, depending on the high-level operations being performed, one or more additional bases can be located in a staging area between and / or adjacent to the tracks 1404 and 1406, each having zero or more trays full of items stacked thereon. Similarly, in some embodiments, multiple sets of destination trays— each on a corresponding wheeled base— can be staged adjacent to the track 1404 or the track 1406, and the topmost tray on each set can be filled, e.g., according to an order manifest and associated fulfillment plan, in a process that is simultaneous and / or sequential.

[0133] In various embodiments, once a source tray is emptied, the system can use the robotic arms 1408 and / or 1410 to move the tray to a staging area and / or the top of a destination tray stack, thereby creating a tray supply to be filled and sent to a final destination and / or exposing the next source tray from which items can be picked to fulfill an order.

[0134] While in Figure 14 In some embodiments, two or more robotic arms can be provided on each track. In some embodiments, two robotic arms on a single track can be used to pick up and move empty trays. For example, in various embodiments, each robotic arm can be used to mechanically engage opposite sides of a tray, either by engaging a structure of the tray or by suction or grasping, and the two arms can be cooperatively operated to maintain control of the empty tray while moving it to a new location, such as a buffer or staging area or the top of a destination tray stack. In some embodiments, one or more robotic arms can be used to move full or partially full trays from the top of a stack, e.g., to a staging area, to expose lower trays in the stack, to add additional items, to rearrange items according to a new or revised plan, etc.

[0135] Cameras 1420 and 1422 and / or cameras 1424 and 1426 mounted on end effectors 1412 and 1414, respectively, are used in various embodiments to generate image data to plan and perform pick / place operations as disclosed herein.

[0136] In various embodiments, the technology disclosed herein enables robotic systems to handle soft or otherwise fragile products in non-rigid packaging (such as loaves of bread packaged in plastic bags or outer packaging) without damaging the items being handled. For example, in various embodiments, loaves of bread can be identified, selected, and robotically moved from a source pallet received from a bakery or storage area to a destination pallet for delivery to a corresponding specific location (such as a retail store) with minimal or no human intervention. In various embodiments, the technology disclosed herein increases the throughput and efficiency of distribution systems, facilities, and processes for bread or other fragile items in soft packaging, for example, compared to purely human and / or less automated operations.

[0137] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A robotic system comprising: Communication interface; and a processor coupled to the communication interface and configured to: receiving, via the communication interface, sensor data associated with a workspace in which one or more robotic elements at least partially controlled by the robotic system are present, wherein the sensor data is received from a plurality of different types of sensors including at least a torque sensor for detecting whether an item has been grasped close to a center of mass and a pressure sensor for detecting air leaks that impair the grasp; determining, based at least in part on the sensor data, an action to be performed in the workspace using one or more robotic elements, the action comprising relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped; actuating a gripping mechanism of the end effector to grasp the article using an amount of force and structure associated with minimizing risk of damage to one or both of the article and its packaging; and while the article is being moved after being successfully grasped using the end effector, evaluating sensor data from each sensor type to determine whether the data is consistent with a continued secure grasp of the article, wherein if a threshold number of sensor types provide data indicating that the grasp is no longer secure, the article is moved to its original source location if the robotic system determines that the grasp is sufficiently secure and the original source location is sufficiently close to do so, otherwise the article is moved to a buffer location; and A control communication is sent to the robotic element via the communication interface to cause the robotic element to perform the action. 2 . The system of claim 1 , wherein the processor is further configured to move the grasped item to a destination location using a robotic element.

3. The system according to claim 1, wherein: The end effector includes a suction-based gripping mechanism.

4. The system according to claim 3, wherein: The end effector includes one or more suction cups.

5. The system according to claim 4, wherein: The one or more suction cups comprise a flexible material that gives way when contact is made with an item.

6. The system according to claim 1, wherein: Sensor data includes image data generated by one or more cameras.

7. The system according to claim 1, wherein: The processor is further configured to move the grasped item to a destination location using the robotic element and place the item at the destination location.

8. The system according to claim 7, wherein: The processor is configured to place the item at least in part by moving the item to a position proximate a destination location in a first operational phase and using sensor data to adjust the position of the item to be more closely adjacent to a structure or a second item adjacent to the destination location.

9. The system according to claim 1, wherein: The processor is configured to determine the action based at least in part on the plan.

10. The system according to claim 9, wherein: The processor is configured to generate the plan based on high-level goals.

11. The system according to claim 10, wherein: The high-level goal includes identification of a plurality of items and a corresponding quantity for each item to be included in a set of items associated with a final destination.

12. The system according to claim 11, wherein The items include different bakery products, and the high-level goal includes a manifest and the quantity of each item to be shipped to a retail store.

13. The system of claim 1, wherein: The processor is further configured to move the grasped item to a destination location using the robotic element and place the item at the destination location, including verifying placement of the item before releasing the item from the grasping mechanism of the end effector by using subsequently received sensor information.

14. The system according to claim 1, wherein: The processor uses the sensor information to verify placement of the article prior to releasing the article from a gripping mechanism of an end effector by, at least in part, using the sensor data to determine a height of the article above a height reference and comparing the determined value to an expected value associated with a successful placement.

15. The system of claim 1, wherein: The end effector includes a plurality of independently actuated sets of one or more suction cups, and the processor is further configured to control the robotic element to simultaneously grasp two or more items, each item using one or more suction cups from the set of independently actuated one or more suction cups.

16. The system according to claim 15, wherein: The processor is further configured to determine an optimal plan for picking and placing the multiple items, including by determining possible combinations of two or more items that can be grasped simultaneously, and determining an optimal plan that includes a mix of single-grasp and multi-grasp pick and place actions.

17. The system of claim 1, wherein: The processor is further configured to determine a strategy for grasping the item using the robotic element.

18. A method for controlling a robotic system, comprising: receiving, via the communication interface, sensor data associated with a workspace in which one or more robotic elements at least partially controlled by the robotic system are present, wherein the sensor data is received from a plurality of different types of sensors including at least a torque sensor for detecting whether an item has been grasped close to a center of mass and a pressure sensor for detecting air leaks that impair the grasp; determining, based at least in part on the sensor data, an action to be performed in the workspace using one or more robotic elements, the action comprising relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped; actuating a gripping mechanism of the end effector to grasp the article using an amount of force and structure associated with minimizing risk of damage to one or both of the article and its packaging; and while the article is being moved after being successfully grasped using the end effector, evaluating sensor data from each sensor type to determine whether the data is consistent with a continued secure grasp of the article, wherein if a threshold number of sensor types provide data indicating that the grasp is no longer secure, the article is moved to its original source location if the robotic system determines that the grasp is sufficiently secure and the original source location is sufficiently close to do so, otherwise the article is moved to a buffer location; and A control communication is sent to the robotic element via the communication interface to cause the robotic element to perform the action.

19. A computer program product embodied in a non-transitory computer-readable storage medium and comprising computer instructions for: receiving, via the communication interface, sensor data associated with a workspace in which one or more robotic elements at least partially controlled by the robotic system are present, wherein the sensor data is received from a plurality of different types of sensors including at least a torque sensor for detecting whether an item has been grasped close to a center of mass and a pressure sensor for detecting air leaks that impair the grasp; determining, based at least in part on the sensor data, an action to be performed in the workspace using one or more robotic elements, the action comprising relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped; actuating a gripping mechanism of the end effector to grasp the article using an amount of force and structure associated with minimizing risk of damage to one or both of the article and its packaging; and while the article is being moved after being successfully grasped using the end effector, evaluating sensor data from each sensor type to determine whether the data is consistent with a continued secure grasp of the article, wherein if a threshold number of sensor types provide data indicating that the grasp is no longer secure, the article is moved to its original source location if the robotic system determines that the grasp is sufficiently secure and the original source location is sufficiently close to do so, otherwise the article is moved to a buffer location; and A control communication is sent to the robotic element via the communication interface to cause the robotic element to perform the action.

Images