Robotic handling of flexible products in non-rigid packaging

Abstract

Techniques are disclosed for performing robotic handling of flexible products in non-rigid packaging. In various embodiments, sensor data associated with a workspace is received. An action to be performed within the workspace using one or more robotic elements is determined, including relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped, actuating a grasping mechanism of the end effector to grasp the item using an amount of force and configuration associated with minimal risk of damage to the item and / or its packaging, and verifying that the item has been securely grasped using sensor data generated after the item has been grasped. Control communications are sent to the robotic elements via a communications interface to cause the robotic elements to perform the action.

Description

[Background technology]

[0001] CROSS-REFERENCE TO OTHER APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 809,398, filed February 22, 2019, entitled "ROBOTIC HANDLING OF SOFT PRODUCTS IN NON-RIGID PACKAGING," which is incorporated herein by reference for all purposes.

[0002] Shipping and distribution centers, warehouses, loading docks, air cargo terminals, large retail stores, and other activities that ship and receive heterogeneous sets of items utilize strategies such as packing and unpacking heterogeneous items into boxes, crates, containers, conveyor belts, pallets, etc. Packing heterogeneous items into boxes, crates, pallets, etc. allows the resulting set of items to be handled by lifting equipment (forklifts, cranes, etc.), allowing the items to be packed more efficiently for storage (e.g., in a warehouse) and / or shipment (e.g., in a truck, cargo hold, etc.).

[0003] In some contexts, items may vary so much in size, weight, density, bulk, rigidity, packaging strength, etc. that they may or may not have attributes that allow any given item or set of items to support the size, weight, weight distribution, etc. of given other items that they may need to pack (e.g., in a box, container, pallet, etc.). When assembling heterogeneous items onto a pallet or other set, the items must be carefully selected and stacked to ensure that the palletized stack does not collapse, tip, or otherwise become unstable (e.g., so that it cannot be handled by a machine such as a forklift), and to avoid damage to the items.

[0004] Some products may be subject to damage such as crushing and / or may be in flexible packaging. For example, bread and bun-like products may be packaged in plastic bags or similar packaging. A robotic gripper can easily crush the bag and / or the product inside. Even a slight crushing of the product or a severely crumpled package can make the item less desirable to the customer.

[0005] Currently, pallets are typically loaded and / or unloaded manually, with human workers selecting items to stack based on, for example, a shipping slip or manifest, and using human judgment and intuition to select, for example, larger, heavier items to be placed at the bottom. However, in some cases, items simply arrive on a conveyor or other mechanism and / or are picked from bins in listed order, resulting in an unstable palletized or otherwise packaged set.

[0006] In the case of bread and other products sold in retail stores, they may need to be distributed to stores in bulk, for example, with the specific combination of items and quantities of each ordered by each store. Bread products are typically delivered to stores by packaging them in stackable trays, loading them onto trucks, and transporting the trays to the stores to load the products onto shelves.

[0007] The use of robotics is made more difficult in many environments due to the variety of items, the variability in the order, number, and combination of items packed, for example, on a given pallet, and the variety in the types and locations of containers and / or feeding mechanisms from which items must be picked to place them on a pallet or other container. In the case of bread and similar products, the need to pick and place arbitrarily fragile product items into flexible packaging (such as plastic bags) makes robotic handling more difficult. [Brief explanation of the drawings]

[0008] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0009] [Figure 1] FIG. 1 is a block diagram illustrating one embodiment of a robotic system for handling flexible products in non-rigid packages.

[0010] [Figure 2] FIG. 1 is a block diagram illustrating one embodiment of a system for controlling a robotic system.

[0011] [Figure 3] 1 is a flow chart illustrating one embodiment of a process for controlling a robotic system.

[0012] [Figure 4] 1 is a flow chart illustrating an embodiment of a process for determining a plan for performing a task using a robot.

[0013] [Figure 5] 1 is a flow chart illustrating one embodiment of a process for picking and placing items using a robot according to a plan.

[0014] [Figure 6A] 1 illustrates an embodiment of an end effector comprising multiple suction cups.

[0015] [Figure 6B] 1 illustrates an embodiment of an end effector comprising multiple suction cups.

[0016] [Figure 6C] 1 illustrates an embodiment of an end effector comprising multiple suction cups.

[0017] [Figure 7]1 is a flow chart illustrating one embodiment of a process for grasping an item with a robotic arm and end effector.

[0018] [Figure 8] 1 is a flow chart illustrating one embodiment of a process for reliably moving an item grasped by a robotic arm and / or end effector to a destination.

[0019] [Figure 9] FIG. 1 is a block diagram illustrating one embodiment of a suction-based end effector.

[0020] [Figure 10] 1 is a flow chart illustrating one embodiment of a process for simultaneously picking and placing items when / if needed to best achieve the goals assigned to the robotic system.

[0021] [Figure 11] 10 is a flow chart illustrating one embodiment of a process for adjusting the final placement of items.

[0022] [Figure 12] 10 is a flow diagram illustrating one embodiment of a process for selecting a destination location for an item.

[0023] [Figure 13] 1 is a flow chart illustrating one embodiment of a process for detecting misplaced items.

[0024] [Figure 14] FIG. 1 illustrates an embodiment of a robotic system for handling flexible products in non-rigid packages. DETAILED DESCRIPTION OF THE INVENTION

[0025] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0026] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0027] Techniques are disclosed for programmatically utilizing a robotic system including one or more robots (e.g., robotic arms with suction cups and / or grippers at their working ends) to palletize / depalletize and / or otherwise pack and / or unpack any set of heterogeneous items (e.g., different sizes, shapes, weights, weight distributions, stiffness, fragility, etc.), including items sold in plastic bags or similar packaging (e.g., bread and similar products).

[0028] In various embodiments, 3D cameras, force sensors, and other sensors are used to detect and determine attributes of items to be picked and / or placed. Items whose type has been determined (e.g., with sufficient confidence, e.g., as indicated by a programmatically determined confidence score) may be grasped and placed using a strategy derived from an item-type-specific model. Unidentifiable items are picked and placed using a strategy not specific to a given item type. For example, a model using size, shape, and weight information may be utilized.

[0029] In various embodiments, a library of item types, their respective attributes, and gripping strategies is used to determine and implement a strategy for picking and placing each item. The library is dynamic in some embodiments. For example, the library may be augmented to add newly encountered items and / or additional attributes learned about the items (such as gripping strategies that did or did not work in a given context). In some embodiments, human intervention may be requested if the robotic system becomes stuck. The robotic system may be configured to observe (e.g., using sensors) and learn (e.g., update the library and / or model) based on how a human teleoperator intervenes, for example, to utilize the robotic arm to remotely pick and place items.

[0030] In some embodiments, the robotic systems described herein may perform processing to collect and store (e.g., add to a library) attributes and / or strategies for identifying and picking / placing items of unknown or newly discovered types. For example, the system may hold items at various angles and / or positions to enable 3D cameras and / or other sensors to generate sensor data for augmenting and / or creating library entries that characterize the item type and store models of how to identify and pick / place items of that type.

[0031] In various embodiments, a high-level plan is developed for picking and placing items on a pallet or other container or location. Strategies are applied, for example, based on sensor information, to implement the plan by picking and placing individual items according to the plan. Deviations from the plan and / or re-planning may be triggered, for example, if the robotic system becomes stuck, if the stack of items on the pallet is detected to have become unstable (e.g., by computer vision, force sensors, etc.), etc. Partial plans may be developed based on known information (e.g., the next N items visible on the conveyor, the next N items on an invoice or manifest, etc.) and adjustments made when more information (e.g., the next item visible, etc.) is received. In some cases, the system is configured to stage (hold) (e.g., set aside within reach) items determined to be unlikely to be suitable for stacking in the layer or container the system is currently building (e.g., a lower (or higher) layer on the pallet or container). As more information becomes known about the next item to be picked / placed, a strategy is generated and implemented that takes into account the staged (buffered) item and the next item.

[0032] For bread and similar products in plastic bags or similar packaging, in various embodiments, the robotic systems disclosed herein use a robotic arm equipped with a suction-type gripper at the working end. For example, in some embodiments, a gripper equipped with two large suction cups is used. In other embodiments, a single suction cup or three or more suction cups may be provided. In various embodiments, the bread is picked from a conveyor, bin, shelf, etc. and placed on a large, stackable plastic tray for delivery. An invoice for each destination may be used to pick and place the item.

[0033] In various embodiments, algorithms, heuristics, and other program techniques are used to generate plans for packing bread items. For example, items having the same type, size, shape, etc. may be placed together for efficient packing. In some embodiments, the planner module may attempt to add items to a tray to minimize gaps, perimeters (e.g., the sum of edges not adjacent to other products and / or sides of the tray), etc.

[0034] In various embodiments, a robotic system for picking and placing pans and similar items as disclosed herein may be implemented, at least in part, using some or all of the techniques illustrated in and / or described in connection with FIGS. 1-14.

[0035] 1 is a block diagram illustrating one embodiment of a robotic system for handling flexible products in non-rigid packages. In the illustrated example, system 100 includes a robotic arm 102 rotatably mounted on a carriage 104 configured for translational movement, e.g., under computer control, along a rail 106. In this example, robotic arm 102 is movably mounted on carriage 104 and rail 106; however, in various other embodiments, robotic arm 102 may be fixed or may be fully or partially movable other than translational movement along a rail, e.g., mounted on a carousel, fully movable on a motor-driven chassis, etc.

[0036] In the illustrated example, the robotic arm 102 has an end effector 108 at its working distal end (the end farthest from the carriage 104). The end effector 108 includes a compliant vacuum (or "suction") cup 110. In various embodiments, the suction cup 110 comprises silicone or another natural or synthetic material that is durable yet compliant enough to at least slightly "give in" when the robotic system 100 makes contact (initially and / or gradually) with an item that it is attempting to grasp using the robotic arm 102 and end effector 108, e.g., by suction (such as a loaf of bread or other soft and / or fragile item in a non-rigid package like a plastic bag or wrapping).

[0037] In this example, the end effector 108 has a camera 112 mounted to the side of the end effector 108. In other embodiments, the camera 112 may be more centrally located, such as on a downward-facing surface of the body of the end effector 108 (in the position and orientation shown in FIG. 1 ). Additional cameras may be mounted elsewhere on the robotic arm 102 and / or end effector 108 (e.g., on the arm segments that comprise the robotic arm 102). Additionally, cameras 114 and 116, which are wall-mounted in this example, provide additional image data that can be used to construct a 3D view of the scene in which the system 100 is positioned and configured to operate.

[0038] In various embodiments, the robotic arm 102 is used to position the suction cup 110 of the end effector 108 over the item to be picked up, as shown, and a vacuum source provides a suction force to grasp the item, lift the item from its source location, and place the item in its destination location.

[0039] In the example shown in FIG. 1 , the robotic arm 102 is configured to be used to pick any items, which in this example are heterogeneous, such as bread items picked from a tray of items received from a bakery, from a source tray 120 and place those items in a destination tray. In the illustrated example, the destination tray 118 and the source tray 120 comprise stackable trays configured to be stacked on a base with wheels on each corner. In some embodiments, the trays 118 and / or 120 may be pushed into position by a human worker. In some embodiments, the trays 118 and / or 120 may be stacked on a motorized and / or robotically controlled base configured to be used, for example, to move a bin stack into a predetermined position to be picked and / or placed, and / or to move a completed stack of trays to a staging area and / or transport area for loading into a delivery vehicle for delivery to a retail store. In some embodiments, other robots not shown in FIG. 1 may be used to push trays 118 and / or 120 into position for loading / unloading and / or onto a truck or other destination for transport, etc.

[0040] In various embodiments, 3D or other image data generated by one or more of cameras 112, 114, and 116 may be used to generate a 3D view of the work area of ​​system 100 and items within the work area. The 3D image data may be used to identify items to be picked / placed, such as by color, shape, or other attributes. In various embodiments, one or more of cameras 112, 114, and 116 may be used to read text, logos, photographs, drawings, images, marks, barcodes, QR codes, or other encoded and / or graphical information or content visible on and / or comprising items within the work area of ​​system 100.

[0041] 1 , in the depicted example, system 100 includes a control computer 122 configured to communicate with elements such as robotic arm 102, carriage 104, effector 108, and sensors (such as cameras 112, 114, and 116, and / or weight, force, and / or other sensors not shown in FIG. 1), in this example via wireless communication (although in various embodiments, via one or both of wired and wireless communication). In various embodiments, control computer 122 is configured to use input from the sensors (such as cameras 112, 114, and 116, and / or weight, force, and / or other sensors not shown in FIG. 1) to observe, identify, and determine one or more attributes of items being loaded and / or unloaded from tray 120 to tray 118. In various embodiments, control computer 122 identifies the item and / or its attributes using item model data in a library stored in control computer 122 and / or accessible to control computer 122, for example, based on image and / or other sensor data. Control computer 122 uses the model corresponding to the item to determine and implement a plan for stacking the item, along with other items, in / on a destination (e.g., tray 118). In various embodiments, the item attributes and / or model are utilized to determine a strategy for grasping, moving, and placing the item at a destination location (e.g., a location where the item has been determined to be placed as part of the planning / re-planning process for stacking the item in / on tray 118).

[0042] In the illustrated example, control computer 122 is connected to an “on-demand” teleoperator 124. In some embodiments, if control computer 122 is unable to continue in fully automated mode, e.g., if a strategy for grasping, moving, and placing an item becomes indeterminable and / or fails such that control computer 122 does not have a strategy for completing the pick and place of an item in fully automated mode, control computer 122 instructs human user 126 to intervene, e.g., by manipulating one or more of robotic arm 102, carriage 104, and / or end effector 108 using teleoperator 124 to grasp, move, and place an item.

[0043] Figure 2 is a block diagram illustrating one embodiment of a system for controlling a robotic system. In various embodiments, the control computer 122 of Figure 1 is implemented as shown in Figure 2. In the illustrated example, the control computer 122 includes a communication interface 202 configured to send and receive one or more wireless, network, and other communications between the control computer 122 and other elements of the robotic system (e.g., the robot arm 102, carriage 104, end effector 108, cameras 112, 114, and 116, and / or robot base 118 and / or 120 in the system 100 of Figure 1).

[0044] In the illustrated example, the control computer 122 includes a planning module 204. The planning module 204 receives invoices, inventory, and spatial / location information (e.g., the location and status of robotic elements, sensors, inventory, empty trays / bins, etc.). The planning module 204 uses that information to determine and / or recalculate plans for picking / placing items from source locations to destination locations, for example, to meet one or more high-level requirements. For example, if a robotic system fulfills orders to deliver bread or other baked goods to retail stores, each according to an invoice or manifest indicating bread items to be included for each order / retail store, in some embodiments, the planning module 204 determines a series of actions to pick bread items from source trays and fill destination trays according to each order, and to load the trays associated with each order or each set of orders into one or more delivery vehicles for transportation to the respective retail store. All or a portion of the information received and / or plans generated by the planning module 206 may be stored in the order / inventory database 206.

[0045] In some embodiments, the planning module 204 may be configured to repeatedly and / or periodically create and / or review the plan. For example, imagery or other sensor data may be received and processed to generate an updated three-dimensional view of the workspace scene and / or to update information regarding which items of inventory are located at which source locations within the workspace. In the bread example, as new trays or stacks of trays arrive at the workspace, e.g., from a bakery, warehouse, etc., the items are identified, and a plan is created, updated, and / or reviewed to move toward achieving a high-level goal (e.g., efficiently fulfilling and transporting all orders) by picking available inventory from its current source location, moving / placing the items into the trays being filled, etc., to fulfill each individual order.

[0046] 2, in the illustrated example, control computer 122 further includes a robot control module 208. In various embodiments, robot control module 208 receives and processes sensor data (e.g., image data from a camera, force and / or torque sensor signals, vacuum / pressure readings, etc.) from a sensor module 210. Sensor module 210 receives input from one or more sensors, buffers the data, and provides input to robot control module 208 in a form usable by control module 208 to control operable elements of a robotic system (e.g., system 100 of FIG. 1 ) being controlled to perform tasks associated with implementing a plan received from plan module 204. In various embodiments, the manipulable elements include one or more of the following: respective sensors and / or actuators (e.g., motors for moving arm segments, valves or other elements for applying vacuum to suction cups, etc.) that make up the robotic arm and / or end effector (e.g., robotic arm 102, carriage 104, and end effector 108 of FIG. 1 ); other sensors in the workspace (e.g., cameras 114 and 116 of FIG. 1 ); and other robotic elements and / or elements that are manipulated or moved under robotic control (e.g., tray stacks 118 and 120 of FIG. 1 ).

[0047] The robot control module 208 controls the movement and other actuation of each robot using a robot model 210 of the one or more robots it is configured to control. For example, the model of the robot arm 102 and carriage 104 of FIG. 1 may be used to determine an efficient and relatively smooth way to actuate the joint motors that make up the arm 102 to rotate the arm 102 on the carriage 104 and / or move the carriage 104 along the rails 106 to pick and place an item from a source location to a destination location, without colliding with humans, other robotic elements, or other obstacles in the workspace, e.g., according to a plan received from the planning module 204.

[0048] In various embodiments, the robot control module 208 maintains and updates a set of item models 212 and / or grasping strategies 214. The item models 212, in various embodiments, describe the attributes of the item to be picked / placed and / or the item's classification or type (size, shape, dimensions, weight, stiffness, packaging type, logo or lettering on the packaging, etc.). In some embodiments, the item models 212 may be used to identify items in the workspace based on, for example, an image, weight, optical code scan, and / or other information.

[0049] In various embodiments, the robot control module 208 uses the grasping strategy 214 to determine a strategy for grasping an item to be moved, for example, from a source tray to a destination tray, based on, for example, the determined item or item type, size, shape, weight, etc., and contextual information (such as the item's location and orientation, what the item is adjacent to, etc.).

[0050] In various embodiments, the robot control module 208 may update one or both of the item model 212 and the grasping strategy 214 over time. For example, in some embodiments, machine learning or other techniques may be used to learn new and / or better grasping strategies over time. For example, frequently failing grasping strategies may be scored lower and / or eliminated, and new strategies may be developed and learned to replace them. In some embodiments, new strategies may be learned by observing how human operators perform teleoperated grasps. The grasping strategies 214 may be specific to an item, a set of items, and / or an item type. Some grasping strategies 214 may be related to the size, shape, dimensions, weight, etc. of unrecognized items.

[0051] The term "gripping" refers to any mechanism for controlling an item firmly enough that the end effector can move the item, either by grasping the item in the case of a gripper-type end effector, or by vacuum / suction by an end effector such as end effector 108 in Figure 1. The techniques disclosed herein may be used with any type of end effector.

[0052] In various embodiments, the robot control module 208 and / or the sensor module 210 receive and process the image data to generate a three-dimensional view of a workspace associated with the robotic system controlled by the control computer 122 and items present in the workspace. In various embodiments, the robot control module 208 and / or the sensor module 210 may be configured to control sensors, such as controlling the position and orientation of cameras present in the workspace, to generate a more complete view of the workspace and items within the workspace.

[0053] In various embodiments, control computer 122 may be configured to control two or more robotic systems in two or more physical locations and / or workspaces simultaneously, in a time-shared manner, etc. In some embodiments, elements shown in FIG. 2 as being comprised in a single physical computer 122 may be distributed across multiple physical or virtual machines, containers, etc.

[0054] In various embodiments, one or more of the planning module 204, the robot control module 208, and the sensor module 210 may comprise one or both of dedicated hardware (e.g., ASIC, programmed FPGA, etc.) and / or functional modules provided by executing software instructions on hardware processors that comprise the control computer 122.

[0055] FIG. 3 is a flowchart illustrating one embodiment of a process for controlling a robotic system. In various embodiments, process 300 of FIG. 3 may be implemented by a computer (such as control computer 122 of FIG. 1 ). In the illustrated example, at step 302, input data indicating high-level goals (e.g., items to be included in each of one or more individual orders) is received. At step 304, a plan for filling one or more of the orders is determined and / or updated. At step 306, a strategy for grasping, moving, and placing items is determined according to the plan. At step 308, remote human intervention is requested, if and for as long as necessary, for example, to perform or complete tasks that cannot be completed in fully automated mode. The process continues until all tasks are completed and the high-level goals are achieved (step 310), at which point process 300 ends.

[0056] FIG. 4 is a flowchart illustrating one embodiment of a process for determining a plan for performing a task using a robot. In various embodiments, step 304 of FIG. 3 is implemented, at least in part, by the process of FIG. 4. In the illustrated example, at step 402, an initialization routine and / or operation is performed to determine the position and pose / state (e.g., in three-dimensional space) of each robotic element, along with items, tools, containers, obstacles, etc., in the workspace. In some embodiments, step 402 may be performed at the start of a series of one or more operations, when the environment changes due to human or other robot intervention, or when data is received indicating that reinitialization may be required (e.g., a failed action, an unexpected condition, etc.). At step 404, imagery and / or other sensor data is received and processed to generate a three-dimensional view of the workspace or other scene or environment. At step 406, the location and contents of a source container (e.g., a tray of bread in the bakery example) are determined. For example, which items are where may be determined using one or more of image data, optical code scanning, recognition of logos or text on packaging, etc. In step 408, the location and status of the destination container is determined. For example, a stack of empty bins, one or more partially filled bins, etc. may be located using image data, RF or other electronic tags, optical codes, or other sensor data. The determined item and container location information, item identification, and other data (e.g., a manifest indicating which items are to be sent to which retail or other store location) are used in step 410 to calculate (or recalculate) a plan for picking and placing items in the workspace to fulfill the order. For example, if one tray full of wheat bread and one tray full of white bread are identified and located within the workspace, and an order is calling for X number of wheat breads and Y number of white breads, in some embodiments, in step 410, a plan may be determined to pick and place the wheat breads and white breads as needed to complete one or more trays on which the required number of each type of bread has been placed.

[0057] In various embodiments, the plan determined in step 410 includes retrieving one or more empty trays as needed, for example, from a staging or buffer area, and arranging / stacking the trays as needed in a loading area to allow items to be placed in the trays. Similarly, the emptied source trays may be moved by a robotic system, for example, to a staging or buffer area for later use in filling subsequently processed orders.

[0058] In various embodiments, the plan determined at step 410 includes determining, for each item to be placed, a location on the destination tray that is appropriate for that item (e.g., a location that is large enough and appropriately sized to accommodate the item). In some embodiments, the plan determined at step 410 takes into account the goal of ensuring efficient use of available tray space, given the items contained in the order (or waiting to be picked / placed to fulfill the order), for example, by packing items on each tray to utilize the optimal amount of surface space. In various embodiments, the plan may be continuously and / or periodically updated at step 410. Updated status information may be used to update the plan. For example, the status information may indicate, for a given tray, which items have already been placed on the tray, their respective positions and orientations, the remaining space available on the tray, and the next (or remaining) N items to be placed to fulfill the order.

[0059] Process 304 of FIG. 4 continues until high-level operations are complete and no further tasks are scheduled (step 412).

[0060] FIG. 5 is a flowchart illustrating one embodiment of a process for picking and placing an item using a robot according to a plan. In various embodiments, step 306 of FIG. 3 is implemented, at least in part, by the process of FIG. 5. In the illustrated example, at step 502, pick / place tasks associated with the plan are received. For example, the robot control module may assign, queue, dequeue, etc., specific pick / place tasks to be performed to progress toward the completion of a series of one or more higher-level operations. At step 504, one or more of the sensor data, item model information, and grasp strategy repository information are used to determine a strategy for grasping the item to be moved. At step 560, an attempt is made to grasp the item according to the strategy. If the grasp is successful (step 508), e.g., if image, weight, and / or other sensor data all indicate successful item grasping, at step 510 the item is moved to the destination location where it was intended to be placed, e.g., according to the previously determined plan. If the grasping attempt is unsuccessful (step 508), a determination is made at step 512 as to whether to retry the grasping, e.g., using the same or a different possible strategy. In some embodiments, retries may be made under fully automated control up to a set or predetermined number of times, or until no further strategies are determined to be available for grasping the item, whichever occurs first. If a retry is attempted (step 512), processing returns to step 504, where a (next) strategy is determined at step 506a for grasping the item using one or more of the sensors, item model, and grasping strategy information. If no further retries are attempted (step 512), human intervention (e.g., via teleoperation, manual execution, etc.) is requested at step 514, and a human worker / operator completes the task (step 510). The process of FIG. 5 then repeats (step 516) until all items have been picked / placed, at which point processing ends.

[0061] FIG. 6A illustrates one embodiment of an end effector including multiple suction cups. In the illustrated example, end effector 602 is movably connected to a robotic arm segment 604. Arm segment 604, in various embodiments, includes internal tubing for providing power and vacuum to end effector 602. In the illustrated example, end effector 602 includes two suction cups, including left-most suction cup 606. In various embodiments, end effector 602 includes internal actuator elements (not shown in FIG. 6A ) configured to independently apply suction force to the left and / or right suction cups. For example, when gripping a smaller item, only one suction cup may be used. Alternatively, two items may be gripped simultaneously, each utilizing one independently (or jointly) controlled suction cup. In some embodiments, the end effector 602 includes a sensor (such as a force sensor or other sensor) attached to the suction cup mount, which in various embodiments is used to detect initial contact with the item to be grasped and / or to determine the identity of the item and / or whether the item was successfully grasped (e.g., by weight). In some embodiments, the end effector 602 includes one or more torque sensors, for example, attached at the "wrist" where the end effector 602 connects to the arm segment 604. In some embodiments, the torque sensor may be used to determine, for example, whether an item was grasped at or near its center of gravity. In some embodiments, the end effector 602 includes a pressure sensor to determine whether the item was successfully grasped by the suction cup, for example, by detecting a good or bad fit between the suction cup and the item and / or package.

[0062] 6B illustrates one embodiment of an end effector comprising multiple suction cups. In the illustrated example, end effector 622 is movably mounted to robotic arm segment 624 and comprises six suction cups arranged in a 2x3 grid. In various embodiments, each of the six suction cups, and / or adjacent subgroups of two or three suction cups therein, may be independently actuated under the control of a robotic control system disclosed herein.

[0063] 6C illustrates one embodiment of an end effector comprising multiple suction cups. In the illustrated example, end effector 642 is movably mounted to robotic arm segment 644 and comprises 18 suction cups arranged in a 3x6 grid. In various embodiments, each of the suction cups, and / or adjacent subgroups thereof, may be independently actuated under the control of a robotic control system disclosed herein.

[0064] 6A, 6B, and 6C include the number and arrangement of suction cups shown, in various embodiments more or fewer suction cups are included, and in some embodiments, gripping structures other than suction cups are included.

[0065] FIG. 7 is a flowchart illustrating one embodiment of a process for gripping an item with a robotic arm and end effector. In various embodiments, process 700 of FIG. 7 may be performed by a computer (e.g., control computer 122) configured to operate a robotic arm and end effector (e.g., robotic arm 102 and end effector 108 of FIG. 1). In some embodiments, process 700 is performed by robot control module 208 of FIG. 2. In the illustrated example, in step 702, image and / or other sensor data is used to move the end effector within proximity of the item to be gripped. For example, the location in three-dimensional space of the surface of a package to be engaged by a suction-type end effector is determined based on the image data, and a trajectory is calculated to move the item-engaging end of the gripper's suction cup within a predetermined distance of the surface. Image data continues to be monitored as the end effector is moved within proximity of the package (or item) surface. In various embodiments, movement is stopped once the end effector has been moved to the position and orientation of the calculated endpoint of the trajectory and / or if image or other sensor data indicates that the end effector has reached the vicinity of the item before reaching the endpoint of the trajectory; for example, movement may be stopped when a force sensor in the end effector detects that the suction cup has made contact with the package / item. Once the end effector has been moved to a position proximate to the item to be grasped (steps 702, 704), the coarse movement (e.g., relatively long distance and / or relatively high speed) of the end effector and arm is stopped in step 706, and the item is attempted to be grasped by one or more of fine movement of the end effector and actuation of a suction or other item engagement mechanism of the end effector.

[0066] In various embodiments, one or more of the following may be performed, present, and / or achieved: Suction-based grasping minimizes the amount of free space required around the item; only the top of the item needs to be accessible. Compliant suction cups reduce contact force and protect the product (i.e., the item being gripped). A pressure sensor to detect when the cup is in intimate contact with the surface of the product. In some embodiments and / or gripping, the wrapping can be tightly gripped without even touching the wrapped product. · Force sensors to detect when force is being applied to the product. Any other combination of sensors to detect the following two gripping conditions: (1) the end effector is in sufficient contact with the product to grip, and (2) the end effector cannot move any closer without risking damaging the product. In some embodiments, these sensors may include proximity sensors and / or limit switches on the suction cup mount. A closed control loop that moves the robot end effector toward the product until either condition (1) or (2) above is encountered.

[0067] A determination is made at step 708 as to whether the item was successfully grasped. For example, a pressure sensor may be used to determine whether the suction grasp was successfully completed. For example, a determination that the item was successfully grasped is made when the required vacuum is achieved. In some embodiments, a determination that the item was successfully grasped may be made based on an attribute of the item (e.g., weight) and whether vacuum is achieved for at least a minimum number of suction cups on the end effector (e.g., the minimum number of cups associated with securely grasping an item of that weight).

[0068] If / when the grasp is determined to be successful (step 708), the item is moved to the associated destination location at step 710. For example, the robotic arm and end effector are moved along a trajectory calculated by the control computer and / or module to move the item to the destination and release the item from the end effector's grip. If the grasp is not (yet) successful (step 708), image and / or other sensor data is evaluated to determine a (new / modified) grasp strategy at step 712. In some embodiments, step 712 includes evaluating whether to attempt a (further) retry; if not, human intervention is initiated. If a new grasp strategy is determined at step 712, the grasp strategy is attempted at step 706, and if successful (step 708), the item is moved to its destination at step 710. Once the item has been moved to its destination (e.g., in step 710 and / or by human intervention if the automatic picking / moving has failed), a determination is made in step 714 as to whether there are more items to be picked up and moved. If there are, then processing proceeds to the next item to be moved in step 716, and a further iteration of process 700 is performed for that item. If there are no more items to be picked up and moved (step 714), processing ends.

[0069] 8 is a flow chart illustrating one embodiment of a process for reliably moving an item grasped by a robotic arm and / or end effector to a destination. In various embodiments, process 800 of FIG. 8 may be performed by a computer (e.g., control computer 122) configured to operate the robotic arm and end effector (e.g., robotic arm 102 and end effector 108 of FIG. 1). In some embodiments, process 800 is performed by robot control module 208 of FIG. 2.

[0070] In various embodiments, a robotic system may grasp an item, for example, by process 700 of FIG. 7 , and then use one or more sensor-based checks to ensure that the grasp is and remains accurate and secure, for example, the robotic system may achieve the following: · Querying databases of SKU-specific or other item information, for example, to determine the weight and / or other attributes of an item. Check pressure sensor readings for air leaks that weaken the grip. Checking force sensor readings to ensure the correct type and quantity of items were grasped. A camera detects the position of the grasped item relative to the end effector, corroborating readings from pressure and force sensors. Verifies that the size of the grasped item matches the size in the SKU or other item database. Torque readings check that the item is gripped near the center of gravity. Upon any (over threshold) discrepancy between sensor readings or failure of any check, the robot returns the grasped object to its original position or a nearby "safe" location for further attempts / inspection. The robot may then continue picking (e.g., other) products.

[0071] 8, in the illustrated example, while an item is being moved (step 802), e.g., after being successfully grasped using a robotic arm and end effector, sensor data from multiple sensors and / or sensor types (e.g., image, force, pressure, etc.) is monitored at step 804. For each sensor / sensor type, at step 806, the sensor data is evaluated to determine whether the data is consistent with a (continued) secure grasp of the item. If all (or a threshold percentage) of the sensors indicate that the grasp remains secure (step 808), then at step 810 the item is moved (and / or continues to be moved) to its destination. If any (or a threshold number) of sensors and / or sensor types provide data suggesting that the grasp is not or no longer secure (step 808), the item is moved to a buffer location (or to the item's original source location, if the robotic system determines that the grasp is secure and / or close enough to move it to that location) in step 812, and once in a safe location, the robotic system determines a grasp strategy and attempts to re-grasp the item. Monitoring the sensors (step 804) and using the sensor data to evaluate whether the grasp is / remains secure (steps 806, 808) continues until the item is successfully moved to its destination (step 814), at which point process 800 ends.

[0072] 9 is a block diagram illustrating one embodiment of a suction-based end effector. In various embodiments, the robot arm end effector 900 of FIG. 9 may be used to implement the end effector 108 of FIG.

[0073] In the depicted example, the end effector 900 comprises a body or housing 902 attached to a robotic arm 904 via a rotatable coupling. In some embodiments, the connection between the housing 902 and the robotic arm 904 may comprise a motor-driven joint controlled by a control computer (such as control computer 118 in FIG. 1 ). The end effector 900 further comprises a suction line or other air line 906 that passes through the robotic arm 904 and into the housing 902 to provide a vacuum source to a suction control module 908. In various embodiments, the control module 908 is connected to a control computer external to the end effector 900 (e.g., control computer 11 in FIG. 1 ) through a communications interface 914, e.g., by wireless and / or wired communication. The control module 908 comprises electronic and / or electromechanical elements operable to provide a suction force to suction cups 910, 912, e.g., to attach the end effector to items picked up, moved, and placed using the end effector 900 by suction.

[0074] In various embodiments, suction control module 908 is configured to apply suction forces independently to cups 910 and 912, allowing only one or the other cup to be used to grasp a given item. In some embodiments, two adjacent items (such as two pans) may be grasped and moved simultaneously, each grasped by a corresponding one of cups 910 and 912.

[0075] In the depicted example, a camera 916 mounted on the side of the housing 902 provides image data of the field of view below the end effector 900. Multiple force sensors 918, 920, 922, 924, 926, and 928 measure the force being applied to the suction cups 910 and 912, respectively. In various embodiments, the force measurements are communicated to an external and / or remote control computer via a communications interface 914. The sensor readings are used in various embodiments to enable the robotic arm 904 and end effector 900 to fit items into place adjacent other items and / or side walls or other structures and / or to detect instability (e.g., insufficient pushback where an item is pushed down while still under suction and in the location where it would be placed and stable). In various embodiments, pairs of horizontally mounted force sensors (eg, 918 and 922, 924 and 928) are positioned at right angles in the xy plane to allow for force determination in all horizontal directions.

[0076] In some embodiments, force sensors may be used to detect initial contact between the suction cups 910, 912 and the item to be grasped. The robotic arm 904 is stopped upon detection of initial contact to avoid damage to the item (such as crushing the bread or crumpling its packaging). A suction force is then applied to grasp the item using suction.

[0077] 10 is a flow chart illustrating one embodiment of a process for simultaneously picking and placing items when / if necessary to best achieve goals assigned to the robotic system. In various embodiments, process 1000 of FIG. 10 may be performed by a computer (such as control computer 122) configured to operate a robotic arm and end effector (such as robotic arm 102 and end effector 108 of FIG. 1). In some embodiments, process 1000 is performed by robot control module 208 of FIG. 2.

[0078] In various embodiments, the robotic systems disclosed herein may achieve the following: · Pick multiple items in one run to increase the throughput of your pick system. - Generate different pick (grasp) postures for different item combinations. Detect combinations of pickable items that fit the current end effector configuration for a firm, tight grip. For example, a gripper (end effector) with two large suction cups may simultaneously pick up items of the same or similar size. Deciding whether to perform a multi-grab based on the current quantity of all pickable items and the customer's needs (e.g., current manifest). Adapt to various system conditions to perform multiple grasps in a safe manner, for example, adjusting the gripper posture to accommodate possible collisions when items are at the edge of the work area. Identify and filter out potentially dangerous cases. Adapt to different gripper configurations: Based on a given gripper configuration (e.g., diameter between gripper penetration points), evaluate the feasibility of multiple grasps and generate different grasp poses. · Perform cost evaluation between possible single and multiple grasps. Evaluate the grasp cost for all possible multiple grasps to achieve the lowest risk in stable multiple grasps.

[0079] 10 , in the illustrated example, item data (e.g., item attribute data, quantity of each item needed to fulfill the current / next order, source / start location of items to be moved, status of source and destination locations (e.g., trays), etc.) is evaluated in step 1002. Single items and supported combinations of items that may be picked and placed as part of a series of operations to fulfill the order are evaluated in step 1004. For example, if two or more of the same item are needed to fulfill an order, adjacent items exist in the source tray, and there is space available (or would be or could be available under the scenario being evaluated during planning), moving two (or more) items in a single task may be considered for inclusion in the plan to fulfill the order. At step 1006, the combination of lowest (overall / total) cost (e.g., time, energy, etc.) and / or lowest risk (e.g., the risk of dropping an item may be considered to be greater by a measurable or predictable amount than moving items one by one) is determined for single-item and / or multiple-item pick / place tasks to fulfill the order. In some embodiments, each cost and risk may have an associated weighting, for example, in a cost function, and a plan that minimizes the cost function may be determined. At step 1008, the items are picked / placed according to the plan determined at step 1006. Processing continues until completed (step 1010).

[0080] In various embodiments, one or more of the steps of process 1000 may be performed continuously and / or periodically to refine the plan determined in step 1006 based on, for example, information that becomes available while the plan is being executed (e.g., a new source tray of items or other source containers is moved into the workspace, human intervention or other unanticipated intervention or event changes the state of any items and / or containers in the workspace, the actual state of a (e.g., destination) tray differs from that expected as a result of a failed pick / place task, etc.).

[0081] FIG. 11 is a flowchart illustrating one embodiment of a process for adjusting the final placement of an item. In various embodiments, the process of FIG. 11 may be performed by a computer (e.g., control computer 122) configured to operate a robotic arm and end effector (e.g., robotic arm 102 and end effector 108 of FIG. 1). In some embodiments, the process of FIG. 11 is performed by robot control module 208 of FIG. 2. In the illustrated example, as an item is being moved (step 1102), the system detects that the item has been moved to a position close to its intended destination location (step 1104). For example, image sensor, force sensor, and / or other sensor data may be used to detect proximity to the final location. In some embodiments, a vertically oriented force sensor may be used to detect when the item contacts the tray bottom or other container surface into which it is to be placed. In step 1106, in response to detecting proximity to the final destination location, a final placement operation is performed to ensure a snug fit before the item is released. For example, horizontal or other force sensors, torque sensors, contact sensors, image sensors, or combinations thereof may be used to assess contact or near-contact of an item with adjacent items and / or structures (such as the sidewalls of a container). The robotic arm and / or end effector may be manipulated to enhance adjacency, such as by ensuring that the item to be placed contacts along any edges and / or surfaces that come flush against the adjacent item according to the pick / place plan. For example, if force sensor readings for sensors associated with the same horizontal access and / or contact edge or surface of the item are not (substantially) the same, the end effector may be rotated and / or the arm moved to reposition the item to a position where the force sensor senses substantially the same force.

[0082] FIG. 12 is a flow chart illustrating one embodiment of a process for selecting a destination location for an item. In various embodiments, the process of FIG. 12 may be performed by a computer (e.g., control computer 122) configured to operate a robotic arm and end effector (e.g., robotic arm 102 and end effector 108 of FIG. 1). In some embodiments, the process of FIG. 12 is performed by robot control module 208 of FIG. 2. In the illustrated example, in step 1202, the condition of a destination tray to which one or more items are to be moved is evaluated. For example, image data and data from previously executed pick / place tasks may be used to determine available slots, spaces, and / or locations available to receive the items to be placed in the destination tray or other container. In step 1204, an optimal slot is selected from the slots determined to be available in step 1202.

[0083] In some embodiments, the optimal slot may be selected according to an algorithm and / or other criteria. For example, in some embodiments, the slot with the shortest perimeter for the items in the destination tray may be selected. In some embodiments, to ensure that the collective set of placement locations is optimal, slots for N items may be determined simultaneously when information about the next N items to be placed is available. In some embodiments, if a previous placement decision affects the availability and / or optimality of a later placement decision, the previously placed item may be moved to a programmatically determined position and / or orientation to make a (more optimal) slot available for the next item to be placed. For example, the decision to place a first item may be made before the system knows which item will be placed next and / or before the system has information about such item. In such a condition, the system may have selected a placement for the previously placed item that would lead to an overall suboptimal situation given the later-placed items. In some embodiments, the system is configured to detect such situations and move and / or change the placement / orientation of a previously placed item if it becomes more / more likely that the item will be placed next or later.

[0084] In some embodiments, if there are no slots available to accept the item at steps 1202 and / or 1204, and in some embodiments, no smaller items that can fit in the tray are available to be picked / placed into the space available and sufficient to accept such items, the system determines that the tray is full and retrieves an empty tray, for example, from a staging area. If the tray stack is not too high and the filled tray and the next new tray to be filled are destined for the same location (e.g., same delivery vehicle / route), the new tray may be stacked on top of the tray determined to be full.

[0085] Once a suitable / optimal slot is selected for placing the item, the item is placed in that location in step 1206, for example, by the processes of one or both of Figures 7 and 11.

[0086] FIG. 13 is a flow chart illustrating one embodiment of a process for detecting misplacement of an item. In various embodiments, the process of FIG. 13 may be performed by a computer (e.g., control computer 122) configured to operate a robotic arm and end effector (e.g., robotic arm 102 and end effector 108 of FIG. 1). In some embodiments, the process of FIG. 13 is performed by robot control module 208 of FIG. 2. In the illustrated example, when an item is placed (step 1302), e.g., moved and placed in its final position but not yet released from the grasp of the end effector, image and / or other sensor data is used to evaluate the placement of the item (step 1304). For example, three-dimensional image data may indicate that the height of the item is higher than expected for a successful placement. For example, the height of the top surface of the item may be expected to be equal to the height of the bottom of the tray plus the expected height of the item. If the top surface is at a higher elevation, such information may suggest that the item was placed completely or partially on top of another item, or that the item may have been crushed or otherwise deformed during the pick / place operation, or that the wrong item may have been placed, etc. If no error is detected (step 1306), the item is released from grip (step 1308), and the arm and end effector are moved to perform the next pick / place task. If an error is detected (step 1306), the item is returned to the starting location, or in some embodiments, to another staging or buffer area, at step 1310, and the pick / place operation is attempted again. In some embodiments, repeated attempts to pick / place an item that lead to the detection of an error may trigger an alert and / or other action to initiate human intervention.

[0087] Figure 14 illustrates one embodiment of a robotic system for handling flexible products in non-rigid packages. In the illustrated example, system 1400 includes robotic arms 1408 and 1410 mounted for computer-controlled movement along rails 1404 and 1406, respectively. Robotic arms 1408 and 1410 terminate in suction-type end effectors 1412 and 1414, respectively. In various embodiments, robotic arms 1408 and 1410 and end effectors 1412 and 1414 are controlled by a robotic control system including a control computer (such as control computer 122 of Figures 1 and 2).

[0088] 14 , robotic arms 1408 and 1410 and end effectors 1412 and 1414 are used to move items, such as bread, from a source tray on wheeled base 1402 to destination trays on wheeled bases 1416 and 1418. In various embodiments, a human and / or robot may position the source tray and wheeled base 1402 at a starting location between rails 1404 and 1406, as shown. Wheeled base 1402 may be advanced through a path formed by rails 1404 and 1406, for example, in the direction of the arrow beginning at the far end of base 1402, as shown. In various embodiments, base 1402 may be advanced using one or both of robotic arms 1408 and 1410, by being pushed manually by one or more human operators, by one or more fixed or unfixed robots, etc., by a conveyor belt or chain type mechanism operating along and / or between rails 1404 and 1406, by a robotically controlled propulsion and / or transport mechanism (such as computer-controlled motorized wheels and brakes) incorporated into base 1402, etc.

[0089] As wheeled base 1402 advances along / between rails 1404 and 1406, and / or while base 1402 is temporarily in a stationary position between rails 1404 and 1406, in various embodiments, the robotic system uses robotic arms 1408 and 1410 and end effectors 1412 and 1414 to move items (such as bread) from a source tray on wheeled base 1402 to a destination tray on wheeled bases 1416 and 1418 according to a plan. For example, the plan may be calculated based on inventory and / or sensor data indicating which one or more types of bread are available in source trays on base 1402 and / or other bases in the work area, and further based on manifest or other data indicating which combinations of bread or other items are to be placed in destination trays for delivery to each respective final delivery location (e.g., a retail store).

[0090] In the illustrated example, once the destination trays on bases 1416 and 1418 are filled, bases 1416 and 1418 are moved outwardly away from rails 1404 and 1406, respectively, e.g., to a staging area and a loading area for loading onto a delivery vehicle.

[0091] 14 shows a single set of source trays on a single base 1402, in various embodiments, one or more additional bases (each having zero or more trays filled with stacked items) may be positioned between rails 1404 and 1406 and / or in a nearby staging area depending on the high-level operations being performed. Similarly, in some embodiments, multiple sets of destination trays, each on a corresponding wheeled base, may be staged adjacent to rail 1404 or rail 1406, with the top tray of each being in the process of being filled simultaneously and / or sequentially, for example, according to a manifest and associated execution plan.

[0092] In various embodiments, once a source tray is empty, the system may use the robotic arms 1408 and / or 1410 to move the tray to a staging area and / or to the top of a stack of destination trays, thereby creating a supply of trays to be filled and sent to a final destination and / or exposing the next source tray from which items can be picked to fulfill an order.

[0093] While FIG. 14 shows a single robotic arm (1408, 1410) positioned on each rail (1404, 1406), in some embodiments, two or more robotic arms may be provided on each rail. In some embodiments, two robotic arms on a single rail may be used to pick up and move empty trays. For example, each arm may be used to engage opposite sides of the tray, either mechanically by engaging structures on the tray or, in various embodiments, by suction or gripping, and the two arms may be coordinated to maintain control of the empty tray while moving it to a new location (e.g., a buffer or staging area, or the top of a stack of destination trays). In some embodiments, one or more robotic arms may be used to move a full or partially full tray from the top of the stack (e.g., to be moved to a staging area) to expose trays below the stack for purposes of adding additional items, rearranging items according to a new or revised plan, etc.

[0094] Cameras 1420 and 1422, and / or cameras 1424 and 1426, mounted on end effectors 1412 and 1414, respectively, are used in various embodiments to generate image data for planning and executing the pick and place operations disclosed herein.

[0095] In various embodiments, the technology disclosed herein enables soft or otherwise fragile products in non-rigid packaging (such as bread packaged in plastic bags or wrap) to be handled by a robotic system without damaging the handled items. For example, in various embodiments, bread may be identified, selected, and moved by a robot from a source tray received from a bakery or storage area to a destination tray for delivery to a corresponding specific location (such as a retail store) with minimal or no human intervention. In various embodiments, the technology disclosed herein enables higher throughput and efficiency compared to purely human and / or partially automated operation of distribution systems, facilities, and processes for, for example, bread or fragile items in soft packaging.

[0096] Although the above embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting. [Application Example 1] A robot system, a communication interface; a processor connected to the communication interface; Equipped with The processor: receiving, via the communication interface, sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system reside; determining an action to be performed within the workspace using one or more of the robotic elements based at least in part on the sensor data, the action including relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped, actuating a grasping mechanism of the end effector to grasp the item using an amount of force and configuration associated with minimal risk of damage to the item and / or its packaging, and confirming that the item has been securely grasped using sensor data generated after the item has been grasped; a system configured to send control communications to the robotic element via the communications interface to cause the robotic element to perform the action; [Application Example 2] A system according to Application Example 1, wherein the processor is further configured to use the robot element to move the grasped item to a destination position. [Application Example 3] A system according to Application Example 1, wherein the end effector is equipped with a suction-based gripping mechanism. [Application Example 4] A system according to Application Example 3, wherein the end effector comprises one or more suction cups. [Application Example 5] A system as described in Application Example 4, wherein the one or more suction cups comprise a compliant material that collapses upon contact with the item. [Application Example 6] The system according to Application Example 1, wherein the sensor data includes image data generated by one or more cameras. [Application Example 7] A system according to Application Example 1, wherein the sensor data includes data generated by one or more of a pressure sensor, a force sensor, a torque sensor, and a contact sensor. [Application Example 8] A system according to Application Example 1, wherein the processor is further configured to use the robot element to move the grasped item to a destination position and place the item at the destination position. [Application Example 9] A system as described in Application Example 8, wherein the processor is configured to place the item in a first stage of operation, at least in part, by moving the item to a position proximate to the destination position and using the sensor data to adjust the position of the item so that it is more closely adjacent to a structure or a second item adjacent to the destination position. [Application Example 10] A system according to Application Example 1, wherein the processor is configured to determine the action based at least in part on a plan. [Application Example 11] The system described in Application Example 10, wherein the processor is configured to generate the plan based on a high-level goal. [Application Example 12] A system as described in Application Example 11, wherein the high-level goal includes identification of multiple items and, for each item, a corresponding quantity to be included in the set of items associated with the final destination. [Application Example 13] The system described in Application Example 12, wherein the items include different bread products and the high-level goal includes a manifest or other list, or items and quantities of each item to be delivered to a retail store or other destination. [Application Example 14] A system as described in Application Example 1, wherein the processor is further configured to use the robot element to move the grasped item to a destination position, place the item at the destination position, and then use received sensor information to confirm the placement of the item before releasing the item from the grasping mechanism of the end effector. [Application Example 15] A system as described in Application Example 1, wherein the processor uses the sensor information to, at least in part, determine the height of the product from a height reference and compare the determined value with an expected value associated with successful placement, thereby confirming the placement of the item before releasing it from the gripping mechanism of the end effector. [Application Example 16] A system as described in Application Example 1, wherein the end effector comprises multiple sets of one or more independently actuated suction cups, and the processor is further configured to control the robot element to simultaneously grasp two or more items using each of one or more sets of the independently actuated set of one or more suction cups. [Application Example 17] A system as described in Application Example 16, wherein the processor is further configured to determine an optimal plan for picking and placing multiple items, such as by determining possible combinations of two or more items that can be grasped simultaneously and determining an optimal plan that includes a combination of single-grasp and multiple-grasp pick-and-place actions. [Application Example 18] A system according to Application Example 1, wherein the processor is further configured to determine a strategy for grasping the item using the robot element. [Application Example 19] A method, receiving, via a communication interface, sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system reside; determining an action to be performed within the workspace using one or more of the robotic elements based at least in part on the sensor data, the action including relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped, actuating a grasping mechanism of the end effector to grasp the item using an amount of force and configuration associated with minimal risk of damage to the item and / or its packaging, and confirming that the item has been securely grasped using sensor data generated after the item has been grasped; sending a control communication to the robotic element via the communication interface to cause the robotic element to perform the action; A method comprising: [Application Example 20] A computer program product embodied in a non-transitory computer-readable storage medium, computer instructions for receiving, via a communications interface, sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system reside; computer instructions for determining an action to be performed within the workspace using one or more of the robotic elements based at least in part on the sensor data, the action including relatively quickly moving an end effector of one of the robotic elements into proximity with an item to be grasped, actuating a grasping mechanism of the end effector to grasp the item using an amount of force and configuration associated with minimal risk of damage to the item and / or its packaging, and verifying that the item has been securely grasped using sensor data generated after the item has been grasped; computer instructions for sending control communications to the robotic element via the communications interface to cause the robotic element to perform the action; A computer program product comprising:

Claims

1. 1. A robotic system comprising: a communication interface; a processor connected to the communication interface; Equipped with The processor: receiving, via the communication interface, sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system reside, the sensor data being received from a plurality of different types of sensors, the plurality of different types of sensors being at least pressure sensors and torque sensors, the sensor data obtained by the pressure sensors indicating whether an air leak exists that weakens the grip on an item, and the sensor data obtained by the torque sensors indicating whether the item has been grasped near its center of gravity; determining an action to be performed in the workspace using the one or more robotic elements based at least in part on the sensor data, the action including moving an end effector of one of the one or more robotic elements into proximity with the item to be grasped and activating (i) one or more suction cups comprising a compliant material that dampens contact forces between the end effector and the item, and (ii) a grasping mechanism of the end effector using the sensor data acquired by the pressure sensor to grasp the item; using the sensor data generated by the plurality of different types of sensors after the item has been grasped to ensure that the item has been securely grasped, at least in part by determining whether a threshold number of the plurality of different types of sensors provides data indicating that the grasp of the item remains secure, and moving the item to a buffer location in response to a determination that the threshold number of the plurality of different types of sensors provides data indicating that the grasp of the item is not or is no longer secure; a robotic system configured to send control communications to the robotic elements via the communications interface to cause the robotic elements to perform the actions.

2. 10. The robotic system of claim 1, wherein the processor is further configured to move the grasped item with the robotic element to a destination location.

3. The robotic system of claim 1 , wherein the end effector comprises a suction-based gripping mechanism.

4. 10. The robotic system of claim 1, wherein the compliant material of the one or more suction cups collapses upon contact with the item.

5. The robotic system of claim 1 , wherein the sensor data includes image data generated by one or more cameras.

6. The robotic system of claim 1 , wherein the sensor data includes data generated by two or more of a pressure sensor, a force sensor, a torque sensor, and a contact sensor.

7. 2. The robotic system of claim 1, wherein the processor is further configured to move the gripped item to a destination location using the robotic element and place the item at the destination location.

8. 8. The robotic system of claim 7, wherein the processor is configured to place the item in a first stage of operation, at least in part, by moving the item to a position proximate to the destination location and using the sensor data to adjust the item's position to be more closely adjacent to a structure or a second item adjacent to the destination location.

9. The robotic system of claim 1 , wherein the processor is configured to determine the action based at least in part on a plan.

10. 10. The robotic system of claim 9, wherein the processor is configured to generate the plan based on a high-level goal.

11. 11. The robotic system of claim 10, wherein the high-level goal includes an identification of a plurality of items and, for each item, a corresponding quantity to be included in a set of items associated with a final destination.

12. 12. The robotic system of claim 11, wherein the items include different bread products and the high-level goal includes a manifest or other list or items and quantities of each item to be delivered to a retail store or other destination.

13. 10. The robotic system of claim 1, wherein the processor is further configured to: use the robotic elements to move the grasped item to a destination location; place the item at the destination location; and use subsequently received sensor data to verify placement of the item before releasing the item from the grasping mechanism of the end effector.

14. 10. The robotic system of claim 1, wherein the processor is further configured to use the robotic elements to move the grasped item to a destination location and place the item at the destination location, and the processor uses the sensor data to verify placement of the item before releasing it from the grasping mechanism of the end effector, at least in part, by using the sensor data to determine a height of the item from a height reference and comparing the determined value to an expected value associated with successful placement.

15. 2. The robotic system of claim 1, wherein the end effector comprises multiple sets of independently actuated one or more suction cups, and the processor is further configured to control the robotic elements to simultaneously grasp two or more items using one or more sets of the independently actuated set of one or more suction cups, respectively.

16. 16. The robotic system of claim 15, wherein the processor is further configured to determine an optimal plan for picking and placing multiple items, such as by determining possible combinations of two or more items that can be grasped simultaneously and determining an optimal plan that includes a combination of single-grasp and multi-grasp pick-and-place actions.

17. The robotic system of claim 1 , wherein the processor is further configured to determine a strategy for grasping the item with the robotic elements.

18. 1. A method comprising: receiving, via a communications interface, sensor data associated with a workspace in which one or more robotic elements controlled at least in part by the robotic system reside, the sensor data being received from a plurality of different types of sensors, the plurality of different types of sensors being at least pressure sensors and torque sensors, the sensor data obtained by the pressure sensors indicating whether an air leak exists that weakens the grip on an item, and the sensor data obtained by the torque sensors indicating whether the item has been grasped near its center of gravity; determining an action to be performed in the workspace using the one or more robotic elements based at least in part on the sensor data, the action including moving an end effector of one of the one or more robotic elements into proximity with the item to be grasped and activating (i) one or more suction cups comprising a compliant material that dampens contact forces between the end effector and the item, and (ii) a grasping mechanism of the end effector using the sensor data acquired by the pressure sensor to grasp the item; using the sensor data generated by the plurality of different types of sensors after the item has been grasped to ensure that the item has been securely grasped, at least in part by determining whether a threshold number of the plurality of different types of sensors provides data indicating that the grasp of the item remains secure, and moving the item to a buffer location in response to a determination that the threshold number of the plurality of different types of sensors provides data indicating that the grasp of the item is not or is no longer secure; sending a control communication to the robotic element via the communication interface to cause the robotic element to perform the action; A method comprising:

19. A computer program product embodied in a non-transitory computer-readable storage medium, computer instructions for receiving, via a communications interface, sensor data associated with a workspace in which one or more robotic elements controlled at least in part by a robotic system reside, the sensor data being received from a plurality of different types of sensors, the plurality of different types of sensors being at least pressure sensors and torque sensors, the sensor data obtained by the pressure sensors indicating whether an air leak exists that weakens the grip on an item, and the sensor data obtained by the torque sensors indicating whether the item is being grasped near its center of gravity; and computer instructions for determining an action to be performed within the workspace using the one or more robotic elements based at least in part on the sensor data, the action including: moving an end effector of one of the one or more robotic elements into proximity with the item to be grasped and activating (i) one or more suction cups comprising a compliant material that reduces contact forces between the end effector and the item, and (ii) a grasping mechanism of the end effector using the sensor data acquired by the pressure sensor to grasp the item; using the sensor data generated by the plurality of different types of sensors after the item has been grasped to ensure that the item has been securely grasped, at least in part by determining whether a threshold number of the plurality of different types of sensors provides data indicating that the grasp of the item remains secure, and moving the item to a buffer location in response to a determination that the threshold number of the plurality of different types of sensors provides data indicating that the grasp of the item is not or is no longer secure; computer instructions for sending control communications to the robotic element via the communications interface to cause the robotic element to perform the action; A computer program product comprising:

20. The robot system according to claim 1, the one or more suction cups include at least a first suction cup and a second suction cup; the first suction cup and the second suction cup are independently controlled; a combination of items available for pickup is detected based at least in part on the sensor data; the action to be performed includes simultaneously moving two or more items based at least in part on a determination that two or more items are needed to fill an order, the two or more items being in close proximity in a source tray from which the two or more items are removed, and sufficient space exists at a destination location for the two or more items.

21. The robot system according to claim 1, the gripping mechanism is actuated to grasp the item by adjusting the force applied to the item until (i) the end effector makes sufficient contact with the item for grasping, or (ii) the end effector cannot approach the item without risking damage, whichever occurs first; A robotic system, wherein the sensor data acquired by the pressure sensor is used to detect (i) whether the end effector is in sufficient contact with the item for the grasping, and (ii) whether the end effector cannot approach the item without risking damage.

Images