Multi-mode robotic end effector

Abstract

To provide an autonomous tray handling line kitting robot that comprises a robotic arm having an end effector with a structure for grasping a tray of items.SOLUTION: Disclosed is an end effector, the end effector including a first gripping mechanism for grasping at least one first object when the robotic end effector is operated in a first mode, and a second gripping mechanism for grasping a second object when the robotic end effector is operated in a second mode. The second gripping mechanism is robotically placed in an inactive state when the robotic end effector is controlled to operate in the first mode.SELECTED DRAWING: Figure 3A

Description

CROSS-REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 253,045, entitled "MULTI-MODE ROBOTIC END EFFECTOR," filed October 6, 2021, which is incorporated herein by reference for all purposes. [Background technology]

[0002] In certain warehouse and similar operations, a series of tasks sometimes referred to as "line kitting" may be performed to assemble stacked trays of items for further distribution (such as delivery to retail stores). Stacks of trays containing the same type of item may be received, and trays are pulled from different homogenous stacks, each having trays of a corresponding type of item, to assemble a mixed stack of trays (e.g., to be sent to a given destination).

[0003] For example, a bakery may bake different types of products, each filling stackable trays with a corresponding uniform type of product (such as a particular type of bread or other baked goods). Stacks of trays may be provided by the bakery, for example, to a distribution center. One stack may include trays holding sliced ​​white bread, another stack may have trays holding whole wheat bread, and yet another tray may have trays holding packages of blueberry cupcakes, etc. Trays may be pulled from various stacks to assemble a (potentially) mixed stack of trays. For example, a stack containing six white bread trays, three whole wheat bread trays, and one blueberry cupcake tray may be assembled, for example, for delivery to a retail store.

[0004] Although the above examples include trays of different types of baked goods, in other line kitting operations, stackable trays may hold other products.

[0005] In a typical approach, the trays are handled by a human worker. The trays may include handles to allow the human worker to grasp and move the tray, for example, by placing the worker's hand on or in the handle. Such tasks by a human worker can cause fatigue or injury, can take a lot of time to complete, and can be prone to error. [Brief explanation of the drawings]

[0006] Various embodiments of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0007] [Figure 1A] FIG. 1 is a block diagram illustrating an embodiment of a robot line kitting system.

[0008] [Figure 1B] FIG. 1 is a block diagram illustrating an embodiment of a robot line kitting system.

[0009] [Figure 2A] FIG. 1 is a state diagram illustrating one embodiment of an automated process for building a stack of trays.

[0010] [Figure 2B] 1 is a flow chart illustrating one embodiment of an automated process for building a stack of trays.

[0011] [Figure 2C] 1 is a flow chart illustrating one embodiment of an automated process for picking and placing items to and from a tray.

[0012] [Figure 3A] FIG. 1 illustrates one embodiment of a robotically controlled tray handling end effector.

[0013] [Figure 3B] FIG. 1 illustrates one embodiment of a robotically controlled tray handling end effector.

[0014] [Figure 3C] FIG. 1 illustrates one embodiment of a robotically controlled tray handling end effector.

[0015] [Figure 4] 1 is a flowchart illustrating a process for operating an end effector to move an object, according to various embodiments.

[0016] [Figure 5A] 10 is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to / from a tray, according to various embodiments.

[0017] [Figure 5B] 10 is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to / from a tray, according to various embodiments.

[0018] [Figure 6A] 10 is a flowchart illustrating a process for operating an end effector in connection with picking or placing a tray or other container, according to various embodiments.

[0019] [Figure 6B] 10 is a flowchart illustrating a process for operating an end effector in connection with picking or placing a tray or other container, according to various embodiments.

[0020] [Figure 6C] 10 is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to / from a tray, according to various embodiments.

[0021] [Figure 6D]10 is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to / from a tray, according to various embodiments.

[0022] [Figure 7A] FIG. 1 illustrates an end effector configured in a first mode, according to various embodiments.

[0023] [Figure 7B] FIG. 1 illustrates an end effector configured in a first mode, according to various embodiments.

[0024] [Figure 7C] FIG. 1 illustrates an end effector configured in a first mode, according to various embodiments.

[0025] [Figure 8A] FIG. 1 illustrates a robotically controlled tray / item handling end effector, according to various embodiments.

[0026] [Figure 8B] FIG. 1 illustrates a robotically controlled tray / item handling end effector, according to various embodiments.

[0027] [Figure 8C] FIG. 1 illustrates a robotically controlled tray / item handling end effector, according to various embodiments.

[0028] [Figure 9A] FIG. 1 illustrates a robotically controlled tray / item handling end effector, according to various embodiments.

[0029] [Figure 9B] FIG. 1 illustrates a robotically controlled tray / item handling end effector, according to various embodiments.

[0030] [Figure 10A]FIG. 1 illustrates a robotically controlled tray / item handling end effector with guide fins, according to various embodiments.

[0031] [Figure 10B] FIG. 1 illustrates a robotically controlled tray / item handling end effector with guide fins, according to various embodiments.

[0032] [Figure 10C] FIG. 1 illustrates a robotically controlled tray / item handling end effector with guide fins, according to various embodiments.

[0033] [Figure 10D] FIG. 1 illustrates a robotically controlled tray / item handling end effector with guide fins, according to various embodiments.

[0034] [Figure 11] 1 is a flow chart illustrating an automated process for placing one or more trays onto a stack according to various embodiments.

[0035] [Figure 12] FIG. 1 illustrates an example stack of trays configured to be stacked in a specific tray orientation.

[0036] [Figure 13] FIG. 1 illustrates an embodiment of a tray handling robot.

[0037] [Figure 14] 10 is a flowchart illustrating a process for selecting a mode in which an end effector is to be operated and operating the end effector in the selected mode, according to various embodiments.

[0038] [Figure 15A] 1 illustrates a bottom view of a suction-based end effector, according to various embodiments.

[0039] [Figure 15B] 1 illustrates a bottom view of a suction-based end effector, according to various embodiments.

[0040] [Figure 15C] 1 illustrates a bottom view of a suction-based end effector, according to various embodiments.

[0041] [Figure 15D] 1 illustrates a bottom view of a suction-based end effector, according to various embodiments.

[0042] [Figure 15E] FIG. 1 illustrates a side view of a suction-based end effector, according to various embodiments.

[0043] [Figure 15F] FIG. 1 illustrates a side view of a suction-based end effector, according to various embodiments.

[0044] [Figure 16] 1 is a flowchart illustrating a process for operating an end effector in connection with picking and placing a set of items, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0045] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments, or any other form the present invention may take, may be referred to herein as technology. In general, the order of steps in a disclosed process may be varied within the scope of the present invention. Unless otherwise noted, components, such as a processor or memory, described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term “processor” refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0046] The following is a detailed description of one or more embodiments of the present invention with reference to figures that illustrate the principles of the invention. While the present invention has been described in connection with such embodiments, it is not limited to any particular embodiment. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters that are well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0047] Autonomous tray handling line kitting robots are disclosed. In various embodiments, the line kitting robots disclosed herein include a robotic arm having an end effector as disclosed herein with a structure for grasping a tray of items. In various embodiments, one or more such robots may operate together in a single workspace to grasp trays from a source stack and move them, singly or in groups, to a destination stack that is constructed according to an invoice, manifest, or other input data indicating the output stack being requested to be constructed. In some embodiments, two or more robots disclosed herein may operate on the same rail or other transport structure. Operations are coordinated to avoid collisions and efficiently utilize all robots to complete a complete line kitting task, such as constructing multiple output stacks, each with a corresponding combination of trays, according to input information (e.g., invoices, manifests, etc.).

[0048] Various embodiments include an end effector device (also referred to herein as an end effector) provided on or connected to a robotic arm for grasping, moving, and placing items or other objects in one or more trays, trays, or other containers. The end effector includes (i) a first gripping mechanism for gripping at least one first object when the robotic end effector is operated in a first mode, and (ii) a second gripping mechanism for gripping a second object when the robotic end effector is operated in a second mode. The second gripping mechanism is robotically placed in an inactive state when the robotic end effector is controlled to operate in the first mode. The second gripping mechanism is robotically placed in an active state when the end effector is controlled to operate in the second mode.

[0049] Various embodiments include a robot end effector comprising a robotically actuated second gripper, a robotically actuated first gripper with first and second elements positioned opposite each other on either side of a central vertical axis of the robot end effector, the robotically actuated second gripper being positioned between the first and second elements, and a robotically actuated retract-extend mechanism configured to place the robot end effector in a first operational mode in which the first gripper is positioned for utilization or a second operational mode in which the second gripper is positioned for utilization.

[0050] As used herein, a first gripping mechanism (which may also be referred to herein as a robotically actuated first gripper) may comprise a suction-based end effector or other gripping mechanism. In some embodiments, the suction-based end effector comprises a plurality of suction cups. The plurality of suction cups may be controlled collectively, or some of the plurality of suction cups may be controlled independently of other portions of the plurality of suction cups. In some embodiments, the suction-based end effector is robotically controlled to pick or place one or more items from a tray or other container within the robot's workspace.

[0051] As used herein, a second gripping mechanism (which may also be referred to herein as a robotically actuated second gripper) may include an end effector with multiple gripper arms that pick up a tray (or other item or container) by gripping the sides or bottom of the tray. In some embodiments, one or more gripper arms of the second gripping mechanism are movable relative to a mount that connects the end effector to a robotic arm. As an example, the one or more movable gripper arms may be robotically controlled to close their grip on the tray (e.g., in connection with picking up the tray) or open their grip on the tray (e.g., in connection with releasing the tray at a destination location). For example, the second gripping mechanism may include an active arm and a passive arm, and the active arm may be robotically controlled to coordinate the gripping / release of the tray.

[0052] Various embodiments include a multi-mode end effector operable in multiple modes, which may include two or more of a first mode, a second mode, and / or a third mode.

[0053] In some embodiments, the first mode includes controlling a suction-based end effector included in the multi-mode end effector. The multi-mode end effector may be controlled to pick and place items from / to a tray or other container, such as in connection with unloading a tray or assembling a kit based on a predetermined manifest (e.g., an order being fulfilled). In some embodiments, when the multi-mode end effector is operated in the first mode, the second gripping mechanism (e.g., an end effector with a gripper arm) may be placed in an inactive state (e.g., retracted into a stowed or stored state), such as to expose the suction-based end effector or to allow the suction-based end effector to better grip an item. For example, in response to determining to operate the multi-mode end effector in the first mode, at least a portion of the second gripping mechanism (e.g., one or more gripper arms) is controlled to transition such portion of the second gripping mechanism to an inactive state (e.g., to move the gripper arms to accommodate the gripper arms in a position that better exposes the suction-based end effector to the item being gripped).

[0054] In some embodiments, the second mode includes controlling an end effector with multiple gripper arms, where such an end effector is included in a multi-mode end effector. The multi-mode end effector may be controlled to pick and place trays or other containers, stack the trays in a tray stack, or remove an empty tray to expose another tray (e.g., to expose items in another tray). In some embodiments, when the multi-mode end effector is operated in the second mode, the second gripping mechanism (e.g., an end effector with multiple gripper arms) may be placed in an active state (e.g., placed in a deployed state), such as to allow the end effector with multiple gripper arms to engage a tray or other container. For example, in the active state, the gripper arms are positioned to provide clearance between a tray engaged by the gripper arms and a suction-based end effector included in the multi-mode end effector. In response to determining to operate the multi-mode end effector in the second mode, at least a portion of the second gripping mechanism (e.g., one or more gripper arms) is controlled to transition such portion of the second gripping mechanism from an inactive state to an active state (e.g., to move the gripper arms to deploy the gripper arms in a position that better exposes the gripper arms for engaging the tray).

[0055] In some embodiments, the third mode includes controlling the multi-mode end effector to pull or push an object (e.g., an item, a container such as a tray, or a cart such as a cart with a stack of trays) using one or more rigid structures attached to the multi-mode end effector. Operating the multi-mode end effector according to the third mode can cause the robotic arm to adjust the position of the object.

[0056] Related art systems for moving containers (e.g., trays, totes, containers, etc.) and items contained within the containers utilize a first gripping mechanism for picking or placing items from / to the container and a second gripping mechanism for moving the container (e.g., an end effector or conveyor, etc.). The related art systems do not include end effectors with the first and second gripping mechanisms (e.g., related art end effectors do not include the first and second gripping mechanisms simultaneously deployed on a particular robotic arm). For example, in some related art systems, a second end effector corresponding to the second gripping mechanism is attached to the robotic arm for moving the container, and the robotic arm is controlled to attach another end effector (e.g., a first end effector corresponding to the first gripping mechanism) or another robotic arm already equipped with a first end effector to pick / place items from the container. In other words, the related art system decouples a first end effector from a robot arm to allow a second end effector to be attached to the robot arm. The related art end effectors do not include multiple end effectors or gripping mechanisms, and are not multi-modal, allowing the end effectors to be operated according to different modes. Therefore, the related art systems or end effectors are inefficient. For example, a related art system may utilize additional robot arms, such that a subset of the robot arms in the system includes a first gripping mechanism and another subset of the robot arms in the system includes a second gripping mechanism, and the different subsets of the robot arms are operated together to perform functions enabled by the first gripping mechanism and functions enabled by the second gripping mechanism.As another example, related art systems may require a first gripping mechanism to be removed (e.g., disconnected) from a robot arm before a second gripping mechanism can be attached, such disconnection and coupling of different end effectors resulting in delays in performing both the functions enabled by the first gripping mechanism and the functions enabled by the second gripping mechanism.

[0057] In some embodiments, the system includes a control computer for controlling the multi-mode end effector to automatically perform operations. The control computer may further be used to control the robotic arm relative to the multi-mode end effector, such that the control computer collectively controls the robotic arm and the multi-mode end effector in connection with performing a set of tasks. Controlling the multi-mode end effector to automatically perform operations may include gripping or moving trays (or other containers or large items) with a gripper arm (e.g., a gripper arm included in the second gripping mechanism), pushing or pulling a stack of trays (e.g., a stack of trays disposed on a dolly or other vehicle) with a gripper arm, picking and / or placing items from / to a tray with the first gripping mechanism (e.g., a suction-based end effector), or otherwise moving smaller items within the workspace, etc.

[0058] In some embodiments, the system determines a set of tasks to be performed (e.g., to achieve a higher-level goal, such as fulfilling a set of orders) and determines the order in which the set of tasks are performed based on a cost function associated with performing each task in the task set. The system may determine the order in which the set of tasks are performed based on a cost associated with transitioning control of a multi-mode end effector between a first mode or a second mode. For example, the system determines the order in which the set of tasks are performed based at least in part on a cost associated with transitioning a second gripping mechanism (e.g., an end effector with multiple gripper arms) between an inactive state and an active state.

[0059] In some embodiments, the system obtains information regarding one or more item attributes of items in the workspace and information regarding the state of the workspace from one or more sensors. The one or more attributes may include an identifier (e.g., barcode, serial number, product number), shape, rigidity, size, weight, an indication of whether the item is fragile, an indication of whether the item has soft or deformable packaging, etc. The information regarding the state of the workspace may include one or more of the following: the number of trays (or other containers, dollies, carts, etc.), the position of the trays within the workspace, an indication of the products or items contained in the trays, the position of another robotic arm (if any), the position of other objects or people in the workspace, etc. In response to obtaining the information regarding the item attributes and the state of the workspace, the system determines a set of M tasks to be performed (e.g., to pick and place items in connection with loading or unloading trays, or in connection with kitting an order according to an invoice, packing slip, etc.) and determines one or more plans for controlling the robotic arms to pick and place items corresponding to the set of M tasks. In some embodiments, the system determines an optimal set of N tasks next to the set of M tasks (e.g., N is less than or equal to M) in connection with performing the set of M tasks. For example, the system determines an optimal order in which the set of M tasks are performed. The optimal set of N tasks or the order of the set of M tasks may be determined based on a cost function or otherwise based at least in part on the state of the multi-mode end effector (e.g., the state of the second gripping mechanism, such as whether the gripper arm is deployed or unretracted). For example, changing the state of the second gripping mechanism (e.g., between an active state and an inactive state) has some associated cost, such as time, energy, or time.Thus, the system can determine an optimal order for completing the set of N tasks so as to minimize the overall cost of performing the set of N tasks, or so as to meet a cost criterion (e.g., so that the overall cost is less than a predetermined cost threshold), or so as to minimize the cost associated with changing the state of the second gripping mechanism while performing the set of N tasks.

[0060] In various embodiments, the tray-handling robotic system disclosed herein comprises a single rail system occupied by multiple robots that cooperate in the fulfillment of trays containing packaged groceries or any other merchandise or other items. The trays may arrive in stacks of various heights and may be stacked in various orientations. In some embodiments, the system is divided into two sides: an input side where homogenous stacks enter, and an output side that is specialized for various customers and / or other destinations and is formed by kitting various products from the input side based on, for example, order lists.

[0061] In some embodiments, multiple robots operate on the same rail or other transport system. For example, two or more robots may operate on the same rail system. Each robot is mounted on a chassis that can be moved along the rail under robotic control, independently of each other. The robots recognize each other and coordinate their movements to optimize order fulfillment. Each robot may use a single multi-mode end effector designed to both grip a tray (e.g., using a secondary gripping mechanism, such as a tray gripper) and pick or place items from / to the tray (e.g., using a primary gripping mechanism, such as a suction-based end effector). Alternatively, the robot may use a tray gripper designed to grip multiple trays at once. In various embodiments, the gripper is modular and can be adapted to a variety of different trays.

[0062] In various embodiments, the robotic system disclosed herein is configured to pick from a stationary stack of trays (or other containers) placed on a dolly (or other cart). One example of such a robotic system is disclosed in U.S. Patent Application No. 16 / 797,359, filed February 21, 2020, entitled "Robotic Handling of Soft Products in Non-Rigid Packaging," the entire contents of which are incorporated herein by reference for all purposes. Another example of such a robotic system is disclosed in U.S. Patent Application No. 17 / 712,915, filed April 4, 2022, entitled "Robotic Tray Gripper," the entire contents of which are incorporated herein by reference for all purposes. Another example of such a robotic system is disclosed in U.S. Patent Application No. 17 / 219,509, filed dated May 1, 2020, entitled "Suction-Based End Effector with Mixed Cup Sizes," the entire contents of which are incorporated herein by reference for all purposes.

[0063] Although the embodiments described herein are provided in the context of picking and placing items from a kitting system or tray, various embodiments may be implemented in a variety of other contexts, such as a palletizing system, a singulation system, etc.

[0064] As used herein, depalletization includes picking an item from a pallet (e.g., from a stack of items on a pallet), moving the item, and placing the item at a destination location (e.g., a conveying structure). Examples of palletization / depalletization systems and / or processes for palletizing / depalletizing sets of items are further described in U.S. Patent Application No. 17 / 343,609, which is incorporated herein by reference in its entirety for all purposes.

[0065] As used herein, singulation of items includes picking items from a source pile / stream and placing the items on a conveying structure (e.g., a partitioned conveyor or similar conveying means). Optionally, singulation may include sorting various items on a conveying structure, such as placing items one by one from the source pile / stream into slots or trays on a conveyor. Examples of singulation systems and / or processes for singulating sets of items are further described in U.S. Patent Application No. 17 / 246,356, which is incorporated herein by reference in its entirety for all purposes.

[0066] Kitting, as used herein, includes picking one or more items / objects from corresponding locations and placing one or more items in predetermined locations such that a set of one or more items corresponds to a kit. Examples of kitting systems and / or processes for kitting sets of items are further described in U.S. Patent Application No. 17 / 219,503, which is incorporated herein by reference in its entirety for all purposes.

[0067] FIG. 1A is a block diagram illustrating one embodiment of a robotic line kitting system. In the illustrated example, system 100 includes source tray stacks 102 and 104 moving from an input end 108 (a staging / loading area) along an input stack transport vehicle (e.g., transport vehicle 106), as provided in this example. Each of source tray stacks 102 and 104 in this example is shown stacked on a wheeled cart or chassis. In various embodiments, source tray stacks 102 and 104 are manually pushed onto transport vehicle 106, which may be a conveyor belt or other structure configured to advance source tray stacks 102 and 104 through a workspace defined by transport vehicle 106. In various embodiments, source tray stacks 102 and 104 may be pushed / pulled onto transport vehicle 106 by a robotic arm (e.g., robotic arm 112 or 114), such as a robotic arm controlled in a third mode in which a multi-mode end effector is used to push / pull the stack of trays. In some embodiments, the chassis or other base structure on which the source trays are stacked is self-propelled. In some embodiments, the source tray stacks 102 and 104 are advanced through / by the transport means 106 under robotic control. For example, the speed and time at which the source tray stacks 102 and 104 are advanced by / through the transport means 106 is controlled to facilitate efficient gripping of trays from the source tray stacks 102 and 104.

[0068] In the illustrated example, a single rail (e.g., rail 110) is disposed along one long side of the transport vehicle 106. In this example, two robots (one with robot arm 112 and the other with robot arm 114) are mounted on rail 110 so as to be independently movable from one another. For example, each robot arm 112, 114 is mounted on a self-propelled chassis that moves along rail 110. In this example, each robot arm 112, 114 terminates in a tray-handling end effector (e.g., end effector 116, 118). In some embodiments, end effector 116 and / or 118 implements end effector 300 of FIGS. 3A-3C, end effector 700 of FIGS. 7A-7C, end effector 800 of FIGS. 8A-8C, end effector 900 of FIGS. 9A-9B, or end effector 1000 of FIGS. 10A-10D.

[0069] In various embodiments, a tray-handling end effector (e.g., end effector 116 or 118) is robotically operated to grasp one or more trays from the source tray stacks 102, 104. In some embodiments, the tray-handling end effector is included in a multi-mode end effector attached to the robotic arms 112, 114. An example of a multi-mode end effector includes end effector 300 in FIGS. 3A-3C. The tray-handling end effector may correspond to a second gripping mechanism of the multi-mode end effector. For example, the tray-handling end effector may include multiple gripper arms, at least some of which are movable to adjust their grip on the trays being picked / placed. In some embodiments, the multi-mode end effector also includes a first gripping mechanism configured to pick and place smaller items (e.g., items contained in one or more trays being moved by the tray-handling end effector). As shown in FIG. 1A , each end effector 116, 118 comprises a lateral member attached to the end of the robotic arm 112, 114. A side member is attached to each end of the lateral member. As shown, at least one of the side members is robotically opened and closed in various embodiments to allow a tray to be grasped (by closing the side member) or released (by opening the side member). In some embodiments, at least one robotically opened or controlled side member is configured to rotate about an axis perpendicular to the length axis of the lateral member. In some embodiments, at least one robotically opened or controlled side member is configured to move along or substantially along / parallel to the length axis of the lateral member.

[0070] In various embodiments, each tray-handling end effector 116, 118 (e.g., the second gripping mechanism of a multi-mode end effector) includes one non-movable ("passive") side member and one movable ("active") side member. In this example, the movable or "active" side member swings open (as shown for end effector 116), e.g., to allow the end effector to be positioned to grip one or more trays, and swings closed (as shown for end effector 118), e.g., to complete gripping of one or more trays. In other examples, the movable or "active" side member is moved in a lateral movement substantially parallel to the length of the lateral member of the multi-mode end effector to which the "active" and "passive" side members are connected or otherwise extend. In other words, the "active" side member is moved in a direction substantially corresponding to the axis of the lateral member to widen or narrow the grip of the second gripping mechanism when applying a force to the tray being picked / placed. In various embodiments, a robotic control system (e.g., a computer (such as control computer 128) controlling the robotic arms 112, 114) controls the end effectors to actuate opening and closing of the end effectors, such as in connection with gripping or releasing a tray. The robotic control system controls the end effectors based at least in part on image data of the workspace and / or one or more sensors included in (or connected to) the corresponding end effectors.In some embodiments, one or more sensors included in (or connected to) a corresponding end effector (i) obtain information indicating whether a gripping mechanism (e.g., an active member of a second gripping mechanism) of the multi-mode end effector is in an open or closed position, (ii) obtain information indicating the extent to which the gripping mechanism is open, (iii) obtain information indicating that the tray (or end effector relative to the tray) is in a position where the multi-mode end effector is controlled to engage at least one side of the multi-mode end effector (e.g., a passive member or structure included in the passive member) with a hole, recess, or handle included in one side of the tray (e.g., the tray to be gripped), and (iv) obtain information indicating that the tray (or end effector relative to the tray) is in a position where the multi-mode end effector (e.g., a passive member or structure included in the passive member) is controlled to engage with a hole, recess, or handle included in one side of the tray. (v) obtain information indicating whether an item is being grasped by a first gripping mechanism (e.g., a suction-based end effector) of a multi-mode end effector; (viii) obtain information indicative of an attribute of the first gripping mechanism (e.g., the pressure between the suction-based end effector and the grasped item); (ix) obtain an indication of whether the first gripping mechanism is engaged with an object; and (x) obtain information indicative of the state of the first gripping mechanism (e.g., information indicative of the state of the suction cups, such as the position of the suction cups when the relative position of the suction cups can be changed to increase or decrease the distance between at least two suction cups).

[0071] In various embodiments, each end effector 116, 118 includes one or more protrusions or similar structures on each side member sized and shaped to fit into holes or other openings on either side of the tray to be grasped, and in various embodiments, to be slid in under robotic control. For example, in some embodiments, protrusions (also referred to herein as "thumbs") on the interior surfaces of the side members are inserted into handles (e.g., holes sized to accommodate a human hand) on either side of the tray, as will be more fully described and explained below.

[0072] In various embodiments, each robotic arm 112, 114 operates fully autonomously and simultaneously to pick trays from the source tray stacks 102, 104 and place them onto a destination tray stack (e.g., destination tray stack 120, 122) in a destination tray stack assembly area across the rails 110 from the transport 106 and source tray stacks 102, 104. The destination tray stacks, in various embodiments, are constructed according to an invoice, manifest, purchase order, or other information. For example, for each of multiple physical destinations (e.g., retail stores), a destination stack associated with that destination (e.g., according to orders placed by the destination) is assembled by selecting trays from each source tray stack 102, 104 and stacking them onto the corresponding destination tray stack 120, 122. The completed destination tray stacks 120, 122 are removed from the destination tray stack assembly area, as indicated by arrow 124, for placement onto a truck, rail car, container, etc., for delivery to a further destination (such as a retail store), for example.

[0073] 1A , in the illustrated example, system 100 includes a control computer 128 configured to wirelessly communicate with the robotic elements comprising system 100, which in various embodiments include one or more of: transport 106; a wheeled chassis (if self-propelled) on which source tray stacks 102, 104 are stacked; robotic arms 112, 114 and / or their respective chassis to which robotic arms 112, 114 are mounted on rails 110; and robotically controlled tray handling end effectors (e.g., end effectors 116, 118). In various embodiments, the robotic elements are controlled by control computer 128 based on input data, such as invoice, purchase order, and / or manifest information, and input status information, such as inventory data indicating what types and / or quantities of products the source tray stack contains.

[0074] In various embodiments, the source tray stacks 102, 104 are inserted into a gate or other entrance / control structure at the input end 108 of the transport vehicle 106. The transport vehicle 106 includes a device (stack mover) that moves the source tray stacks 102, 104 along the rails 110 in a manner that optimizes throughput and minimizes robot displacement, for example, by minimizing the distance and / or frequency that the robot arms 112, 114 must move along the rails 110 to grasp the source trays and place them on their respective destination stacks. The source tray stacks 102, 104 may arrive with trays of different orientations / weights / and weight distributions. The system 100 uses force and moment control to operate the robot arms 112, 114 to gently but firmly insert thumbs or other protrusions into the trays, and to plan the robot's movements and tray trajectories to avoid collisions with the robot or the environment. In various embodiments, each robotic arm 112, 114 operates in a very narrow space, approximately 2.5 meters wide, and has a very light footprint. The robot utilizes its entire workspace to intelligently plan movements that optimize its grasp. The robot recognizes the need to perform orientation changes when avoiding obstacles and acts accordingly. The robot coordinates with other robots on the rail 110 to navigate to the correct output (e.g., destination tray stack 120, 122) corresponding to the correct customer. The robot then utilizes advanced force control and interaction with the environment to find the appropriate placement strategy. The cycle then begins again.

[0075] 1A , system 100 includes a 3D camera 126. In various embodiments, system 100 includes multiple 3D cameras (or other cameras) (e.g., camera 126) and uses the image and depth data generated by such cameras to generate a three-dimensional view of at least relevant portions of the workspace and scene (e.g., the scene / condition shown in FIG. 1A ). In some embodiments, a camera (e.g., camera 126) is used to identify the tray contents in source trays that make up the tray stack, for example, by recognizing the size, shape, packaging, and / or labeling of such items, and / or by recognizing the shape, color, dimensions, or other attributes of the source stack trays themselves, and / or by reading barcodes, QR codes, radio frequency tags, or other image-based or non-image-based information on or emitted by the trays.

[0076] In various embodiments, image data generated by a camera (such as camera 126) is used to move a robotic arm and end effector to a position near the tray or stack of two or more trays to be grasped and picked from the source stack, and / or to a position near the destination where the tray will be placed (e.g., on top of the corresponding destination stack). In some embodiments, force control is used to complete the final stages of the pick / grasp episode and / or place episode, as described more fully below.

[0077] 1A shows a single camera (e.g., camera 126) mounted in the workspace of system 100, in various embodiments, multiple cameras or other sensors or a combination thereof are statically mounted in the workspace. Additionally or alternatively, one or more cameras or other sensors are mounted on or near each robotic arm 112, 114 (e.g., on the arm itself and / or on the end effector 116, 118, etc.) and / or on a structure that moves with the robotic arms 112, 114 as they are moved along rail 110.

[0078] FIG. 1B is a block diagram illustrating an embodiment of a robotic line kitting system. FIG. 1B shows an example overhead view of a workspace in which the system 100 of FIG. 1A can operate. In the illustrated example, robotic arms 112, 114 travel along a common rail (e.g., rail 110) as in FIG. 1A to access and pick trays from a source stack 140 traveling along the transport vehicle 106 and place the trays onto corresponding destination stacks 142 in a destination stack assembly area across the rail 110 from the source stack 140 and the transport vehicle 106. In this example, a human worker manually loads the source stacks onto the transport vehicle 106; however, in some embodiments, a robotic worker performs all or part of the tasks, for example, according to a programmatically generated plan to fulfill a set of orders each associated with a corresponding destination. Once the destination stacks 142 are completed, they are removed from the destination stack assembly area, as indicated by the arrow at the top of FIG. 1B, which corresponds to arrow 124 in FIG. 1A.

[0079] 1A and 1B, each tray contains only one type of tray, but in other embodiments and applications, source and destination trays with mixed items may be manipulated to assemble a destination stack of trays as disclosed herein. Similarly, in the example shown in FIGS. 1A and 1B, each source stack of trays contains only trays of the same type and contents, but in other embodiments and applications, a source tray stack may contain mixed trays and / or item types. For example, control computer 128 is provided with information indicating which types of trays are in which positions in each source tray stack and uses that information, along with manifests or other information indicating the required contents of each destination tray stack, to build the required destination tray stacks by picking the required trays from their corresponding positions on the source tray stacks and adding the trays to the corresponding destination stacks.

[0080] FIG. 2A is a state diagram illustrating one embodiment of an automated process for building a stack of trays. In various embodiments, processing according to state diagram 200 is performed by a control computer (such as control computer 128 of FIG. 1A). In the illustrated example, a planning state, process, and / or module 202 generates and dynamically updates a plan to build an output stack of trays by using a robotic device disclosed herein to pick trays from a source stack of homogeneous or heterogeneous trays according to a set of orders, invoices, manifests, etc., and build destination stacks each having one or more types of trays. The planning state, process, and / or module 202 receives feedback indicating which destination tray stacks have been completed, which source stacks of trays have been moved into the workspace, and / or other state and context information that can be used to continue updating the plan for picking and placing (stack) trays to build the destination stack. In state 204, the process controlling a given robotic device (e.g., in the example shown in FIG. 1A , robot arms 112 and / or 114 and associated end effectors 116 and 118) determines the next set of one or more trays to move from a source stack to a destination stack according to the planning state, process, and / or current overall plan received from module 202. For example, the robot may decide to pick up one, two, or more trays from the source stack and add them to the destination stack (or start a new destination stack). The robot enters state 206, where a strategy and plan is determined for one or more of moving one or more trays into position to pick them up, picking up the trays, and / or starting to move the trays towards the destination stack location, and the robot moves into place to pick up the trays.Once one or more trays have been grasped, the robot enters state 208 where the trays are moved along a planned (and, if necessary, dynamically adapted) trajectory to the vicinity of the destination stack (e.g., hovering over the destination stack and / or the location or structure where the destination stack will be built). In state 210, the robot places the tray on the destination stack. In some embodiments, state 210 includes force-controlled manipulation to ensure that the tray is securely placed on the destination stack, for example, by moving (or attempting to move) the tray back and forth (or side to side, if applicable) to ensure that any interconnect structures are aligned and properly interdigitated (e.g., tabs on the bottom of the tray are positioned to fit into corresponding recesses in the sidewall of the tray under which it will be placed, etc.). Once the tray is determined to be securely placed, the robot releases the tray and re-enters state 204, where the next set of one or more trays is determined to be picked from the corresponding source stack and moved to the corresponding destination stack, for example, according to the planning state, process, and / or overall planning information received from module 202. In various embodiments, the robotic system disclosed herein continues to cycle through states 204, 206, 208, and 210 of FIG. 2A until all destination stacks have been built.

[0081] FIG. 2B is a flow chart illustrating one embodiment of an automated process for building a stack of trays. In various embodiments, a process or module controlling one or more tray-handling robots performs process 220 of FIG. 2B. In various embodiments, process 220 of FIG. 2B is performed by a process or module executing on a control computer (such as control computer 128 of FIG. 1A). In some embodiments, process 220 is performed in connection with gripping trays using a second gripping mechanism of a multi-mode end effector (e.g., a gripping mechanism with a gripper arm, etc.). In the illustrated example, at step 222, it is determined that a particular set of one or more trays will be moved from a source stack to a destination stack. In some embodiments, the robot arm has an end effector (e.g., a second gripping mechanism) that supports picking and placing only one tray at a time. In other embodiments, the robot has an end effector that can grip a stack of two or more trays, for example, by gripping the bottom tray of the stack to be gripped. At step 224, a strategy for navigating to and gripping the trays is determined. For example, the robot plans and executes a set of operations to move the end effector to a position above or other nearby location of the tray to be grasped. As another example, the robot plans and executes actions to control the end effector to grasp the tray. The robot controls the end effector (e.g., a multi-mode end effector) to change modes in connection with grasping a tray or an item from the tray (e.g., to control the end effector to use a first gripping mechanism or a second gripping mechanism based at least in part on whether the end effector is grasping a tray or an item from the tray, etc.). A strategy for grasping the tray is determined and executed. In step 226, a plan (e.g., a trajectory) for moving the tray to the destination stack is determined and executed.The trajectory / plan takes into account obstacles in the workspace (e.g., other stacks) and potential collisions with other robotic devices (e.g., another pick / place robot operating in the same workspace (e.g., robot arms 112, 114 of FIG. 1A)). At step 228, a strategy for placing the tray onto the corresponding destination stack is determined and executed. At step 230, the results of the pick / place operation are reported, for example, to a planning process or module. Subsequent iterations of steps 222, 224, 226, 228, and 230 are repeated until processing is determined to be complete (e.g., all destination stacks have been completed) at step 232.

[0082] FIG. 2C is a flow chart illustrating one embodiment of an automated process for picking and placing items to / from a tray. In some embodiments, process 250 is performed by system 100 of FIG. 1A. In some embodiments, process 250 is performed in connection with picking and placing items to / from a tray using a first gripping mechanism of a multi-mode end effector (e.g., a gripping mechanism with suction cups for suction-based gripping, etc.). In various embodiments, a process or module controlling one or more tray-handling robots performs process 220 of FIG. 2B. In various embodiments, process 220 of FIG. 2B is performed by a process or module executing on a control computer (e.g., control computer 128 of FIG. 1A).

[0083] In the illustrated example, in step 252, it is determined that a particular set of one or more trays will be moved from a source location to a destination location. For example, the system determines to retrieve items from a source location (e.g., a kitting shelf, conveyor, etc.) and place the items in a tray or other container. As another example, the system determines to pick items from a tray and place the items in a destination location (e.g., a conveyor, chute, other container, etc.). In some embodiments, the robot arm has an end effector (e.g., a first gripping mechanism such as a suction-based end effector) that supports picking and placing of only one item at a time. In other embodiments, the robot has an end effector that can grip multiple items (e.g., by gripping each item using a different portion of the suction cups of the suction-based end effector).

[0084] In step 254, a strategy for moving to and grasping the item is determined. For example, the robot plans and executes a set of operations to move the end effector (e.g., a suction-based end effector of a multi-mode end effector) to a position above or other nearby location of the item to be grasped. As another example, the robot plans and executes actions to control the end effector to grasp the item. The robot controls the end effector (e.g., a multi-mode end effector) to change modes in connection with grasping a tray or an item from a tray (e.g., to control the end effector to use a first gripping mechanism or a second gripping mechanism based at least in part on whether the end effector is grasping a tray or an item from a tray, etc.). A strategy for grasping the item is determined and executed.

[0085] A plan (e.g., a trajectory) for moving the item to the destination location is determined and executed in step 256. The trajectory / plan takes into account obstacles in the workspace (other items, stacks of trays, etc.) and potential collisions with other robotic devices (such as other pick / place robots (e.g., robot arms 112, 114 in FIG. 1A) operating in the same workspace).

[0086] In step 258, a strategy for placing the item in the corresponding destination location (eg, destination tray, conveyor, etc.) is determined and executed.

[0087] The results of the pick / place operation are reported, for example, to a planning process or module, at step 260. Subsequent iterations of steps 252, 254, 256, 258, and 260 are repeated until processing is complete (e.g., all items have been picked and placed (e.g., the items correspond to a manifest such as an order or packing slip, or the tray where the items are picked is empty, or the tray where the items are placed is full)) as determined at step 262.

[0088] FIG. 3A illustrates one embodiment of a robotically controlled tray handling end effector. In some embodiments, the end effector 300 is implemented in connection with the system 100 of FIG. 1A, such as by the robotic arms 112, 114. The end effector 300 is a multi-mode end effector with at least two gripping mechanisms (e.g., a first gripping mechanism and a second gripping mechanism). In some embodiments, the end effector 300 is robotically controlled to operate according to different modes based on the task being performed, etc. For example, the end effector 300 operates in a first mode with a first gripping mechanism (e.g., a suction-based end effector used to pick / place items). As another example, the end effector 300 operates in a second mode with a second gripping mechanism (e.g., an end effector with a gripper arm used to pick / place trays). As another example, the end effector 300 operates in a third mode in which structures on the end effector 300 are used to push / pull a tray or stack of trays.

[0089] In the illustrated example, end effector 300 includes multiple gripping mechanisms. In some embodiments, end effector 300 includes (i) a first gripping mechanism corresponding to suction-based end effector 314 and (ii) a second gripping mechanism including a gripper arm (e.g., a side member). The different gripping mechanisms included in end effector 300 are utilized for different functions or in different modes. Suction-based end effector 314 includes one or more suction cups 314a, 314b, 314c, and 314d. In some embodiments, end effector 300 is robotically controlled to grip an object (e.g., a tray, items in the tray, etc.) based on selectively controlling one or more of the first and second gripping mechanisms.

[0090] As shown in FIG. 3A , the end effector 300 includes a cross member 302 for a first gripping mechanism and / or multiple elements for a second gripping mechanism. For example, the end effector 300 includes a cross member 302 to which a side member 304 is fixedly attached and which is hinged or otherwise movably attached to allow the side member 306 (e.g., the active side member) to be moved to an open position that facilitates moving the end effector 300 into position for gripping a tray. An active side thumb 308 is disposed on (or comprises an integral part or feature of) the inner surface of the side member 306. In some embodiments, both gripper arms (e.g., side members) are movable relative to the cross member 302, such as to widen or narrow the grip between the gripper arms.

[0091] According to various embodiments, the side members 306 are movable within a predetermined range of motion. By way of example, the end effector 300 includes one or more stop mechanisms (e.g., stops, switches, etc., or a combination thereof) that limit the movement of the side members 306 within the predetermined range of motion. The end effector 300 includes an open position stop mechanism that prevents the side members 306 from moving in an open direction beyond an open position threshold (e.g., 130 degrees relative to the plane / vector along which the lateral member 302 extends, or 30-50 degrees relative to the closed position, where the active member 306 is substantially perpendicular to the plane / vector along which the lateral member 302 extends). The end effector 300 includes a closed position stop mechanism that prevents the side members 306 from moving in a closed direction beyond a closed position threshold (e.g., approximately 90 degrees relative to the plane / vector along which the lateral member 302 extends). Various values ​​may be selected for the open position threshold and / or the closed position threshold. In some embodiments, the open position threshold is set based at least in part on the environment in which the robot to which the end effector 300 is connected operates. As an example, when multiple robots operate within a relatively close range, the range of motion of the side member 306 is based at least in part on the distance between the robots or between the zones in which various robots (e.g., adjacent robots) operate. As the side member 306 moves from a closed position to an open position, the side member 306 extends in the x-direction. Furthermore, the longer the distance the side member 306 can move from the closed position to the open position, the longer the time required for the robotic system to control the side member 306 to open / close in conjunction with picking / placing a tray. Therefore, limiting the range of motion of the side member 306 (e.g., to an open position threshold sufficient to allow the end effector to easily pick up one or more sets of trays) allows the robotic system to operate more efficiently in proximity to other robots (e.g., other robots that are autonomously picking, moving, and placing trays).

[0092] In some embodiments, the open and / or closed position thresholds are configurable, e.g., one or more stop mechanisms are configurable and set based on the desired open and / or closed position threshold configuration.

[0093] The active side thumb 308 and corresponding structure on the inner surface of the side member 304 (not visible in FIG. 3A ) have a size and shape suitable for insertion into gripping or other recesses or holes on either side of a tray to be gripped by the end effector 300. In various embodiments, the thumb 308 is removable and replaceable, e.g., replaced as it wears with use, or replaced with a thumb having a different shape, size, material, etc. suitable for gripping a different type of tray. The active side thumb 308 is fixedly attached to the side member 306, e.g., to prevent rotation of the thumb 308 (e.g., while engaged with the tray handle). For example, the active side thumb is attached to the side member 306 at three attachment points. Various other attachment configurations or numbers of attachment points may be implemented. As shown in the three-view diagram on the right side of FIG. 3A , in the illustrated example, the thumb 308 has convex surfaces 308a-d on each of its four sides. In various embodiments, convex surfaces 308a-d facilitate the insertion of thumb 308 into a handle or other hole or recess in the side of a tray being grasped using force and moment control. In some embodiments, the convex surfaces are used in conjunction with active force control and orientation impedance control to ensure a gentle and secure final grasp, where the active side is fully within the tray. For example, even if alignment is imperfect, convex surfaces 308a-d engaging the side or edge of the hole allow the remainder of thumb 308 to more easily slide more fully into the hole. Flat surface 308e at the base of the thumb is closest to the inner wall of the side member 304, 306 to which the thumb is attached, and in various embodiments, allows misalignment between end effector 300 and the tray being grasped to be corrected and / or alignment to be fine-tuned. For example, in a pick episode, the thumb of side member 304 (e.g., the passive side member) is moved to a position near a handle or other hole on one side of the tray being grasped. Convex surfaces 308a-d are used under force control to slide the thumb into the hole to some extent, and flat surface 308e near the base of the thumb better aligns the passive side with the tray before closing side member 306.

[0094] 3A , in the illustrated example, the end effector 300 includes a force sensor 310 attached to a cross member 302 and a bracket 312 for attaching the end effector 300 to a robotic arm. In some embodiments, the end effector 300 is attached to the robotic arm via a pin inserted through a hole in the bracket 312, allowing the end effector 300 to freely swing and / or rotate under robotic control, e.g., using one or more motors, about the longitudinal axis of the pin. In various embodiments, the force sensor 310 detects forces / moments experienced by the end effector 300 in the x, y, and / or z directions. The force sensor 310 may have a single-axis overload of at least +10,000 N of force in the x or y direction (e.g., Fxy) and / or a single-axis overload of at least +30,000 N of force in the z direction (e.g., Fz). The force sensor 310 may have a single-axis overload of at least +1000 Nm of torque in the x or y direction (e.g., Txy) and / or a single-axis overload of at least +1000 Nm of torque in the z direction (e.g., Tz). In some embodiments, the force sensor 310 has a single-axis overload of about +18000 N of force in the x or y direction (e.g., Fxy) and / or a single-axis overload of about +48000 N of force in the z direction (e.g., Fz), and a single-axis overload of about 1700 Nm of torque in the x or y direction (e.g., Txy) and / or a single-axis overload of about 1900 Nm of torque in the z direction (e.g., Tz).

[0095] In various embodiments, the side members 304 are fixedly attached to the cross member 302. The fixed attachment of the side members 304 allows forces and moments acting on the end effector 300 (e.g., the side members 304) to propagate through the frame of the end effector (e.g., the cross members 302 and the side members 304) to the force sensor 310. For example, the fixed attachment of the side members 304 prevents forces and moments from being translated into movement of other parts of the end effector (such as the active member 306) when the active member 306 is actuated to move the thumb 308 into engagement with the tray handle (e.g., inserting the thumb 308 into the tray handle).

[0096] 3B illustrates one embodiment of a robotically controlled tray handling end effector. The end effector 300 includes a second gripping mechanism that is controlled (e.g., during a second operational mode of multi-mode operation) to grip an item with gripper arms (e.g., side members 304, 306). In some embodiments, the end effector 300 is controlled to move one or more of the gripper arms to open their grips to allow the end effector 300 to move into position to grip an object (e.g., a tray), and to move one or more of the gripper arms to close their grips on the object to be gripped.

[0097] In the state shown in FIG. 3B , the active side member 306 is opened to an open position by, for example, a pneumatic or hydraulic piston, a motor, or other motive force and a structure (not shown) housed within the cross member 302 in FIG. 3B . Vector / direction 316 represents an example of a closed position (e.g., a closed position threshold). In various embodiments, the closed position is a configuration in which the side member 306 forms a normal vector with respect to the cross member 302. For example, the closed position threshold is 90 degrees (or substantially 90 degrees) relative to the direction along which the cross member 302 is aligned. As shown in FIG. 3B , the side member 306 is moved to an open position. When the side member 306 is moved to the open position, the angle between the side member 306 and the vector / direction 316 is represented by angle 313. According to various embodiments, the open position threshold corresponds to a configuration in which angle 313 is between 35 degrees and 50 degrees. In some embodiments, the open position threshold corresponds to a configuration in which angle 313 is between 40 degrees and 50 degrees. In some embodiments, the open position threshold corresponds to a configuration where angle 313 is between 40 and 45 degrees.

[0098] In various embodiments, the robotic system controls the side member 306 (e.g., controls an actuator to move the side member 306) based at least in part on information obtained by one or more sensors, such as a sensor on the side member 306 (e.g., the thumb 308 of the side member 306), a sensor on the side member 304 (e.g., the thumb of a passive side member), a camera or other sensor on or around the robot to which the end effector 300 is connected (e.g., to capture information about the robot's workspace), or any combination thereof. The side member 306 is controlled according to a plan to grasp, move, and / or place one or more sets of trays and the information obtained from the one or more sensors. The side member 306 is further controlled according to obstacles in the robot's workspace, such as another tray stack (e.g., an adjacent stack), another robot working to move a tray from another tray stack (or the same tray).

[0099] In various embodiments, the tray picking operation disclosed herein is smooth, gentle, precise, and resistant to uncertainty and disturbances. In various embodiments, a pick episode using a second gripping mechanism (e.g., gripping a tray with a gripper arm) includes one or more of the following: Descending from a hover attitude (above the stack) to a target attitude (adjacent to the tray) in conjunction with height checks to improve tray handle location estimation and dynamic targeting. Use of the end effector's active side surface to control for any uncertainty in the rail orientation, whether due to tray misalignment or human error. In various embodiments, after moving to a hover posture, the robot descends into position to align with the handle, and force control is used to ensure perfect (or substantially perfect) alignment of the rail orientation during this descending movement. This is quite likely because the gripper fits the length of the tray between itself almost perfectly, and any misalignment could lead to contact. This contact is guaranteed to occur on the diagonally facing active side panel, which means the robotic system can effectively use contact between the gripper and a misaligned tray to adjust its position using force control. Using a three degree of freedom (3DOF) force controller (e.g., based on sensor readings from force sensor 310) to locate the passive side (tray handle) slot and begin inserting the passive side thumb into the slot using the convex surface of the thumb (e.g., one or more of surfaces 308a-d, depending on which engages the tray). In some embodiments, a 6DOF controller is used to perform XYZ force control to ensure the thumb is inserted, and XYZ axis moments to ensure the plane of the passive side panel is flush against the plane of the tray exterior surface. In some embodiments, one or more sensors in the side member 304 (or in the thumb of the side member 304) are used to obtain information related to the position of the tray, such as information indicating the position of the second side member relative to the first tray, information indicating that the first tray is in a position where the end effector is controlled to engage the passive side structure with a hole, recess, or handle provided in the first structure (e.g., to detect that the tray is in proximity to the tray, such as at the entrance of a gripper, to detect that the end effector 300 is properly positioned to begin the process of engaging the tray with the side member 304), information indicating that the first tray is in a position where the passive side structure will engage with a hole, recess, or handle provided in the first structure, or any combination thereof. · Utilize the flat end of the thumb (e.g., 308e) to adjust for any misalignment. When all is well (e.g., upon determining that the side members 304 and / or the active side member are properly positioned to grip the tray), close the active side (e.g., 306) with force / moment control to account for any remaining orientation or position uncertainty in the tray's pose, and lift the tray to perform a sanity check on the grip quality (e.g., predicted weight, balanced, and forces and moments consistent with an otherwise good grip). In some embodiments, when the gripper condition is deemed good, the active side is closed with force / moment control enabled to improve and correct any remaining orientation / position errors, which ensures gentle manipulation of the tray. The robot safely aborts the pick if it detects any anomalies with the weight or quality of the trays in the stack or the quality of the stacking itself.

[0100] According to various embodiments, the end effector 300 is controlled to actuate the second gripping mechanism between an active state (e.g., a deployed state) and an inactive state (e.g., a retracted state). As one example, when the end effector 300 is controlled to operate in a first mode (e.g., to grasp an item from a tray using the first gripping mechanism), the second gripping mechanism is actuated to be configured in the inactive state. During operation in the first mode, the end effector 300 is transitioned to an inactive state in which one or more elements of the second gripping mechanism are moved to enable the first gripping mechanism to grasp an object (e.g., an item in a tray). As another example, when the end effector 300 is controlled to operate in a second mode (e.g., to grasp a tray using the second gripping mechanism), the second gripping mechanism is actuated to be configured in the active state. During operation in the second mode, the end effector 300 is transitioned to an active state in which one or more elements of the second gripping mechanism are moved to allow the gripper arm to engage a tray or other object gripped by the second gripping mechanism.

[0101] 3C illustrates one embodiment of a robotically controlled tray handling end effector. The end effector 300 includes a second gripping mechanism that is controlled (e.g., during a second operational mode of multi-mode operation) to grip an item with gripper arms (e.g., side members 304, 306). In some embodiments, the end effector 300 is controlled to move one or more of the gripper arms to open their grips to allow the end effector 300 to move into position to grip an object (e.g., a tray), and to move one or more of the gripper arms to close their grips on the object to be gripped.

[0102] In some embodiments, during operation of the end effector 300 in the first mode, the end effector is transitioned to an inactive state in which elements (e.g., gripper arms) are moved to a fully stored (retracted) state. As shown in Figure 3C, the side members 304, 306 are placed in an active state in which the side members 304, 306 are fully retracted, allowing the suction-based end effector 314 (e.g., a first gripping mechanism such as a suction-based end effector) to grip an item.

[0103] Vector / direction 316 illustrates an example of a closed position (e.g., closed position threshold) corresponding to end effector 300 operating in the second mode (e.g., gripper arm positioned in the active state). In various embodiments, the closed position is a configuration in which side member 306 forms a normal vector (or a substantially normal vector) with respect to cross member 302, extending in a direction away from the portion of cross member 302 attached to the robot arm. For example, the closed position threshold is 90 degrees (or substantially 90 degrees) relative to the direction along which cross member 302 is aligned. As shown in FIG. 3C , side members 304, 306 are moved to an open position (e.g., retracted). When side members 304, 306 are moved to the open position, the angle between side member 306 and vector / direction 316 is represented by angle 315. According to various embodiments, the open position threshold corresponds to a configuration in which angle 313 is between 145 degrees and 225 degrees. In some embodiments, the open position threshold corresponds to a configuration in which angle 313 is between 180 degrees and 225 degrees.

[0104] 4 is a flowchart illustrating a process for operating an end effector to move an object, according to various embodiments. In some embodiments, process 400 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 400 is performed by system 100 of FIG. 1A, or the like.

[0105] In step 402, a decision is made to operate an end effector (e.g., a multi-mode end effector) to pick / place an object. In some embodiments, the object may be a tray, a container, a tote, a box, an item (e.g., an item that may be contained in a tray), etc.

[0106] At step 404, a mode in which the end effector will be operated is determined. The system selects from a plurality of modes a mode according to which the end effector will be operated. In some embodiments, the system determines whether to operate the end effector in a first mode according to which a first gripping mechanism (e.g., a suction-based end effector) is used to grip an object, and / or whether to operate the end effector in a second mode according to which a second gripping mechanism (e.g., an end effector with multiple gripper arms) is used to grip an object.

[0107] At step 406, a determination is made as to whether the end effector is operated in the first mode. In response to determining at step 406 that the end effector is operated in the first mode, process 400 proceeds to step 408. Conversely, in response to determining at step 406 that the end effector is not operated in the first mode, process 400 proceeds to step 412.

[0108] At step 408, a plan for picking / placing an object using the suction-based end effector is determined. In response to determining to operate the end effector in the first mode, the system determines a plan (or strategy) for grasping an object, such as an item contained in a tray or other container, and for placing the object at a destination location (e.g., a tray, a conveyor, a shelf, etc.). In some embodiments, in response to determining to operate the end effector in the first mode, the system controls the end effector to transition the second gripping mechanism to an inactive state (e.g., the gripper arm is moved to a retracted position). The determined plan for gripping the object may include an operation for transitioning the second gripping mechanism to an inactive state.

[0109] In step 410, the suction-based end effector is controlled to pick and place the object at the destination location. The system controls the suction-based end effector to activate a suction mechanism to apply a suction force between a suction cup of the suction-based end effector and the object to be grasped. The system controls the suction mechanism based at least in part on feedback received by a sensor that detects the suction force (or other attribute of adhesion between the suction cup and the object). In some embodiments, controlling the suction-based end effector to pick and place the object includes controlling a robotic arm to which the multi-mode end effector is attached to pick and place the object using the suction-based end effector of the robotic arm.

[0110] At step 412, a plan is determined for picking / placing an object using an end effector with a gripper arm. In response to determining to operate the end effector in the second mode, the system determines a plan (or strategy) for gripping an object, such as a tray (e.g., a tray in a stack of trays, etc.). In some embodiments, in response to determining to operate the end effector in the second mode, the system controls the end effector to transition the second gripping mechanism to an active state (e.g., the gripper arm is moved to a deployed position). The determined plan for gripping the object may include an operation for transitioning the second gripping mechanism to an active state.

[0111] In step 414, the end effector with the gripper arms is controlled to pick and place the object at the destination location. The system controls the end effector with the gripper arms (e.g., a second gripping mechanism) to actuate movement of one or more of the gripper arms to grasp the object to be grasped (e.g., a tray). For example, the system controls movement of an active side member to engage the object. The system controls the end effector with the gripper arms based at least in part on feedback received by a sensor that detects positioning of one or more gripper arms (or thumbs of such arms) relative to the object to be grasped. In some embodiments, controlling the end effector with the gripper arms to pick and place the object includes controlling a robotic arm to which the multi-mode end effector is attached to grasp and pick / place the object using the gripper arms of the robotic arm.

[0112] At step 416, a determination is made as to whether process 400 is complete. In some embodiments, process 400 is determined to be complete in response to a determination that there are no more objects (e.g., trays, items) to be moved, that the trays held by the task table are empty (e.g., in the case of an unloading operation), that the trays held by the task table are full (e.g., in the case of a loading operation), that a user has shut down the system, that an administrator has indicated that process 400 is to be paused or stopped, etc. In response to a determination that process 400 is complete, process 400 ends. In response to a determination that process 400 is not complete, process 400 returns to step 402.

[0113] 5A is a flowchart illustrating a process for operating an end effector in connection with picking or placing items to / from a tray, according to various embodiments. In some embodiments, process 500 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 500 is performed by system 100 of FIG. 1A, or the like.

[0114] At step 502, a decision is made to operate the end effector (e.g., a multi-mode end effector) in a first mode. In some embodiments, the system determines to operate the multi-mode end effector in the first mode in connection with determining that the object to be grasped is an item to be picked / placed from / to a tray or otherwise determining that the object will be grasped with a suction-based end effector.

[0115] At step 504, information is obtained from one or more sensors. The information indicates whether one or more of the gripper arms are in an active state or an inactive state (or an intermediate state between the inactive state and the inactive state). In some embodiments, the system utilizes the information corresponding to the positioning of the gripper arms in connection with controlling the gripper arms (or second gripping mechanisms) to transition between the active state and the inactive state depending on the mode in which the multi-mode end effector is operated.

[0116] At step 506, a determination is made as to whether the gripper arm is disposed in an inactive state. In response to determining at step 506 that the gripper arm is not in an inactive state (or determining that the gripper arm is in an active state), process 500 proceeds to step 508, where the configuration of the gripper arm is adjusted. For example, the system controls the gripper arm to move (or continue to move) to an inactive state (e.g., to a retracted position). In some embodiments, the inactive state corresponds to the gripper arm being disposed in a threshold retracted state, such as within an angular range between the gripper arm and a side member (e.g., the gripper arm is considered to be in an inactive state if it is within a threshold retraction of the gripper arm, even if the gripper arm is not fully retracted). Process 500 repeats steps 504-508 until the system determines that the gripper arm is in an inactive state.

[0117] In response to determining in step 506 that the gripper arm is in an inactive state, process 500 proceeds to step 510 where the system determines to engage an item (such as an item in a tray or other source location (e.g., a shelf, conveyor, etc.)).

[0118] In step 512, the system controls to adjust the position of the suction-based end effector (e.g., first gripping mechanism). The system controls to position the suction-based end effector to engage with the item to be gripped. For example, the system moves the robot arm and end effector to a position where the suction cups on the suction-based end effector engage with the item.

[0119] In step 514, the system grasps an item with the suction-based end effector using suction control. The system activates a suction mechanism to apply a suction force between one or more suction cups (e.g., included in the suction-based end effector) and the grasped item. In some embodiments, the suction-based end effector is controlled to grasp multiple items (e.g., to simultaneously move multiple items to respective destination locations).

[0120] At step 516, information is obtained from one or more sensors. The information indicates whether the suction-based end effector is engaged with the item to be grasped. For example, the system obtains information regarding the suction force between the suction cups of the suction-based end effector and the item to be grasped.

[0121] At step 518, the system determines whether the item is engaged. For example, the system determines whether the item is securely grasped by the suction-based end effector. In response to determining at step 518 that the item is not securely grasped (e.g., the suction force between the item and the end effector is less than a threshold suction force, or the item is not engaged with the suction-based end effector), process 500 returns to step 514, where the system uses suction control to adjust / ensure the engagement / grasping of the item with the suction-based end effector. Process 500 repeats steps 514-518 until the system determines that the item is securely grasped by the suction-based end effector.

[0122] In step 520, the item is moved to the destination location and the suction-based end effector is controlled to place the item. In some embodiments, the system controls the robot arm to move the item to (or near) the destination location and then controls the suction-based end effector to release the item at the destination location. For example, the system controls the suction-based end effector to reduce / eliminate the adhesive force between the suction-based end effector and the item.

[0123] 5B is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to or from a tray, according to various embodiments. In some embodiments, process 550 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 550 is performed by system 100 of FIG. 1A, or the like.

[0124] In step 552, the system determines to operate the end effector in a first mode. In some embodiments, step 552 corresponds to or is similar to step 502 of process 500 of Figure 5A.

[0125] Information is obtained from one or more sensors at step 554. In some embodiments, step 554 corresponds to or is similar to step 504 of process 500 of Figure 5A.

[0126] At step 556, a determination is made as to whether the gripper arm is disposed in an inactive state. In some embodiments, step 556 corresponds to or is similar to step 506 of process 500 of FIG. 5A. In response to determining at step 556 that the gripper arm is not in an inactive state (or determining that the gripper arm is in an active state), process 500 proceeds to step 558, where the configuration of the gripper arm is adjusted. In some embodiments, step 558 corresponds to or is similar to step 508 of process 500 of FIG. 5A. Process 550 repeats steps 554-558 until the system determines that the gripper arm is in an inactive state.

[0127] In step 560, the system determines to engage an item in a tray or other container (or an item from a source location). The system determines to engage an item based on a manifest (e.g., order, packing slip, etc.).

[0128] In step 562, the system controls to adjust the position of the suction-based end effector (e.g., first gripping mechanism) in an inactive state. In some embodiments, step 562 corresponds to or is similar to step 512 of process 500 of FIG. 5A.

[0129] In step 564, the system uses suction control to grasp the item with the suction-based end effector. In some embodiments, step 564 corresponds to or is similar to step 514 of process 500 of Figure 5A.

[0130] Information is obtained from one or more sensors at step 566. In some embodiments, step 566 corresponds to or is similar to step 516 of process 500 of Figure 5A.

[0131] At step 568, the system determines whether the item is engaged. In some embodiments, step 568 corresponds to or is similar to step 518 of process 500 of FIG. 5A. In response to determining at step 568 that the item is not securely grasped (e.g., the suction force between the item and the end effector is less than a threshold suction force, or the item is not engaged with the item), process 500 returns to step 564, where the system uses suction control to adjust / ensure the engagement / grasping of the item with the suction-based end effector. Process 550 repeats steps 564-568 until the system determines that the item is securely grasped by the suction-based end effector.

[0132] At step 570, a determination is made as to whether one or more other items are to be grasped by the suction-based end effector. For example, the system determines whether the suction-based end effector is simultaneously moving multiple items to their respective destination locations. In response to determining at step 570 that one or more other items are to be grasped by the suction-based end effector (e.g., due to simultaneous movement / placement), process 550 returns to step 560, and process 550 repeats steps 560-570 until the system determines that no more items are to be grasped by the suction-based end effector.

[0133] In step 572, the item is moved to a destination location and the suction-based end effector is controlled to place the item. In some embodiments, step 572 corresponds to or is similar to step 520 of process 500 of Figure 5A.

[0134] 6A is a flowchart illustrating a process for operating an end effector in connection with picking or placing a tray or other container, according to various embodiments. In some embodiments, process 600 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 600 is performed by system 100 of FIG. 1A, or the like.

[0135] A decision is made to operate the end effector (e.g., a multi-mode end effector) in a second mode at step 602. In some embodiments, the system determines to operate the multi-mode end effector in the second mode in connection with determining that the object to be grasped is a tray to be picked and / or placed, such as in a stack of trays, or otherwise determining that the object will be grasped with an end effector having a gripper arm.

[0136] At step 604, information is obtained from one or more sensors. The information indicates whether one or more of the gripper arms are in an active state or an inactive state (or an intermediate state between the inactive state and the inactive state). In some embodiments, the system utilizes the information corresponding to the positioning of the gripper arms in connection with controlling the gripper arms (or second gripping mechanisms) to transition between the active state and the inactive state depending on the mode in which the multi-mode end effector is operated.

[0137] At step 606, a determination is made as to whether the gripper arms are disposed in an inactive state. In response to determining at step 606 that the gripper arms are not in an active state (or that the gripper arms are in an inactive state), process 600 proceeds to step 608, where the configuration of the gripper arms is adjusted. For example, the system controls the gripper arms to move (or continue to move) to an active state (e.g., to a deployed position). In some embodiments, the active state corresponds to the gripper arms being disposed in a threshold deployment state, such as within an angular range between the gripper arms and the side members (e.g., the gripper arms are considered to be in an active state if they are within a deployment threshold of the gripper arms, even if they are not fully deployed). As an example, referring to FIG. 3C , the threshold deployment state may correspond to the side members 304, 306 being within a range of 30 degrees to −30 degrees relative to vector 316. 3C, the threshold deployment state may correspond to the side members 304, 306 being within a range of 15 degrees to -15 degrees relative to the vector 316. The process 600 repeats steps 604-608 until the system determines that the gripper arm is in an inactive state.

[0138] In response to determining in step 606 that the gripper arm is in an inactive state, process 600 proceeds to step 610 where the system determines to engage an object (e.g., one or more trays) with a second gripping mechanism (e.g., a gripper arm).

[0139] In step 612, the system controls to adjust the position of an end effector having a gripper arm (e.g., a second gripping mechanism). The system controls to position the end effector having the gripper arm to engage with an item to be gripped. For example, the system moves the robot arm and end effector to a position where the gripper arm of the end effector engages with an object (e.g., a tray).

[0140] In step 614, the system controls the end effector to grip a tray with a gripper arm (e.g., an end effector with a gripper arm). The system actuates one or more of the gripper arms to apply a force between the gripper arm and the tray to be gripped. In some embodiments, the end effector with gripper arms is controlled to grip multiple items (e.g., to simultaneously move multiple items to respective destination locations). As an example, the system controls an active arm (e.g., an active gripper arm that is movable relative to a lateral member of a multi-mode end effector) to close and insert the thumb of the active arm into the gripping hole of the tray using force control.

[0141] In step 616, the system (e.g., a robot) tests its grip on the tray, and if the grip is determined to be tight in step 618, the robot moves the tray to its destination (e.g., process 600 proceeds to step 622). If the grip is not tight, as determined in step 616, the grip is adjusted in step 622 and tested again in step 616. For example, the robot may place the tray back on the source stack, release the tray, and attempt a new grip. Alternatively, the robot may place the tray at least partially on the source stack and attempt to adjust the grip without fully releasing the tray, for example, by using force control to attempt to insert the passive and / or active side thumbs, respectively, more fully into the tray.

[0142] In step 620, the tray is moved to the destination location and the end effector is controlled to place the tray (e.g., the gripper arm is controlled to move / release the tray). In some embodiments, the system controls the robotic arm to move the item to (or near) the destination location, and then controls the end effector to release the tray at the destination location.

[0143] According to various embodiments, processes 625 and 650 of process 600 are performed in conjunction with steps 612-614 of process 600.

[0144] 6B is a flowchart illustrating a process for operating an end effector in connection with picking or placing a tray or other container, according to various embodiments. In some embodiments, process 625 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 625 is performed by system 100 of FIG. 1A, or the like. Process 625 is performed in connection with grasping an item, such as a tray.

[0145] According to various embodiments, a side member (e.g., a passive side member, such as side member 304 of end effector 300) includes one or more sensors. The one or more sensors included on the side member are configured to acquire information regarding the position of a structure (e.g., a tray) relative to the position of the end effector (or, in particular, the passive side member). Examples of information acquired by the one or more sensors include (i) acquiring information indicating that the tray (or the end effector relative to the tray) is in a position where the end effector is controlled to engage at least one side of the end effector (e.g., a passive member or a structure included on the passive member) with a hole, recess, or a on one side of the tray (e.g., the tray to be grasped); (ii) acquiring information indicating that the tray (or the end effector relative to the tray) is in a position where the end effector (e.g., a passive member or a structure included on the passive member) is located. The robotic system utilizes the information acquired from the one or more sensors in connection with positioning the passive side member (or, in general, the end effector). In some embodiments, the robotic system uses information obtained from one or more sensors, along with information obtained from the force sensor, to control an end effector (e.g., a thumb provided on a passive side member for engaging a tray).

[0146] In various embodiments, the end effector includes a first sensor configured to acquire information indicating that the tray is in a position where the end effector is controlled to engage a passive side structure (e.g., a thumb located on the passive side structure) with a hole, recess, or handle included in a structure of the tray. The first sensor is located on the passive side member, such as at or near a distal end of the passive side member (e.g., near the bottom or distal end of a fin on the passive side member). When the end effector is moved near the tray, the robotic system uses the information acquired from the first sensor in connection with moving the end effector to engage the tray with a structure on the passive side structure (e.g., a thumb located on the passive side member). For example, the robotic system uses the information acquired from the first sensor to roughly position the end effector (e.g., to determine whether the tray is between the side members of the end effector, etc.).

[0147] In various embodiments, the end effector includes a second sensor configured to acquire information indicating that the tray is in a position where the passive side structure is engaged with a hole, recess, or handle provided in the structure (e.g., a structure on the side of the tray). The second sensor is located on the passive side member, such as near a structure on the passive side member (e.g., near a thumb on the passive side member or near the top of a fin on the passive side member). When the end effector is moved near the tray, the robotic system uses the information acquired from the second sensor in connection with moving the end effector to engage the tray with a structure on the passive side structure (e.g., a thumb located on the passive side member). For example, the robotic system uses the information acquired from the second sensor to fine-tune the positioning of the end effector.

[0148] At step 626, information is obtained from the first sensor, the information indicating whether a passive arm (passive side member) of the end effector is near the tray.

[0149] In step 628, the robotic system determines whether to engage the passive arm. For example, the robotic system uses information obtained from the first sensor to determine whether to engage the tray with the passive arm. The robotic system determines to engage the tray with the passive arm in response to determining that the plan for moving the tray indicates that the tray is to be picked and placed at a destination location and that the tray is near the end effector. In some embodiments, the system uses information obtained from the first sensor to determine whether the tray is between the passive arm and the active arm of the end effector.

[0150] In response to determining in step 628 that the passive arm is not engaged (e.g., with the tray), process 625 proceeds to step 630 where the position of the passive arm is adjusted. The robotic system controls the robot arm to move the end effector, such as closer to the tray. Process 625 then returns to step 626.

[0151] In response to determining in step 628 that the passive arm is engaged (e.g., with a tray), process 625 proceeds to step 632, where the robotic system uses force control to engage the passive arm thumb with the tray. For example, the robotic system uses force control to engage the passive arm thumb into a structure of the tray (e.g., a hole, recess, handle, etc.).

[0152] At step 634, information is obtained from the second sensor. The information indicates whether the passive arm thumb is engaged with a structure on the tray.

[0153] In step 636, the robotic system determines whether the thumb of the passive arm is engaged with the tray.

[0154] In response to determining in step 636 that the passive arm thumb is not engaged with the tray, the process 625 returns to step 632, and steps 632-636 are repeated. For example, the robotic system further controls the end effector to move to engage a structure on the tray with the passive arm thumb.

[0155] In response to determining in step 636 that the passive arm thumb is engaged with the tray, process 625 proceeds to step 638 with an indication that the passive arm is engaged with the tray. For example, if process 625 was called by step 612 of process 600, step 628 provides an indication to the robotic system (e.g., a process running on the robotic system) that the passive thumb arm is engaged with the tray and process 600 proceeds to step 606.

[0156] FIG. 6C is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to / from a tray, according to various embodiments. In some embodiments, process 650 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 650 is performed by system 100 of FIG. 1A, or the like. Process 650 is performed in connection with grasping an item, such as a tray. Process 650 is performed in connection with grasping an item, such as a tray (or multiple trays). For example, process 650 is performed in connection with steps 610-614 of process 600.

[0157] According to various embodiments, the end effector (e.g., the side member, the active arm, or both gripper arms) includes one or more sensors that obtain information regarding the position of the active arm or arms (e.g., the side members 304, 306 of the end effector 300). For example, the system uses the information to determine whether the gripper arms are positioned in an active state (e.g., a deployed state) or an inactive state (e.g., a retracted state). The system can control the end effector (e.g., a multi-mode end effector) to operate in different modes or otherwise transition to different states (e.g., an active state, an inactive state, etc.). In some embodiments, the end effector (e.g., the side member, the active arm, or both gripper arms) includes a sensor that detects whether the active arm is in an open or closed position. For example, the sensor is a mechanical limit switch configured to obtain information indicating whether the active side member is in an open or closed position. As another example, the sensor is a mechanical limit switch configured to obtain information indicative of whether the corresponding gripper arm is in a deployed position or a retracted state (or an intermediate state between fully deployed and fully retracted, etc.). In some embodiments, the end effector (e.g., a side member or an active arm) includes a sensor that detects the extent to which the gripper arm is in the open or closed position (e.g., the sensor determines a particular orientation of the gripper arm or a particular position of the gripper arm between (including) the open and closed positions). As another example, the sensor is an optical sensor. The optical sensor is configured to obtain information indicative of whether the active side member is in the open or closed position. As another example, the sensor is an optical sensor. The optical sensor is configured to obtain information indicative of whether the corresponding gripper arm is in a deployed position or a retracted state (or an intermediate state between fully deployed and fully retracted, etc.).The optical sensor may also be configured to obtain information indicative of the degree to which the active side member is open (e.g., whether the active arm is partially open, such as halfway between an open position and a closed position). The robotic system uses one or more sensors (e.g., a sensor that obtains information about the position of the active arm) to control actuation of an actuator that moves the gripper arm to move (e.g., to move the active arm between a closed position and an open position).

[0158] In some embodiments, the end effector (e.g., the active arm of the end effector) includes one or more sensors used to detect whether the active arm (e.g., the thumb of the active arm) is engaged with the tray (e.g., a structure on the tray, such as a hole, recess, or handle). For example, the end effector includes sensors 315a and / or 315b of the end effector 300 shown in FIG. 3C.

[0159] In step 652, information indicating whether the active arm is open / closed is obtained from a sensor (e.g., a sensor provided on a lateral member or active member on the end effector). The robotic system uses the sensor to obtain information regarding the position of the active arm.

[0160] In step 654, the robotic system determines whether the active arm is in an open position. For example, the robotic system determines whether the active arm is fully open (e.g., to an open position threshold). As another example, the robotic system determines whether the active arm is open enough to grasp the tray (e.g., if an adjacent stack prevents / restricts the robotic system from fully opening the active arm).

[0161] In response to determining in step 654 that the active arm is not in the open position, process 650 proceeds to step 656, where the position of the active arm is adjusted. For example, the position of the active arm is adjusted to allow the end effector to grasp the tray (e.g., to ensure tray clearance when the end effector is controlled to grasp the tray). The robotic system controls the robotic arm to move the active arm to open it further or to open it completely. Process 650 then returns to step 652.

[0162] In response to determining in step 654 that the active arm is in the open position, process 650 proceeds to step 658, where the robotic system determines to engage a tray. For example, the robotic system determines to control an actuator to move the active arm to engage the tray with the active arm (e.g., a structure on the active arm, such as the active arm thumb).

[0163] The configuration of the active arm is adjusted to a closed position using force control in step 660. In response to determining to engage a tray, the robotic system controls the actuator to move the active arm to the closed position.

[0164] In step 662, information is obtained from a sensor indicating whether the active arm thumb is engaged with a structure on the tray. For example, the robotic system obtains information from sensors 315a and / or 315b of the end effector 300 shown in FIG. 3C.

[0165] At step 664, a determination is made as to whether the active arm thumb is engaged with the tray. In some embodiments, the robotic system uses information obtained from the sensor (e.g., information indicating whether the active arm thumb is engaged with a structure on the tray) to determine whether the active arm thumb is engaged with the tray (e.g., whether the active arm thumb is inserted into a hole, recess, or handle on the tray). In some embodiments, the robotic system further obtains information from a force sensor and utilizes information regarding the force acting on the end effector in connection with determining whether the active arm thumb is engaged with the tray.

[0166] In response to determining at step 664 that the active arm thumb is not engaged with the tray, process 650 returns to step 660 to further adjust the configuration of the active arm. Process 650 repeats steps 660, 662, and 664 until the robotic system determines that the active arm thumb is engaged with the tray.

[0167] In response to determining in step 664 that the active arm thumb is not engaged with the tray, process 650 proceeds to step 666 where an indication is provided that the active arm is engaged with the tray. For example, if process 650 was called by step 612 of process 600, step 666 provides an indication to the robotic system (e.g., a process executing on the robotic system) that the active thumb arm is engaged with the tray and process 600 proceeds to step 614.

[0168] 6D is a flowchart illustrating a process for operating an end effector in connection with picking or placing an item to / from a tray, according to various embodiments. In some embodiments, process 675 is performed by system 100 of FIG. 1A and / or robot 1300 of FIG. 13. Process 675 is performed in connection with grasping an item, such as a tray.

[0169] At step 677, a decision is made to place one or more trays. The system determines that one or more trays are to be placed at a destination location. For example, the system determines to create a stack of trays by placing one or more trays on top of another tray. As another example, the system determines to move a tray at the top of a stack of trays in response to determining that the top tray is empty (e.g., to expose items in a tray below the top tray).

[0170] In step 679, the configuration of the active arms of the end effector is adjusted to an open position using force control. For example, the second gripping mechanism of the end effector is robotically placed in an active state, and the end effector is actuated to move the active arms of the second gripping mechanism in association with placing one or more trays with the second gripping mechanism. In some embodiments, the system controls the end effector to move the multiple gripper arms in association with releasing a grip on the trays.

[0171] In step 681, information indicating whether the active arm is open or closed is obtained from one or more sensors. In some embodiments, the system determines whether the gripper arm is in an active or inactive state.

[0172] In step 683, a determination is made as to whether the active arm is open (or retracted). For example, the system determines whether the gripper arm is in an active or inactive state. The system determines whether the active arm is open or closed based at least in part on information obtained from one or more sensors indicating whether the active arm is open or closed.

[0173] In response to determining in step 683 that the active arm is not open, process 675 proceeds to step 685 where the configuration of the active arm is adjusted. The system controls actuation of the end effector (e.g., the active arm) to move the active arm to an open position. Process 675 then returns to step 681 and repeats steps 681-685 until the system determines that the active arm is open.

[0174] In response to determining in step 683 that the active arm is open, process 675 proceeds to step 687 where information indicating whether the active arm thumb is engaged with a structure on the tray is obtained from one or more sensors.

[0175] A determination is made as to whether the thumb of the active arm is engaged with the tray at step 689. In some embodiments, the system determines whether the thumb of the active arm is engaged with the tray based at least in part on information obtained from one or more sensors indicating whether the active arm thumb is engaged with a structure on the tray.

[0176] In step 691, the system provides an indication that the active thumb has been detached from the tray. In some embodiments, the system provides an indication that the gripper arm has been detached from the tray (e.g., the tray has been released). The system can provide the indication to the process that called operation 675 (e.g., step 620 of operation 600).

[0177] 7A illustrates an end effector configured in a first mode, according to various embodiments. In the illustrated example, a multi-mode end effector 700 is used to pick an item from a tray 720. In response to determining to pick an item from the tray 720, the system determines to operate the multi-mode end effector in a first mode in which side members 704 and 706 are moved to an inactive state. Moving the side members 704 and 706 to the inactive state includes moving the side members sufficiently to expose the suction-based end effector 714 (e.g., to allow the suction-based end effector 714 to engage / grasp the item). In the illustrated example, the side members 704 and 706 are positioned in an inactive state (e.g., a retracted state) in which the side members 704 and 706 are opened approximately 180 degrees relative to their positions in the active state. In various embodiments, the side members 704 and 706 are opened greater than 180 degrees (eg, such that the side members 704 and 706 form an acute angle with respect to the top surface of the cross member 702).

[0178] 7B illustrates an end effector configured in a first mode, according to various embodiments. In the illustrated example, the multi-mode end effector 700 is positioned near an item 724 (e.g., the suction-based end effector 714 is engaged with the item 724). The system robotically controls the robotic arm to which the multi-mode end effector 700 is attached to move the suction-based end effector 714 to a source position for the item 724. The system robotically controls the suction-based end effector 714 to apply an adhesive force with the item 724. For example, the system activates a suction control of the suction-based end effector 714 (or the multi-mode end effector 700) to form an adhesive force between at least one suction cup of the suction-based end effector 714 and the item 724. The system determines whether the item 724 is securely gripped before moving the robotic arm and / or the multi-mode end effector 700 to move the item 724.

[0179] 7C illustrates an end effector configured in a first mode, according to various embodiments. In the illustrated example, the multi-mode end effector is picking an item 724 from a tray 720. In some embodiments, in response to determining that the suction-based end effector 714 has a firm grip on the item 724, the system controls the robotic arm to move the item 724 to a corresponding destination location.

[0180] FIG. 8A illustrates a robotically controlled tray / item handling end effector, according to various embodiments. In various embodiments, the end effector (such as end effector 300 of FIGS. 3A and 3B ) comprises the structure shown in FIG. 8A . In the illustrated example, end effector 800 comprises a cross member 802 and side members 804 and 806, each having thumbs 805 and 808 configured to insert into a handle or other hole or recess on a first side of a tray. As shown, side members 804 and 806 include tabs or brackets 810 and 822 attached to the inside top of side members 804 and 806, which, in this example, are positioned to align with through-hole 812 (or a set of through-holes) through cross member 802. Various other configurations or attachments of side members 804 and 806 to cross member 802 may also be implemented. In some embodiments, side thumbs 805 and 808 have different profiles. For example, side thumb 808 has a steeper curvature or profile than side thumb 805. As another example, side thumb 808 has a greater height than side thumb 805.

[0181] According to various embodiments, end effector 300 is a multi-mode end effector. For example, end effector 800 is controlled to operate in a first mode in which suction-based end effector 830 is used to grasp an object, and in a second mode in which a second gripping mechanism including side members 804 and 806 is used to grasp an object. In the illustrated example, suction-based end effector 830 is connected to the bottom surface of side member 802. suction-based end effector 830 includes suction cups 832, 834, 836, and 838. In some embodiments, suction cups 832, 834, 836, and 838 are controlled together (e.g., a single control is used to activate each of suction cups 832, 834, 836, and 838). In some embodiments, suction cups 832, 834, 836, and 838 are controlled individually or in subgroups. For example, the system may control suction cups 832 and 834 together and separately control suction cups 836 and 838 together. Independent control of at least some of suction cups 832, 834, 836, and 838 allows multi-mode end effector 800 to grasp and simultaneously move multiple items (e.g., to place the items at respective destination locations).

[0182] FIG. 8B illustrates a robotically controlled tray / item handling end effector, according to various embodiments. In various embodiments, the end effector (such as end effector 300 of FIGS. 3A-3C) comprises the structure shown in FIG. 8B. In the illustrated example and state, end effector 800 of FIG. 8A is shown assembled (in a deployed position corresponding to an active state). Shoulder bolt 814 (or hinge pin or similar structure) is shown inserted through through-hole 812 and tab / bracket 810. Side member 804 may be similarly connected to cross member 802. A pneumatic or hydraulic cylinder (e.g., cylinder 816) is mounted (e.g., by a pivot bracket or other bracket) to an internal surface within cross member 802.

[0183] In various embodiments, the cylinder 816 (e.g., a pneumatic cylinder) and end rod 818 comprise a cushioned, two-way pneumatic cylinder. Actuation of one or more movable side members (e.g., side members 804, 806) is performed by actuating the cylinder 816. The end rod 818 of the cylinder 816 is connected to the side members 804, 806. The side member 806 is rotatably connected to the cross member 802 via a pivot joint formed by inserting a shoulder bolt 814 into a hole 812 and a tab or bracket 810, and is pushed / pulled by the pneumatic cylinder to close / open the side member 806, respectively. The side member 804 is similarly connected to the cross member 802 and may be similarly controlled to move (e.g., transition between an active state and an inactive state). Actuation of the cylinder 816 is controlled by a single, four-way, two-position solenoid in various embodiments.

[0184] 8C is a diagram illustrating a robotically controlled tray / item handling end effector, according to various embodiments. In the illustrated example, side members 804 and 806 are transitioned to an inactive state (e.g., a retracted position). Controlling end effector 800 to move side members 804, 806 enables end effector 800 to utilize suction-based end effector 830.

[0185] 9A is a diagram illustrating a robotically controlled tray / item handling end effector, according to various embodiments. In the illustrated view, tray handling end effector 900 includes cross member 902, side member 904, and side member 906. End effector 900 is attached to a robot arm (not shown) via force sensor 910 and pivot bracket 912. In the illustrated state, end effector 900 is operated in a second mode in which side members 904 and 906 are in an active state and are used to grasp an object (such as a tray 914). For example, thumbs (not shown) on side members 904, 906 may each be inserted into corresponding holes (not shown) in both trays 914.

[0186] In the illustrated example, the end effector 900 further includes guide fins 918 and 920 attached along the bottom edges of the side members 904 and 906, respectively. In various embodiments, the guide fins 918 and 920 extend along all or a substantial portion of the bottom edges of the side members 904 and 906. As shown, each guide fin 918 and 920 has a flared shape at the bottom, such that the distance between the respective bottom edges of the guide fins 918 and 920 is greater than the width of the tray 914 and the distance between the inner surfaces of the side members 904 and 906 when in the closed position, as shown.

[0187] 9B is a diagram illustrating a robotically controlled tray / item handling end effector, according to various embodiments. In some embodiments, the guide fins 918, 920 are correspondingly rotatably connected to the side members 904, 906. As an example, when the side members 904, 906 are moved to an inactive state (e.g., to a retracted position), the guide fins 918, 920 are controlled to rotate relative to the side members 904, 906 to further retract the guide fins 918, 920. In the illustrated example, the side members 904, 906 are in the inactive state and the guide fins 918, 920 are further retracted (e.g., the guide fins 918, 920 are rotated toward the center of the side member 902 (e.g., from an extended (deployed) position when the side members 904, 906 are in the active state)).

[0188] Operating the end effector 900 in a first mode in which the side members 904 , 906 are moved to an inactive state exposes the suction-based end effector 930 for grasping an item (eg, items 922 , 924 , 926 ) from the tray 914 .

[0189] FIG. 10A illustrates a robotically controlled tray / item handling end effector with guide fins, according to various embodiments. In the illustrated example, the tray handling end effector 1000 includes a cross member 1002 and side members 1004 and 1006. The end effector 1000 is attached to a robot arm (not shown) via a force sensor 1010 and a set of pivot brackets (not shown). In the illustrated state, the end effector 1000 is gripping a tray 1014. For example, the side thumbs (not shown) of the side members 1004 and 1006 may each be inserted into corresponding holes (not shown) in both of the trays 1014. While the example end effector 1000 includes only the side member 1006 that is movable (e.g., using a pneumatic piston, etc.), both side members 1004 and 1006 may be rotatably connected to the cross member 1002.

[0190] In the illustrated example, the end effector 1000 further includes guide fins 1018 and 1020 attached along the bottom edges of the side members 1004 and 1006, respectively. In various embodiments, the guide fins 1018 and 1020 extend along all or a substantial portion of the bottom edges of the side members 1004 and 1006. As shown, each guide fin 1018 and 1020 has a flared shape at the bottom, such that the distance between the respective bottom edges of the guide fins 1018 and 1020 is greater than the width of the tray 1014 and the distance between the inner surfaces of the passive member and the side members 1004 and 1006 when in the closed position, as shown.

[0191] 10A, the end effector 1000 is used to position (e.g., place) the tray 1014 on top of the tray 1016. For example, the tray 1016 may be the top tray on the destination stack to which the tray 1014 is being added.

[0192] In some embodiments, the end effector 1000 includes one or more vehicle gripper modules 1021 a or 1021 b on the side members 1004, 1006 or the guide fins 1018, 1020. The vehicle gripper modules 1021 a or 1021 b may include an inner surface (e.g., a surface that engages a tray or vehicle (such as a dolly)) that has a relatively higher friction than the inner surfaces of the side members 1004, 1006 or the guide fins 1018, 1020. The one or more vehicle gripper modules 1021 a or 1021 b may be shaped or configured to stably grip a vehicle (e.g., a dolly) when the end effector is controlled to cause the one or more vehicle gripper modules 1021 a or 1021 b to engage such vehicle.

[0193] In the illustrated example, the end effector 1000 comprises a suction-based end effector 1030. For example, the end effector 1000 is a multi-mode end effector that can be selectively operated in multiple modes (e.g., a first mode in which the suction-based end effector 1030 is used to grasp an object, a second mode in which the side members 1004, 1006 are used to grasp an object, and / or a third mode in which the end effector 1000 is controlled to pull / push a tray, cart, or other object).

[0194] 10B illustrates a robotically controlled tray / item handling end effector with guide fins, according to various embodiments. In the example and configuration illustrated in FIG. 10B, the guide fins 1018, 1020 facilitate the use of force control to align and place tray 1014 on tray 1016, as shown. In various embodiments, the guide fins 1018, 1020 have a degree of compliance that facilitates tray placement. The guide fins 1018, 1020 are designed with flex characteristics to enhance operating speed, tolerance, and precision.

[0195] In various embodiments, a tray-placement episode by a single-robot tray-grabbing robot disclosed herein is smooth, gentle, precise, and resistant to uncertainty and jitter. In various embodiments, a placing episode includes one or more of the following: · Use force control to descend from a hovering position above the destination stack (eg, as shown in FIG. 10A) and make initial contact with a guide fin (eg, one or both of guide fins 1018, 1020). Guide fins 1018, 1020 help guide the tray being placed into a position that is more aligned with the tray into which it is being placed, and the guide fins 1018, 1020 also provide a feedback signal via a load cell (e.g., force sensor 1010) to adjust the position of the tray arriving at the top of the stack. Using force control, the robot makes stable contact with the destination stack and gently inserts a tray (e.g., 1014) onto it.

[0196] 10C and 10D illustrate a robotically controlled tray / item handling end effector with guide fins, according to various embodiments. In the example shown in FIGS. 10C and 10D, the end effector 1000 further includes a sensor 1022 and / or a sensor 1024. The robotic system utilizes the sensor 1022 and / or the sensor 1024 in connection with guiding the end effector 1000 in grasping the tray 1014, such as for controlling the end effector 1000 to engage the tray 1014 with structures on one or both of the side members 1004, 1006.

[0197] The robotic system uses the sensor 1024 to detect, for example, whether the tray 1014 is near the end effector 1000 so that the robotic system can finely control the movement of the end effector to engage the tray with a structure on the side member 1004 (e.g., a thumb on the side member 1004). In some embodiments, the sensor 1024 obtains information indicating that the tray 1014 is in a position where the end effector 1000 is controlled to engage a hole, recess, or handle provided in the tray 1014.

[0198] The robotic system uses the sensor 1022 to detect whether the tray 1014 is engaged by the side member 1004 (e.g., by a structure (such as a thumb) on the side member 1004). In some embodiments, the sensor 1024 acquires information indicating the tray 1014 is present in a position where a passive side structure would engage with a hole, recess, or handle provided on the tray 1014. As shown in FIG. 10D , the system determines that the robotic system determines that the tray 1014 is present in a position where a passive side structure would engage with a hole, recess, or handle provided on the tray 1014 when (i) the information acquired from the sensor indicates that a structure is adjacent to the sensor (e.g., light is reflected toward the sensor 1022), or (ii) the information acquired by the sensor 1024 indicates that no structure is present proximate the sensor 1022 (e.g., no light is reflected toward the sensor 1024).

[0199] In some embodiments, the robotic system utilizes information obtained by sensors 1022 and / or 1024 in connection with determining whether to control actuators to move one or both of the side members 1004, 1006 to grasp the tray 1014 (e.g., to engage the thumbs of the side members 1004, 1006 with the tray 1014).

[0200] The end effector 1000 includes one or more rigid structures (such as rigid structures 1040a, 1040b of side members 1004, 1006) on one or more side members (e.g., gripper arms). In some embodiments, the system uses rigid structures 1040a and / or 1040b to control the robot and / or end effector 1000 to move a dolly (or other cart, etc.), push or pull a tray, etc. As an example, the end effector 1000 includes rigid structure 1040 in addition to or instead of vehicle gripper modules 1021a or 1021b.

[0201] 10A-10D are provided in the context of gripping a tray, side members 1004 and 1006 can be used to grip a variety of other objects. For example, side members 1004 and 1006 can be used as gripper arms (e.g., gripper arms used similar to pincers to grip items). Gripper arms can be used to pick up boxes or other items.

[0202] FIG. 11 is a flowchart illustrating an automated process for placing one or more trays onto a stack, according to various embodiments. In various embodiments, process 1100 of FIG. 11 is performed by a control computer (such as control computer 128 of FIG. 1A ) configured to control one or more single-robotic tray-handling robots as disclosed herein. In the illustrated example, in step 1102, position control is used to position one or more trays to be placed, e.g., on top of a destination stack. For example, 3D camera and / or other image data is used to determine the location and orientation of the destination stack, the robot is moved (e.g., along a rail) to a position near the destination stack, and then the robot arm is manipulated to place the tray above the destination stack. In step 1104, force control is used to engage the top of the destination stack. For example, referring to the example shown in FIGS. 10A and 10B , tray 1014 is lowered until the bottom edge of one or both of guide fins 1018, 1020 just touches tray 1016 at the top of the destination stack. In step 1106, force control is used to guide a tray (e.g., tray 1014) onto the top of a destination stack (e.g., on top of tray 1016). In step 1108, the robot tests to determine if the tray being placed is fully and properly aligned and securely inserted onto the top of the top tray in the destination stack.

[0203] In various embodiments, the insertion episode (e.g., step 1108) functions to ensure stability of the tray at the top of the stack and its proper insertion. After adjustments in the z-axis (up and down) and y-axis (the axis along the rail with the tray slot, e.g., perpendicular to the page as shown in FIGS. 9A and 9B), the robot performs routing to ensure stability in a third direction (x-axis, the horizontal axis as shown in FIGS. 9A and 9B) as well. In various embodiments, the insertion episode includes: When placed in a forward offset position relative to the stack, gently pull the tray back to the top of the slot. A series of force movements that separate and smoothly move the trays. If force feedback indicates that the tray is over-slotted, a corrective action is taken to reverse the slot. For example, if the system fails to find a notch at the rear of the tray, and an equivalent notch exists at the front of the tray, the system will attempt to "slot" against that notch by reversing the direction of the slotting action. The quality of the slots is verified by moving the tray back and forth and analyzing the resulting force signals. In various embodiments, the robot is trained and / or manually trained to recognize force signals indicative of a good (or bad) insertion.

[0204] If the tray is determined to be securely inserted (step 1110), the robot releases the tray (e.g., by opening one or more of the side members and pulling the thumbs out of the holes where they are inserted) and the process ends. If not (step 1110), the placement is adjusted at step 1112 and tested again at step 1108. If the tray cannot be verified as securely placed after a configured number of attempts, in various embodiments the system prompts a human operator for assistance, e.g., remotely or manually.

[0205] FIG. 12 illustrates an example stack of trays configured to be stacked in a particular tray orientation. In the illustrated example, destination stack 1200 includes tray 1202 stacked on top of tray 1204. Tray 1206 is added to the top of stack 1200. Trays 1202, 1204, and 1206 each include a pair of differently shaped recesses on the top of the tray (e.g., recesses 1210 and 1214 on the top of tray 1202, one on a first side and the other on the opposite side) and corresponding protrusions on the bottom of the tray (e.g., protrusions 1208 and 1212 on the bottom of tray 1206). As shown in FIG. 12 , protrusion 1208 has a size and shape to fit into recess 1214, while protrusion 1212 has a size and shape to fit into recess 1210. However, as shown, tray 1206 has been inverted so that protrusion 1208 is positioned over mismatched recess 1210 and protrusion 1212 is positioned over mismatched recess 1214.

[0206] In various embodiments, the tray handling robotic system disclosed herein learns and / or trains to recognize force sensor readings and / or profiles associated with misalignments such as those shown in Figure 12. For example, after attempting to place tray 1206 on tray 1202, the system detects that tray 1206 is not securely inserted onto tray 1202, associated with tray 1206 being inverted, as shown in Figure 12. In various embodiments, in response to detecting an incorrect orientation such as that shown in Figure 12, the system lifts the tray (e.g., 1206), rotates tray 1206 180 degrees about the z (up-down) axis, and resumes the attempt to place the tray on top of the destination stack.

[0207] Figure 13 illustrates one embodiment of a tray-handling robot. Robot 1300 performs (or is used to perform) operation 400 of Figure 4, operation 500 of Figure 5A, operation 550 of Figure 5B, operation 600 of Figure 6A, operation 625 of Figure 6B, process 650 of Figure 6C, operation 675 of Figure 6D, and / or operation 1400 of Figure 14.

[0208] In various embodiments, one or more robots (such as robot 1300 of FIG. 13) may be included in the robotic tray handling systems disclosed herein (e.g., robot arms 112, 114 of FIGS. 1A and 1B). In the illustrated example, robot 1300 includes a robotic arm 1302 and a tray-handling end effector mounted on a chassis 1306 (e.g., a carriage) configured for robotically controlled movement along rails 1308 and 1310. An upper structure including vertical supports 1312 and 1314 and an upper frame 1316 provides mounting locations for 3D cameras 1318, 1320, 1322, and 1324. In various embodiments, one or more 3D cameras are located near the base of the robot.

[0209] 1A and 1B, a source tray stack is provided on one side of rails 1308 and 1310 (e.g., the far side of rail 1308 as shown), and a destination tray stack is constructed on the opposite side of rails 1308 and 1310 (e.g., the near side of rail 1310 as shown). A pair of cameras (e.g., 1318, 1320) on the source tray stack side and a pair of cameras (e.g., 1322, 1324) on the destination tray stack side are used to provide views of the relevant portions of the workspace near which the robot 1300 is positioned and operating.

[0210] In various embodiments, the image data is used to perform one or more of: avoiding collisions with other robots, tray stacks, and other items present in the workspace; planning a trajectory; and positioning the end effector 1304 and / or a tray gripped by the end effector 1304 to at least an initial position under position control. The end effector 1304 is a multi-mode end effector with a first gripping mechanism (e.g., a suction-based end effector) and a second gripping mechanism (e.g., an end effector with a gripper arm). The end effector is robotically controlled to operate in one of several different operational modes, such as a first mode in which an object is gripped using the suction-based end effector, a second mode in which an object is gripped using the end effector with a gripper arm, and a third mode in which the end effector 1304 is used to push or pull an object (e.g., a stack of trays, a cart, a dolly, etc.). Various other modes may also be implemented.

[0211] In some embodiments, cameras 1318, 1320, 1322, and 1324 are included in a vision system used to control the robotic tray handling system disclosed herein. In some embodiments, the vision system is designed to be self-calibrating. The robot utilizes markers placed at one of its joints and makes the markers visible to cameras (e.g., cameras 1318, 1320, 1322, and 1324) in the system that recognize the markers and perform pose estimation to understand its pose in a world coordinate system. The robot plans its movements using collision avoidance to move the markers close to the cameras to achieve high-quality calibration.

[0212] In some embodiments, a single manual process is performed after the automatic calibration to further ensure the quality of the process. The point cloud is overlaid on the simulated graphics of the system, and a human operator performs matching of the rendered graphics of the robot and environment in the simulator against the point cloud seen by a camera mounted on the robot. An additional verification procedure is also performed to verify the perceived depth of objects of known height in the world frame (coordinates).

[0213] In some embodiments, the systems disclosed herein self-calibrate their own dimensions. The robot traverses rails to locate pick-and-place locations and uses force control to find the coordinates of input / output slots. Updates are performed dynamically. For example, in some embodiments, the system uses specially designed calibration movements (including force control) to find the exact location of each input and output surface (where the stack of trays resides), called the "layout," and internally updates the layout values ​​over time to account for changes (uneven ground, misaligned peripherals, etc.). The robot, in various embodiments, performs these updates dynamically throughout its life.

[0214] In some embodiments, the vision system approximates the pose of the target tray or target destination stack to check the robot's target motion, and the vision system scheduler ensures simultaneous checks when possible and when both input and output targets are in view.

[0215] 14 is a flowchart illustrating a process for selecting a mode in which an end effector is to be operated and operating the end effector in the selected mode, according to various embodiments. In some embodiments, process 1400 is performed by system 100 of FIG. 1A and / or robot 1300 of FIG. 13.

[0216] In step 1402, a decision is made to operate an end effector (e.g., a multi-mode end effector) to pick / place an object. In some embodiments, the object may be a tray, a container, a tote, a box, an item (e.g., an item that may be contained in a tray), etc.

[0217] In step 1404, a mode in which the end effector will be operated is determined. The system selects from a plurality of modes according to which the end effector will be operated. In some embodiments, the system determines whether the end effector will operate in (i) a first mode in which a first gripping mechanism (e.g., a suction-based end effector) is used to grip an object, (ii) a second mode in which a second gripping mechanism (e.g., an end effector with multiple gripper arms) is used to grip an object, or (iii) a third mode in which a multi-mode end effector configuration is used to push or pull an object (e.g., a stack of trays, a cart, a dolly, a tray, items in a tray, etc.).

[0218] At step 1406, a determination is made as to whether the mode in which the end effector is operated is the first mode. In response to determining at step 1406 that the mode in which the end effector is operated is the first mode, process 1400 proceeds to step 1408. Conversely, in response to determining at step 1406 that the end effector is not operated in the first mode, process 1400 proceeds to step 1412.

[0219] In step 1408, a plan for picking / placing an object using the suction-based end effector is determined. In response to determining to operate the end effector in the first mode, the system determines a plan (or strategy) for grasping an object, such as an item contained in a tray or other container, and for placing the object at a destination location (e.g., a tray, a conveyor, a shelf, etc.). In some embodiments, in response to determining to operate the end effector in the first mode, the system controls the end effector to transition the second gripping mechanism to an inactive state (e.g., the gripper arm is moved to a retracted position). The determined plan for gripping the object may include an operation for transitioning the second gripping mechanism to an inactive state.

[0220] In step 1410, the suction-based end effector is controlled to pick and place the object at the destination location. The system controls the suction-based end effector to activate a suction mechanism to apply a suction force between a suction cup of the suction-based end effector and the object to be grasped. The system controls the suction mechanism based at least in part on feedback received by a sensor that detects the suction force (or other attribute of adhesion between the suction cup and the object). In some embodiments, controlling the suction-based end effector to pick and place the object includes controlling a robotic arm to which the multi-mode end effector is attached to pick and place the object using the suction-based end effector of the robotic arm.

[0221] At step 1412, a determination is made as to whether the mode in which the end effector is operated is the second mode. In response to determining at step 1412 that the mode in which the end effector is operated is the second mode, process 1400 proceeds to step 1414. Conversely, in response to determining at step 1412 that the end effector is not operated in the first mode, process 1400 proceeds to step 1418.

[0222] In step 1414, a plan is determined for picking / placing an object using an end effector with a gripper arm. In response to determining to operate the end effector in the second mode, the system determines a plan (or strategy) for gripping an object, such as a tray (e.g., a tray in a stack of trays, etc.). In some embodiments, in response to determining to operate the end effector in the second mode, the system controls the end effector to transition the second gripping mechanism to an active state (e.g., the gripper arm is moved to a deployed position). The determined plan for gripping the object may include an operation for transitioning the second gripping mechanism to an active state.

[0223] In step 1416, the end effector with the gripper arms is controlled to pick and place the object at the destination location. The system controls the end effector with the gripper arms (e.g., a second gripping mechanism) to actuate movement of one or more of the gripper arms to grasp the object to be grasped (e.g., a tray). For example, the system controls movement of an active side member to engage the object. The system controls the end effector with the gripper arms based at least in part on feedback received by a sensor that detects positioning of one or more gripper arms (or thumbs of such arms) relative to the object to be grasped. In some embodiments, controlling the end effector with the gripper arms to pick and place the object includes controlling a robotic arm to which the multi-mode end effector is attached to grasp and pick / place the object using the gripper arms of the robotic arm.

[0224] In step 1418, a decision is made to operate the end effector in a third mode. As one example, the system decides to operate the end effector in response to determining that the item to be grasped is best suited to the end effector's configuration rather than a suction-based end effector. As another example, the system decides to operate the end effector in the third mode in response to determining that the item will be slightly moved or shifted, or that a stack of trays or cart / trolley will be moved / pushed.

[0225] At step 1420, a plan for pushing and / or pulling the object is determined. The object may be a stack of trays, a cart, a dolly, an item, etc. in the workspace. In response to determining to operate the end effector in the third mode, the system determines a plan (or strategy) for moving the object based on nudging or pushing / pulling the object with a portion of the end effector (rigid structure, hook, etc.). The determined plan for grasping the object may include an operation to transition the second grasping mechanism to an active state.

[0226] The end effector is controlled to push / pull an object at step 1422. In some embodiments, the system controls a robotic arm to which the multi-mode end effector is attached to engage an item using a portion of the multi-mode end effector (e.g., a rigid structure, a hook, etc.), and controls the robotic arm to push / pull an object using the multi-mode end effector.

[0227] At step 1424, a determination is made as to whether process 1400 is complete. In some embodiments, process 1400 is determined to be complete in response to a determination that there are no more objects, trays, or carts to be moved (e.g., picked or placed), that totes or other containers corresponding to a manifest (e.g., order) have been built / packed, that a user has exited the system, that an administrator has indicated that process 1400 is paused or stopped, etc. In response to a determination that process 1400 is complete, process 1400 ends. In response to a determination that process 1400 is not complete, process 1400 returns to step 1402.

[0228] 15A is a diagram illustrating a bottom view of a suction-based end effector, according to various embodiments. In some embodiments, the system 100 of FIG. 1 implements an end effector 1500. According to various embodiments, the end effector 1500 is a suction-based end effector. The end effector 1500 may be included in a multi-mode end effector. For example, the end effector 1500 corresponds to a first gripping mechanism of a multi-mode end effector.

[0229] 15A, an end effector 1510 having a square face 1502 (e.g., a square base) includes a plurality of suction cups, and at least a first subset of the plurality of suction cups is different from a second subset of the plurality of suction cups. For example, suction cup 1504 is larger than suction cup 1506. As another example, suction cup 1504 has a diameter that is larger than the diameter of another suction cup (such as suction cup 1506) on end effector 1500.

[0230] An actuation mechanism (not shown) operably connected to the end effector 1500 actuates suction for one or more of the suction cups of the end effector 1500. In some embodiments, the actuation mechanism actuates a first suction cup independently of actuation of a second suction cup. In some embodiments, the suction cups are actuated according to the suction cup set to which the suction cup belongs. The actuation mechanism actuates one or more of the suction cups on the end effector 1500 based at least in part on a plan (e.g., a grasping strategy included in a plan for a singulation operation, a plan for a kitting operation, etc.).

[0231] 15B is a diagram illustrating a bottom view of a suction-based end effector, according to various embodiments. In some embodiments, the system 100 of FIG. 1 implements the end effector 1520. In some embodiments, the system 100 of FIG. 1 implements the end effector 1520. According to various embodiments, the end effector 1520 is a suction-based end effector. The end effector 1520 is included in a multi-mode end effector. For example, the end effector 1520 corresponds to a first gripping mechanism of a multi-mode end effector.

[0232] In some embodiments, the end effector 1520 includes one or more movable suction cups. The positioning of the movable suction cups is controlled based on an item to be grasped by the end effector 1520, or based on multiple items to be grasped. For example, the positioning of the movable suction cups is controlled based on a plan for grasping one or more items using the end effector 1520. The suction cups are moved relative to the surface of the end effector 1520 to increase the distance between at least two suction cups on the end effector 1520, such as in connection with enabling the end effector 1520 to grasp two different items to enable simultaneous grasping / movement of the items. The suction cups may also be moved relative to the surface of the end effector 1520 to decrease the distance between at least two suction cups on the end effector 1520, such as in connection with enabling the end effector 1520 to grasp two different items to enable simultaneous grasping / movement of the items.

[0233] In the illustrated example, end effector 1520 includes three sets of suction cups: first set 1525, second set 1530, and third set 1535. First set 1525 includes suction cups 1527 and 1529, second set 1530 includes suction cups 1532 and 1534, and third set 1535 includes suction cups 1537 and 1539.

[0234] In some embodiments, at least two of the first set 1525, the second set 1530, and the third set 1535 are controlled independently of one another (e.g., suction forces may be applied independently to different sets of the three suction cup sets). In some embodiments, at least two of the first set 1525, the second set 1530, and the third set 1535 are controlled together (e.g., a collective control is used to apply suction forces to the suction cups of such at least two sets). In some embodiments, the suction force for any one of the three suction cup sets may be controlled independently for the various suction cups within such set, or may be controlled by subset. For example, with respect to the first set 1525, suction cup 1527 is controlled independently from suction cup 1529.

[0235] The end effector 1520 is controlled to move one or more suction cups in the various suction cup sets.

[0236] 15C is a diagram illustrating a bottom view of a suction-based end effector, according to various embodiments. In some embodiments, the system 100 of FIG. 1 implements the end effector 1550. In some embodiments, the system 100 of FIG. 1 implements the end effector 1550. According to various embodiments, the end effector 1550 is a suction-based end effector. The end effector 1550 is included in a multi-mode end effector. For example, the end effector 1550 corresponds to a first gripping mechanism of a multi-mode end effector.

[0237] In the illustrated example, in contrast to the end effector 1520 of FIG. 15B , the first set 1525 of suction cups and the third set 1535 of suction cups have been moved. For example, the end effector 1550 is controlled to shift suction cups 1527, 1529, 1537, and 1539 toward the periphery of the face of the end effector 1550. In some embodiments, the end effector 1550 is controlled to move (e.g., shift) a portion of the suction cups on the end effector 1550. For example, the end effector 1550 is controlled to shift suction cup 1527, suction cup 1529, or both, while maintaining suction cups 1532, 1534, 1537, and 1539 in their normal positions.

[0238] 15C , shifting the suction cups 1527 and 1529 in the first set 1525 increases the distance between suction cups 1527 and 1532 and the distance between suction cups 1529 and 1534. Thus, the end effector 1550 may be controlled to increase its grip (or gripping range) to facilitate grasping of larger items (e.g., to better position the suction cups across the surface of a larger item) or to allow the end effector 1550 to grasp multiple items simultaneously (e.g., a first item is grasped by the first set 1525 and a second item is grasped by the second set 1530).

[0239] 15D is a diagram illustrating a bottom view of a suction-based end effector, according to various embodiments. In some embodiments, the system 100 of FIG. 1 implements the end effector 1575. In some embodiments, the system 100 of FIG. 1 implements the end effector 1575. In some embodiments, the system 100 of FIG. 1 implements the end effector 1575. According to various embodiments, the end effector 1575 is a suction-based end effector. The end effector 1575 is included in a multi-mode end effector. For example, the end effector 1575 corresponds to a first gripping mechanism of a multi-mode end effector.

[0240] In the illustrated example, in contrast to the end effector 1520 of FIG. 15B , the first set 1525 of suction cups and the third set 1535 of suction cups have been moved. For example, the end effector 1575 is controlled to shift suction cups 1527, 1529, 1537, and 1539 toward an inner portion of the end effector 1575 (e.g., the suction cups are moved closer to the second set 1530). In some embodiments, the end effector 1575 is controlled to move (e.g., shift) a portion of the suction cups on the end effector 1575. For example, the end effector 1575 is controlled to shift suction cup 1527, suction cup 1529, or both, while maintaining suction cups 1532, 1534, 1537, and 1539 in their normal positions.

[0241] 15D, shifting the suction cups 1527 and 1529 in the first set 1525 reduces the distance between suction cups 1527 and 1532 and the distance between suction cups 1529 and 1534. Thus, the end effector 1575 may be controlled to narrow the grip (or gripping range) to facilitate grasping of smaller items (e.g., to better position the suction cups across the surface of smaller items).

[0242] In some embodiments, a first set of one or more suction cups are moved to increase the distance between such suction cups and the center of the suction base end effector, while a second set of one or more suction cups are moved to decrease the distance between such suction cups and the center of the suction base end effector. Referring to Figures 15C and 15D, the first set 1525 is positioned (e.g., moved outward) as shown in Figure 15C, while the third set 1535 is positioned (e.g., moved inward) as shown in Figure 15D.

[0243] 15E and 15F are side views of a suction-based end effector, according to various embodiments. In some embodiments, the suction-based end effector is implemented as a first gripping mechanism for a multimode end effector (such as multimode end effector 300 of FIGS. 3A-3C, multimode end effector 800 of FIGS. 8A-8C, multimode end effector 900 of FIGS. 9A-9B, and multimode end effector 1000 of FIGS. 10A-10D). For example, the multimode end effector utilizes the suction-based end effector to grip an object in connection with operating in a first mode.

[0244] In the depicted example, the suction-based end effector 1580 comprises a plurality of suction cups 1527, 1532, 1535. In some embodiments, the suction-based end effector 1580 is robotically controlled to change the configuration or relative positioning of at least a portion of the plurality of suction cups 1527, 1532, 1535. For example, the suction-based end effector 1580 comprises an actuation mechanism 1588 configured to change the position / configuration of the suction cups 1535. The plurality of suction cups 1527, 1532, 1535 are mounted to mounting plates 1584, 1586, and in response to actuation, such as by control signals sent by a control computer, the actuation mechanism 1588 is actuated to move the mounting plate 1586 relative to the mounting plate 1584, thereby changing the configuration / position of one portion of the suction cups (e.g., suction cup 1586) relative to another portion of the suction cups (e.g., suction cups 1527, 1532). 15F , the suction base end effector 1580 includes sliders 1592, 1594 along which the mounting plate 1586 (and thus the suction cup 1535) moves. The sliders 1592, 1594 may be pistons or channels that provide support for the mounting plate 1586 and allow it to traverse when the actuation mechanism 1588 is actuated. In some embodiments, the sliders 1592, 1594 are pneumatic sliders that are controlled to change the configuration / position of one portion of the suction cup (e.g., suction cup 1586) relative to another portion of the suction cup (e.g., suction cups 1527, 1532), such as by moving the mounting plate 1586 relative to the mounting plate 1584.

[0245] 15E and 15F illustrate actuation mechanism 1588 as a pneumatic piston, various other actuation mechanisms may be implemented. Examples of other actuation mechanisms include a rack and pinion arrangement, a motor, etc.

[0246] 16 is a flowchart illustrating a process for operating an end effector in connection with picking and placing a set of items, according to various embodiments. In some embodiments, process 1600 is performed in connection with controlling end effector 300 of FIGS. 3A-3B. In some embodiments, process 1600 is performed by system 100 of FIG. 1A, or the like. Process 1600 is performed in connection with grasping a set of items, such as by using different modes of operation of a multi-mode end effector.

[0247] In step 1602, a decision is made to move a set of N objects using the multi-mode end effector. In some embodiments, the system determines the set of N objects to be moved based on a manifest or order corresponding to a kit of items to be assembled / collected for shipment. Some of the N objects may be items contained in one or more trays in the workspace of the robotic arm to which the multi-mode end effector is attached. Another portion of the N objects may be one or more trays in the workspace, such as an upper tray in a stack of trays that is empty, an upper tray that is emptied while the multi-mode end effector is controlled to grasp an item from the upper tray, or an upper tray that is moved to expose another tray from which an item is to be grasped. The system determines the set of N objects based at least in part on information obtained by one or more sensors deployed in the workspace.

[0248] In step 1604, the order in which the set of N objects are moved is determined based at least in part on a cost function that is based at least in part on one or more of: (i) different motion modes in which the various objects are grasped, (ii) respective destination positions of the objects, (iii) respective origin positions of the objects, (iv) positions of other objects or structures within the workspace, (v) predicted trajectories for moving the objects, (vi) costs of transitioning the multi-mode end effector to operate according to different modes, etc.

[0249] In some embodiments, the system determines a set of tasks to be performed (e.g., to achieve a higher-level goal, such as fulfilling a set of orders) and determines the order in which the set of tasks are performed based on a cost function associated with performing each task in the task set. The system determines the order in which the set of tasks are performed based on a cost associated with transitioning control of a multi-mode end effector between a first mode or a second mode. For example, the system determines the order in which the set of tasks are performed based at least in part on a cost associated with transitioning a second gripping mechanism (e.g., an end effector with multiple gripper arms) between an inactive state and an active state.

[0250] In step 1606, a first subset of the N objects to be grasped is selected based on the order. In some embodiments, the first subset of the N objects is selected according to an initial operating mode of the multi-mode end effector. For example, the order is determined based on a cost function such that the cost associated with moving items includes the cost of transitioning the multi-mode end effector between different operating modes / states. As an example, the first subset of the N objects are selected to be grasped according to the same operating mode of the multi-mode end effector. For example, the first set of N objects are items to be grasped from a tray using a suction-based end effector to avoid changing the state of the multi-mode end effector (e.g., transitioning the gripper arm between an inactive and an active state) while grasping various items in the first subset of the N objects.

[0251] As another example, a first subset of the N objects is selected to be gripped according to the same operating state of the multi-mode end effector. For example, the state of the gripper arm of the multi-mode end effector is the same when operating in the second mode as when operating in the third mode. The second mode may include gripping an item using the gripper arm, and the third mode may include pushing or pulling an object (e.g., a cart, a stack of trays, etc.) using a structure / hook on the multi-mode end effector (e.g., on the gripper arm). Thus, the subset of N objects includes objects moved according to the second mode and objects moved according to the third mode.

[0252] At step 1608, information is obtained from one or more sensors. The information indicates whether one or more of the gripper arms are in an active state or an inactive state (or an intermediate state between the inactive state and the inactive state). In some embodiments, the system utilizes the information corresponding to the positioning of the gripper arms in connection with controlling the gripper arms (or second gripping mechanisms) to transition between the active state and the inactive state depending on the mode in which the multi-mode end effector is operated.

[0253] At step 1610, a determination is made as to whether the gripper arm is configured in the correct state. The correct state corresponds to the state in which the gripper arm will be configured when moving a corresponding subset of the N objects. For example, if a subset of the items are to be gripped using a suction-based end effector, the correct state for the gripper arm is an inactive state (e.g., a retracted position). As another example, if a subset of the objects are to be gripped using the gripper arm, the correct state for the gripper arm is an active state (e.g., a deployed position). In response to determining at step 1610 that the gripper arm is not in the correct state, process 1600 proceeds to step 1612, where the configuration of the gripper arm is adjusted. For example, the system controls the gripper arm to move (or continue to move) to the modified state. Process 1600 repeats steps 1608-1612 until the system determines that the gripper arm is in the correct state.

[0254] In response to determining in step 1610 that the gripper arm is in the correct state, process 1600 proceeds to step 1614 where the system determines that the selected subset of items will engage with an object, such as an item in a tray or other source location (e.g., a shelf, a conveyor, etc.) if the selected subset of items are being moved using the first mode, or a tray if the selected objects are being moved using the second mode.

[0255] In step 1616, the system controls to adjust the position of the multi-mode end effector. The system controls to position the multi-mode end effector to engage the object to be grasped. For example, the system moves the robot arm and end effector to a position where a suction cup on the multi-mode end effector engages the object.

[0256] In step 1618, the system controls the multi-mode end effector to grasp an object with the multi-mode end effector. The system actuates a gripping mechanism to grasp the object. For example, if the multi-mode end effector is used to grasp an object using a suction-based end effector, the system actuates a suction mechanism to apply a suction force between one or more suction cups (e.g., included in the suction-based end effector) and the object to be grasped. For example, if the multi-mode end effector is used to grasp an object using an end effector with gripper arms, the system actuates a mechanism to reposition one or more gripper arms to grasp the object.

[0257] In step 1620, information is obtained from one or more sensors, the information indicating whether a suction-based end effector is engaged with an item to be grasped, whether a gripper arm (e.g., a thumb of the gripper arm) is engaged with a tray to be grasped, etc.

[0258] At step 1622, the system determines whether the object is engaged. For example, the system determines whether the object is securely grasped by the multi-mode end effector. In response to determining at step 1622 that the object is not securely grasped (e.g., the adhesive force between the item and the end effector is less than a threshold adhesive force or the item is not grasped by the gripper arm), process 1600 returns to step 1618, which controls the system to grasp the object using an appropriate gripping mechanism. Process 1600 repeats steps 1618-1622 until the system determines that the item has been grasped. In some embodiments, the multi-mode end effector is used to grasp a set of items at a time (e.g., for simultaneous movement to respective destination locations), and the repetition of steps 1618-1622 is used to determine whether each of the items in the set being moved is securely grasped.

[0259] In response to determining in step 1622 that the object is firmly grasped, process 1600 proceeds to step 1624 where the object is moved to a destination location and the grasping of the object (e.g., by the multi-mode end effector) is controlled to place the object (e.g., to release the object at the destination location).

[0260] For example, when a multi-mode end effector is operated in a first mode, the system controls the robot arm to move an item to (or near) a destination location, and then controls the suction-based end effector to release the item at the destination location. The system controls the suction-based end effector to reduce / eliminate the adhesive force between the suction-based end effector and the item.

[0261] For example, when the multi-mode end effector is operated in the second mode, the system controls the robot arm to move an object to (or near) a destination location and then controls a second gripping mechanism (e.g., one or more of the gripper arms) to release the object at the destination location.

[0262] At step 1626, a determination is made as to whether one or more other objects in the appropriate object subset are moved. For example, the system determines whether any additional objects are moved while the multimode end effector is configured in a particular state before transitioning the state of the multimode end effector to move another subset of objects.

[0263] In response to determining at step 1626 that one or more other objects in the appropriate object subset are to be moved, process 1600 returns to step 1614 and repeats steps 1614-1626 until the system determines that there are no more objects to be moved. Conversely, in response to determining at step 1626 that there are no more objects to be moved in the appropriate object subset, process 1600 proceeds to step 1628.

[0264] In step 1628, the system determines whether additional subset objects are moved using the multi-mode end effector. For example, the system determines whether additional subset objects are used using another mode of the multi-mode end effector. The other subset objects are moved using the multi-mode end effector with a different gripper arm configuration / state than the previous subset objects.

[0265] In response to determining in step 1628 that an additional subset of objects is to be moved using the multi-mode end effector, process 1600 proceeds to step 1630 where a next subset of objects is selected and the multi-mode end effector is controlled to change operational modes. For example, the system controls the multi-mode end effector to transition the state of the gripper arm. Process 1600 repeats steps 1608-1630 until no more subsets of the set of N objects are to be moved.

[0266] Although the foregoing embodiments are described in connection with picking, moving, and placing one or more trays, various other receptacles or containers may be implemented, including bags, boxes, pallets, crates, etc.

[0267] Various example embodiments described herein are described with reference to flowcharts. While the examples may include some steps performed in a particular order, according to various embodiments, various steps may be performed in different orders and / or various steps may be combined into a single step or performed in parallel.

[0268] Although the above-described embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. A robot end effector, a robotically actuated second gripper; and a robotically actuated first gripper having first and second elements positioned opposite each other on either side of a central vertical axis of the robot end effector, and the robotically actuated second gripper positioned between the first and second elements; a robotically actuated retract-extend mechanism configured to place the robot end effector in a first mode of operation in which the first gripper is positioned for use or in a second mode of operation in which the second gripper is positioned for use; A robot end effector comprising:

2. 2. The robot end effector of claim 1, the robot-actuated second gripper is placed in a robotically inactive state when the robot end effector is controlled to operate in the first mode; The robot end effector, wherein the robotically actuated second gripping mechanism is placed in a robotically active state when the end effector is controlled to operate in the second mode.

3. 2. The robot end effector of claim 1, wherein placing the robot end effector in the first mode exposes at least a portion of the robot-actuated first gripper for engagement by the robot-actuated first gripper with a first object.

4. 10. The robot end effector of claim 1, wherein the robotically actuated second gripper is configured to grasp a tray or other container.

5. 10. The robot end effector of claim 1, wherein the robotically actuated first gripper is configured to grasp at least one first object contained in a tray or other container.

6. 2. The robot end effector of claim 1, the robot end effector is configured to be connected to a robot arm; The robot end effector, wherein the first element and the second element correspond to gripper arms configured to engage two or more sides of the object or a bottom surface of the object.

7. 7. The robot end effector of claim 6, wherein placing the robot end effector in the first mode includes rotating at least one of the gripper arms to a stowed state, wherein rotating the at least one gripper arm exposes at least a portion of the robot-actuated first gripper for engaging an object.

8. 10. The robot end effector of claim 1, wherein the robotically actuated first gripper comprises a plurality of suction-based gripping mechanisms and one or more actuation mechanisms for applying suction forces to the plurality of suction-based gripping mechanisms.

9. 9. The robot end effector of claim 8, wherein the robotically actuated first gripper is configured to grasp multiple first objects at once.

10. 10. The robot end effector of claim 9, wherein a first subset of the plurality of suction-based gripping mechanisms is configured to be controlled independently of a second subset of the plurality of suction-based gripping mechanisms.

11. 11. The robot end effector of claim 10, wherein the first subset of gripping mechanisms are controlled to grip a first subset of one or more first objects, and the second subset of gripping mechanisms are controlled to grip a second subset of the one or more first objects.

12. 9. The robot end effector of claim 8, the one or more actuation mechanisms are configured to receive one or more signals from a control computer and operate in response to at least one of the one or more signals; The one or more actuation mechanisms are determined according to a grasping strategy for grasping one or more first objects in response to at least one of the one or more signals.

13. 13. The robotic end effector of claim 12, wherein at least some of the plurality of suction-based gripping mechanisms comprise extendable suction cups.

14. 14. The robotic end effector of claim 13, wherein the extendable suction cup is controlled based at least in part on at least one of the one or more signals.

15. 10. The robot end effector of claim 1, further comprising one or more structures configured to engage an object or cart and push or pull the object or cart.

16. 16. The robot end effector of claim 15, wherein the one or more structures are located on the robotically actuated second gripper.

17. 10. The robot end effector of claim 1, further comprising: a cross member configured to be coupled to the robotic arm; the first element is coupled to the cross member at a first distal end and configured to mechanically engage a first recess on a first side of an object to be grasped; the second element is coupled to the cross member at a second distal end opposite the first distal end and configured to mechanically engage a second recess on a second side of the object to be grasped.

18. 18. The robot end effector of claim 17, wherein the first robotically actuated gripper is coupled to the cross member at a location between the first distal end and the second distal end.

19. 20. The robot end effector of claim 17, further comprising a sensor configured to obtain information regarding the position of one or both of the first element or the second element.

20. 20. The robot end effector of claim 19, wherein the sensor is a mechanical limit switch configured to obtain information indicative of whether the one or both of the first element or the second element is in a deployed position corresponding to the second mode or a stowed position corresponding to the first mode.

21. 20. The robot end effector of claim 19, wherein the sensor is an optical sensor configured to obtain information indicative of whether the first element or the one or both of the elements is in a deployed position corresponding to the second mode or a stowed position corresponding to the first mode.

22. 18. The robot end effector of claim 17, one or both of the first element and the second element are movable relative to the cross member; the one or both of the first element and the second element are configured to be robotically moved between a deployed position corresponding to the second mode and a stowed position corresponding to the first mode.

23. 10. An autonomous tray-handling robotic system comprising the robotic end effector of claim 1, further comprising: a memory configured to store data indicative of a set of output stacks to be constructed, each output stack including an associated set of objects; a processor coupled to the memory and configured to control operation of one or more robots, each of the one or more robots configured to grasp, move, and place one or more first objects at a time according to a plan to repeatedly pick one or more first objects from source stacks of objects to build the set of output stacks, such as by building each output stack by sequentially placing first or second objects picked from one or more corresponding source stacks onto the output stack; The system, wherein each of the robots comprises a robot arm and the robot end effector configured to grasp, move, and place the one or more first objects without assistance from another robot.

24. 1. A method comprising: determining, by one or more processors, to grasp the object with a robotic arm configured with a robotic end effector; Determining a strategy for grasping the one or more objects, wherein determining a strategy for grasping the one or more objects comprises: determining to operate the robot end effector in a first operational mode or a second operational mode; controlling the robot end effector based at least in part on the strategy; Equipped with the robot end effector includes a robotically actuated retract-extend mechanism configured to place the robot end effector in the first mode or the second mode; controlling the robot end effector based at least in part on the strategy includes controlling the robotically actuated retract-extend mechanism to place the robot end effector in the first mode or the second mode based at least in part on the strategy.

25. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for determining, by one or more processors, to grasp an object with a robotic arm configured with a robotic end effector; computer instructions for determining a strategy for grasping the one or more objects, the computer instructions including determining to operate the robot end effector in a first operational mode or a second operational mode; computer instructions for controlling the robot end effector based at least in part on the strategy; Equipped with the robot end effector includes a robotically actuated retract-extend mechanism configured to place the robot end effector in the first mode or the second mode; controlling the robot end effector based at least in part on the strategy includes controlling the robot-actuated retract-extend mechanism to place the robot end effector in the first mode or the second mode based at least in part on the strategy.

26. 1. A system comprising: a robotic arm configured with a robotic end effector, the robotic end effector including a robotically actuated retraction-extension mechanism configured to place the robotic end effector in a first operational mode or a second operational mode; a control computer configured to control the robotic arm to grasp an object; Equipped with The control computer determining to grasp the object using a robotic arm configured with a robotic end effector; determining a strategy for grasping the one or more objects; determining whether the robot end effector is to operate in the first mode or the second mode; a system configured to control the robot end effector based at least in part on the strategy, wherein controlling the robot end effector based at least in part on the strategy includes controlling the robot-actuated retract-extend mechanism to place the robot end effector in the first mode or the second mode based at least in part on the strategy.

27. 27. The system of claim 26, wherein the robot end effector comprises: a robotically actuated second gripper; and a robotically actuated first gripper having first and second elements positioned opposite each other on either side of a central vertical axis of the robot end effector, and the robotically actuated second gripper positioned between the first and second elements; a robotically actuated retract-extend mechanism configured to place the robot end effector in a first operational mode in which the first gripper is positioned for use or a second operational mode in which the second gripper is positioned for use; A system comprising:

28. A robot end effector, a set of suction-based gripping mechanisms configured to grip one or more objects when an adhesive force is applied thereto; a robotically controlled actuation mechanism configured to move at least a first subset of suction-based gripping mechanisms to change the relative positions of the first subset of suction-based gripping mechanisms and a second subset of suction-based gripping mechanisms; A robot end effector comprising:

29. 30. The robotic end effector of claim 28, wherein the set of suction-based gripping mechanisms comprises a plurality of suction cups.

30. 29. The robot end effector of claim 28, wherein varying the relative positions of the first subset of suction-based gripping mechanisms and the second subset of suction-based gripping mechanisms varies a distance between at least one of the first subset of suction-based gripping mechanisms and at least one of the second subset of suction-based gripping mechanisms.

31. 29. The robotic end effector of claim 28, wherein the robotically controlled actuation mechanism comprises a pneumatically controlled piston that, when actuated, changes the relative positions of the first subset of suction-based gripping mechanisms.

32. 30. The robotic end effector of claim 28, wherein the robotically controlled actuation mechanism is controlled based on one or more control signals received from a control computer.

33. 33. The robotic end effector of claim 32, wherein the control computer determines to vary the relative positions of the first subset of suction-based gripping mechanisms and the second subset of suction-based gripping mechanisms based at least in part on a strategy for gripping a particular object.

34. 34. The robot end effector of claim 33, wherein the control computer determines to increase a distance between at least one of the first subset of suction-based gripping mechanisms and at least one of the second subset of suction-based gripping mechanisms based at least in part on determining that the size of the particular object exceeds a threshold distance.

35. 30. A multi-mode robot end effector comprising the robot end effector of claim 28, wherein the multi-mode robot end effector is configured to utilize the robot end effector of claim 28 in connection with operating the multi-mode robot end effector in a first mode.

Images